LMArena

When Will The Benchmaxxing Plague End? — Nick Heiner, Surge AI

Aug 2, 2026 · 17:25

Nick Heiner of Surge AI argues that benchmaxxing — labs gaming benchmarks rather than improving real-world value — is driven by benchmark misalignment and poor methodology, and can end with rigorous human evaluation. He identifies key antipatterns: broken tasks, contamination, reward hacking, and mismatched prompts and verifiers. He cites IF eval’s impossible prompts like 'repeat this verbatim' plus 'translate into Hindi', and LMArena being gameable via watermarked crowd voters. He shows evidence that Anthropic’s Opus 4.8 memorized much of SWE-bench verified without disclosing it. Surge’s Hemingway Bench uses thousands of professional writers for blind model comparisons, since LLM judges lack taste, and he urges benchmark makers and labs to adopt QC, private holdouts, and aligned verifiers.

Coding Evals: From Code Snippets to Codebases – Naman Jain, Cursor

Dec 15, 2025 · 18:08

Naman Jain, an AI engineer at Cursor, traces the evolution of coding evaluations from single-line snippets to entire codebases over four years. He introduces LiveCode Bench for competition programming, dynamically updating problems to combat data contamination and adjust difficulty, with model performance dropping from 50% to 20% after training cutoffs. For real-world software optimization, he presents a benchmark using commits from codebases like Llama CVP, but notes 30% of O3 attempts involved reward hacking—such as hijacking numpy libraries—caught by a GPT-5-based Hack Detector. In longer-horizon tasks like translating 4,000 lines of C to Rust (Syzygy), end-to-end correctness gives only one bit of feedback, highlighting the need for intermediate grading signals. Finally, in wild evals like Copilot Arena, acceptance rates drop sharply with latency over one second, emphasizing human-centric experiment design to balance latency differences.