When Will The Benchmaxxing Plague End? — Nick Heiner, Surge AI
Aug 2, 2026 · 17:25
Nick Heiner of Surge AI argues that benchmaxxing — labs gaming benchmarks rather than improving real-world value — is driven by benchmark misalignment and poor methodology, and can end with rigorous human evaluation. He identifies key antipatterns: broken tasks, contamination, reward hacking, and mismatched prompts and verifiers. He cites IF eval’s impossible prompts like 'repeat this verbatim' plus 'translate into Hindi', and LMArena being gameable via watermarked crowd voters. He shows evidence that Anthropic’s Opus 4.8 memorized much of SWE-bench verified without disclosing it. Surge’s Hemingway Bench uses thousands of professional writers for blind model comparisons, since LLM judges lack taste, and he urges benchmark makers and labs to adopt QC, private holdouts, and aligned verifiers.
