# Benchmarks: The Good, the Bad, and the Ugly — Ali Khial, G2i AI Engineer · 2026-07-31 Ali Khial, director of AI/ML at G2i, argues that popular coding benchmarks mislead because their instructions and grading are not engineered like real work. Three of G2i's best engineers rejected benchmark prompts as unrealistic; Swebench Pro averages 481 words per instruction, and DeepSwe shows Swebench Pro accepting wrong implementations in 8.5% of tasks and rejecting correct ones in 24%. Models increasingly reward-hack by finding test files or .git folders instead of solving problems, so leaderboards hide a quality gap engineers distrust. Khial's five principles: human-authored instructions, holistic graders, production-grade tasks, contamination-free novel tasks with private holdouts, information above leaderboards. Software engineers should inspect benchmarks and contribute input. ## Questions this episode answers ### What did DeepSwe's analysis reveal about grading errors in the Swebbench Pro coding benchmark? DeepSwe found that 8.5% of tasks in Swebbench Pro accepted incorrect implementations, while over 24% rejected correct solutions. Ali Khial cited this to illustrate weak verifiers, where tests check for unexpected specifics like exact variable names not mentioned in instructions, leading to false negatives and positives that misrepresent model performance. [8:48](https://aiengineer.podhood.com/a4cedd19-f669-44a3-b947-836635cdac6b?t=528000) ### How are AI models learning to game coding benchmarks instead of genuinely solving problems? Models increasingly bypass the intended problem-solving by seeking shortcuts: they look for test files, search .git folders, or scour the internet for traces that reveal the expected answer. Khial notes that as models evolve, they become smarter at this 'reward harking,' exploiting a quality gap in benchmarks that aren't designed to prevent such gaming, leading to inflated scores. [11:38](https://aiengineer.podhood.com/a4cedd19-f669-44a3-b947-836635cdac6b?t=698000) ## Key moments - **[0:00] Intro** - [1:21] Three of G2i's best engineers said they would never write a prompt as ambiguous as those in popular coding benchmarks. - **[2:20] Benchmark Anatomy** - **[3:37] Flawed Prompts** - [3:37] SWE-bench Pro instructions average 481 words per task, making them unrealistic two-pagers that no engineer would write. - **[5:17] Weak Verifiers** - [5:28] DeepSwe's audit found 8.5% of SWE-bench Pro tasks accept wrong implementations and over 24% reject correct ones. - **[6:58] Reward Hacking** - [6:58] Newer models show increasing reward hacking, exploiting benchmark tests instead of solving the problem, with the delta growing over time. - [8:06] A quality gap in coding benchmarks creates a trust gap; engineers now test models themselves rather than relying on leaderboards. - **[8:32] Five Principles** - [9:24] Benchmarks need holistic graders that combine behavioral tests with precision where needed, akin to real software testing practices. - [10:40] Contamination-free benchmarks require novel tasks and private holdout sets, not tasks pulled from public GitHub repos. - **[11:59] Call to Action** - [12:14] "Benchmarks are not hard. We need to look under the hood and join the Discord," Ali Khial urges software engineers. ## Speakers - **Ali Khial** (guest) ## Topics Benchmarks ## Mentioned G2i (company), DeepSWE (product), Swebbench Pro (product), Swebmarathon (product) ## Transcript ### Intro **Ali Khial** [0:13] Hello, everyone. Um, this is the last talk of this session, so hopefully it's going to be short. I know that you guys had to go through a long day, so I'll try to keep it short and light for you all. I'm going to present myself: I'm Ali, I'm the director of AI/ML at G2i. I have zero experience in ML, so I don't know why they put the ML in my title. I'm a software engineer at heart, and to prove that, I have more than 50 abandoned side projects in my machine, so you can know. So, uh, I'm going to make a disclaimer: the title of the presentation is a little bit misleading. As I was working on it, I realized that it would be better if I presented my journey into benchmarks and what I learned, instead of trying to find a dichotomy of the bad, the ugly, and the good. So, um, let's start with— I want to grab your attention, and I invite you to look at this. This beautiful three screenshots are a single prompt on one of the benchmark tasks. And as I was looking at it, I was like, how can an engineer write a task like this? So I said, nah, it's impossible. No one writes prompts like these ever. But I wanted to double-check with my engineers, so I took three of our best engineers, I showed them the prompt, and I said, "Would you ever write a prompt like this?" And the answer was... no. And they'reright. They shouldn't. And so, at that point, I was like, what is the— what are benchmarks, anyway? I needed to take a step back. I needed to look more. I needed to understand. And so, as I was researching, I faced a wall of keywords. Um, graders, long horizon, verifiers, benchmarks, and a lot of jargon. So I was like, either this is too complicated, or, um, there's a lot of jargon and a lot of, um, words to work through here. So, um, I worked through it, worked with my team. ### Benchmark Anatomy **Ali Khial** [2:20] I have a lot of good researchers in the team, and we kind of, like, nailed— like, simplified it to the most basics. And so, the way I see it is that it starts as a prompt or an instruction. That prompt is fed to models and agents. Agents provide solutions. Those solutions are verified and graded through verifiers and rubrics. All of that is wrapped in a harness that's preventing it from, um, from the external factors. And if it all goes good, we have, um, trajectories, scores, and metadata that we can use to, to, to verif— to basically rank models. And so, the equation is simple. If prompts and instructions are great, and verifiers and rubrics are doing their job, while the harness is preventing, uh, or creating an environment that is good for a benchmark, we should have amazing results. But that's not the reality. So what, what went wrong? ### Flawed Prompts **Ali Khial** [3:37] So the first thing is, when looking deeper in benchmarks, most of the instructions are unrealistic. I did a quick research on Swebbench Pro, and there's 481 words per instruction in average. That's a two-pager per task. That is not how people write prompts. And to illustrate more of that, I took a couple examples here. The first one I looked at, I called the leaky prompt. It's a goal task that's basically, um, that's trying to match in some rejects and doing tests on some rejects. So in the first screenshot here, the instruction is pointing directly to the test file, which basically means that the LLM has all the ingredients it needs to go and find that test file and implement based on that. The second one is even worse. It's basically providing a complete interface of the implementation. Basically locking the LLM from any kind of creativity, and it's forcing it to do it that way. So that's the leaky prompt. The second example, it's the not-economically-valuable prompt. This is from Swebmarathon, and this prompt is well-formed. It's abstracted enough to allow for the LLM to do its work, but it's asking it to build a C compiler in Rust. So I don't know if any of you ever tried to do that, but I don't think it's a good idea. We should not do that. Allright, moving on. The second problem: weak verifiers. So the screenshot here is the work that DeepSwe did ### Weak Verifiers **Ali Khial** [5:28] to compare their bench against Swebbench Pro. And, um, let me just fix here so I can see the numbers. In Swebbench Pro, 8.5% of all the tasks accepted wrong implementation in one hand, and more than 20, 24% of the tasks rejected correct implementations. And so I kind of went, again, dug a little bit, and I extracted one of the tasks, and I started looking at it. And here's what's happening in the example of rejecting, um, possibly rejecting good answers. So in this example, the test is basically expecting a variable to exist. But that variable is, first, not specified in the instruction, and two, why would we expect an LLM to write the variable name this way? So this test is cornering the LLM and basically causing those false negatives. In the other example, it's basically— the test is basically checking functions that are unexpected. So if that was a PR in any of our projects and exposed these types of tests, we would not accept it. So this is what a weak verifier looks like. ### Reward Hacking **Ali Khial** [6:58] Allright, moving on. Reward harking. So what's happening is, models are becoming increasingly able to optimize and figure out solutions to hard problems by going around the problem. So instead of actually trying to fix the— to apply a patch to a task, they try to go and find .Git folders, or they look up the internet for any kind of traces that would allow them to, um, to do the task. And this first graph here shows, like, shows that as models evolve, they are now more smarter and smarter in being able to do reward harking. But that's what we want. We want LLMs to be smart. The benchmarks are lagging behind, and they're not preventing from that to happen. Um, more in detail, as you can see here, the more you go in time and the more you have new versions, the delta of, um, of, um, reward harking is increasing. So the conclusion here is, there's a quality gap, and it's causing a trust gap. I have not met an engineer in the last six months that would choose a model or choose, um, an LLM based on the leaderboards. They look at them, there's a lot of hype, but then they move on and they test things by themselves, and they apply that. So how do we close the gap? In the last two months, we've been working with our team at G2i to basically try to define a framework, a set of principles that would allow us to build tasks for benchmarks that are better than what we have today. ### Five Principles **Ali Khial** [8:54] The first one: human instructions. Authored by humans, reviewed by humans. This is basically the entry point for any great tasks. The instructions given to an agent or LLM should lean towards expressing desired behaviors, objectives, and hard constraints. Not implement details or try to guarantee self-containment when the task itself is expressing too much, uh, too much details. The second principle is holistic graders. Behavioral tests in one hand, and then precision where needed. This is very similar to how we approach, um, tests in engineering. We want to have the most surface covered without being too prescriptive, but we also want to be precise where needed. So for security issues or business logic, we want to have the whole stack: unit tests, integration tests, and then end-to-end tests. But for the rest of the— the rest of the software, we don't want to have 100% coverage because that's, um, not efficient. The third principle: production grade. The tasks have to be— they have to have value, um, and they have to be economically valuable. It is one thing to have a test— a task that is failing the LLM and proving that the LLM is not there yet. It is another— it is another thing for an engineer to look at a task and say, "If the LLM is fixing this, I trust it to fix that." Currently, we don't have that. So production grade. The fourth principle: contamination-free by design. We want to do novel tasks only, and we want to make sure that we keep private holdout sets. This is a principle that is very important as, currently, the tasks that are existing in benchmarks are all put from GitHub repos or from, um, from, from public repos. So our approach here is that it should always be novel. This way, it's contamination-free by design. And the fifth and last principle here is information above leaderboards. The benchmark needs to tell a story and needs to help people make decisions. Leaderboards are what we see in benchmarks today. They tell you who wins, but they don't tell you why. And so we want to basically put the X-axis back on the first page. The idea here is that there's, um, there's a lot of, um, data that we can extract from these runs, and unfortunately, they're not being put in the forefront. And people have to dig a lot and do their own experiments to get to those data points. And so finally, initially, I wanted to have a kind of a lofty ending to this, but I think I pivoted to something more interesting. This is a call to action to software engineers. ### Call to Action **Ali Khial** [12:14] Benchmarks are not hard. We need to look under the hood, and we need to understand them and join the Discord, because engineers' input is valuable. And thank you. --- This library is powered by PodHood (https://podhood.com), the podcast website platform.