Benchmarks: The Good, the Bad, and the Ugly — Ali Khial, G2i

Hello, everyone. Um, this is the last talk of this session, so hopefully it's going to be short. I know that you guys had to go through a long day, so I'll try to keep it short and light for you all.

I'm going to present myself: I'm Ali, I'm the director of AI/ML at G2i. I have zero experience in ML, so I don't know why they put the ML in my title. I'm a software engineer at heart, and to prove that, I have more than 50 abandoned side projects in my machine, so you can know.

So, uh, I'm going to make a disclaimer: the title of the presentation is a little bit misleading. As I was working on it, I realized that it would be better if I presented my journey into benchmarks and what I learned, instead of trying to find a dichotomy of the bad, the ugly, and the good.

So, um, let's start with— I want to grab your attention, and I invite you to look at this. This beautiful three screenshots are a single prompt on one of the benchmark tasks. And as I was looking at it, I was like, how can an engineer write a task like this?

So I said, nah, it's impossible. No one writes prompts like these ever. But I wanted to double-check with my engineers, so I took three of our best engineers, I showed them the prompt, and I said, "Would you ever write a prompt like this?"

And the answer was... no.

And they'reright. They shouldn't. And so, at that point, I was like, what is the— what are benchmarks, anyway? I needed to take a step back. I needed to look more. I needed to understand. And so, as I was researching, I faced a wall of keywords.

Um, graders, long horizon, verifiers, benchmarks, and a lot of jargon. So I was like, either this is too complicated, or, um, there's a lot of jargon and a lot of, um, words to work through here. So, um, I worked through it, worked with my team.

I have a lot of good researchers in the team, and we kind of, like, nailed— like, simplified it to the most basics. And so, the way I see it is that it starts as a prompt or an instruction.

That prompt is fed to models and agents. Agents provide solutions. Those solutions are verified and graded through verifiers and rubrics. All of that is wrapped in a harness that's preventing it from, um, from the external factors. And if it all goes good, we have, um, trajectories, scores, and metadata that we can use to, to, to verif— to basically rank models.

And so, the equation is simple. If prompts and instructions are great, and verifiers and rubrics are doing their job, while the harness is preventing, uh, or creating an environment that is good for a benchmark, we should have amazing results.

But that's not the reality. So what, what went wrong?

So the first thing is, when looking deeper in benchmarks, most of the instructions are unrealistic. I did a quick research on Swebbench Pro, and there's 481 words per instruction in average. That's a two-pager per task. That is not how people write prompts.

And to illustrate more of that, I took a couple examples here. The first one I looked at, I called the leaky prompt. It's a goal task that's basically, um, that's trying to match in some rejects and doing tests on some rejects.

So in the first screenshot here, the instruction is pointing directly to the test file, which basically means that the LLM has all the ingredients it needs to go and find that test file and implement based on that. The second one is even worse.

It's basically providing a complete interface of the implementation. Basically locking the LLM from any kind of creativity, and it's forcing it to do it that way. So that's the leaky prompt. The second example, it's the not-economically-valuable prompt. This is from Swebmarathon, and this prompt is well-formed.

It's abstracted enough to allow for the LLM to do its work, but it's asking it to build a C compiler in Rust. So I don't know if any of you ever tried to do that, but I don't think it's a good idea.

We should not do that.

Allright, moving on. The second problem: weak verifiers. So the screenshot here is the work that DeepSwe did

to compare their bench against Swebbench Pro. And, um, let me just fix here so I can see the numbers.

In Swebbench Pro, 8.5% of all the tasks accepted wrong implementation in one hand, and more than 20, 24% of the tasks rejected correct implementations. And so I kind of went, again, dug a little bit, and I extracted one of the tasks, and I started looking at it.

And here's what's happening in the example of

rejecting, um, possibly rejecting good answers. So in this example, the test is basically expecting a variable to exist. But that variable is, first, not specified in the instruction, and two, why would we expect an LLM to write the variable name this way?

So this test is cornering the LLM and basically causing those false negatives. In the other example, it's basically— the test is basically checking functions that are unexpected. So if that was a PR in any of our projects and exposed these types of tests, we would not accept it.

So this is what a weak verifier looks like.

Allright, moving on. Reward harking. So what's happening is, models are becoming increasingly able to optimize and figure out solutions to hard problems by going around the problem. So instead of actually trying to fix the— to apply a patch to a task, they try to go and find .Git folders, or they look up the internet for any kind of traces that would allow them to, um, to do the task.

And this first graph here shows, like, shows that as models evolve, they are now more smarter and smarter in being able to do reward harking. But that's what we want. We want LLMs to be smart. The benchmarks are lagging behind, and they're not preventing from that to happen.

Um, more in detail, as you can see here, the more you go in time and the more you have new versions, the delta of, um, of, um, reward harking is increasing.

So the conclusion here is, there's a quality gap, and it's causing a trust gap. I have not met an engineer in the last six months that would choose a model or choose, um, an LLM based on the leaderboards.

They look at them, there's a lot of hype, but then they move on and they test things by themselves, and they apply that.

So how do we close the gap? In the last two months, we've been working with our team at G2i to basically try to define a framework, a set of principles that would allow us to build tasks for benchmarks that are better than what we have today.

The first one: human instructions. Authored by humans, reviewed by humans. This is basically the entry point for any great tasks. The instructions given to an agent or LLM should lean towards expressing desired behaviors, objectives, and hard constraints. Not implement details or try to guarantee self-containment when the task itself is expressing too much, uh, too much details.

The second principle is holistic graders. Behavioral tests in one hand, and then precision where needed. This is very similar to how we approach, um, tests in engineering. We want to have the most surface covered without being too prescriptive, but we also want to be precise where needed.

So for security issues or business logic, we want to have the whole stack: unit tests, integration tests, and then end-to-end tests. But for the rest of the— the rest of the software, we don't want to have 100% coverage because that's, um, not efficient.

The third principle: production grade. The tasks have to be— they have to have value, um, and they have to be economically valuable. It is one thing to have a test— a task that is failing the LLM and proving that the LLM is not there yet.

It is another— it is another thing for an engineer to look at a task and say, "If the LLM is fixing this, I trust it to fix that." Currently, we don't have that. So production grade.

The fourth principle: contamination-free by design. We want to do novel tasks only, and we want to make sure that we keep private holdout sets. This is a principle that is very important as, currently, the tasks that are existing in benchmarks are all put from GitHub repos or from, um, from, from public repos.

So our approach here is that it should always be novel. This way, it's contamination-free by design. And the fifth and last principle here is information above leaderboards. The benchmark needs to tell a story and needs to help people make decisions.

Leaderboards are what we see in benchmarks today. They tell you who wins, but they don't tell you why. And so we want to basically put the X-axis back on the first page. The idea here is that there's, um,

there's a lot of, um, data that we can extract from these runs, and unfortunately, they're not being put in the forefront. And people have to dig a lot and do their own experiments to get to those data points.

And so finally, initially, I wanted to have a kind of a lofty ending to this, but I think I pivoted to something more interesting. This is a call to action to software engineers.

Benchmarks are not hard. We need to look under the hood, and we need to understand them and join the Discord, because engineers' input is valuable.

And thank you.