Your Agent Evolved. Your Evals Didn't. — Ameya Bhatawdekar, Braintrust

Hello everyone, my name is Ameya Bhatawdekar, and I'm the Field CTO at Braintrust. Uh, Braintrust is an evals and observability platform that helps AI teams build and improve their AI with confidence. So,

I'm sure all of you, if not, you know, I-I'm sure everyone here has built some application over the last couple of years that has a model at the center of it,right? Some sort of a chatbot or an AI agent, or some system that's doing batch processing using AI at the heart of it.

And I'm sure all of you, over that time span, have done significant, uh, changes to that application. You have either rewritten that application entirely, or you have, like, done some pretty complex surgery on your application. And the way it looks now, compared to how it looked when you started, is likely very, very different.

And I think everyone's probably, uh, experienced the same pattern, which is, like, how building a demo with AI is really easy, but making it production quality is really hard. The same way, when you're evolving your AI application and making significant changes to it, it can be very, very challenging,right?

And, uh, the challenge is not because you built it the wrong way. The challenge is because the system around you is evolving and changing so dynamically, so rapidly. You know, the models are changing, the way your users use your application changes, the data that your application works with changes, and all of those things require you to continually make changes to your applications.

And so if you look at, you know, the rate at which the models have evolved over the last couple of years, it's truly astonishing. Like, every few months there's a new release, and that locks unlocks a, you know, a ton of new capabilities, a ton of new features that were not present in the previous generation of the models,right?

We have started seeing, like, models that got really good at working with tools, models getting really good at handling very long context. Uh, we started seeing models generate code that can be reliably and safely executed in, uh, sandboxes.

We have seen memory systems becoming very sophisticated and practical. And so each of those was not a minor upgrade. It was not an incremental change to the previous state of art. It was a step function change,right? And so now we are moving from this era of, like, re uh, sort of iterating on improving our applications to re-platforming our applications because everything is changing so dramatically.

So, why can't you just drop in a new model and ask expect your system to work? Um, well, the models, the previous system that you built was built with some assumptions around the existing limitations and the constraints that the models had,right?

Your previous systems were built to account for the fact that your models weren't really as good at tool calling, for example. And so your system implemented a bunch of logic to make it work with those limitations. And so when you drop in a new model,right, uh, you are not able to tap into the new capabilities, the new state of the art, without really restructuring your systems in a pretty dramatic way,right?

And so in order to capture that kind of capability, the new unlock, you have to re-architect.

And so as you re-architect,right, um, what happens is, uh, models evolve. So you've got to go in and change your application architecture, do a lot of work on, on getting it to work with the new models. But that means that you also now have to update your evals, the way you ensure that your system is going to operate reliably,right?

Because every new, um, uh, unlock is potentially also giving you new surface area where things can go wrong. And so your evals now have to adapt and evolve to your new architecture. And so, you know, architecture follows model updates, and your evals have to follow your architecture.

So, as I talked through the various generations of the AI systems architectures and how you do, uh, you know, what that architecture is and how the evals have to evolve with those architectural changes, I want to ground it in a real example.

And so what I want to talk about is, um, on the subsequent slides, I'll share a bunch of notional evals, but I want them to be grounded in a real agent. In this case, we are going to look at this SRE agent.

The SRE agent is able to not only read information, but it's able to take actions on and update systems. So it can, you know, roll back a deployment or, uh, escalate it to a human, page someone. Uh, so it has access to read tools and write tools.

So let's see how this system would have evolved through the various generations of AI architectures. So let's start with the simplest case,right? This is how a lot of AI applications started about three years ago. This is a single prompt, a single model call.

You have one input, one model call, one output. And so the focus of evaluations was on the final answer quality,right? Did you get the correct answer in terms of, uh, accuracy and factuality? Um, or did the, uh, model hallucinate something?

Did it make up stuff? Or did it reference, uh, old, non uh, the, the previous knowledge that it had been trained on and not the latest, uh, information related to that subject? Um, so in this case, um, you were really lo focusing primarily on the final answer.

That was your unit of evaluation. And so the approach was you would put together a golden data set. You will create a bunch of various scorers that were looking at, um, encoding your definition of what good looks like that then you could evaluate the answers against.

And this was great. This was a good way to get started. It was narrow because there's no tool calling, there's no orchestration, there's no, uh, retrieval, no other steps. It's just a simple call to the model. Uh, but the next iteration of this was the chain.

This is where you started doing a set of steps before you actually made the model call,right? Uh, the typical RAG application looked like it took the user input, it parsed some information from the user output input, it then used that to retrieve information, then generate the context, and then hand it over to the model.

And then the model synthesizes reasons on that information, synthesizes an answer, and you evaluate the answer. But there's a number of other places where things could go wrong. Yeah, your parser could extract the wrong information. It could retrieve the wrong context.

The model could struggle with the context. Like, in the early days, even though the model windows were the context window sizes were increasing, the models struggled to, um, reason over large context. So context stuffing could be an issue for the model performance.

And so now you had multiple, uh, areas of failure. And so you needed to eval all of these steps in order to figure out, like, where things went wrong and how to act upon it. But this was kind of very, um, what I would call very limited.

Like, it did things a very specific way all the time,right? And so in late, mid-late 2023, early '24, the ReAct paper became really popular. And so folks were looking at building, um, model, um, in a loop, running a model in a loop where it could, uh, reason and act, uh, in a stepwise way.

So the model could make tool calls. It could then understand what the tool calls returned, uh, reason on that data, and then figure out what the next step was so that it could then continue to run this in a loop till the user intent was finally satisfied or the model ran out of the iteration budget,right?

And so this was great because it now gives you, gives the model, the AI system, a lot more flexibility. It's not pinned down to operating in a very specific workflow. It now is able to reason on the various intents, and it's able to self-organize, self-orchestrate, and complete the user tasks.

Unfortunately, the models of that era were not as robust as they needed to be. So, you know, models struggled with tool callings. They got the arguments wrong. The models struggled with orchestration, so they called the wrong tools. The models still had challenges with reasoning.

They weren't necessarily doing a great job of, you know, dealing with long context. So you had things like context collapse. And so while the idea was, like, really, really exciting, um, it fell short of delivering on the actual promise.

And so what does, what do you do when your model can't be controlled,right? You take the control, and you bake that control into the system that you're building around the model. And so teams started moving towards these kind of workflow graphs,right?

Um, they started building the orchestration and the execution and planning logic into the, the system itself, either as a graph or as a state machine. And so you took control of the orchestration while you allowed the models to operate at the node level.

And that way you got a lot more, uh, reliability and predictability in how your AI was going to operate across those various intents. But then the problem is you are now building a system that is designed to work for a specific set of intents, for a specific types of use cases.

And as you start hand, you know, the system starts interacting with, with instances that are outside that distribution, the sys-system starts struggling with that,right? You expect, um, you know, a certain set of applications or, uh, user interactions to work well because they can be fulfilled by the orchestration that you have designed.

But when your, uh, the user intent needs to be, it requires other things to happen beyond what's specified in the orchestration. The system can start, um, you know, breaking at the seams. And, uh, in order to do that, folks were now building a lot more complexity into their orchestration logic.

And so you're building these special, uh, branches and way you handle special intents in the complex graph that described your system. And so what that means is, like, you had now a ton of different surfaces for failure. So you now had to deal with, uh, you know, uh, dealing with, uh, branch consistency and branching logic failures.

You had to deal with things like the contracts between the nodes not working out well. Uh, you had to deal with the limitations of, uh, the nodes that were, you know, built for a specific set of use cases.

So, you know, there were classifier nodes, for example, and they could make mistakes. And so you could now have a significant amount of, um, you know, areas where you could, uh, where the system could fail. And so your evals now have to not only look at, um, you know, the overall orchestration, but they now have to, you have to have node-level evals.

You have to, uh, make sure that you have evals for testing your branching logic or, uh, uh, you know, how you do retry loops. There's a lot of complex behaviors of the system that now need to be evaluated in addition to all the other things that you were evaluating before.

So the graphs are kind of popular, like, in, in late '24, early '25. And so a lot of systems were now implemented using certain frameworks, and they were now in production. Uh, but then, um, Anthropic and OpenAI launched some amazing new model capabilities mid-late '25.

And what that was like, tool calling became extremely reliable. We started, uh, seeing, uh, much better orchestration control. Uh, the models were able to plan a lot more effectively, accurately. They were able to manage long-horizon tasks. They were able to do a much better job of introspecting and course correcting.

And so, like, as things went a little off track, the models were able to, you know, understand that and bring the execution back on track. And so what that meant was a lot of these, uh, uh, graph-based systems were not able to take advantage of these new capabilities.

They were still running into some of those, like, brittleness issues that the new model state-of-art had unlocked. And so, um, we started looking at building out, um, the ReAct loop again. That's, that started working. And so now you had this new AI systems that could effectively, reliably work in a loop.

They could make those tool calls. They could figure out the next step, and then they could, uh, essentially go in and, um, fulfill the user intent. But the way they worked was very, it had a high degree of variance.

So every trajectory for the same input, if you ran it a couple of times, you would see, you know, dramatically different trajectories while yielding theright answer. And so now there's a lot of variance that you have to deal with.

So now instead of just focusing on a specific eval, the unit of eval was no longer just one eval. Now you're looking at doing, um, an analysis of the distribution of the evals. You're taking the same eval, you're running it multiple times, you're running it k times, and you're ensuring that, uh, you get a statistically relevant signal from that eval.

So now new metrics like, uh, pass at k and pass raised to k or pass wedge k, these were the new metrics that suddenly started to make a lot of sense. Pass at k is like, if you take the same that eval and you run it k times, does it succeed at least once?

And that is a measure of its capability. And pass wedge k is like, if you run that eval multiple times, how many times of those k instances does it run successfully? That's a measure of its, uh, reliability. And so now you can understand whether your system with a high pass at k, uh, you know, is reliable by seeing how it, you know, by measuring the pa-pass wedge k metric, for example.

So this gives you a lot more, um, you know, uh, understanding of, like, how your system is working, what the failure sources are, and how you work on those,right? And then more recently, what we've seen is, um, there's a big shift from it's your system is not just a model running in a loop,right?

It becomes a product system. It's that there's a model in the loop that's augmented by a lot of peripheral components. You know, you have a memory system that is able to provide robust memory storage and memory, um, retrieval capabilities, uh, within a session, cross sessions.

Uh, models can tap into this memory to, you know, improve upon their runs in subsequent instances by learning from previous runs. For example, you've got robust code execution, uh, sandboxes now. And so you can run model-generated code reliably, robustly on, uh, uh, during, uh, execution.

You've got, um, MCP and skill, uh, directories that the model can now tap into, and you can, you know, weave in extensibility. You now have things like a skills repository or a skills systems that can be used to continually augment the, the, the capabilities of models through, you know, symbolic instructions.

And so, uh, now, you know, like, uh, these systems are getting pretty complex. And as a result, uh, you know, if you are continuing to, to use the evals from the previous generation, you're going to get sort of a partial coverage of your system.

You're not going to see, uh, how your system is fragile in ways because of the unlock, because of the new surface that you have, uh, you know, uh, unlocked in your new system. So

what that means is, um, just reflecting back on the pattern is, like, you know, all of these model innovations resulted in, in, you know, corresponding shift in the architectures. And so, so you've seen these waves of architecture. And then what's needed is, like, your evals to be congruent, congruent with that architecture,right?

Uh, because ultimately it's the evals that are sort of your durable asset that describe how your system is supposed to work. And as you go through these generational shifts, that's a good way to ensure that, you know, your system, your user, users experience your system in a way that things that we're working on not broken, but it's unlocked a bunch of new capability.

And so everyone's seen this, you know, diagram of this flywheel. Everyone's sort of, like, bought into it conceptually,right? The idea of harvesting data from production to inform your evals so that your evals are reflective of the real world.

I think that all makes sense,right? And, and this is the way that, you know, teams that are doing a great job at building and shipping and improving their AI systems, they, they, they follow this workflow pretty religiously. Um, so I've talked to a lot of teams, and I think while there is a general acceptance that, yeah, you need to run that workflow, um, in practice, a lot of teams don't do that.

Their evals are somewhat static. And even if you're not changing your AI agent architecture, you're, you know, by not really being disciplined about running that, that workflow, that flywheel, you are now getting stagnant evals that are not being as effective in helping you measure and improve the quality of your AI.

And especially as you go through this generational shift, it's really important that you need a mechanism to not only harvest data from production in a way that shows you failures that you are looking out for because you have defined what good looks like as part of your evals, but you also want something to shine a light on the new failure types,right?

The system is going to fail in new and novel ways, in ways that you might not have anticipated. And you now need to start harvesting that data in a meaningful way. And you want to do this again as, as part of the flywheel.

And so this is where you need systems to come in and, uh, shine a light on things that are broken in ways that you had anticipated, but also broken in a way, in ways that you had not anticipated.

And this is really important. So I'm going to quickly talk a little bit about, like, how we do this in Braintrust. So Braintrust provides all the components that you need to run this flywheel. We've got evals, we've got observability, we have ways in which you can get insights from your production data to harvest, uh, new eval cases that you can then pass off to the, to the team that they can then use to hill climb and improve your AI system.

But Topics is a really cool feature. What Topics does, it does a cluster analysis on all of your production data. And so the idea over here is now you are able to find new categories of failure that you had not anticipated.

So your system is now able to look at all what's going on in production, and it's able to now start surfacing these new failure modes that tell you, here's a new, new failure, uh, you know, um, uh, situation that you hadn't thought about, and you didn't have any guardrails in place, or you didn't have any evals in place.

And so now it's really easy for teams to expand the set of their evals to now cover those kind of new failures. And so this is, this is a pretty exciting, uh, capability in Braintrust that enables these teams to continually not only get new failure examples for known failure modes, but more importantly, as they make these systemic architectural changes, they're able to also understand the new ways in which your system is going to fail and build out effective data sets from production data.

So I think the takeaway for today's talk is that the models will keep on changing. Uh, I, I don't think we are going to see any slowdown. I don't think we have hit a plateau yet. I think there are lots of unlocks that are coming down, um, this road.

Um, and as a result, you will be making significant changes to your AI systems. You know, you'll be doing a lot of surgery on your AI agents in the coming months, years. And so it's really important that you have a robust workflow system in place to ensure that as you make those changes, as you incorporate these new models into your systems, that your systems continue to get better at doing new things, but also continue to work well for the things that they were doing before.

And so building out, like, a robust evals discipline, uh, with theright tools and theright automation and theright systems becomes paramount to manage these generational changes. And so ultimately what you want is, um, to really, uh, index on that flywheel and make it part of your workflow so that, uh, you know, the ability to improve incrementally when the changes in the system are incremental and the ability to improve your system in a, in sort of a step function way are both supported by your evals.

So with that, I want to say thank you.