# From Agent Traces to Agent Simulations — Rustem Feyzkhanov, Snorkel AI AI Engineer · 2026-07-25 Rustem Feyzkhanov of Snorkel AI argues that every company needs a private agent benchmark built from production traces to reliably evaluate, release, and improve agents. He explains that public benchmarks like WebArena only measure pass rate, whereas companies care about cost per solved task, latency, and policy compliance. To construct such benchmarks, he describes using Docker containers that replicate production tools, databases, and APIs, with multistep tasks and simulated users. Verifiers combine deterministic checks and LLM judges to assess final state, trace, and artifacts. He warns of edge cases like reward hacking and missing fixtures, and recommends treating benchmarks as software with a dedicated CI pipeline. Ultimately, benchmarks should be part of an agent ops loop that connects observability traces to experiments and release gates. ## Questions this episode answers ### Why can't I just use public benchmarks like Swe-bench or TerminalBench to evaluate my AI agent? Public benchmarks focus on generic domains and only report pass rates, but in production you need to measure cost per solved task, latency, and retries on your specific tools, APIs, and policies. A private benchmark lets you compare the full agent stack—prompt, model, skills—in an environment that mirrors your production, ensuring apples-to-apples comparisons before shipping. [3:06](https://aiengineer.podhood.com/1a08654f-8c94-4ebc-a61f-ae97be300374?t=186000) ### How do you build a simulation environment that accurately replicates production for an AI agent? The environment is defined by a set of files including an instruction Markdown, Docker containers for the agent’s tools and services, an Oracle solution to verify solvability, and verifiers. You can mock API services and use an LLM to simulate user interactions. The environment must be as close to production as possible without running full production infrastructure, often using snapshots and sidecar containers. [6:34](https://aiengineer.podhood.com/1a08654f-8c94-4ebc-a61f-ae97be300374?t=394000) ### How can you integrate agent simulations into a CI pipeline to prevent regressions? Treat the benchmark as software with its own CI pipeline: pin dependencies, run Oracle solutions to confirm tasks are solvable, check that verifiers work, and run multiple agent attempts to tag task difficulty. Any change to the agent stack triggers a re-run on the benchmark, which serves as a release gate, ensuring fixes do not introduce regressions and allowing optimization for cost or latency. [13:07](https://aiengineer.podhood.com/1a08654f-8c94-4ebc-a61f-ae97be300374?t=787000) ## Key moments - **[0:00] Intro** - [0:30] Rustem Feyzkhanov: 'Every company needs a benchmark.' - **[1:17] Simulation** - [1:17] Snorkel AI runs millions of agent simulations per month to build reliable benchmarks. - **[3:08] Public Limits** - [3:08] 'Public benchmark is useful to orient and build your prior, but your private benchmark is useful to ship.' - **[4:01] Private Benchmarks** - [4:10] When releasing agents to production, you care about cost per solved task, latency, and number of retries—not just pass rate. - **[4:52] Environments** - [5:13] A benchmark becomes part of agent evaluation, integration testing, and training set to improve the agent. - **[6:34] Task Design** - [7:01] Each simulation task includes an Oracle solution to verify the task is solvable before agents attempt it. - [8:59] You cannot put a real user in a simulation task; simulate the user with an LLM that mimics human behavior. - **[10:54] Verifiers** - [11:55] Agents can reward-hack simulation environments if they realize they're being tested, so benchmarks must be robust. - **[13:02] CI Pipeline** - [13:07] Benchmark tasks are software: treat them as code with a separate CI pipeline for dependency and Oracle checks. - [14:24] An anti-pattern is fixing agent failures by stuffing prompts with 'never do this'; simulation lets you fix the right layer. - [15:10] In production, maintain two loops: benchmark expansion from observability traces and simulation runner for release gates. - **[15:19] Ops Loop** - **[16:51] Q&A** - [16:53] Q: How many examples should a benchmark include and what's the ideal train/test split? A: Use an 80-20 train/validation split with a stand-alone dataset. - [17:49] Q: What data should go into a benchmark? A: Include both bread-and-butter use cases and edge cases like tool failures. - [19:16] Q: When should you use LLM vs. human experts for verifiers? A: Use human experts to review cases where there is disagreement between agent and verifiers. ## Speakers - **Rustem Feyzkhanov** (guest) ## Topics Agent Evaluation, Benchmarks ## Mentioned Snorkel AI (company), Arise (product), CoaBench (product), Docker (product), Harbor format (product), Qwen (product), SweBench (product), TerminalBench (product) ## Transcript ### Intro **Rustem Feyzkhanov** [0:13] Okay, hey—uh, thanks everyone for coming, and I know this is the last session before lunch, so thanks for staying here. Let's make it smooth and with good vibes. Just as Dad said, and thanks Dad for the introduction, and for inviting me. So, yeah, my name is Rustem, I'm leading the AI platform team at Snorkel, and today I want to tell you how to turn agent traces into agent simulations, and why this becomes the next stage for agent evaluations. So, three main things that I want you to take away from my talk is: every company needs a benchmark. It's the only way to reliably evaluate, release, and improve your agents. It has to be as close to production as possible. It has to mimic your real tools, real API services, policies, and workflows. And finally, it has to be part of your agentic lifecycle. It's not a static benchmark; it's a constantly populated dataset from your production traces. So, why is Snorkel AI giving this talk? We're a data-as-a-service company, and we're basically selling benchmarks, and we're producing benchmarks at scale, and for us, benchmark construction is an engineering discipline. ### Simulation **Rustem Feyzkhanov** [1:33] We run millions of agent simulations per month, and we learned how to do, like, environment-built at scale, working using both agents and subject matter experts to build reliable benchmarks that are close to production and specific domains. So, a lot of the time when people say about agent evaluation, they're focused on traces. And traces are very useful. The usual traces, like you can see an example on the screen, it basically shows, okay, here is the input prompt, here are the actions that agent took, and here is the agent output. And then evaluation can analyze it, eval, and say, okay, was agent successful or not? Was there any edge case? Uh, so it is useful to find failures in production, but it's hard to test different variants. You can run A/B testing, and that's one way of checking different agent configurations, but it's hard to make sure that everything is repeatable because you will get different database state, different tool versions, and so on, so you never fully compare apples to apples. A fine simulation turns traces into repeatable experiments. Now you take production traces, you construct tasks, and then you can run simulation benchmark, um, with different agent configuration offline, and you can compare agents using different metrics. Not just success rate, but cost, latency, and retries, and you can run those in parallel. But you can ask, okay, but why do we need it? Like, we already have public benchmarks. The challenge with public benchmarks is that usually they are focused on very specific domains. For example, Swebench is focused on, like, fixing GitHub issues, TerminalBench will focus on agent running in terminal, and CoaBench will focus on computer-use agent. ### Public Limits **Rustem Feyzkhanov** [3:24] In your case, you want your benchmark to be focused on your company's domain, both from a perspective of use cases and in terms of tooling that your agent has, whether it follows the policies that your company uses, and whether you get full production environment. Basically, public benchmark is useful to orient and build your prior, but your private benchmark is useful to ship. And a lot of the time, public benchmarks, they're specifically focused on pass rate. Every time you see a new model release, you see performance, like pass rate on different benchmarks, which makes sense because it tells us, like, about the frontier, how good is the, like, new model. ### Private Benchmarks **Rustem Feyzkhanov** [4:10] But when you release agent to production, you actually care about more metrics. You care about cost per solving the task. You care about latency. You care about number of retries. And by running, uh, evaluation offline within simulations, you can effectively compare apples to apples and iterate on agent. In this case, you can test the full stack of your agent. You're not just checking, okay, is model, uh, one is one model performs better than the other model. You check even, like, thinking talk, like thinking level. You can change the prompt. You can tune the full harness and skills tools available to agent because in your production, you don't care about the model, you care about the full system. And here you can configure and test the full system while keeping environment and, uh, evaluators the same between runs. ### Environments **Rustem Feyzkhanov** [5:13] And that raises a point about, like, what benchmark is in this case because first of all, you can use it to first release the agent. You can make sure that it works. You can handle edge cases well. You can select the optimal model, and you can debug the traces. That's how you make sure that it works. Next, you can make itright. You can put it as a release gate for your agent and verify that any change to agent stack didn't introduce regression suddenly, and iterate on harness. And finally, you can optimize it. You can tune it for better cost or latency, or you can use the traces to even do error training. Uh, later I will say I will share a link to our website where we have example how we used simulation environments to fine-tune small Qwen model to match performance of large Qwen model for specific tasks. Here, effectively, that becomes like a trifecta of use cases. Like, for you, benchmark becomes part of agent evaluation, becomes part of integration test for agent for release, and it becomes also training set for agent to improve it. So I hope I explained why you need benchmark. So now let's take a look, like, how can you actually construct it at scale for your company. Like, what is the anatomy of the benchmark task? If we take a step back, like, what is the sequence of running the benchmark? ### Task Design **Rustem Feyzkhanov** [6:38] It's straightforward. Agent gets input prompt. It interacts with the environment trying to solve the task with APIs, MCP tools, database files. Then it produces the output: trace, final state of the environment, and artifacts, basically output files. Then we run verifiers, and we can produce the metrics. How was agent able to solve the task? How well did it do? And so on. The important second part of the task is Oracle. When Oracle solution runs, it runs through the whole sequence, but just instead of running the real agent, it runs Oracle, and it and we construct Oracle ourselves to make sure that task is solvable in the first place because if it's not solvable, agent won't be able to solve it. So Oracle is important part of the task. If we look at the anatomy of the, like, of the benchmark task, how it looks in terms of files, we can look at one of the most popular formats nowadays, Harbor format, which was done by the same team who maintains TerminalBench. In this case, basically, it's just three set of files. So basically, what agent sees and interacts with: instruction MD, classic markdown file, environment, which you can see Docker file, could be Docker composing because you have multiple Docker containers. You have something that agent doesn't see, which are which is Oracle solution and verifiers. And finally, you have some metadata. It may look very straightforward. Yeah, I'm saying, like, okay, simulation environment is just Docker file and a bunch of stuff, but it is useful because now you have repeatable way of running experiments and agent simulation. So now let's dive deeper into two main parts of benchmark. First is environment. The main challenge with environment is that effectively it has to be mini production, but you don't want to run full production for every experiment. So you want to make sure that your database, API service, tools, and files match production. Just as, like, previous speaker mentioned, like, that you don't want your agent to know that it's running within simulation. So it has to be real. One thing though, you cannot put real user in your simulation task. So you can simulate the user. In this case, that becomes effectively LLM with its own prompt, which with additional context that can mimic human behavior and interaction with your system. So in the system all these things exist in the environment, and verifiers just interact with the environment afterwards. There are certain patterns to make sure that you can, uh, organize environment this way. And think about it as, like, how you construct integration tests. Basically, how you give effectively dev environment to your agent. You don't run the full production database. You have a certain snapshot. You can run site container site cars in this case. So your agent runs in one main environment, but there are other containers which contain API services, databases, MCP tools, and so on available to your agent. You don't need to have, like, full production API services. You can mock them. I already mentioned simulated users. Uh, and one important piece here is multistep. To handle long horizon tasks, if your agent needs to handle tasks that span hours, you want to ensure that you have intermediate, like, steps, and for each step you have separate prompt, separate verifiers, and you can finish simulation early if you see agent failing. And basically, that enables you to simulate long running horizon tasks. So next part is verifiers. Like, in traditional sense, usually when people speak about verifiers, you just verify the output. You get agent output. You verify it. That's it. That's why let's say how coding works. Like, we just verify the output code with tests and so on. Here, it's more complex. The way agent interacts with the simulation, we get a lot of different data. We get the world, basically how environment changed and the final environment state. What is your database state? What are the API responses? What are user replies? And so on. And your verifier analyzes final state, trace, and artifacts. So how can you analyze it? Effectively, there are multiple ways to do it. ### Verifiers **Rustem Feyzkhanov** [11:04] You can have deterministic checks, basically, and that can work really well for things like final output or tool calls where, like, it's very easy to check whether it was correct or not. Sometimes you can use LLM as a judge or even harness as a judge or agent as a judge to evaluate basically whether the trace quality was successful, whether planning of the agent was correct, and so on. In this case, it really depends on the use case. So you can use one or another or both depending on what's what works better. And finally, it's important to keep in mind that you can use subject matter expert to review some of the traces, some of the outputs. Not for everything, but for cases where you see discrepancy in agent behavior and where you want human involvement. So final piece, how can we kind of organize everything together as part of agent release and agent improvement process? So can we just start? Not just yet. Can something go wrong with benchmark task? 100%. Agent can try to reward hack simulation environment because it can understand that it's in simulation and it can hack it. Task could be too simple and, like, or verifiers could be too broad, and in this case, agent will always pass even if it does something incorrectly. It could be that agent always fail because verifiers are incorrect. Or it could be that agents not perform in a stable way and, like, you have high variation of agent success. So all of these are effectively edge cases that you need to catch during your benchmark development because benchmark development is an art on its own. We saw already, like, hundreds of benchmarks appear over the course of last years, but this is something that the culture and engineering discipline that needs to be built in each engineering team that needs to ship AI agents to production. ### CI Pipeline **Rustem Feyzkhanov** [13:07] Because as you saw, effectively, benchmark is software. It's code. It's files. You need to treat it as such. You need to have a separate CI pipeline for it. And you can check pretty obvious things, like, for example, making sure that all dependencies are pinned or, like, your base image is correct or you don't have any missing fixtures. Then you can run Oracle solution and make sure that it passes. Or, like, if you don't run Oracle, that verifiers fail. You can run several agent runs on the task and verify that, okay, it is solvable and it is hard for agent. You can tag the task whether it's simple, medium, or hard depending on how much time how many times the agent is successful. And finally, you can approve it to make it part of your benchmark. In this case, the process for improving agent becomes straightforward. You establish baseline, like, on your benchmark. You run evaluation data set. You see the failures, where it doesn't perform well. You change one thing. You rerun experiment and where you can use something like Arise to record your experiments. And then once you fix it, you rerun the full experimentation again. And then you can finally release to production. What it unlocks is to make sure that you fix issues correctly. There is a bit of anti-pattern in the industry where, like, folks try to fix things in the prompt, and they populate the prompt with things like, never do this or only do that. Never output this critical that, which is one way of handling it for sure. But with simulation, you control the full stack. You can evaluate the full stack, and you can make sure that fix leaves in correct place. You don't push everything to the prompt. You fix harness if you want if you have context overload. Or you can if there is a missing procedure, you put it into skill. Or, like, if you need to have specific output schema, you put it as part of structured output. So once agent is in production, effectively, you will have two loops. One loop for benchmark expansion. You take observability traces traces from observability basically using something like Arise. You record failures and use failures to build your benchmark further. ### Ops Loop **Rustem Feyzkhanov** [15:31] Then you have simulation runner that will run experiments on you on extended benchmark or on new agent config. We'll record these experiments. Then you can use it as a release gate whether agent performs significantly well and finally release it to production. In this case, it's important that you have system like, it's important to understand to make sure that your observability piece and experimentation piece are connected because they're part they're two sides of the same coin. So yeah, to summarize my talk, everyone needs every company needs a benchmark. Traces are useful for, like, finding edge case in production, but simulation helps to test what would happen. And finally, you want your benchmark to be part of your agent ops loop. Thank you very much for your time. And I think maybe I will have answer for one question, but we'll be happy to answer any questions outside. And please check our booth if you have any questions about benchmarks. **Guest** [16:49] Allright. **Guest 2** [16:50] Couple questions. ### Q&A **Guest** [16:52] Allright. **Guest 3** [16:53] A great talk. I have a question about structuring the benchmark. How many examples you should include ideally, and do you split them into, like, a train test split? If so, like, how do you recommend structuring examples across those two splits? **Rustem Feyzkhanov** [17:11] Great question. So the question was basically how can we structure benchmark for training and validation. And just as previous speakers shared, basically, they also had a pattern of, like, having train val split. This is very close to traditional machine learning where we want to have standalone data set that agent didn't see, basically where we can verify agent config. I think the classic approach applies when, like, you want to have 80-20%. Always depends on the use case, but you do want to have a standout data set that agent didn't see through the experimentation process. **Guest** [17:45] Awesome. Next question. **Guest 2** [17:49] So when you create the benchmarks, what data do you include in the benchmark? Do you include production runs that you've handpicked into the benchmark? Or if your agent is not yet in production, do you create data sets or problems to solve for the agent in the benchmark? And if you're if you're, like, handpicking problems for the agent to solve, how do you make sure you have, like, enough coverage similar to what you'd see in production? **Rustem Feyzkhanov** [18:20] Great question. So the question is how to ensure coverage. And distribution of benchmark is a very important piece. So effectively, you want to keep make sure, like, you have both. You have bread and butter use cases, basically covering all main use cases that work, but also making sure you have edge cases, basically how you can make sure, like, your agent can handle edge case when tools fail, like, there is problem with database and so on. So you want to have both. Basically, think about it as integration tests. You have happy path, but you also have edge cases. **Guest 2** [18:50] Would you simulate it with LLMs or would you create handcrafted? **Rustem Feyzkhanov** [18:54] Yeah. Great question. Basically, simulate versus handcrafting. I think basically, I mean, people use agents to write code. So a lot of things here can be automated. So what is handcrafted kind of changes. Like, the most important piece that you can provide is, like, the built environment once and then build the context for basically that mimics your production. **Guest 2** [19:16] You mentioned sometimes you use LLM or sometimes you use a human expert to build verifiers. Can you elaborate on that? What's the best practice over there? **Rustem Feyzkhanov** [19:29] Great question. So in our case, basically, we have a lot of subject matter experts, so we do things at scale. So the important piece here is to you don't need subject matter experts to review everything, but you specifically want to find cases where there is disagreement between agent and different verifiers. If you think that basically task was supposed to be solved, but agent somehow marks it as, like, not solved correctly or, like, it marks that trace wasn't optimal enough. And this is where you want to have subject matter expert that can basically tune the agent that does the review. --- This library is powered by PodHood (https://podhood.com), the podcast website platform.