# SimulationMaxxing: How we ship agents 20× faster — Aman Gupta (Nubank) + Shreya Rajpal (Snowglobe) AI Engineer · 2026-07-29 Shreya Rajpal (CEO of Snowglobe) and Aman (Principal ML Engineer at Nubank) argue that generating evaluation data in simulation instead of waiting on production data lets Nubank ship AI agents 20× faster. Nubank serves 135 million customers and has five agents in production, with TNPS approaching human quality. Snowglobe points at the agent, generates thousands of grounded multi-turn conversations (e.g., persona Maria Souza ordering a credit card), and pipes results into evals. Human review found simulated conversations comparable to real ones 80% of the time, enabling the team to catch regressions before production and improve one agent's self-service rate by 4%. The tight ship-observe-simulate-repeat loop also lets them test open-source models against frontier models in days instead of weeks, because the eval bottleneck is gone. ## Questions this episode answers ### How do AI agent simulations work, and why did Nubank adopt them for testing? Shreya Rajpal explains that Snowglobe's SDK wraps the agent with mocked tools and generates detailed synthetic personas, complete with account context and consistent state. Running these personas against the agent yields realistic multi-turn conversations for evaluation, bypassing the slow manual curation and risky production testing that previously bottlenecked development. [7:21](https://aiengineer.podhood.com/f254a186-f0e2-45cb-b4ce-83befd115b17?t=441000) ### What speed improvement did Nubank see by using simulations, and how did customer satisfaction change? Aman states that generating eval data via simulation cuts the agent shipping timeline from weeks to under a day—a 20x acceleration. Over a few quarters, the TNPS (customer satisfaction score) for five production agents improved dramatically, with many now exceeding human quality; one agent's TNPS even 2X'd. [1:09](https://aiengineer.podhood.com/f254a186-f0e2-45cb-b4ce-83befd115b17?t=69000) ### How reliable are simulated conversations compared to real customer interactions, according to Nubank's tests? Aman reports high correlation between simulation and production data in evaluation metrics. Domain experts conducted human review and found that simulated conversations provided usable evaluation data 80% of the time, for both mature and newly built agents. This validation gave the team confidence to trust simulations for decision-making. [11:08](https://aiengineer.podhood.com/f254a186-f0e2-45cb-b4ce-83befd115b17?t=668000) ### What risks did simulations help Nubank catch, and how did they impact self-service efficiency? Aman shares that simulations caught a regression that would have otherwise reached production, and identified an issue that could have reduced the self-service rate. By catching these early, the team improved SSR by 4% in one case and overall made agents more efficient without sacrificing customer satisfaction. [12:01](https://aiengineer.podhood.com/f254a186-f0e2-45cb-b4ce-83befd115b17?t=721000) ## Key moments - **[0:00] Results** - [0:13] Nubank ships agents 20x faster using simulations for eval data - [1:32] Nubank's five AI agents improved TNPS and now approach or exceed human support quality - **[2:54] Data Bottleneck** - [4:29] Multi-turn agent evaluation data is far more expensive than single-turn QA or structured ML data - **[6:29] Simulations** - [6:29] Nubank verified in production: simulations cut agent release cycle from weeks to hours - **[7:21] Sim Mechanics** - **[8:44] Sim Example** - [8:44] A grounded simulation: persona 'Maria Souza' orders a credit card with consistent fake address and card data - **[10:11] Improvement Loop** - [10:23] Nubank's self-improvement loop: ship agent, observe, simulate with evals, optimize agent harness, and ship - **[11:08] Sim Quality** - [11:25] 80% of Nubank domain experts confirm simulations produce usable data comparable to real conversations - **[12:01] Derisking** - [12:17] One Nubank agent's TNPS doubled after adopting simulation and robust eval systems - [13:14] Simulation helped Nubank improve self-service rate by 4% in one case without sacrificing customer satisfaction - **[13:33] Model Testing** - [13:33] Testing open-source models through simulation saved Nubank multiple weeks of evaluation effort - **[14:27] Takeaways** ## Speakers - **Aman** (guest) - **Shreya Rajpal** (guest) ## Topics Agent Evaluation ## Mentioned Nubank (company), Snowglobe (company), Snowglobe SDK (product) ## Transcript ### Results **Shreya Rajpal** [0:13] Hi everybody, my name is Shreya. I am the CEO of Snowglobe, and we have with us Aman, who is a Principal Machine Learning Engineer at NU. And this talk is going to be about SimulationMaxxing and how you can ship, or how Nubank ships agents 20x faster using simulations. **Aman** [0:33] Hey everyone, I'm Aman. So let me talk about Nubank at a glance. We are the leading digital bank in Latin America. We have 135 million customers in Brazil, Mexico, and Colombia, and we are launching in the U.S. real soon. Our quarterly revenue crossed 5 billion in Q1 2026. Our NPS, our customer level is very high. And we are the perfect company for using AI agents for customer support, where human experience, our fanatical customer care people, and AI agents together solve customer issues in a fast, empathetic, and correct manner. AI handles a lot of routines end to end. Humans focus on the hardest and long-tail cases, and together we aim to delight our customers. So as Shreya said, this talk is only about one thing really. If you generate your eval data in sim, instead of waiting on production data, you can ship agents 20x faster. And we'll give you evidence for that. So let's start with the results directly. So this is, you know, the average of TNPS, which is a measure of customer satisfaction for five of our AI agents in production. And at the beginning, they were not so great, but now, you know, with a few months of work and a few quarters' worth of effort, we've been able to massively increase the TNPS and customer love for our AI agents. And many of them are approaching human quality, and this data is a bit stale, as many of them are exceeding human quality. So we are at the stage where we are actually able to show proof that this actually works in production. Here is a QR code for our KDD paper in case you want to check it out. It's going to be presented in Korea in August. **Shreya Rajpal** [2:22] Awesome. So we opened with results, and you know, it's really about this journey of how do you implement theright systems for evaluation in order to, you know, be able to achieve those results. So in this talk, we basically split it up into these three sections. The first is why evals are so important and essential, but why they're also the bottleneck from being able to, you know, do a lot of high-throughput experimentation to get the results that Aman was showing earlier. And then why the second part of this talk is about why simulated data works as well as collected data and helps you circumvent a lot of this bottleneck that we're going to talk about. ### Data Bottleneck **Shreya Rajpal** [2:54] And then the third part is really digging deep into the systems and the findings that we had by implementing, you know, this framework at scale in Nubank. Awesome. So evals, we know this is like a whole talk track dedicated at this conference to evals. Evals are absolutely critical for building good agents. And evals are really only about two things,right? There's metrics and there's data. Metrics, again, I hope you attended many of the amazing talks yesterday on the evals track, but metrics are really, while they're challenging, we have a playbook for how to build metrics that are really well aligned with, you know, the rubrics that we care about,right? Which is you essentially use, you know, LLM as a judge style, you know, classifiers, and you align it with human judgment and getting human data. And you can iteratively build on it using, you know, auto optimization and auto prompt tuning techniques. The thing that's a bottleneck, and that still remains very challenging and unsolved, is what is the data that you're actually computing these metrics on. And that process is very time-consuming and very expensive. Specifically so for agents. So once again, people have been talking about, you know, if you're around like machine learning era, circa, you know, 2018, it would be like ML work is 85% data work,right? So data has always been challenging, but with agents, the level of sophistication that data requires is just so much more expensive. So here's examples, you know, of what structured ML data looked like, and what even early era of AI data for, you know, chatbots or single-turn QA, which was, you know, much more manageable and tractable. And you could still think of it as these structured rows. But now for multi-turn agents, each data point is a trajectory with a lot of internal tool calls that all need state, etc. And that becomes very, even one data point is expensive to generate and annotate,right? But scaling it up on a sample set becomes even harder. So how teams primarily get eval data today is two methods. One is either manual, or the second is production traces. And both of those are painful, but for different reasons,right? Manual authoring, I mean, this one's obvious. It is so time-consuming, especially the kind of data that I was showing you on the previous slide. You have to hand plan, you know, state updates. You have to hand plan the trajectory that you want to take this, you know, agent data point in. And synthetic, the state for these data points has to stay consistent across all tool calls, etc. that you make,right? Production traces, in comparison, are almost free. You don't have to pay for them. You're going to get them anyway. But the cost is that you're testing on real live users every time you're testing it. So for example, if you want to run, you know, many parallel experiments at scale, which is often something you want to do in a machine learning setting, it becomes very hard to do that in prod. **Aman** [5:46] So for those of you who build agents, you know how difficult building a new agent and maintaining an existing agent is. So if you want to change the agent harness, you want to change the prompt, the tools, anything around the LLM that powers your agent, that can take a few hours in the best case. But as Shreya mentioned, running offline evals on hand-curated data can take a few days. And then the worst part is if you run an A/B test in production and monitor regressions, see if you get a statistically significant improvement of a previous version, that can take forever because customer feedback is through some kind of a feedback form, and it can be sparse, it can be noisy, and it can take forever to know, am I moving in theright direction or not. ### Simulations **Aman** [6:29] So I'm here to tell you that we have verified in production that yes, simulations circuit this timeline, short circuit this timeline. From a few weeks, you can go to less than a day, sometimes even a few hours, a few minutes. And as I mentioned earlier, you know, with simulations, it takes a few hours. Without simulation, it takes a few hours to change the harness and offline evals. And all of this becomes super fast. Now, with simulation, the agent harness change still takes a few hours. Running offline evals becomes a lot easier because you have a simulated user to fall back on. You're not waiting on hand-curated data. You're not waiting on, you know, people to create data for you. And the best part, if you want to know that, am I doing well in production? Is this version going to do well? That can take just a few hours. So this has been a real game changer for us at Nubank. **Shreya Rajpal** [7:21] Allright. So we talked about, you know, like simulations shortening the release cycle, but what is mechanically a simulation? You know, it's existed in robotics. It's existed in, you know, a lot of different domains. But simulations for agent evals, you know, mean a specific thing. ### Sim Mechanics **Shreya Rajpal** [7:36] And this is really where Snowglobe enters and, you know, sets up a lot of that core infrastructure for you. So simulations are essentially, you know, it works by first like wrapping your agent, you know, using a, like pointing the Snowglobe SDK at your agent without requiring any code changes. And, you know, you can grok what are the tools that you really need to mock in order to make the simulation really work. The second input is about how you want to drive or steer your simulation. So, you know, who are the personas? What are the use cases? What are the data points that we're really simulating? Once you have those inputs ready, Snowglobe essentially runs the simulation. And on the output, you end up getting, you know, first just a data set. So you get like thousands of multi-turn conversations against your real agents. These conversations have, you know, tools mocked, etc., have all of the bells and whistles needed to be actually, you know, useful for evaluating your agent already ready. And then you run like judges on top of it so that on this data set, you get, you know, per-turn information about how your agent is really behaving. And it then like pipes directly into, you know, your evaluation pipeline so that you can really get data on demand. This is an example of what a concrete output of simulation is like,right? So this is as an example, like one simulated persona and then the corresponding conversation or data point for that simulated persona. So here, for example, this is totally synthetic, a user named Maria Souza. ### Sim Example **Shreya Rajpal** [9:02] You know, she wants to order a credit card. This is who she is. She's 34, a designer, first-time credit card customer. In addition to, you know, this basic information about this synthetic user, we also generate, you know, this essential grounding data. As an example, this fake address, a fake credit card, etc., that stays consistent across your agent execution so that, you know, your agent can actually be evaluated against that. And then you also have things like tone, voice, how do they really interact? And then, oops. And then when you actually run this persona through the agent, you end up getting, you know, this conversation that's consistent with the simulated intent,right? So you see, for example, that this user has very curt one-line messages in terms of how they interact because this is how the persona was set up. Or that, you know, when the agent wants to, when the agent runs its tool, when Maria enters the chat, the agent is actually able to get a valid output mocked by the simulator that, you know, is able to identify who the user is, where the address is, the address can be verified, etc. And then finally, all of this is also grounded in like account data as well. **Aman** [10:11] So for those of us building real agents, you know how hard things are. You don't know if your agent is good. You don't know if your evals are good. You have no idea what's happening. And that was me a few quarters ago. ### Improvement Loop **Aman** [10:23] Since the theme of this year's conference is self-improving agents, so we are also kind of jumping on the bandwagon. So this is our self-improvement loop. You ship your agent, you observe, you create robust evals. We have spent a lot of time creating robust evals using automated prompt optimization with algorithms like JEPA. And then you also do simulation. You run simulation and you pipe the simulation data and real data through your evals. And that gives you rich signal to then optimize your agent harness. And once you verify that agent harness optimization is good, you ship. And this has enabled us to do many, many, many A/B tests for a given agent. And we have many, many such agents across the company. So this has been a real game changer for us in terms of improving our agents. ### Sim Quality **Aman** [11:08] And the proof, as I already showed you, is that, you know, the quality of our agents is going up pretty dramatically. This is some data on how simulation really tracks real production data. So when you run simulations, you want to compare, is my simulation comparable to production data? So we looked at sim quality via evals. So for, you know, quality of with evals from sim and real data, the correlation is pretty high. And we had human review done where 80% of our domain expert labels confirmed that sims give us usable data, not just for mature agents, but also for greenfield agents. So this has been pretty useful for us in bringing up a new agent from scratch and also taking an existing agent and massively improving it. And as I mentioned, you know, simulation is not just about improving, and Shreya also talked about this. We are derisking a lot of things that could have made to production. For instance, we caught a regression with simulation that could have made it to production, but simulation caught it. And at the same time, we also caught in another agent an issue which could have lowered our self-service rate, making it less efficient. ### Derisking **Aman** [12:17] And all of this has resulted in pretty large improvements. For instance, one of our agents, the TNPS has 2X'd thanks to simulation, robust evals, and investing in something super principled rather than, you know, throwing something at the wall and see what sticks. It's been pretty awesome to see simulation saving us so much time. So now the team uses Snowglobe and tries different ideas. And they're able to short circuit launching A/B tests. They don't launch until they're happy with sim output. So imagine if you're running 10 A/B tests a quarter for your agent or probably more if you have more data. Now you can just run a bunch of stuff through sim and launch just one A/B test, which kind of short circuits launching the first five or six. And at the same time, this is often discussed in customer service circles that TNPS can come at the cost of self-service rate. Sometimes there's a trade-off. But I'm happy to report that we are not compromising on self-service rate. With sim, we are also able to pretty dramatically improve SSR across the board. But in one particular case, we were able to improve it by 4%. All of this makes us a much more efficient company and makes sure that we keep our customers really happy and we keep our SSR rates and TNPS rates very, very high. ### Model Testing **Aman** [13:33] This is one example that I am most excited about. As you know, all of you know, open-source models have been on a tear recently. So many of them have come out. So we've been trying to test various open-source models in our agent harnesses and see which ones are the best. And with sim, we have the perfect recipe to just throw a bunch of ideas, open-source models at our agent harness and eval and sim setup and see which model versions, you know, really, really work for us. This has saved us multiple weeks of effort. And probably underselling it, it's been pretty awesome. And we have A/B tests running as I speak, figuring out are these open-source models as good as, you know, the frontier models. So sim has been super, super helpful in helping us figure out kind of, you know, spreading our cards a bit. ### Takeaways **Shreya Rajpal** [14:27] Awesome. So the core three takeaways from this talk,right, is about where eval is today and how you can really remove a lot of bottlenecks to it. So we lied when we said earlier that there's just one thing you should take away from it. That one thing is still essential, but there's a few key downstream things that you can unlock if, you know, you adopt it, which is the first is if you generate your evaluation data in simulation rather than solely relying on production to get signal on how, you know, different agents are performing, you'll be able to undercut or you'll be able to short circuit a lot of the bottleneck in releasing, you know, versions of your agents much faster. The second is in order for any of these gains to really be unlocked, you know, you really need to close out the sim to real gap. So you need to, you know, set up like offline, online, human review kind of metrics to really understand how sim performs vis-à-vis real production data that you've seen so that you are able to kind of like trust the results of these simulations. And the third is, again, there's so much excitement around, you know, like auto research, self-improving agents, RSI, etc. In an enterprise setting, when you're building an agent, it really does come down to two things, data and metrics. If you have aligned metrics that are able to really catch the signals you care about and you have a reliable way of generating data that those metrics can give you signal on, it's then very easy to put together a loop of an agent that, you know, continuously improves itself from like feedback it receives from all of these places, which, you know, again, is like where the future of this field is heading. Awesome. That's us. Thank you for coming for the talk. There's a few like contact links and details for us, you know, if you guys want to get in touch. And otherwise, we'll be around here if you guys want to ask us any questions. --- This library is powered by PodHood (https://podhood.com), the podcast website platform.