# AI Evals for Cross-Functional Teams — Nachiket Paranjape & Swaroop Chitlur Haridas, DoorDash AI Engineer · 2026-08-28 DoorDash GenAI platform team's Swaroop Chitlur Haridas and Nachiket Paranjape argue evals stopped being an engineering harness and became a cross-functional effort spanning strategy and operations, product, operations, and engineering. They describe a continuous loop — trace, sample, annotate, calibrate — and an API-first platform that lets non-engineers use Codex or Claude Code to vibe code their own annotation UIs. Judge prompts are calibrated self-serve through a UI showing original and optimized prompts side by side, using JetPa, so product managers and operators can run optimization loops without engineering. Per-annotation cost fell sharply at DoorDash scale, and variation in who owns judge prompts across teams is treated as a sign the org is still learning. ## Questions this episode answers ### Who owns which part of quality in DoorDash's AI evals? Nachiket Paranjape says evals are a team sport: strategy and operations folks set priorities and the quality bar, product people turn requirements into rubrics and workflows, operations teams run annotations, and engineering teams like DoorDash's provide APIs, telemetry, datasets, and judges. Combining those roles is what ships quality AI products through the evals platform. [4:34](https://aiengineer.podhood.com/b84672bd-dca2-490c-bea4-2ef627c61d24?t=274000) ### How were DoorDash's non-engineers able to build their own annotation UIs? Nachiket Paranjape explains that because the GenAI platform is API-first, strategy and operations staff could use coding agents like Codex or Claude Code to vibe code their own annotation UIs. They built different UIs for image annotation and manual testing, but the underlying patterns were similar, so stable APIs let operators self-serve instead of waiting for the platform team. [9:18](https://aiengineer.podhood.com/b84672bd-dca2-490c-bea4-2ef627c61d24?t=558000) ### How does DoorDash's self-serve judge calibration work? Nachiket Paranjape says DoorDash made judge calibration self-serve by putting the logic behind a UI. A product manager or operator sets configs, chooses a model like Gemini or Claude/OpenAI models, and runs the loop: start with a judge prompt, use the JetPa library to optimize, elevate the result as the LLM judge. The UI shows original and calibrated prompts side by side for trust. [11:36](https://aiengineer.podhood.com/b84672bd-dca2-490c-bea4-2ef627c61d24?t=696000) ### What results did DoorDash see from its self-serve annotation platform? Nachiket Paranjape reports a lot of reduction in the spend at per annotation cost. With thousands of rows needing annotation every week, DoorDash scale made it expensive, and the self-serve platform increased velocity and lowered the cost paid to annotators. Teams could iterate faster and calibrate their own judges, leading to very high velocity. [13:39](https://aiengineer.podhood.com/b84672bd-dca2-490c-bea4-2ef627c61d24?t=819000) ## Key moments - **[0:00] Intro** - [0:51] Swaroop Chitlur Haridas: DoorDash's GenAI platform team helps every product team balance accuracy, latency, and cost through shared model and agent gateways. - **[2:12] Evolution** - [3:05] DoorDash's eval platform shifted from UI-first to API-first to workflow-first so non-engineers, engineers, and coding agents can all operate it. - **[3:51] Team sport** - [4:01] Nachiket Paranjape: "Evals is not just an engineering harness; it is a cross-functional effort that adds domain-specific knowledge to AI quality." - [4:34] At DoorDash, StratOps set the quality bar, PMs turn it into rubrics, operations run annotations, and engineering supplies APIs, telemetry, datasets, and judges. - **[5:26] The loop** - [5:26] DoorDash's eval quality loop: trace, sample, annotate, build golden datasets, calibrate judges, monitor, and rinse and repeat. - **[6:12] Two surfaces** - [6:14] Nachiket Paranjape: DoorDash's eval platform separates telemetry (traces/scores via MCP/SDK/API) from workflow (annotation, golden datasets, judge calibration). - **[7:47] Annotation** - [9:49] DoorDash's stable eval APIs let StratOps teams use Codex or Claude code to vibe-code their own annotation UIs — Nachiket Paranjape. - **[10:29] Calibration** - [11:36] DoorDash's self-serve judge calibration UI lets PMs run JetPa prompt optimization with Gemini/Claude/OpenAI and review original vs calibrated prompts side by side. - [13:00] Swaroop Chitlur Haridas: DoorDash lets each team decide whether StratOps, PMs, or engineering owns the judge prompt, treating org design as still improving. - **[13:39] Impact** - [14:13] DoorDash's self-serve eval annotation reduced per-annotation cost and sped up loops for thousands of weekly rows. ## Speakers - **Nachiket Paranjape** (guest) - **Swaroop Chitlur Haridas** (guest) - **Swaroop Chitlur Haridas** (guest) ## Topics LLM-as-Judge, Evaluation Frameworks, Vibe Coding ## Mentioned Doordash (company), OpenAI (company), Claude (product), Claude Code (product), Codex (product), Gemini (product), GenAI Platform (product), JetPa (product), LLM gateway (product), agent gateway (product) ## Transcript ### Intro **Swaroop Chitlur Haridas** [0:14] Good afternoon, everyone. Thanks for, uh, coming for a post-lunch talk; I always appreciate that. My name is Swaroop, and here's my teammate, Nachiket. We are here on behalf of the DoorDash GenAI platform team, and we kind of wanted to share our evals journey. It started as, you know, evals is another engineering thing, but then it slowly— we realized it evolved into a cross-functional effort, and we kind of want to share our story here. So, what is this team? This team is a GenAI platform team. We are a horizontal team that helps all other product teams. So product teams at DoorDash build on top of the infrastructure and the primitives that we provide. And we see our USP, and the value that we provide is that we help product teams balance these three forces, which is accuracy, latency, and cost. Initially, we applied this in terms of models, but if you think about it, it also applies to agents. And the way we achieve this is we have primitives and building blocks. So, for example, we have an LLM gateway where you can easily switch between different models and try the latest and greatest. We have an agent gateway where you can connect to tools and other agents, and we help solve authentication, agent identity, and other things in a central place, which our security team can bless. Similarly, we pair the LLM gateway with open weights models hosting. Of course, cost is a number one concern these days, and we kind of invested in open weights models and have seen significant impact already. And maybe we'll talk about that in a future conference. The fourth pillar is evals, and that's the part that we would want to share today. When we started talking to product teams internally at DoorDash, there were varying distinct needs across teams. We had a consumer discovery and shopping assistant team. For those who attended Rago's talk earlier today, you will see the need for session-level quality judgments, then the personalization ML, then you needed a way to scale up human judgment. ### Evolution **Swaroop Chitlur Haridas** [2:38] And with multi-agent systems, we needed trajectory-based evals. Now, the question is, how do you cater to all these different needs under a common platform? And as we spoke to these teams, we realized, like, we needed to empower the people who are the domain experts. And in our case, that was strategy and operations folks, it was product managers, it was even labeling partners, and not only engineers. So we kind of started with, like, okay, we have to be UI first, and this was the guidance we had from Andy Fang, our co-founder, as well. So we had UIs for non-engineers to contribute. Then we kind of evolved to also being API first, so that engineers can also build and not be blocked on the central platform, and they can build their own systems. And then, of course, with the coding agents, now we have become workflow first, where we kind of empower SNO and PMs to also being able to navigate the platform and run operations as well. So with that context, I'll hand it off to Nachiket to talk about how we went about delivering this. **Nachiket Paranjape** [3:51] Cool. Thanks, Swaroop. And thanks, everyone, for joining us. I know France is playingright now, and I promise you this will be better than that. I'm kidding. So, as Swaroop was saying, evals is not just an engineering harness; it is a cross-functional effort across different pillars, across different teams, that actually helps us add all the domain-specific knowledge into our— into the quality of the AI itself. ### Team sport **Nachiket Paranjape** [4:19] So from your traces to your datasets, from, you know, scoring mechanisms, this is all basically a team sport. We all have to play and help improve the quality of AI. So, going a little bit deeper into the same aspect, we have different teams at DoorDash who help us actually improve the quality of AI. So you're going to have your strategy and operations folks who are going to set priorities, set the quality bar that you want to aim for. You're going to have your product people who are going to translate these requirements into rubrics, workflows. You're going to have your operations teams running annotations. You're going to have your engineering teams like us providing APIs, telemetry, datasets, judges, all, you know, the cool things. And combining all these together is what a recipe is for actually making sure that you are shipping quality AI products through an evals platform. So we've tried to boil this down into, sort of, you know, like a continuous iteration loop. Soright from tracing, you know, having a tracing solution, viewing your sessions, your traces, to sampling them down, you know, to a very small set that you actually want to look at, annotating these with the domain-specific expertise that you bring in with the different teams I mentioned, reviewing those, then creating those golden datasets, which are going to be, you know, your golden datasets that you want to measure or calibrate against. ### The loop **Nachiket Paranjape** [5:56] And then, of course, like, you know, monitoring this over a period of time, and then, you know, rinse and repeat, go through the whole loop again. So this is, in our experience, has been, you know, like a good sort of continuous loop for, you know, shipping quality AI. ### Two surfaces **Nachiket Paranjape** [6:12] On the platform level, we have two surfaces. So we have the telemetry layer, where we have all our traces, our scores, observations. That is also sort of the plane where users are able to access these traces using an MCP, using an SDK, using our APIs. And then we have the workflow layer. This is where a lot of our StratOps, our product teams, operate on the platform. So this is where all the annotation tasks are set. You know, this is where they review their golden datasets, create their judges, calibrate their judges, and so on. So maybe today we'll go through, you know, these sort of four different modules or pillars of our platform step by step. So again, first one, tracing and sampling, which is actually capturing what your agents, what your LLMs are actually, you know, outputting, for the lack of better words, and actually viewing those. Now, in order to also power this whole platform, we have, I think as Swaroop mentioned, we have gone in an API-first approach. What that has allowed us to do is have these stable APIs that actually, you know, and then, you know, build UIs on top of that. So all our scores, our datasets, these are all powered by very stable APIs that our team owns. So all your API access, including, you know, like an SDK access, is basically powered by this single plane. Again, ### Annotation **Nachiket Paranjape** [7:47] going back and, you know, like just refreshing your memory, step one, capture your traces, capture your sessions, measure your scores. Then you want to start almost, you know, like adding all your judgment, your context, your domain knowledge, and then calibrating your judges is what we have seen as the whole life cycle. Step two is on the annotation side. So you obviously are capturing a lot of your agentic behavior, your sessions, your traces, but you actually want to see what are some places where things went well and what are some places where things did not go well. This is where you can actually titrate your, you know, and actually look inside what's actually happening at the session level and annotate these datasets. And as Swaroop mentioned, we have a lot of use cases. We talk to multiple different teams who have various ways of annotating their datasets, and it's almost hard for a platform team to, you know, build like a UI specific for each use case. And, you know, to give you an example, it's usually going to be an annotator who's going to annotate these datasets. So the platform team is, you know, in charge of the APIs. We have a strategy and ops person who's actually deciding what to annotate, and then you have an annotator who's actually going to annotate your dataset. So we took this approach. Everybody has, you know, access to coding agents, and we actually doubled down on that API-first approach. So because we had these APIs, we were actually able to enable our StratOps teams to use something like a Codex or a Claude code and vibe code their own annotation UIs. So we had different use cases. I think we had a talk from Raghav before. We had image annotation use cases. We had some, you know, manual testing use cases. What stood out to us was the underlying patterns were similar. So if we are API-first, we can actually enable our partners to simply vibe code these UIs for annotation. So it's like a very simple example there, you know, of a vibe coded UI. Looks pretty clean, does the job, and you get, you know, the annotation that you need. This is basically like a menu from a restaurant. It's, you know, nothing crazy. But the point I want to make here is that what helped us was to give this workflow in the hands of the operators so that they can actually build their own vibe coded annotation UIs. ### Calibration **Nachiket Paranjape** [10:29] So moving on, once you have these annotation UIs, you obviously want to, you know, calibrate your judge prompts. You obviously have some LLM as a judge metric that you're tracking. You want to now start improving that with these golden datasets. In order to do that, you know, we have a pretty simple process. You're going to start with, you know, some judge prompt, take a look at, you know, what exactly do you want to measure from the output. You know, have something simple. You're going to have your baseline scores where you're going to simply run those LLM judges on your traces, and then you're going to have that optimization loop. So we use the JetPa library, which is a pretty commonly used library out there for prompt optimization. And once, you know, the iteration loop is complete, our partner teams are happy. They're going to then elevate that judge prompt as their LLM as a judge. Now, even while doing that, LLM as a judge as a concept, the whole prompt calibration concept might be straightforward to a lot of folks, but it is still like a pretty new and evolving field. And what we wanted to do was really reduce the friction of back and forth with an engineering team. So we tried to really remove all the complicated logic and make this into a self serve UI. So the screenshot that you actually see is what actually exists. So, you know, like a product manager or an operator is going to come to our UI. They're going to set some of these configs on the platform and then actually run the calibration loop themselves. So they don't have to worry about the different settings that they need to worry about, what are the different, you know, tweaks that they need to do. And they can actually, like, you know, run a calibration loop using any model of their choice. I think in this example, I have Gemini. They can run it using, you know, any of the Claude or the OpenAI models too. The other important piece was actually making this reviewable. You know, again, a lot of this is a closed box where you can't really, it's hard to see what's actually happening. So the second piece that we built was actually giving them visualization and visibility into what's actually happening. So on the left, you can see we, and this is like one of the good examples where we saw like a significant amount of improvement in the judge prompt. And we actually show the, you know, the previous, the original system prompt and the calibrated prompt to our partners so that they are also able to gain that trust as we build this. **Swaroop Chitlur Haridas** [13:00] Yeah, just wanted to add to that is this enables different configurations and different teams. In some teams, you have seen the strategy and operations folks own the prompt. You have seen some teams where the product manager owns the prompt. You have seen some teams where engineering owns the prompt. So this gives the flexibility for teams to design and evolve because we are all learning. So even the org design is improving, and we are enabling that. **Nachiket Paranjape** [13:24] Yeah, that's a good point. I think the overall idea was to, you know, build something which is as self serve as possible so that, you know, people aren't always necessarily blocked by our team helping them out. And then finally, you know, the quality loop in practice. You know, as we've been going through this exercise, we've seen a lot of improvements happening to our product as well. So, you know, for example, Swaroop mentioned we started with the UIs. We are, you know, now API and workflow first. ### Impact **Nachiket Paranjape** [13:51] We're trying to reuse a lot of the existing infrastructure that already existed at DoorDash, and that's helped us get a long way. Now, some of, we've seen obviously like, you know, really good results. I think a very good result that we do like to call out is we actually did see a lot of reduction in the spend at per annotation cost. As you all can imagine, we do have, you know, thousands of rows that need to get annotated every week, and it can get pretty expensive at DoorDash scale. And having this self serve annotation platform really helped us reduce, increase the velocity and reduce the cost that we were actually spending with these annotators to annotate the data for us. Obviously, this resulted in faster loops. Teams were able to iterate faster. They were able to, you know, calibrate their own judges in a completely self serve way. And thus, it has resulted in moving with a very, very high velocity. So finally, I just wanted to, you know, quickly touch on this slide again. The eight steps, you know, continuous loop, which is, you know, you have your traces, you want to look at your traces, your sessions, you want to sample it down to a size which you are comfortable with. You want to start annotating your datasets. You really want to start making the data better with the human knowledge that exists and the domain knowledge that exists, and then calibrate your workflows, calibrate your agents, calibrate your LLM judges with this golden dataset, and then repeat this whole cycle over a period of time to, you know, to ship reliably and ship with high quality. Yeah, we have four minutes left. Thank you once again. I think that was the last slide. Thanks for attending. And if there's any questions, we'd be happy to hang out after the talk or even happy to answer them now. --- This library is powered by PodHood (https://podhood.com), the podcast website platform.