# The Future of Evals: From LLM as a Judge to Agent as a Judge — Aparna Dhinakaran, Arize AI AI Engineer · 2026-07-24 Aparna Dhinakaran, co-founder of Arize AI, argues that as agents evolved from simple prompts to complex systems with tool calls, reasoning, and long-horizon tasks, evals must evolve too—from deterministic checks to LLM as a judge, and now to agent as a judge. She reveals that the top teams run over 3,800 different evaluators, yet classical LLM-as-judge evals fail to catch subtle failures in agents that generate unique trajectories per user. Arize's new tool, Signal, is a long-running agent that reads traces, discovers patterns like inefficient tool loops, and can even open a PR to fix issues. The episode traces this arc from static checks to adaptive analysis, emphasizing that the future of evals requires all three layers to handle the complexity of modern agents. ## Questions this episode answers ### What is "agent as a judge" in AI evaluation and why does it represent the future of evals? Aparna Dhinakaran describes agent as a judge as an adaptive, dynamic analysis method where an agent evaluates another agent. Unlike LLM-as-a-judge's fixed rubrics, it handles agents that take fundamentally different execution trajectories each time, catching subtle failures like inefficient loops or forgotten context that static evaluations miss. She argues teams need all three: deterministic checks, LLM-as-a-judge, and agent-as-a-judge. [3:33](https://aiengineer.podhood.com/784d61df-d5b3-4514-a339-1acebda98963?t=213000) ### How has the approach to evaluating AI agents evolved as the agents themselves became more complex? Aparna Dhinakaran explains that in 2023, agents were simple prompt-answer systems, and evals were deterministic checks. As frontier models added tool calls, reasoning, and deep research, agents became multi-step loops with sub-agents. Each capability jump made prior evals insufficient, leading from deterministic checks to LLM-as-a-judge, and now to agent-as-a-judge for dynamic, unpredictable trajectories. [2:06](https://aiengineer.podhood.com/784d61df-d5b3-4514-a339-1acebda98963?t=126000) ### What is Arize's "Signal" and how does it implement agent-as-a-judge? Aparna Dhinakaran announces the release of Signal, a long-running agent that reads traces sent in, discovers patterns of issues, and identifies problems classical evals can't, such as a tool being called repeatedly in an inefficient loop. Signal then analyzes these failures and can open a pull request to provide a fix, making it an active evaluation agent. [4:42](https://aiengineer.podhood.com/784d61df-d5b3-4514-a339-1acebda98963?t=282000) ### Why are traditional LLM-as-a-judge evals insufficient for modern AI agents? Aparna Dhinakaran notes that modern agents create fundamentally different trajectories for each user interaction, so fixed-rubric LLM-as-a-judge evals fail to catch failures like context forgetting, not knowing when a task is complete, or getting stuck in loops. These dynamic failures require an eval method that adapts, which is why agent-as-a-judge is necessary. [3:12](https://aiengineer.podhood.com/784d61df-d5b3-4514-a339-1acebda98963?t=192000) ## Key moments - **[0:00] Intro** - [1:09] Arize runs over 100 million evals per month; top teams deploy 3,800+ evaluators across 12 average eval jobs - **[1:59] Agent Evolution** - [1:59] Evals broke as agents evolved from answering prompts in 2023 to running long-horizon loops with tool calls, reasoning, and sub-agents by 2024 - **[2:41] New Failure Modes** - **[3:33] Agent as Judge** - [3:33] "What if the best way to evaluate an agent was actually with an agent?" - **[4:28] Future of Evals** - [4:42] Arize releases Signal, an agent-as-judge that reads traces, discovers subtle failure patterns like inefficient loops, and can open a PR to apply fixes - **[5:14] Join Us** ## Speakers - **Aparna Dhinakaran** (guest) ## Topics Agent Evaluation, LLM-as-Judge ## Mentioned Anthropic (company), Arize (company), OpenAI (company), Snorkel (company), Turnbench (company), Uber (company), Alex (product), Signal (product) ## Transcript ### Intro **Aparna Dhinakaran** [0:13] Awesome. Well, hey everyone. My name is Aparna, one of the founders of Arize. We work with some amazing teams to help them build evals, and we have an incredible lineup of talks for you all today at the evals track. It's happening in room 2005, and there's going to be amazing speakers from Turnbench and Uber and Snorkel, kind of all happening after this. But today I'm here to talk to you about the future of evals. Evals have gone from the new skill that every PM and every AI engineer has to learn, to the thing that every serious AI team is betting on. We've been really fortunate to get to work with some of the best AI teams in the world, so we get a front-row seat into not just what's happening when they're building their actual agents and before they actually ship, but actually the evals the teams are running on their live production agent via their traces. A little bit of some stats for you guys: we run over 100 million evals every month. The average team runs about 12 different eval jobs, with the top teams running over 3,800 different evaluators. And offline evals, online evals, they each have their own place. But today what I'm actually going to talk to you about is the teams that are running evals on their traces. This is actually what's helping teams figure out what's working, catch their failures, and that's the type of data you need to fuel your continual learning loops. And the industry kind of agrees. I mean, all the CPOs of Anthropic, OpenAI, all, you know, GDB, you have Gary Tan saying, "Evals are everything you need." And the whole industry kind of agrees. So we added evals, they catch all the failures. ### Agent Evolution **Aparna Dhinakaran** [1:59] Right? Here's the problem. While we were building all of these first-gen evals, the thing that we were actually evaluating has changed underneath us. In 2023, it was about just answering a prompt. In 2024, we started to see all the frontier models. They've added tool calls, they've added reasoning, they've added deep research. Now what we have is teams running loops on real-world data with sub-agents kicked off on long-horizon tasks. Every one of these was actually a massive jump in complexity, and we didn't just make the problem harder, we actually got a fundamentally different type of problem. ### New Failure Modes **Aparna Dhinakaran** [2:41] What that meant is that as these systems got more complex, so did the way that they actually fail. We're really lucky because we have our own agent that we've built, Alex, that lives in our UI, and we get to kind of get to feel this pain ourselves. Every time the frontier labs added new functionality, we added it to our agent. And now Alex has much longer memory. It has the ability to create dynamic UIs. It can go search across an enormous volume of traces. But we also realized that it would forget context. It wouldn't know when something was done. Sometimes it would just get stuck in these loops. And the key thing here is that the classical LLM as a judge evals that probably many of you have written in this room just weren't enough for us to be able to catch all the types of failures that we were experiencing. I mean, it's just fundamentally different,right? You have a deterministic flow, and now what we have is literally every time a user interacted with Alex, it would create a new UI. That's a fundamentally different trajectory. So this led to our really big revelation: what if the best way to evaluate an agent was actually with an agent? ### Agent as Judge **Aparna Dhinakaran** [3:55] Does it mean that all of the ways that we did evals with deterministic evals, with LLM as a judge, classic evals, doesn't matter anymore? But it just means that we have a different type of tool to solve a different type of problem. Agent as a judge is about adaptive dynamic analysis. LLM as a judge just gives you a fixed rubric with these fixed scores. It's what everyone's doing. But when your agent's doing completely different trajectories every time a user puts in data, it just means that you need a fundamentally different type of eval. My take is that most teams today are doing the first two, but the future of evals is actually having all three. And today I'm actually excited to share we've released Agent as a Judge to help our teams on their eval journey. ### Future of Evals **Aparna Dhinakaran** [4:42] We've released Signal. Signal is actually a long-running agent that can read traces sent in, discover patterns of issues. It can figure out types of problems that a classical LLM as a judge eval just would never be able to do with these deterministic rubrics. It's helped us figure out very subtle failures that you wouldn't even think of doing, such as something going on in a loop for multiple times. It was calling the same tool for a repeatedly long time. The trajectory was inefficient. And actually, what this does is, because it has all that analysis, it can go put up a PR and put up a fix. So if you want to learn more, come to our booth. We'reright by the OpenAI booth. ### Join Us **Aparna Dhinakaran** [5:25] We'll give you a demo. We'll show you a bit more about it. We're also, like I said, taking over the evals track. So come to room 2005. We're going to be talking a lot about the future of evals and what they look like. And if you just want to hang out with our team, we're throwing a viewing party for the USA World Cup game tonight. So check out the Luma and register to come join us. Awesome. Thank you all so much. --- This library is powered by PodHood (https://podhood.com), the podcast website platform.