Alex

The Future of Evals: From LLM as a Judge to Agent as a Judge — Aparna Dhinakaran, Arize AI

Jul 24, 2026 · 6:06

Aparna Dhinakaran, co-founder of Arize AI, argues that as agents evolved from simple prompts to complex systems with tool calls, reasoning, and long-horizon tasks, evals must evolve too—from deterministic checks to LLM as a judge, and now to agent as a judge. She reveals that the top teams run over 3,800 different evaluators, yet classical LLM-as-judge evals fail to catch subtle failures in agents that generate unique trajectories per user. Arize's new tool, Signal, is a long-running agent that reads traces, discovers patterns like inefficient tool loops, and can even open a PR to fix issues. The episode traces this arc from static checks to adaptive analysis, emphasizing that the future of evals requires all three layers to handle the complexity of modern agents.

LLM Observability, Evaluation, Experimentation Platform — Dat Ngo, Arize

Jun 7, 2026 · 16:32

Dat Ngo from Arize AI explains that observability and evaluation are essential for debugging nondeterministic LLM agents, where code no longer audits behavior—telemetry does. He outlines five flavors of eval signal—LLM as judge, human feedback, golden datasets, deterministic checks, and business metrics—and describes running them at different scopes: single span, multispan, trajectory, and session. Arize’s open-source Phoenix runs as a single container without Kubernetes, while the enterprise Arize AX adds Alex, an AI that scans traces for latency and errors and creates evals automatically. The goal is to automate the entire observability loop, letting developers focus on improvement rather than manual monitoring.

Ship Real Agents: Hands-On Evals for Agentic Applications — Laurie Voss, Arize

May 14, 2026 · 2:04:18

Laurie Voss, head of developer experience at Arize AI, delivers a hands-on workshop on evaluating agentic applications using Arize Phoenix, demonstrating that choosing the right eval matters more than tuning it: a correctness eval scored 0 out of 13 on the same financial analysis agent that a faithfulness eval scored 13 out of 13, because the model doesn't know the current year and cannot verify forward-looking data. He walks through building a complete eval pipeline from scratch—starting with tracing a Claude Haiku-based financial agent, reading and categorizing traces to identify root causes, then implementing code evals, built-in LLM-as-a-judge evals, and a custom actionability rubric with labeled examples. Voss emphasizes the importance of meta-evaluation to validate judge accuracy and introduces Phoenix experiments to prove prompt changes actually improve scores, not just vibes. Practical tips include using the impact hierarchy (data quality > prompting > model selection > hyperparameters) and the value of regression evals for safe model upgrades. The workshop closes with cost-aware evaluation, pairwise evaluation, and reliability scoring as next steps beyond the foundations…

How we solved Context Management in Agents — Sally-Ann Delucia

May 10, 2026 · 16:17

Sally-Ann Delucia of Arize explains how her team solved context management for their AI agent Alex, which analyzes trace data from Arize's observability platform. She details the failure of naive truncation and LLM summarization, and the success of smart truncation preserving head/tail with a retrievable memory store. Long sessions are handled with evals that test context at every 10 turns, and heavy tasks are offloaded to sub-agents to keep main context lean. She notes that Claude Code uses a similar truncation strategy, and emphasizes that context engineering matters more than prompt engineering for agent success.

Build a Prompt Learning Loop - SallyAnn DeLucia & Fuad Ali, Arize

Jan 6, 2026 · 52:08

SallyAnn DeLucia and Fuad Ali from Arize present prompt learning, a method to continuously improve AI prompts by using feedback from evals and human annotations. They argue that agent failures often stem from weak instructions rather than weak models, and that adding rules to system prompts can yield 15% improvements on SWE-Bench without fine-tuning or architecture changes. The talk compares prompt learning to GEBA, noting it achieves better results in fewer loops by leveraging text explanations. A case study shows Kline's performance improved from 30% to 45% on SWE-Bench-Lite after optimizing prompts. The workshop demonstrates building an optimization loop that ingests input-output pairs, evaluates them, and iteratively refines the system prompt using OpenAI.