Signal

From Signal to PR: Anatomy of a Self-Improving Agent — Jason Lopatecki, Arize

Jul 24, 2026 · 20:36

Jason Lopatecki demonstrates how Arize's Signal agent transforms observability from human-clicked dashboards into telemetry for self-fixing systems. The key unlock is pulling production traces and logs as files into a repo—Claude Code works magically with files, not dashboards—so the agent sees the exact code path software took. Signal runs periodically or event-based in a sandbox (VPC-deployed for Uber, Booking), using composable skills to gather context, find root cause, and create a PR. Lopatecki argues you should trace and log ten times more because agents can read that smoke. For the question 'why not just point Claude Code at data?', he explains skills must be well-designed to fetch and format the right data into files. On evals, they run as LLM judges layered on production traces, pre-processing information for the agent to catch known failures and create new evaluators.

The Future of Evals: From LLM as a Judge to Agent as a Judge — Aparna Dhinakaran, Arize AI

Jul 24, 2026 · 6:06

Aparna Dhinakaran, co-founder of Arize AI, argues that as agents evolved from simple prompts to complex systems with tool calls, reasoning, and long-horizon tasks, evals must evolve too—from deterministic checks to LLM as a judge, and now to agent as a judge. She reveals that the top teams run over 3,800 different evaluators, yet classical LLM-as-judge evals fail to catch subtle failures in agents that generate unique trajectories per user. Arize's new tool, Signal, is a long-running agent that reads traces, discovers patterns like inefficient tool loops, and can even open a PR to fix issues. The episode traces this arc from static checks to adaptive analysis, emphasizing that the future of evals requires all three layers to handle the complexity of modern agents.