From Agent Traces to Agent Simulations — Rustem Feyzkhanov, Snorkel AI
Jul 25, 2026 · 20:24
Rustem Feyzkhanov of Snorkel AI argues that every company needs a private agent benchmark built from production traces to reliably evaluate, release, and improve agents. He explains that public benchmarks like WebArena only measure pass rate, whereas companies care about cost per solved task, latency, and policy compliance. To construct such benchmarks, he describes using Docker containers that replicate production tools, databases, and APIs, with multistep tasks and simulated users. Verifiers combine deterministic checks and LLM judges to assess final state, trace, and artifacts. He warns of edge cases like reward hacking and missing fixtures, and recommends treating benchmarks as software with a dedicated CI pipeline. Ultimately, benchmarks should be part of an agent ops loop that connects observability traces to experiments and release gates.
