TerminalBench

From Agent Traces to Agent Simulations — Rustem Feyzkhanov, Snorkel AI

Jul 25, 2026 · 20:24

Rustem Feyzkhanov of Snorkel AI argues that every company needs a private agent benchmark built from production traces to reliably evaluate, release, and improve agents. He explains that public benchmarks like WebArena only measure pass rate, whereas companies care about cost per solved task, latency, and policy compliance. To construct such benchmarks, he describes using Docker containers that replicate production tools, databases, and APIs, with multistep tasks and simulated users. Verifiers combine deterministic checks and LLM judges to assess final state, trace, and artifacts. He warns of edge cases like reward hacking and missing fixtures, and recommends treating benchmarks as software with a dedicated CI pipeline. Ultimately, benchmarks should be part of an agent ops loop that connects observability traces to experiments and release gates.

Task Fidelity Scaling Laws — Kobie Crawdord, Snorkel

Jun 2, 2026 · 20:40

Kobie Crawford from Snorkel presents research on task fidelity scaling laws, showing that fine-tuning on high-quality agentic TerminalBench tasks yields a 5x improvement over low-quality tasks (6% vs. 1% uplift) with the same model, compute, and task count. Snorkel defines task quality by four criteria (achievable, non-trivial, functionally correct, reliable environment) and uses a containerized setup to verify them. Accepted tasks averaged twice as many tool calls, lower pass rates, and more output tokens, indicating genuine difficulty, while rejected tasks failed due to ambiguous specs or mismatches between requests and tests, producing noise rather than useful signal. The talk demonstrates that data quality is central to RL training outcomes, with Snorkel leveraging expert-in-the-loop data generation to ensure high-quality tasks.