Snorkel AI

Beyond Static Intelligence: Evaluating Continual Learning — Parth Asawa, UC Berkeley

Aug 12, 2026 · 20:30

Parth Asawa, a UC Berkeley PhD student, argues that standard LLM evaluations, which reset memory between tasks, fail to measure continual learning. He introduces Continual Learning Bench 1.0, a benchmark spanning six domains including database exploration and sales prediction, using a 'gain' metric that compares stateful versus stateless performance to isolate learning from base model strength. The benchmark requires headroom, shared latent structure, and a learning signal. Initial results show vanilla in-context learning tops the leaderboard over more elaborate context management systems on reward, gain, and cost. Asawa highlights failure modes like a forecasting model that overpredicts, corrects, then reverts, and a notepad system that dismisses relevant cohort definitions. He advocates for designing continual learning as a first-order requirement, potentially as a single training phase, rather than retrofitting existing models.

From Agent Traces to Agent Simulations — Rustem Feyzkhanov, Snorkel AI

Jul 25, 2026 · 20:24

Rustem Feyzkhanov of Snorkel AI argues that every company needs a private agent benchmark built from production traces to reliably evaluate, release, and improve agents. He explains that public benchmarks like WebArena only measure pass rate, whereas companies care about cost per solved task, latency, and policy compliance. To construct such benchmarks, he describes using Docker containers that replicate production tools, databases, and APIs, with multistep tasks and simulated users. Verifiers combine deterministic checks and LLM judges to assess final state, trace, and artifacts. He warns of edge cases like reward hacking and missing fixtures, and recommends treating benchmarks as software with a dedicated CI pipeline. Ultimately, benchmarks should be part of an agent ops loop that connects observability traces to experiments and release gates.

Computer-Use 2.0: Agents Just Got Multi-Cursor — Francesco Bonacci, Cua

Jul 15, 2026 · 16:41

Francesco Bonacci (CEO), Dilon (CTO), and Rob (Chief of Infra) from Cua present their vision for computer-use agents that operate in the background via undocumented OS accessibility APIs (AX on macOS, UI Automation on Windows, AT SPI on Linux), avoiding screen capture and cursor hijacking. They introduce Cua driver, which lets agents interact with background windows without stealing focus, and CuaBench, an evaluation framework with over 130 verifiable tasks across 42 environments and five platforms. Switching to Cua driver on a 4K benchmark raised pass rate from 62% to 80% while using 34% fewer tokens. Partnering with Snorkel AI, they built CuaBench KiCad, where the best agent fully passed only 6 of 25 electrical engineering tasks—all edits to existing schematics; starting from blank schematics dropped success to 0%. Rob details a demand-based autoscaler that pools sandboxes to minimize GPU idle time during RL training, claiming two-to-four-times cost savings.

The Art & Science of Benchmarking Agents — Vincent Chen, Snorkel AI

Jun 4, 2026 · 23:25

Vincent Chen, a research fellow at Snorkel AI, argues that the ability to measure AI has fallen behind the ability to build it, and benchmarks must shape future capabilities rather than just measure past progress. Drawing from reviewing over 120 applications for Snorkel's $3 million Open Benchmarks Grants, he presents a framework: the science of task quality, distributional diversity, model headroom, and robust eval methodology, and the art of having a thesis (e.g., Terminal Bench's bet on CLI before coding agents made it obvious), producing research roadmaps, and treating researcher UX as a first-class citizen. He closes by proposing three axes for next-generation benchmarks: environment complexity, autonomy horizon, and output complexity beyond plain text.