Snorkel

The Future of Evals: From LLM as a Judge to Agent as a Judge — Aparna Dhinakaran, Arize AI

Jul 24, 2026 · 6:06

Aparna Dhinakaran, co-founder of Arize AI, argues that as agents evolved from simple prompts to complex systems with tool calls, reasoning, and long-horizon tasks, evals must evolve too—from deterministic checks to LLM as a judge, and now to agent as a judge. She reveals that the top teams run over 3,800 different evaluators, yet classical LLM-as-judge evals fail to catch subtle failures in agents that generate unique trajectories per user. Arize's new tool, Signal, is a long-running agent that reads traces, discovers patterns like inefficient tool loops, and can even open a PR to fix issues. The episode traces this arc from static checks to adaptive analysis, emphasizing that the future of evals requires all three layers to handle the complexity of modern agents.

Everything Is a Rollout — Alex Shaw + Ryan Marten, Terminal-Bench, Harbor, Laude Institute

Jul 24, 2026 · 21:11

Alex Shaw presents Harbor, an open-source framework for evaluating and optimizing AI agents through sandboxed environments, arguing that agent development is a form of machine learning requiring empirical evaluation. He contrasts this with traditional software engineering, showing how agents' behavior is best treated as a black box artifact. Harbor provides a common format for specifying agentic tasks, enabling parallel rollouts across any model, sandbox, and task. Shaw outlines four evaluation use cases: assessing agents building internal products, using external APIs, powering product features, and automating processes. He highlights adoptions by companies like Cognition, Scale, and Poolside, and notes Harbor's role in benchmarks like Frontier Suite and Rune Bench for Runescape. The framework also supports training via SFT and reinforcement learning, with integration partners like Tinker and LangChain.

Stop Making Models Bigger, Make Them Behave — Kobie Crawford, Snorkel

Jun 10, 2026 · 20:56

Kobie Crawford of Snorkel explains how a 4B parameter model fine-tuned via RL for under $500 outperformed Qwen 3 235B on financial analysis tool use. The key was training tool discipline—inspecting schemas and self-correcting errors—not deeper reasoning. Single-table training alone boosted multi-table FinQA benchmark from 13.9% to 26.6%, and breaking evals into rubrics identifies which behaviors to fix.

Task Fidelity Scaling Laws — Kobie Crawdord, Snorkel

Jun 2, 2026 · 20:40

Kobie Crawford from Snorkel presents research on task fidelity scaling laws, showing that fine-tuning on high-quality agentic TerminalBench tasks yields a 5x improvement over low-quality tasks (6% vs. 1% uplift) with the same model, compute, and task count. Snorkel defines task quality by four criteria (achievable, non-trivial, functionally correct, reliable environment) and uses a containerized setup to verify them. Accepted tasks averaged twice as many tool calls, lower pass rates, and more output tokens, indicating genuine difficulty, while rejected tasks failed due to ambiguous specs or mismatches between requests and tests, producing noise rather than useful signal. The talk demonstrates that data quality is central to RL training outcomes, with Snorkel leveraging expert-in-the-loop data generation to ensure high-quality tasks.

Insights from Snorkel AI running Azure AI Infrastructure: Humza Iqbal and Lachlan Ainley

Feb 8, 2025 · 20:46

Humza Iqbal of Snorkel AI explains how the company uses Azure AI infrastructure powered by NVIDIA GPUs to fine-tune foundation models for enterprise customers, achieving better performance per dollar by switching from A100s to H100s. He details their distributed training stack (PyTorch, Horovod, NFS) and lessons learned such as balancing node count for batch size and monitoring GPU utilization to avoid networking or data-loading bottlenecks. A cost comparison found two H100s outperformed four A100s on both training and inference, enabling faster iteration through more synthetic data. Azure's dedicated VMs, reliable NFS throughput, and flexible capacity allowed Snorkel to scale experiments from single-node to dozens of GPUs. Future work includes programmatic preference signals and multimodal retrieval algorithms, all planned on Azure.