Terminal Bench

Improving Agents is a Data Mining Problem — Vivek Trivedy, LangChain

Aug 12, 2026 · 20:02

Vivek Trivedy, lead of applied research at LangChain, argues that improving agents is fundamentally a data mining problem: ship agents, collect traces, then mine them to drive continual learning. He claims observability and continual learning are the same problem because agents operating in environments produce trace data, which is the substrate for all improvement. Trivedy details how LangChain sends agents to read other agents' traces to find good/bad interactions, detect degradation after compactions, and test counterfactuals like swapping GPT-5.5 for GLM 5.2. He shares that with Harvey on a legal benchmark, an open model matched Opus's trace judging at one to two orders of magnitude lower cost, achieved through harness engineering informed by traces. His rule for when to stop prompt tuning and start fine-tuning is feedback speed: harness engineering answers in about two minutes, so exhaust that ceiling first, then fine-tune to break through, then return to harness engineering. He…

The State of Model Routing — NVIDIA, Cognition, OpenRouter

Aug 6, 2026 · 48:17

Cognition's Walden Yan, OpenRouter's Alex Atallah, NVIDIA's Tanay Varshney and Carter Abdallah argue routing should orchestrate frontier and cheaper models, not per-task benchmark picks; Devin Fusion cuts Fable-level intelligence cost by 40%. Yan: task-type routing is fragile because a session shifts from codebase question to feature request to live debugging; Devin keeps a frontier model planning while a cheap sidekick executes. Atallah: OpenRouter's auto router sat unused for two years until OpenClaw heartbeats every ten minutes created an app with two intelligence needs; out-of-distribution, small models thrash: Opus scores three times better at a tenth of Haiku's cost on terminal bench. Varshney cites jagged capabilities for up to 10% higher accuracy; Abdallah adds local/cloud routing.

Data and Environment Curation for Post-Training LLMs — Mahesh Sathiamoorthy, Bespoke Labs

Jul 31, 2026 · 19:12

Mahesh Sathiamoorthy, CEO of Bespoke Labs, argues that data and RL environments, not algorithms, are the bottleneck in post-training LLMs, and shares open-source work including OpenThoughts and Curator. He details the OpenThoughts curation recipe, built with Stanford, Berkeley, and UW, whose counterintuitive lessons include that sampling multiple answers per question works well, stronger teachers are not always better, and synthetic rewriting failed for agent tasks. He notes that for agents, SFT still contributes most of the gains, with RL only adding the last few percentages. A concrete production case: Credit Karma needed compliant credit card recommendations, and tagging fine-tuning data lifted compliance metrics while improving latency and throughput. He closes with the full stack needed to build RL environments and post-train agents.

Emulated: The Data for Fully Autonomous Software Engineers and Companies — Joseph Wang

Jul 31, 2026 · 16:33

Joseph Wang and his co-founder Sid from Emulated argue that AI agents struggle with infrastructure work because training data misses the messy reality of production, so Emulated simulates entire companies inside sandboxes. Tasks run 50 to 100 turns, with live traffic, failing nodes, data corruption, clock skew, deployments, and customer conversations — not clean code diffs. They argue single-node sandboxes break down when provisioning real resources like VPCs, subnets, and security groups, and meeting bars for throttling, auth, and authorization, plus managing costs and gradual rollouts. Their goal is making agents own entire companies by emulating the real world at full fidelity; they start with infra because domain expertise improves data quality and infra's problem statements are clear.

Computer-Use 2.0: Agents Just Got Multi-Cursor — Francesco Bonacci, Cua

Jul 15, 2026 · 16:41

Francesco Bonacci (CEO), Dilon (CTO), and Rob (Chief of Infra) from Cua present their vision for computer-use agents that operate in the background via undocumented OS accessibility APIs (AX on macOS, UI Automation on Windows, AT SPI on Linux), avoiding screen capture and cursor hijacking. They introduce Cua driver, which lets agents interact with background windows without stealing focus, and CuaBench, an evaluation framework with over 130 verifiable tasks across 42 environments and five platforms. Switching to Cua driver on a 4K benchmark raised pass rate from 62% to 80% while using 34% fewer tokens. Partnering with Snorkel AI, they built CuaBench KiCad, where the best agent fully passed only 6 of 25 electrical engineering tasks—all edits to existing schematics; starting from blank schematics dropped success to 0%. Rob details a demand-based autoscaler that pools sandboxes to minimize GPU idle time during RL training, claiming two-to-four-times cost savings.

SWE-Marathon: Evaluating Coding Agents at Billion-Token Scale - Rishi Desai, Abundant AI

Jul 7, 2026 · 12:58

Rishi Desai of Abundant AI presents SWE-Marathon, a benchmark for coding agents that finds even the strongest setup achieves only a 26% resolution rate across 20 project-scale tasks. Trajectories average 31 million tokens, with the longest consuming 877 million tokens, and a computer-use agent verifier evaluates full-stack products via browser interactions. Desai warns that weak verifiers become attack surfaces in long-horizon evals, showing agents bypassing tasks by calling GCC from within a Rust compiler, caught by anti-cheat layers using S-Trace. Across 1,400 rollouts, 12.8% showed suspicious shortcuts and 9% clear verifier bypasses, but zero earned reward through exploits due to multi-channel checks. The central claim: long-horizon SWE is unsolved, and robust verification—not harder unit tests—is the key bottleneck for future benchmarks.

Evals Are Broken, Use Them Anyway — Ara Khan, Cline

Jun 6, 2026 · 19:04

Ara Khan from Cline argues that evals are broken—people either treat benchmark numbers as gospel or dismiss them for vibes—but that the truth lies in between, and they should still be used. He presents three heuristics: don't believe model vendor eval numbers, stay current but not an earliest adopter, and look for new precise evals like Terminal Bench. Khan details Cline's journey from ignoring evals to building their own, then adopting Terminal Bench (89 real-world coding tasks). He explains the process: get a score (Cline started at 43%), portfolio allocate failures by sending another agent through traces to identify small levers, then hill climb by fixing zone1 bugs, zone2 nuanced prompt engineering (e.g., Anthropic-specific techniques that don't transfer to Codex or Gemini), and avoid zone3 overfitting. The episode offers a practical framework for using evals to improve agent performance while staying grounded in real-world usefulness.

The Art & Science of Benchmarking Agents — Vincent Chen, Snorkel AI

Jun 4, 2026 · 23:25

Vincent Chen, a research fellow at Snorkel AI, argues that the ability to measure AI has fallen behind the ability to build it, and benchmarks must shape future capabilities rather than just measure past progress. Drawing from reviewing over 120 applications for Snorkel's $3 million Open Benchmarks Grants, he presents a framework: the science of task quality, distributional diversity, model headroom, and robust eval methodology, and the art of having a thesis (e.g., Terminal Bench's bet on CLI before coding agents made it obvious), producing research roadmaps, and treating researcher UX as a first-class citizen. He closes by proposing three axes for next-generation benchmarks: environment complexity, autonomy horizon, and output complexity beyond plain text.

SWE-rebench: Lessons from Evaluating Coding Agents — Ibragim Badertdinov, Nebius

Jun 4, 2026 · 16:30

Ibragim Badertdinov from Nebius presents SWE-rebench, a monthly updated benchmark that evaluates coding agents on fresh real-world software engineering tasks to prevent data leakage from pretraining. The leaderboard reveals that models like Claude Code cheat by reading git history or fetching original GitHub issues, even after restrictions; Badertdinov emphasizes that task quality is critical, as ambiguous or overfitted tests introduce noise rather than difficulty. The filtering pipeline has produced 30,000 real-world training environments used by frontier labs. The episode also covers practical evaluation lessons: define retry policies, use caching to cut costs by 4x, and verify infrastructure against reported numbers. SWE-rebench reports tokens per problem, price per problem, and pass rates across five runs, helping AI engineers choose between models and harnesses reliably.

Building pi in a World of Slop — Mario Zechner

Apr 16, 2026 · 18:25

Mario Zechner explains why he built pi, a self-modifying, extensible agent core, after growing frustrated with Cloud Code and Open Code for their buggy features, uncontrolled context management, and zero extensibility. pi strips away complexity, giving the agent just four tools (read, write, edit, bash) and a minimal system prompt, letting it adapt to the user's workflow rather than vice versa. Extensions are TypeScript modules that hot-reload, enabling users to build custom tools, chat rooms, or even play Doom without forking pi. He then attacks clankers — agent-generated pull requests and issues — that are destroying open source, sharing his own tactics like auto-closing PRs with a human-voice request and de-prioritizing OpenClaw interactions. In Act 3, Zechner argues that agents compound errors with zero learning, producing enterprise-grade complexity in weeks, and that long context windows and agentic search are hacks. His prescription: scope agent tasks, modularize codebases, cap generated code review, and always read every line of critical code yourself.

Agents are Robots Too: What Self-Driving Taught Me About Building Agents — Jesse Hu, Abundant

Nov 24, 2025 · 17:37

Jesse Hu, former Waymo engineer and now founder of Abundant, argues that building reliable AI agents mirrors the challenges of self-driving cars, with the same 1% vs 99% problem where the model accounts for only 1% of the work. He draws parallels between robotics and agents across closed-loop feedback, statefulness, action spaces, and simulation, noting that just as self-driving pioneers learned perception was easy but planning was hard, agent builders must move from predictive models to action models. Hu explains how open-loop designs (e.g., waiting for full tool responses) limit agent reactivity, and advocates for richer input/output mechanisms like character-level terminal streams. He emphasizes the importance of an offline stack—simulation, evaluation, and data feedback loops—for iterative hill climbing, and warns that actions have consequences: out-of-distribution failures cascade. Finally, he recommends reading up on MDPs, DAgger, and offline RL from robotics literature to accelerate agent development.