Rethinking Environments for Long-Horizon Work — Rayan Garg, Theta Software
Aug 1, 2026 · 21:15
Rayan Garg and Gurveer of Theta Software argue that progress on long-horizon AI agents depends on environment and verifier design, not benchmark headlines. They define long horizon via METR-style human time thresholds, such as 50% success on tasks taking humans 16 hours, alongside noisy model units like tokens and steps, and say real difficulty comes from sequential complexity—where a bad early query cascades—rather than artificially chained subtasks. They explain why soft verifiable work needs judge models with detailed rubrics, and why judges must be agents themselves: able to inspect final environment state (e.g. GitHub/CloudWatch logs) with read-only permissions, query long trajectories through sub-agents, and avoid collapsing valid solution spaces. They critique finance benchmarks GDPVal, ToolBench, and Apex agents for average human hours far below METR thresholds, saturation, narrow breadth, and weak reward signals. Theta's own finance tasks average 15 human hours across a 50-task sample, take models long trajectories, and still require mean@5 evaluation.
