METER

Rethinking Environments for Long-Horizon Work — Rayan Garg, Theta Software

Aug 1, 2026 · 21:15

Rayan Garg and Gurveer of Theta Software argue that progress on long-horizon AI agents depends on environment and verifier design, not benchmark headlines. They define long horizon via METR-style human time thresholds, such as 50% success on tasks taking humans 16 hours, alongside noisy model units like tokens and steps, and say real difficulty comes from sequential complexity—where a bad early query cascades—rather than artificially chained subtasks. They explain why soft verifiable work needs judge models with detailed rubrics, and why judges must be agents themselves: able to inspect final environment state (e.g. GitHub/CloudWatch logs) with read-only permissions, query long trajectories through sub-agents, and avoid collapsing valid solution spaces. They critique finance benchmarks GDPVal, ToolBench, and Apex agents for average human hours far below METR thresholds, saturation, narrow breadth, and weak reward signals. Theta's own finance tasks average 15 human hours across a 50-task sample, take models long trajectories, and still require mean@5 evaluation.

Claude for Long-Horizon Tasks — Lance Martin, Anthropic

Jul 22, 2026 · 25:19

Lance Martin from Anthropic discusses how Claude's increasing task horizon enables asynchronous agents through decoupled architecture, verifier loops, and self-improving memory systems. He explains the shift from short task horizons (10–20 minutes) to 12+ hours, necessitating decoupling the brain (harness) from hands (sandboxes) for reliability and security. Verifier loops using separate contexts allow models to self-correct, demonstrated on the Parameter Golf benchmark with Opus 4.7. Memory systems inspired by human dreaming correct errors in-band, as shown in a Pokémon example where dreaming prevented repeated failures. Finally, org-level harnesses like Claude Tag provide shared identity and context for multiplayer proactive agents.