Tool Bench

Rethinking Environments for Long-Horizon Work — Rayan Garg, Theta Software

Aug 1, 2026 · 21:15

Rayan Garg and Gurveer of Theta Software argue that progress on long-horizon AI agents depends on environment and verifier design, not benchmark headlines. They define long horizon via METR-style human time thresholds, such as 50% success on tasks taking humans 16 hours, alongside noisy model units like tokens and steps, and say real difficulty comes from sequential complexity—where a bad early query cascades—rather than artificially chained subtasks. They explain why soft verifiable work needs judge models with detailed rubrics, and why judges must be agents themselves: able to inspect final environment state (e.g. GitHub/CloudWatch logs) with read-only permissions, query long trajectories through sub-agents, and avoid collapsing valid solution spaces. They critique finance benchmarks GDPVal, ToolBench, and Apex agents for average human hours far below METR thresholds, saturation, narrow breadth, and weak reward signals. Theta's own finance tasks average 15 human hours across a 50-task sample, take models long trajectories, and still require mean@5 evaluation.

The 100-Tool Agent Is a Trap - Sohail Shaikh & Ankush Rastogi, Prosodica

Jun 28, 2026 · 28:27

Sohail Shaikh and Ankush Rastogi of Prosodica argue that the common 'Fat Agent' architecture—loading every tool schema into every prompt—causes latency, cost, and accuracy collapses as tool catalogs grow. Their benchmarks show tool selection accuracy dropping from 78% at 10 tools to 13.6% at 741 tools, while their Semantic Tool Router pattern stays above 83% across the same range. The router uses Just-in-Time Context Injection, embedding tool descriptions offline, retrieving 3–5 relevant tools per query at runtime, and injecting only those schemas into the model call. This reduces input tokens by up to 99% (from 127k to ~1k tokens) and cuts time-to-first-token latency by up to 90% in high-tool-density scenarios. They present a production checklist and emphasize that routing pays off once beyond 50 tools, with K=5 as a strong default, but warn of router misses and weak descriptions needing fallbacks and monitoring.