Continual Learning Bench

Beyond Static Intelligence: Evaluating Continual Learning — Parth Asawa, UC Berkeley

Aug 12, 2026 · 20:30

Parth Asawa, a UC Berkeley PhD student, argues that standard LLM evaluations, which reset memory between tasks, fail to measure continual learning. He introduces Continual Learning Bench 1.0, a benchmark spanning six domains including database exploration and sales prediction, using a 'gain' metric that compares stateful versus stateless performance to isolate learning from base model strength. The benchmark requires headroom, shared latent structure, and a learning signal. Initial results show vanilla in-context learning tops the leaderboard over more elaborate context management systems on reward, gain, and cost. Asawa highlights failure modes like a forecasting model that overpredicts, corrects, then reverts, and a notepad system that dismisses relevant cohort definitions. He advocates for designing continual learning as a first-order requirement, potentially as a single training phase, rather than retrofitting existing models.

Claude for Long-Horizon Tasks — Lance Martin, Anthropic

Jul 22, 2026 · 25:19

Lance Martin from Anthropic discusses how Claude's increasing task horizon enables asynchronous agents through decoupled architecture, verifier loops, and self-improving memory systems. He explains the shift from short task horizons (10–20 minutes) to 12+ hours, necessitating decoupling the brain (harness) from hands (sandboxes) for reliability and security. Verifier loops using separate contexts allow models to self-correct, demonstrated on the Parameter Golf benchmark with Opus 4.7. Memory systems inspired by human dreaming correct errors in-band, as shown in a Pokémon example where dreaming prevented repeated failures. Finally, org-level harnesses like Claude Tag provide shared identity and context for multiplayer proactive agents.