InstructGPT

What's Next After RLHF? — Diogo Almeida, TypeSafe AI

Jul 31, 2026 · 18:05

Diogo Almeida, a GPT-4 co-author and founder of TypeSafe AI, argues that RLHF optimized models for human approval, making them superb assistants but unreliable for autonomous work. He contrasts assistance with automation, noting RLHF's reward model encourages confident overpromising — like ChatGPT praising an audio file of farts as music. The next era is not Claude Code (still assistance-native), but real automation via RLVR-style methods focused on calibrated decision-making, echoing Sutton's bitter lesson that the task matters more than data. He defends pre-training as phenomenal, blaming post-training's asymmetric reward model for hallucinations. TypeSafe is rebuilding the AI stack for reliability and automation.

Scaling to Long Horizons — Ross Taylor & Chengxi Taylor, General Reasoning

Jul 31, 2026 · 18:07

Ross Taylor and Chengxi Taylor, co-founders of London-based RL company General Reasoning, argue that scaling to long horizons demands a mindset shift and better simulation, not just bigger context windows. Ross recounts how Galactica's 2022 thinking tokens were prescient, but its base-model demo backfired, while RLHF made LLMs products; his Meta team's PPO with verifiable rewards worked, but DeepSeek-R1 later showed better base models were key. Chengxi details long-horizon obstacles: sparse rewards, credit assignment, and 1M token limits, solved by value models that reduce variance and enable bootstrapping. Kelly Bench gave frontier models $100K to trade Premier League matches; all lost money, exposing how little current environments simulate real competition. Pipeline RL trades off off-policy staleness (8 steps okay) against GPU utilization, and they point to openreward.ai, with 350+ environments, for long-horizon RL.

Real AI Agents Need Planning, Not Just Prompting - Yuval Belfer

Jun 3, 2025 · 7:58

Yuval Belfer of AI21 Labs argues that LLMs alone still fail at instruction following, as shown by GPT-4.1's struggles in 2025, and that true AI agents require dynamic planning, not just prompting. He critiques ReAct for lacking look-ahead, contrasting it with AI21 Maestro's planner and smart execution engine, which uses best-of-n sampling, candidate pruning, and replanning. On AIfeval, Maestro pushes GPT-4.0, Claude Sonnet 3.5, and R3 Mini to near-perfect scores; on internal requirement satisfaction benchmarks, it improves over single LLM calls despite higher runtime and cost. Belfer advises starting simple (SLMs, ReAct) and escalating to planning only for complex tasks, inviting listeners to join the Maestro waitlist.

Building Blocks for LLM Systems & Products: Eugene Yan

Nov 2, 2023 · 17:24

Eugene Yan presents practical patterns for building LLM systems and products, emphasizing evals as the foundation. He argues that automated evals—starting with as few as 40 domain-specific questions—enable faster iteration and safer deployment, while academic benchmarks like MMLU may not fit real tasks. For retrieval-augmented generation, he warns that LLMs perform worse when the answer is in the middle of retrieved documents, and even with perfect retrieval, accuracy tops out at 75%. Guardrails against hallucination can be built via natural language inference at the sentence level or by sampling multiple summaries to check consistency. Finally, he highlights UX design—like Copilot’s accept/reject or Midjourney’s upscale/vary—as a way to collect implicit feedback that builds a data flywheel for evals and fine-tuning.