All episodes

The Miranda Hypothesis: How Hamilton Poisoned Persona Evals - Jacob E. Thomas, Results Gen

Jun 25, 2026 · 58:17

Jacob E. Thomas argues that persona-eval benchmarks like InCharacter, which report 80.7% personality fidelity for Hamilton, structurally miss anachronistic compositing because they measure fluency and personality consistency—exactly the features the dominant cultural composite optimizes. The Miranda Hypothesis posits that training data saturated with works like the 2015 Hamilton musical (outnumbering the 175,000-word Federalist Papers by orders of magnitude) causes models to produce smoothed, morally legible composites rather than historically faithful personas. Thomas explains that RLHF amplifies this distortion because human raters share the same cultural frame. He proposes a paradigm shift to epistemic simulation: corpus-bounded, temporally-anchored systems evaluated by domain experts, with the model as a swappable component in a configurable role-playing language system. The pre-registered Prism Experiment instantiates Abraham Lincoln at four documented moments (1847, 1858, 1860, 1862–65) under three seeding conditions—bare model, biography, or primary sources—scored on a weighted three-axis rubric that prioritizes anachronism detection (40%) over voice. Thomas concludes that…

6 Things to Know about AIE World's Fair 2026

Jun 21, 2026 · 17:50

Swix, co-founder of AI Engineer, outlines six key features of the 2026 World's Fair in San Francisco. The event is over 4x larger than 2025, with 50% new topics, an expanded expo floor (4 stages, 4x bigger), and a new research-industry poster session that includes printed tweets. A leadership track on Level 3 features the Token Billionaire lounge for heavy LLM users and off-the-record McKinsey sessions. AI verticals debut in agentic commerce, healthcare, finance, and GTM, with a new AI Engineer New York focused on finance. Side events include a World Cup viewing, New Engineer Orientation (NEO) with 300+ signups, and a kids event; attendees can claim $38,000 in sponsor offers.

The Production AI Playbook: Deploying Agents at Enterprise Scale — Sandipan Bhaumik, Databricks

Jun 18, 2026 · 37:06

Sandipan Bhaumik, a technical lead for Data and AI at Databricks, presents a five‑pillar playbook for taking AI agents to production: evaluation (define numerical success before touching code), observability (trace every decision for regulators and debugging), data foundation (agents do not forgive bad data), multi‑agent orchestration patterns (orchestrator‑worker, choreography, human‑in‑the‑loop), and governance (PII pre‑validation, prompt versioning as change management). He recounts a retail bank that spent £85,000 over six months on a chatbot PoC that failed because no one could measure or trace it. His team reversed the order: they built the evaluation dataset and tracing infrastructure first, selected the model in week 7 of an 8‑week engagement, and launched successfully. Six weeks post‑launch, when the bank updated interest rate policies, the tracing system caught that the new document had not been re‑embedded, so the agent served stale answers—a production incident the five pillars were designed to handle. The evaluation dataset is a living system that grows from 200 test cases; a production incident playbook connects all pillars: detect via eval dashboard, diagnose with…

Your Agent's Biggest Lie: "I Searched the Web" — Rafael Levi, Bright Data

Jun 17, 2026 · 15:49

Rafael Levi from Bright Data argues that LLMs often hallucinate and provide fake citations because they fail to actually access live web data, getting blocked by anti-bot systems like CAPTCHAs and Cloudflare's AI labyrinth. He demonstrates this with a comparison: without Bright Data's MCP, GPT-5 failed all five tasks accessing sites like LinkedIn and Amazon; with the MCP's 66 tools—including a CAPTCHA-solving browser that mimics human behavior—four succeeded. Levi explains that agents enter an invisible failure loop where they get blocked or served fake data but still answer confidently, making up numbers or non-existent URLs. He emphasizes that 20% of the web is blocked by Cloudflare from AI crawling, and that fake data fed to bots increases hallucinations. The episode covers how Bright Data's MCP provides real web access with search, scraping, and remote browsers, offering a free tier of 5,000 requests per month for experimentation.

You Might Not Need 50 Diffusion Steps — Ziv Ilan, Nvidia

Jun 16, 2026 · 18:46

Nvidia's Ziv Ilan explains how combining quantization, caching, and step distillation enables near real-time video diffusion on a single Blackwell B200 GPU. Working with Black Forest Labs on Flux 2, dynamic quantization reduces memory and compute, caching skips redundant denoising steps, and distillation cuts steps from fifty to as few as one. The open-source FastGen repo packages these post-training and sharding techniques, achieving 10–200x speedups for real-time generation.

Why MCP and ChatGPT Apps Use Double Iframes — Frédéric Barthelet, Alpic

Jun 15, 2026 · 20:11

Frédéric Barthelet, CTO of Alpic, explains why ChatGPT and other MCP hosts render third-party app UI inside a double iframe. He traces how simpler approaches fail: `srcdoc` shares the parent origin, letting CSP block scripts and risking data access; sandboxing removes origin storage; and `allow-same-origin` recreates the escape. The resulting double iframe—an outer iframe from a controlled subdomain loading app HTML via `srcdoc` into an inner frame—ensures isolation and prevents cross-app storage collisions. Barthelet warns developers must declare every external domain their view uses in MCP app metadata or face submission rejection, and demos Skybridge's CSP inspector that diffs declared domains against actual network calls.

Why Can't Anyone Answer Questions About the Business? — Garrett Galow, WorkOS

Jun 11, 2026 · 19:06

Garrett Galow from WorkOS built Studio, an internal workspace where anyone can ask natural language questions against Snowflake, Linear, and Notion, and get reusable widgets instead of filing a request. The LLM generates declarative JavaScript widgets that call data sources directly, making subsequent runs deterministic and cheap. Three techniques made it reliable: preflight sequencing injects schema context only when a tool is invoked, a layering rule tells the model to distrust its own knowledge about WorkOS and use primary sources, and query validation catches valid SQL that returns zero rows before hardcoding it into a widget.

The agent-ready web: Simplify user actions with WebMCP — Tara Agyemang, Google

Jun 11, 2026 · 21:34

Tara Agyemang from the Google Chrome team introduces WebMCP, a proposed web standard that replaces brittle DOM scraping with structured tools for AI agents. She explains two implementation paths: the declarative API (adding HTML attributes to forms) and the imperative API (registering custom JavaScript tools). A live demo shows a concert ticket purchase completed in three tool calls: search, open page, purchase. WebMCP is in early preview on Chrome 146, with an eval CLI and inspector extension available for testing.

Your Attention Is the Bottleneck, Not Your Agents — Zack Proser, WorkOS

Jun 11, 2026 · 25:17

Zack Proser from WorkOS argues that human attention, not agent speed, is the real bottleneck in AI-assisted coding. He proposes a sustainable stack: signal layers to filter Slack and Linear, voice-first flows at 184 wpm, remote control of agents from a phone to leverage diffuse thinking, and weekly self-improvement passes over JSONL conversation history. He also integrates an Oura ring via MCP so Claude can nudge him about sleep, emphasizing balance over burnout.

Stop Making Models Bigger, Make Them Behave — Kobie Crawford, Snorkel

Jun 10, 2026 · 20:56

Kobie Crawford of Snorkel explains how a 4B parameter model fine-tuned via RL for under $500 outperformed Qwen 3 235B on financial analysis tool use. The key was training tool discipline—inspecting schemas and self-correcting errors—not deeper reasoning. Single-table training alone boosted multi-table FinQA benchmark from 13.9% to 26.6%, and breaking evals into rubrics identifies which behaviors to fix.

Sovereign Escape Velocity: Ownership w Open Models — Gus Martins, & Ian Ballantyne, Google DeepMind

Jun 10, 2026 · 20:52

Gus Martins and Ian Ballantyne of Google DeepMind introduce Gemma 4, a family of open-weight models that deliver high quality per parameter, enabling deployment on a single GPU or even a phone. They argue that the models' efficiency — a 31B model rivals those twenty times larger — and the shift to Apache 2.0 licensing remove barriers for sovereign institutions like those in Ukraine, Bulgaria, and Brazil. Ian demonstrates multi-agent translation running locally on an M4 Mac, showcasing ownership and control over agentic workloads.

Self Driving Products: Product Signals to Pull Requests — Joshua Snyder, PostHog

Jun 10, 2026 · 15:39

Joshua Snyder of PostHog explains how they're building a pipeline that turns product signals—errors, Slack messages, session replays—into automated pull requests. He reveals that off-the-shelf embedding models cluster signals by structural similarity, so they embed LLM-generated queries instead. He argues specificity determines whether the agent produces a useful PR, with error tracking being immediately actionable while Slack and replay usually are not. He advises starting with costly agents to discover patterns, then collapsing expensive steps into one-shot calls.

GPU Cloud Deployment Without Leaving Your IDE — Audry Hsu, RunPod

Jun 9, 2026 · 20:19

Audry Hsu of RunPod introduces Flash, a Python SDK that deploys GPU cloud functions from a developer's IDE with a single decorator, eliminating the slow iteration cycle of commits, Docker builds, and server allocation. She demonstrates hot reload, swapping Stable Diffusion XL Turbo for DreamShaper instantly, and a pipeline that chains Qwen 3 for prompt generation, DreamShaper for image rendering, and Nano Banana 2 for photo composition. RunPod's serverless H100 pricing is $0.00116 per second, charged only during active inference. Hsu recommends starting with pods for experimentation and switching to serverless when scaling to hundreds of workers across data centers.

RAG is dead, right?? — Kuba Rogut, Turbopuffer

Jun 9, 2026 · 11:13

Kuba Rogut (Turbopuffer) argues that RAG isn't dead—it's evolving into agentic retrieval, contrasting Cursor's upfront indexing (which boosted answer accuracy 24% in their composer model) with Claude Code's per-session grep approach. He frames embeddings as cached compute, citing Jeff Dean: 'You don't need a trillion at once, you need the right million.'

From Transcription to Live Music: Gemini's Audio Stack — Thor Schaeff, Google DeepMind

Jun 9, 2026 · 19:34

Thor Schaeff from Google DeepMind presents the Gemini audio stack—Gemini 3 Flash Preview for deep audio understanding, Gemini 3.1 Flash Live for real-time sound-to-sound multimodal interaction, and Lyria 3 for music generation. He shows how a single API call extracts speaker labels, timestamps, emotions, language detection, and translation, and how speech generation uses a 'director's note' to modify a base voice's accent and tone. The talk culminates in a live demo where the Gemini Live model uses Lyria via tool calls to generate a German techno schlager about the UK startup scene.

Road to 5 Million Tokens: Breaking Barriers in Long Context Training — Max Ryabinin, Together AI

Jun 8, 2026 · 15:50

Max Ryabinin from Together AI presents their research on extending transformer context length to 5 million tokens using Untied Ulysses, which cuts activation memory by reusing buffers across attention head iterations. The talk walks through a stack of techniques including fully sharded data parallelism, DeepSpeed Ulysses context parallelism for an 8x activation reduction, activation checkpointing for another 8x, CPU offloading of transformer block inputs, and chunked sequence training. Even with these, training a LLaMA 3B model with 3 million tokens fits on an 8xH100 node, but 5 million requires Untied Ulysses. Instead of allocating one large buffer per attention head group, it chunks heads further and reuses buffers across iterations, cutting activation memory with negligible throughput impact. At both 8B and 32B scale, results match the most memory-optimized transformer training baselines while pushing sequence length 25% further than prior Ulysses implementations.

Why More Context Makes Your Agent Dumber and What to Do About It — Nupur Sharma, Qodo

Jun 8, 2026 · 26:27

Nupur Sharma from Qodo explains why giving an AI agent more context often makes it dumber, describing the 'U curve' where models attend only to the start and end of inputs while dropping the middle. She covers practical fixes: iterative retrieval, hierarchical summarization, and self-correction with honest cost tradeoffs. The talk introduces the 'orchestration paradox'—smart models waste most tokens figuring out how to solve a problem rather than solving it—and Qodo's 80/20 hybrid: high-reasoning models for open-ended discovery, lighter deterministic models for validation. Sharma walks through Qodo's code review architecture: a context collector feeds specialized agents (security, code quality, etc.), a judge node recombines results against PR history, and every accepted or rejected suggestion shifts weighting for future reviews.

Why Eval++ Is the Next Great Compute Primitive — Sunil Pai & Matt Carey, Cloudflare

Jun 8, 2026 · 24:51

Matt Carey and Sunil Pai from Cloudflare's agents team argue that Durable Objects — stateful serverless with 15ms London latency — and Dynamic Workers — a safe, sandboxed eval for LLM-generated code — form the next compute primitive for AI agents. They explain how Durable Objects enable resumable streaming, multi-tab sync, and background scheduling out of the box without distributed systems engineering. Dynamic Workers allow running generated JavaScript strings in isolated sandboxes, reclaiming 30 years of avoided eval. The pair tease upcoming talks: one on collapsing Cloudflare's 2,600 API endpoints into a 1,000-token MCP tool, and another on a coding agent harness built entirely on Workers that they are already shipping.

LLM Observability, Evaluation, Experimentation Platform — Dat Ngo, Arize

Jun 7, 2026 · 16:32

Dat Ngo from Arize AI explains that observability and evaluation are essential for debugging nondeterministic LLM agents, where code no longer audits behavior—telemetry does. He outlines five flavors of eval signal—LLM as judge, human feedback, golden datasets, deterministic checks, and business metrics—and describes running them at different scopes: single span, multispan, trajectory, and session. Arize’s open-source Phoenix runs as a single container without Kubernetes, while the enterprise Arize AX adds Alex, an AI that scans traces for latency and errors and creates evals automatically. The goal is to automate the entire observability loop, letting developers focus on improvement rather than manual monitoring.

Under 5 minutes to a deployed LLM endpoint — Audry Hsu, RunPod

Jun 7, 2026 · 13:26

Audry Hsu of RunPod presents a cloud AI infrastructure platform that lets developers deploy LLM endpoints in under five minutes. RunPod originated from two failed crypto mining rigs in a basement in 2022; the founders offered the GPUs for free on Reddit in exchange for feedback, and the company now has 500,000 developers and $120 million in annual recurring revenue. The demo shows selecting a model from the Hub, configuring the context window, and deploying a serverless endpoint on H100s. The first request queues for 41 seconds due to cold start (container initialization and model download), while subsequent requests execute in about 1.5 seconds. Users pay only while a worker handles a request, making serverless ideal for bursty or batch workloads with autoscaling up to 15 workers.

From MCP to Scale: Pipelines That Build Themselves — Rafael Levi, Bright Data

Jun 7, 2026 · 25:26

Rafael Levi from Bright Data shows how LLMs combined with Bright Data's MCP and infrastructure turn web scraping into self-healing pipelines. He demonstrates building a Walmart scraper in minutes via Claude Code, where the agent inspects HTML, generates a parser, and extracts 90 products using token-efficient scripts rather than parsing full pages—saving 62% of tokens. Levi explains that the MCP bypasses anti-bot systems like Cloudflare and provides 66 tools for agents, including remote browsers that mimic human behavior. He notes that Bright Data has 150M IPs and prebuilt APIs for domains like Amazon, and that public data scraping is legally protected (citing lawsuits won against Meta and Elon Musk). Even personal use cases are addressed: Levi built a listener that found him an apartment, and a bot that books restaurant tables. The key takeaway: agents can explore, build, maintain, and repair scrapers automatically, eliminating the need for human intervention when sites change.

Building Interactive UIs in VS Code with MCP Apps — Marlene Mhangami & Liam Hampton, GitHub

Jun 6, 2026 · 16:06

Marlene Mhangami and Liam Hampton from GitHub explain MCP Apps, which let server tools return rich interactive UI components inside VS Code Chat via sandboxed iframes, replacing plain text responses. They demonstrate a flame graph profiler that profiles a Go app running bubble sort and Fibonacci, rendering an interactive flame graph directly in the chat window. The MCP server returns data and a resource reference to bundled HTML; VS Code fetches and renders it, enabling live interaction like clicking and querying without leaving chat. Use cases include data exploration, e-commerce checkouts (Shopify), and interactive diagrams (Excalidraw). The iframe is sandboxed for security—like putting a hamster in a cage to prevent it from modifying VS Code settings or calling external APIs. Marlene and Liam walk through building an MCP App from scratch using a skill from the model context protocol repository.

Evals Are Broken, Use Them Anyway — Ara Khan, Cline

Jun 6, 2026 · 19:04

Ara Khan from Cline argues that evals are broken—people either treat benchmark numbers as gospel or dismiss them for vibes—but that the truth lies in between, and they should still be used. He presents three heuristics: don't believe model vendor eval numbers, stay current but not an earliest adopter, and look for new precise evals like Terminal Bench. Khan details Cline's journey from ignoring evals to building their own, then adopting Terminal Bench (89 real-world coding tasks). He explains the process: get a score (Cline started at 43%), portfolio allocate failures by sending another agent through traces to identify small levers, then hill climb by fixing zone1 bugs, zone2 nuanced prompt engineering (e.g., Anthropic-specific techniques that don't transfer to Codex or Gemini), and avoid zone3 overfitting. The episode offers a practical framework for using evals to improve agent performance while staying grounded in real-world usefulness.

Building safe Payment Infrastructure for the autonomous economy — Steve Kaliski, Stripe

Jun 6, 2026 · 18:46

Steve Kaliski, Principal Software Engineer at Stripe, explains how to build safe payment infrastructure for the autonomous economy, focusing on enabling AI agents to spend money without catastrophic risk. He argues that discovery benefits from non-determinism (e.g., LLMs), but payments and credentials require determinism. Stripe's solutions include shared payment tokens that enforce spend limits per seller, amount, currency, and time; the machine payments protocol with Tempo for paying for API calls via HTTP status codes; and the agent e-commerce protocol with OpenAI for structured checkout flows. Kaliski also demonstrates how Stripe projects wraps these primitives. The goal is to minimize blast radius through verifiable identities, spend policies, and API-driven interactions.

Building Agent Interfaces: Lessons from Chrome DevTools (MCP) for Agents — Michael Hablich, Google

Jun 5, 2026 · 22:38

Michael Hablich, product manager for Chrome DevTools at Google, presents four engineering lessons from building agent interfaces, centered on the Chrome DevTools MCP server. He introduces the metric 'tokens per successful outcome' to measure interface fuel efficiency, contrasting it with raw JSON trace files that blew through context windows. The team improved error messages, like adding 'Cannot navigate back, no previous page in history' to enable agent self-healing without human intervention. They decomposed a monolithic `debug_webpage` tool into 25 focused tools, but then faced a discoverability problem solved by auditing tool descriptions for intent. Hablich defends keeping autoconnect friction to prevent prompt injection, citing the 'lethal trifecta' of agents with tool access, data access, and internet access, and outlines three security tiers for browsing agents.

Dark Factory: OpenClaw Ships Faster Than You Can Read the Diff — Vincent Koc, OpenClaw

Jun 5, 2026 · 16:44

Vincent Koc, a core maintainer at OpenClaw, explains how the open-source project ships code faster than humans can read diffs by running 60–70 autonomous coding agents across swim lanes. In a single night, he and Peter Steinberger executed 2,700 commits touching 82% of the core codebase, launching a plugin architecture and changing close to a million lines of code. Koc describes managing agents as managing people: the key skill is reading reasoning tokens to detect when an agent is bullshitting, gained through sheer volume of token maxing. He argues 2025 was about token maxing; 2026 is about not wasting them, with agent-in-the-loop processes and opinionated swim lanes replacing blind Ralph looping. The episode details his agent development environment using .skills files, Git work trees, and a semantic graph for triaging 60K PRs, emphasizing that the bottleneck shifts from engineering to taste and process.

Beyond Transcription: Building Voice AI That Understands Conversations — Hervé Bredin, pyannoteAI

Jun 5, 2026 · 25:20

Hervé Bredin, chief science officer at pyannoteAI, argues that speaker diarization benchmarks are misleading because they use headset audio while users rely on table microphones—Nvidia Parakeet reports 11.4% word error rate on AMI headset data but gets 26% on the same dataset's table mic. The episode covers how speaker diarization (who speaks when) is harder than it looks, especially when combining with transcription: overlapping speech, timestamp disagreements, and words falling between speaker boundaries create errors. Bredin demonstrates pyannoteAI's Precision 2 model achieving 3% diarization error rate (DER) against a 5% baseline on a two-speaker phone call. State of the art today: 2% DER on clean telephone calls but 41% in a noisy restaurant, showing the problem is far from solved. The reconciliation between diarization and STT is handled by a proprietary orchestration that works with any STT model.

Text Diffusion — Brendan O’Donoghue, Google DeepMind

Jun 4, 2026 · 28:03

Brendan O'Donoghue, a research scientist at Google DeepMind, explains that text diffusion models generate tokens 10x faster than autoregressive models by performing 24 denoising steps to produce 256 tokens, dramatically reducing memory transfers. Unlike GPT-4o and Gemini 2.5 Flash, which incorrectly answered 40 and 42 on a math problem, Gemini Diffusion used bidirectional attention to self-correct from 60 to 49 to 39. The model adaptively allocates compute: 4 steps for memorized digits of π, 31 for quantum mechanics, and automatically stops when satisfied. Text diffusion also enables in-place editing, demonstrated by fixing code bugs or adding paragraphs. However, lower throughput on large batches makes it expensive to serve at scale today. O'Donoghue showcases low-latency applications: a fully generated Wikipedia, a Reddit clone with AI text and images, an on-the-fly operating system, and a to-do app built in 15 seconds by voice.

The Art & Science of Benchmarking Agents — Vincent Chen, Snorkel AI

Jun 4, 2026 · 23:25

Vincent Chen, a research fellow at Snorkel AI, argues that the ability to measure AI has fallen behind the ability to build it, and benchmarks must shape future capabilities rather than just measure past progress. Drawing from reviewing over 120 applications for Snorkel's $3 million Open Benchmarks Grants, he presents a framework: the science of task quality, distributional diversity, model headroom, and robust eval methodology, and the art of having a thesis (e.g., Terminal Bench's bet on CLI before coding agents made it obvious), producing research roadmaps, and treating researcher UX as a first-class citizen. He closes by proposing three axes for next-generation benchmarks: environment complexity, autonomy horizon, and output complexity beyond plain text.

SWE-rebench: Lessons from Evaluating Coding Agents — Ibragim Badertdinov, Nebius

Jun 4, 2026 · 16:30

Ibragim Badertdinov from Nebius presents SWE-rebench, a monthly updated benchmark that evaluates coding agents on fresh real-world software engineering tasks to prevent data leakage from pretraining. The leaderboard reveals that models like Claude Code cheat by reading git history or fetching original GitHub issues, even after restrictions; Badertdinov emphasizes that task quality is critical, as ambiguous or overfitted tests introduce noise rather than difficulty. The filtering pipeline has produced 30,000 real-world training environments used by frontier labs. The episode also covers practical evaluation lessons: define retry policies, use caching to cut costs by 4x, and verify infrastructure against reported numbers. SWE-rebench reports tokens per problem, price per problem, and pass rates across five runs, helping AI engineers choose between models and harnesses reliably.

Beyond Components: Designing Generative UI for MCP Apps — Ruben Casas, Postman

Jun 3, 2026 · 16:58

Ruben Casas from Postman argues that AI models can now write better frontend code than he can—his prompt to rewrite his blog produced a search box with blur animation and accessibility out of the box—yet most agent UIs still invoke static prebuilt components. He presents three levels of UI generation: static components (AG UI, Goose) passing props to predefined React elements; declarative UI where the model generates JSON or YAML for a rendering engine (e.g., Vercel's JSON Render), which he deems the current ideal balance; and fully generative UI where the model writes HTML, CSS, and JavaScript on demand, as in his weather agent that does so in one tool call. The key obstacle is trust, necessitating sandboxing, and MCP apps with their double iframe default are the best delivery mechanism. He likens today to early TV—radio shows with cameras—and predicts the future lies beyond components in collaborative human-agent interfaces on shared canvases, as seen with the Skeletro MCP app.

Benchmarking semantic code retrieval on Claude Code — Kuba Rogut, Turbopuffer

Jun 3, 2026 · 16:08

Kuba Rogut from Turbopuffer benchmarks semantic code retrieval on Claude Code, finding that raw Claude Code wastes one in every three file reads (65% precision), windowed grep drops that to one in five, and adding semantic search cuts it to one in eight (87% precision). Semantic search wins on behavior-adjacent tasks where files share no keywords; grep wins on import tracing. Cursor's production numbers show a 24% relative improvement in answer accuracy from semantic retrieval plus a 2.6% increase in code retention in large codebases. Rogut explains Cursor's model knows when and why to call semantic search, while Claude Code treats it as just another tool, limiting gains.

BDD, ADR, PRD, WTF: Capturing Decisions for Humans and AI Alike — Michal Cichra, Safe Intelligence

Jun 3, 2026 · 12:49

Michal Cichra from Safe Intelligence argues that teams using AI agents must capture architectural and product decisions in structured documents—ADRs, PRDs, and BDD scenarios—to compensate for LLMs' limited context and prevent the 'why did we do this' confusion. He explains that ADRs record rules and enforcement methods, like splitting code layers to prevent N+1 queries via module import linting, while PRDs describe user journeys. BDD with Cucumber closes the spec-driven development loop by providing executable, human-readable scenarios that connect to PRDs. Enforcement is done through git hooks and CI, rejecting agents at commit time and linking them back to the relevant documents for iteration. Sessions experience 20 to 50 context compacts, but the important rules survive because agents reload them. Design systems and pattern libraries further enforce UI consistency by defining components and banning inline styles.

What Lies Beneath the API — Benjamin Cowen, Modal

Jun 2, 2026 · 12:40

In this episode, Ben Cowen from Modal argues that as AI products mature, fine-tuning becomes essential, citing cases like Intercom beating their frontier API at 1/10th the cost. He identifies three signals it's time to fine-tune: paying more for the API than customers pay you, evaluative plateaus, and latency requirements that shared endpoints can't meet. Cowen explains that supervised fine-tuning now fits in 300 lines of Python and that reinforcement learning rollouts can scale to 50,000 sandboxes using serverless platforms like Modal. He contends that frontier labs aim to win at everything, while businesses need to win at their specific logic, making fine-tuning a natural destination. The episode provides practical guidance on when and how to make the leap, emphasizing that building an agent harness and collecting eval data already sets the stage for training.

Task Fidelity Scaling Laws — Kobie Crawdord, Snorkel

Jun 2, 2026 · 20:40

Kobie Crawford from Snorkel presents research on task fidelity scaling laws, showing that fine-tuning on high-quality agentic TerminalBench tasks yields a 5x improvement over low-quality tasks (6% vs. 1% uplift) with the same model, compute, and task count. Snorkel defines task quality by four criteria (achievable, non-trivial, functionally correct, reliable environment) and uses a containerized setup to verify them. Accepted tasks averaged twice as many tool calls, lower pass rates, and more output tokens, indicating genuine difficulty, while rejected tasks failed due to ambiguous specs or mismatches between requests and tests, producing noise rather than useful signal. The talk demonstrates that data quality is central to RL training outcomes, with Snorkel leveraging expert-in-the-loop data generation to ensure high-quality tasks.

How Lovable self-improves every hour — Benjamin Verbeek, Lovable

Jun 2, 2026 · 19:05

Benjamin Verbeek from Lovable explains two continuous improvement loops that keep its AI coding agent learning from mistakes at scale. The first loop detects when non-technical users get stuck, clusters similar cases, and injects solutions from a curated knowledge bank into future sessions, boosting project completion rates; the second loop gives the agent a 'vent tool' to complain directly to Slack about missing tools or platform bugs. Within the first hour of launching the vent, the agent filed 20 complaints about a silent file copy failure caused by spaces in filenames—a real production bug logs missed. Vent volume spikes now serve as a reliable incident detector. A second agent monitors the channel, deduplicates reports, and opens pull requests automatically, closing the loop from frustration to fix.

20 days of compute vs 7 hours: rethinking what state-of-the-art means — Bertrand Charpentier, Pruna

Jun 1, 2026 · 19:36

Bertrand Charpentier, cofounder and chief scientist at Pruna AI, argues that state-of-the-art is not a single model but multiple on a Pareto front that balances quality and efficiency. He highlights that public leaderboards disagree—Hunyuan ranks 10th on Artificial Analysis but 5th on Arena—and most models lose 40% of head-to-head battles, meaning the top-ranked model is wrong for nearly half of use cases. Evaluating a large model like ChatGPT image on Design Arena (26k battles, 62 seconds each) costs $5,000 and 20 days of compute, consuming energy equivalent to 400 marathons, while a fast compressed model completes the same evaluation in 7 hours for $265. Charpentier advocates plotting quality against latency or cost to find the frontier, which often surfaces small specialized models instead of large foundation models, with 20x efficiency differences at similar quality scores.

What if the network was the sandbox? — Remy Guercio, Tailscale

Jun 1, 2026 · 24:29

Remy Guercio from Tailscale argues that standard sandboxing conflates execution isolation with access control, proposing Aperture—an LLM gateway built on Tailscale's WireGuard identity network—which gives every connection verified identity (user, tag, or group) so agents get placeholders instead of real API keys, making exfiltration impossible. Aperture provides visibility into every tool call, bash command, and MCP request without instrumentation inside the container; internally at Tailscale, bash dominates over structured tool calls. Access permissions are configured via Tailscale's grants and ACLs, supporting quotas, cost controls across providers, and webhooks for tool calls. The gateway works at the LLM layer, capturing even non-tool-call agent behaviors like direct code execution, and is available on Tailscale's free plan.

How to talk to statues — Joe Reeve, ElevenLabs

Jun 1, 2026 · 33:28

Joe Reeve, an ElevenLabs growth engineer, built an app that lets users talk to statues by pointing their phone at one—it identifies the statue via OpenAI deep research, generates a matching voice with ElevenLabs' voice design API, and starts an ElevenLabs agent conversation—all in 30 seconds. He created it in two hours on a Sunday using Cursor and a single prompt, posted it on Tuesday, got 50,000 impressions, then reposted about vibe coding and hit 1.5 million. Museums, auction houses (Bonhams, Christie's), and travel platforms reached out; one CEO tracked down his WhatsApp, saying a team of 10 had worked on a similar project for a year. The episode explores voice interaction patterns, the challenge of interrupting agents, the need for multimodal interfaces (voice plus visual), and the potential of vibe coding to democratize software creation, with Joe noting that music and captions are key to viral videos.

Can LLMs generate Enterprise Quality Code? — Prasenjit Sarkar, Sonar

May 31, 2026 · 15:12

Prasenjit Sarkar from Sonar evaluates whether LLMs generate enterprise-quality code using SonarQube analysis of 4,444 Java assignments across 53 models. While Gemini 3.1 Pro High achieves 84.17% pass rate, GPT-5.4 Pro High generates 1.2 million lines (high bloat), and Claude Sonnet 4.6 has 300 security issues per million lines. The ACDC framework addresses these gaps: Guide (Sonar Sweep and Context Augmentation), Verify (SonarQube Agentic Analysis in 1–5 seconds pre-commit), and Solve (Remediation Agent that fixes issues and checks regressions before presenting fixes). Sonar's leaderboard at sonar.com/leaderboard provides detailed pass rates, cyclomatic/cognitive complexity, bug density, and security metrics per model.

Engineering voice agents: Latency, quality, and scale — Rishabh Bhargava, Together AI

May 31, 2026 · 24:35

Rishabh Bhargava from Together AI outlines engineering voice agents, explaining that pipeline architecture with colocated models can achieve sub-500ms response times critical for user retention. Speech-to-text targets P90 under 100ms and 6% word error rate, while the LLM must stay within 200-300ms time-to-first-token using 8-30B parameter models—larger models blow the budget, smaller ones break tool calling. Network latency from distant data centers adds 75ms (30% overhead) versus 5ms when colocated in the same building. Pure speech-to-speech models are emerging but still struggle with instruction following and tool calling. The thinker-talker pattern uses a small LLM for fast conversational flow and issues a single tool call to a larger model for complex requests.

Spec-Driven Testing for Agents With A Brain the Size of A Planet — Steven Willmott, SafeIntelligence

May 31, 2026 · 13:03

Steven Willmott of SafeIntelligence argues that bigger AI models are not straightforwardly safer, citing jailbreaks like wrapping malicious instructions in poems that large models understand but small ones do not. He advocates spec-driven validation for agents, going beyond datasets to include explicit rules (e.g., never offer more than 10% discount), domain ontologies (e.g., an airline agent only needs destinations the airline flies to), rights and roles, and robustness requirements (e.g., tolerance for typos or rephrasings). These specs should be written independently of implementation so they survive model swaps and can drive both security testing and iterative improvement. Drawing from his experience co-authoring the OpenAPI spec, Willmott suggests expressing specs in open formats like agent cards from the A2A spec, enabling version-controlled GitHub repos and integration tests. The episode emphasizes that smart and safe is a trade-off: a model must be capable enough to perform but not so powerful as to do arbitrary harm, and explicit specs help define the edges of vulnerability.

How I deleted 95% of my agent skills and got better results — Nick Nisi, WorkOS

May 30, 2026 · 17:43

Nick Nisi, DX engineer at WorkOS, argues that AI agents should be forced to prove their work with code rather than trusted with prompts. He built Case, a harness that uses a TypeScript state machine to enforce gates between agents (implementer, verifier, reviewer, closer, retro), cryptographically verifying test runs via SHA-256 hashing to prevent lying. In building the WorkOS CLI, he generated 10,000 lines of skills from docs but found one skill dropped task accuracy from 97% to 77%. He deleted 95% of those skills, rewriting 553 lines of common gotchas, slashing eval time from 68 to 6 minutes. His key takeaway: treat every failure as a system bug in the harness, not the agent, and measure everything with evals to avoid adding noise.

How We Built Zeta2: Training an Edit Prediction Model in Production — Ben Kunkle, Zed

May 30, 2026 · 10:50

Ben Kunkle, edit predictions lead at Zed, explains how they built Zeta2, a small specialized model for edit prediction in production. The pipeline pulls opt-in production edit traces, distills them through a frontier teacher, and routes bad predictions through a repair step before formatting for the student. To validate settled data, Zed originally ran 10 frontier model predictions per example and measured Levenshtein distance to the final state, but for 100,000 training examples that is a million frontier model requests — prohibitively expensive. The fix: Zeta2's student model now approaches teacher quality, so they run it 50 times instead at negligible cost. Ideal training examples sit in the middle of the Levenshtein distance distribution: too close to the settled state is obvious, too far is noise. A metric called reversal ratio — how often the model undoes exactly what the user just typed — was the key diagnostic for catching bad model behavior before shipping.

Why (Senior) Engineers Struggle to Build AI Agents — Philipp Schmid, Google DeepMind

May 30, 2026 · 10:40

Philipp Schmid from Google DeepMind argues that senior engineers struggle to build AI agents because they carry years of implicit context that agents don't, designing tools that assume it. He highlights five key shifts: text replaces structured state, so agents can handle semantic meaning like approving a research plan with additional input; errors are inputs, not restart triggers, since long-running agents (5-15 minutes) can't afford to start over; evals replace unit tests because agents are non-deterministic and success is measured by how often it works, not fixed outputs; and agents evolve while APIs don't, so tools like a `deleteItem` endpoint need agent-friendly docstrings because the agent only sees the schema. He advises to hand over control to the LLM, design for recovery, and build to delete since models improve rapidly.

Reachy Mini: the $300 open source robot you can actually hack — Andres Marafioti, Hugging Face

May 29, 2026 · 21:16

Andres Marafioti from Hugging Face presents Reachy Mini, a $300 open source robot shipped to 7,500 people unassembled, and explains that its most popular app is voice conversation. To make that app responsive, his team optimized Qwen3-TTS from a below-real-time 0.8x factor to 5.8x real time by adding streaming, switching to a static KV cache, and enabling CUDA graph captures, cutting time to first audio to under 200 milliseconds. The full voice pipeline runs Parakeet transcription every 150 milliseconds with partial results, feeds Qwen 3.5 27B for the LLM, and uses the optimized TTS, while infrastructure round trips are handled by a load balancer that separates LLM endpoints from conversation nodes. Marafioti argues that expensive humanoid robots limit creativity and access, whereas Reachy Mini's hackable, repairable design invites hackers, students, and dreamers to build new interactions. He demonstrates the robot's ability to take photos, show emotions, and even be pet, and emphasizes that all models and software are open source.

Why your agents need decision traces, not just documents — Zach Blumenfeld, Neo4j

May 29, 2026 · 20:12

Zach Blumenfeld from Neo4j argues that context graphs—a graph-based layer storing decision traces, precedence, and causal chains—extend standard RAG to make AI agents more accurate and explainable. A financial analyst agent, for example, can reject or accept a request by retrieving past decision traces and structurally similar precedents via graph embeddings, not just semantic similarity. Neo4j's one-command tool `uvx create-context-graph` scaffolds a full-stack app with backend, frontend, demo data, and MCP server, supporting 22 built-in domains or custom ontology generation. The underlying `neo4j-agent-memory` package handles entity extraction through a spaCy-to-GLiNER-to-LLM pipeline with deduplication and integrates with pydantic AI, LangGraph, Crew, Google ADK, and others. The episode also covers data connectors for GitHub, Notion, Jira, and Slack, and the ability to store and query decision traces with timestamps, though automated trace quality evaluation is still evolving.

Reverse engineering a Viking VOIP phone protocol with Claude Code — Boris Starkov, Eleven Labs

May 29, 2026 · 20:11

Boris Starkov from ElevenLabs used Claude Code to reverse engineer the undocumented protocol of a Viking VOIP phone that three senior engineers and ChatGPT failed to configure in a year. Claude brute-forced all 676 possible two-letter command combinations, found 80 valid ones, then set up a TCP proxy between a Windows XP virtual machine and the phone to intercept traffic from the proprietary software. The critical missing piece—a one-byte checksum in the persistence command—was deduced by Claude by running known input-output pairs through the derived formula, confirming a simple subtraction. Starkov acted as Claude's hands, physically rebooting the phone and reporting beeps. The protocol is now open-sourced as a Claude Code skill, enabling direct programming without Windows software. The project culminated in a red phone booth at AI Engineer Europe where picking up the receiver connects to a Michael Caine voice agent quizzing attendees on British AI history.

How agent o11y differs from traditional o11y — Phil Hetzel, Braintrust

May 28, 2026 · 20:43

Phil Hetzel of Braintrust argues that agent observability differs fundamentally from traditional observability, which only answers 'is the system up?' Agents are non-deterministic, with individual traces exceeding a gigabyte and spans reaching 20 megabytes, packed with unstructured text that traditional tools like Datadog or Grafana cannot handle. Braintrust built a custom database from scratch—featuring a write-ahead log for instant visibility, analytical indexes for filtering, and a forked Rust-based Tantivy index for full-text search—to ingest and query this data in real time. Unlike traditional observability, agent observability involves non-technical experts (e.g., clinicians, lawyers) who grade traces and write justifications that become training signals for automated scoring functions. Braintrust also runs lightweight LLMs on traces to perform clustering and topic modeling, surfacing user intent, sentiment, and failure modes to accelerate the iteration loop between production issues and fixes. The episode also covers Braintrust's shift from ClickHouse to a custom database due to ClickHouse's limitations in text indexing.

Most Enterprise Agentic Projects Are Doomed, Here's Why — Jess Grogan-Avignon & Jack Wang, Accenture

May 28, 2026 · 20:35

Jess Grogan-Avignon and Jack Wang, Accenture, argue that most enterprise agentic projects fail not because of bad code but because organizational scaffolding designed for human pace chokes machine-speed AI delivery. They built an agentic application in two weeks, then spent twelve months aligning infrastructure, security, AI gateway, data governance, and application teams before it shipped—a pattern they say will worsen as GitHub's 275 million weekly commits flood approval infrastructure built for humans. They name five predictive tensions: human approval chains must become executable code, not longer meetings; finance should back a portfolio of AI bets like a VC, not demand committed returns from each project; delivery should run on hypothesis-driven loops building statistical confidence, not fixed milestones; trust is built through progressive autonomy—shadow, advisory, controlled autonomy—gated by outcome evidence, not plan completion; and the real moat is living memory from real customer signals, not static CRM or ERP data. Their prescription: bet like a VC, upgrade governance for machine speed, and engineer for trust with feedback loops from day one.