Sonnet

Anthropic's Applied AI team on the Evolution of Agentic Surfaces

Aug 11, 2026 · 31:24

Gagan Bhat and Isabella Kai He of Anthropic's Applied AI team explain how agentic surfaces evolved from the Messages API to the Claude Agent SDK and now Claude-managed agents, arguing that harnesses encode assumptions about model limits that go stale as models improve. They detail decoupling the brain (agent loop) from the hands (tool execution sandbox), which cut time to first token by 60% at P50 and over 90% at P95, and made failures recoverable via durable session logs. The session log also powers observability, context recovery, and 'dreaming,' a batch process that rewrites agent memory for self-improvement. They cover production lessons: keeping credentials in vaults, self-hosted sandboxes for VPC control, MCP tunnels for private servers, and the 'outcomes' feature that uses a grader agent to enforce success criteria.

Benchmarking Coding Agents on New vs Legacy Codebases — Denys Linkov, Wisedocs

Aug 8, 2026 · 18:08

Denys Linkov of Wisedocs, whose medical-claims ML pipeline ran across ten legacy repos, argues his team's six-month monorepo refactor was worthwhile even as coding agents improve fast. A refactor task that took o3 three hours and ten major mistakes now takes about one-fifth the time: Sonnet 4.6 needed one extra iteration, Opus 4.8 nearly one-shot it. Yet GPT 5.5 extra high 'completed' the job in 10 minutes 22 seconds, writing 2,000 lines of scaffolding with models missing and admitting no deployment or bootstrap command. So Linkov reads METR's task-length curve at 80-90% success, not 50%; an hour-long agent run at coin-flip odds wastes the hour and your attention. The payoff was social too: commit velocity never flattened, months-long features ship in under a week, and developers now volunteer across the monorepo.

Claude for Long-Horizon Tasks — Lance Martin, Anthropic

Jul 22, 2026 · 25:19

Lance Martin from Anthropic discusses how Claude's increasing task horizon enables asynchronous agents through decoupled architecture, verifier loops, and self-improving memory systems. He explains the shift from short task horizons (10–20 minutes) to 12+ hours, necessitating decoupling the brain (harness) from hands (sandboxes) for reliability and security. Verifier loops using separate contexts allow models to self-correct, demonstrated on the Parameter Golf benchmark with Opus 4.7. Memory systems inspired by human dreaming correct errors in-band, as shown in a Pokémon example where dreaming prevented repeated failures. Finally, org-level harnesses like Claude Tag provide shared identity and context for multiplayer proactive agents.

Claude Fable, Claude Tag, and Anthropic's Culture — Cat Wu & Thariq Shihipar ft Simon Willison

Jul 15, 2026 · 51:30

This episode features Anthropic's Cat Wu and Thariq Shihipar discussing Claude Code, Claude Tag, and Claude Fable, arguing that these tools have shifted engineering from slow spec-driven processes to rapid, ambitious building. Thariq notes that with each model generation, delegation increased, and Claude Fable now enables one-shot features. Cat says engineers now need product taste over execution, as timelines shrink from six months to a week. Claude Tag, a proactive multiplayer agent, lands 65% of product engineering PRs internally by monitoring channels and remembering preferences. The team reduced Claude Code's system prompt by 80% for frontier models by removing examples and hard constraints, relying on model judgment. Auto mode, used internally since January, mitigates prompt injection through thousands of evals and Sonnet classifiers. Cultural hacks include default-public channels and a 'don't negotiate against ourselves' mindset, leading to ambitious builds like Thariq's Claude-powered video editing and a Street Fighter game.

State of the Union: Why Local, Why Now — NVIDIA, Osmantic, Roboflow, EXO Labs, @matthew_berman

Jul 11, 2026 · 44:29

Nader Khalil (NVIDIA), Joseph Nelson (Roboflow), Alex Cheema (Exo Labs), Matthew Berman, and Ahmad Osman (Osmantic, r/LocalLLaMA) argue that local AI is now useful, driven by stronger open models and better hardware. They cite inflection points like Llama 2, DeepSeek v3, and GLM 5.2, which closed the gap with frontier cloud models. Sovereignty and control are key: enterprises need to choose their own model versions and avoid lock-in. Specialized models, such as Roboflow's fine-tuned vision models for deep-sea fish discovery, outperform general ones for specific tasks. Optimization is critical: EXO Labs achieved 10x performance on the DGX Spark by tuning existing NVIDIA kernels. The panel emphasizes that simplicity remains a barrier—most users need point-and-click solutions—and advocates for open-source AI to ensure freedom and innovation.

Everything we knew about software has changed — Theo Browne, @t3dotgg ​

Jul 8, 2026 · 16:02

Theo Browne argues that AI model evolution—from Sonnet 3.5’s tool-calling to Opus 4.5's long-running tasks and Mythos's orchestration—requires engineers to think bigger and wider. He compares current developer habits to skeuomorphism in iOS 7, urging rejection of legacy constraints like Git's inability to commit environment files and terminal-centric workflows. Browne introduces a shifted tier system: what was a startup is now a side project; a Markdown file running on a cron job can replace a company's product. His own PR triage service became a Markdown file updated daily via cron. He advocates building breadth over depth—architecting products so users can extend features, enabling small teams to compete with AWS or Salesforce. 'If your idea doesn't feel stupid, it's because your idea is not big enough,' he concludes.

We Cut 94% of AI Coding Tokens With a Local Code Index - Rajkumar Sakthivel, Tesco

Jun 28, 2026 · 10:43

Rajkumar Sakthivel and his friend Faz built Code Context Engine (CCE) to cut AI coding costs after their bill jumped from £15 to £200 in one month. They found that 45,000 tokens were sent per query but only 5,000 were needed, with 90% of cost being input context. Their solution is a local retrieval layer that uses AST-aware chunks, combined vector and keyword search, and a weighted scoring heuristic (50% meaning, 30% keyword, 20% recency) that runs in 0.4 milliseconds. On a FastAPI test, tokens per question dropped from 83k to 4.9k—a 94% reduction—with 90% accuracy. They emphasize that fixing the input, not the model, yields the biggest savings, and their open-source tool shares a single index across Claude Code, Cursor, Copilot, and Codex.

Task Fidelity Scaling Laws — Kobie Crawdord, Snorkel

Jun 2, 2026 · 20:40

Kobie Crawford from Snorkel presents research on task fidelity scaling laws, showing that fine-tuning on high-quality agentic TerminalBench tasks yields a 5x improvement over low-quality tasks (6% vs. 1% uplift) with the same model, compute, and task count. Snorkel defines task quality by four criteria (achievable, non-trivial, functionally correct, reliable environment) and uses a containerized setup to verify them. Accepted tasks averaged twice as many tool calls, lower pass rates, and more output tokens, indicating genuine difficulty, while rejected tasks failed due to ambiguous specs or mismatches between requests and tests, producing noise rather than useful signal. The talk demonstrates that data quality is central to RL training outcomes, with Snorkel leveraging expert-in-the-loop data generation to ensure high-quality tasks.

Agentic Evaluations at Scale, For Everybody — Nicholas Kang & Michael Aaron, Google DeepMind

May 25, 2026 · 20:03

Nicholas Kang and Michael Aaron from Google DeepMind's Kaggle team argue that AI evaluations are broken due to being scattered, stale, and lacking transparency, citing a competing lab publishing inflated results by using custom compaction settings. They introduce four solutions: hackathons to channel community expertise, a standardized agent exam that returned 500+ submissions in its first week without promotion, a Game Arena where models play poker, chess, and werewolf for an ELO rating that cannot saturate, and an open benchmarks platform. A wastewater treatment plant engineer in Turkey built a novel safety benchmark from 20 years of field experience. They note that on SWE-Bench Pro, six frontier models land within a couple of percentage points, but the harness shifts performance by 22%, complicating comparisons. Challenges include high cost (400,000 poker hands for statistical significance), maintaining community engagement, and dealing with fast model deprecation cycles.

Fast Models Need Slow Developers — Sarah Chieng, Cerebras

May 22, 2026 · 18:02

Sarah Chieng from Cerebras argues that the 20x speed increase of models like Codex Spark (1,200 tokens/sec vs. 40-60 for Sonnet/Opus) forces developers to rethink workflows, or risk generating technical debt at unprecedented scale. She presents a practical playbook: validation and linting become free at every step, so must run continuously; developers can generate 75 component variations across five sub-agents and cherry-pick the best; and with context filling in 30 seconds instead of ten minutes, a four-file external memory system (agents, plan, progress, verify) maintains continuity between sessions. Chieng emphasizes real-time collaboration with the model rather than spawning agents and walking away, and advocates using a slower planner model with a fast executor to orchestrate agents effectively.

How Building with AI Can Double the Throughput of Your Engineering Team — Brian Scanlan, Intercom

May 15, 2026 · 21:49

Brian Scanlan of Intercom explains how treating Claude Code like a new hire—onboarding it to a 15-year-old Rails monolith, writing skills for every recurring task, and connecting it to production systems—doubled engineering PR throughput in under a year. By going all in on one platform instead of letting everyone choose their own tool, Intercom achieved a 17.6% automatic PR approval rate with SOC 2 sign-off, and its CI infrastructure collapsed under the volume. The key principle: give agents problems, not tasks. Scanlan recounts a security incident where Claude automatically pulled files, ran analysis, and handed back next steps in two minutes using a skill he didn't know existed. The talk emphasizes that all engineering work—debugging, testing, planning—should be agent-first, and that companies must invest in platform-level adoption and continuous skill improvement.

Lessons from Trillion Token Deployments at Fortune 500s — Alessandro Cappelli, Adaptive ML

May 12, 2026 · 18:35

Alessandro Cappelli, co-founder of Adaptive ML, argues that 95% of GenAI pilots fail to reach production because they rely on proprietary models or instruction fine-tuning, which lack systematic feedback integration. Reinforcement learning (RL) is the only post-training technique that mathematically incorporates defects, business metrics, and production signals to continuously improve models. RL enables smaller, cheaper, faster models that enterprises can own—critical for scaling use cases like AT&T’s transcript summarization or Manulife’s agents. For agents, RL naturally fits because it was designed for environments; synthetic data is generated as a byproduct of environment training, not a prerequisite. Reward signals come from business KPIs (e.g., containment rate) or LLM judges defined by human rubrics in hours, not weeks. Adaptive ML’s Adaptive Engine abstracts away RL complexity (orchestrating four models for PPO) and provides pre-built recipes to industrialize model deployment.

Viktor: AI Coworker That Lives in Slack — Fryderyk Wiatrowski

May 11, 2026 · 19:30

Fryderyk Wiatrowski, co-founder of Viktor, explains how his AI employee lives in Slack—no web UI—participating in channels and threads like a teammate, inheriting integrations from whoever connected them first, and handling tasks that take ten minutes. He details the challenges of scaling a personal agent to a company agent: memory management across hundreds of users, managing Slack's complex input surface (threads, DMs, edits, emoji reactions), and preventing context leakage between channels. He shares that swapping the underlying model from Opus to GPT-5.4 caused user backlash due to personality differences, and describes the need to earn proactivity trust to avoid security alarms. Viktor's advantage is shared context—only one person needs to connect an integration for the whole team—but warns against giving it personal email access, as illustrated by a customer story. The episode argues that a great AI coworker requires helping get work done, knowing the company context, and being friendly.

Vibe Engineering Effect Apps — Michael Arnaldi, Effectful

May 7, 2026 · 1:43:04

Michael Arnaldi demonstrates that cloning the Effect library's repository into a project, rather than relying on prompts, is the most effective way to give coding agents the context they need to build reliably with Effect. Starting from an empty repo, he sets up a Bun, Vitest, and TypeScript project, adds the Effect repo as a git subtree, and creates an agents.md to guide the agent. He then uses the agent to research patterns, resulting in a fully functional Todo HTTP API with OpenAPI docs, SQLite persistence, and tests — all from scratch in under two hours. The workshop stresses that treating the library code as part of the project, combined with strict diagnostics and pattern files, makes agents effective even in unfamiliar codebases. Arnaldi also discusses the importance of workflow systems like Effect Cluster for long-running AI processes, where server failures become likely.

Full Walkthrough: Workflow for AI Coding — Matt Pocock

Apr 24, 2026 · 1:36:30

Matt Pocock presents a hands-on workshop on building a full AI-assisted coding workflow, arguing that software engineering fundamentals—not hype—make agents effective. He introduces the 'smart zone' and 'dumb zone' of LLMs (performance drops after ~100k tokens) and the 'Memento problem' (agents forget between sessions). His process starts with a 'Grill Me' skill that relentlessly questions the user until shared understanding is reached, then produces a PRD without reading it, slices work into vertical 'tracer bullet' issues, and runs agents AFK using TDD. He advocates designing codebases with deep, testable modules and shows Sandcastle, a TypeScript library for parallel agent execution with separate implementer (Sonnet) and reviewer (Opus). The workshop transforms ambiguous briefs into shippable features while keeping humans in the loop for QA and taste.

From Vibe Coding To Vibe Engineering – Kitze, Sizzy

Dec 14, 2025 · 25:28

Kitze, founder of Sizzy, argues that "vibe engineering"—actively steering AI agents with technical knowledge—trumps passive "vibe coding" for building real software. He shares how Cursor's Composer 1 let him port Benji.so and Glink to Next.js 16 with Monorepo in under a week, reviving near-dead projects. Kitze insists LLMs excel at React because humans are bad at it, and that repetitive code is fine since agents don't care. He warns against giving AI tools to juniors without oversight, and predicts the bottom of the job market will thin as agents replace interns. The talk ends with a pitch for "vibe code fixers" as a new role, maintaining legacy systems like cowboy coders.

Hard Won Lessons from Building Effective AI Coding Agents – Nik Pash, Cline

Dec 12, 2025 · 14:18

Nik Pash, head of AI at Cline, argues that frontier models have made clever agent scaffolds obsolete—capability now beats engineering tricks. He insists that real model improvement comes from benchmarks and RL environments, not from RAG or indexing systems. Pash details Cline's RL environment factory, which converts real-world coding tasks into training data by qualifying tasks, reconstructing environments, and defining pure outcome verifiers. He announces Clinebench, an open-source benchmark built from actual software development captured via Cline's provider, designed to measure and improve models on real tasks rather than synthetic puzzles. Pash urges the community to contribute by using Cline on open-source projects, turning model struggles into benchmark candidates.

Building Cursor Composer – Lee Robinson, Cursor

Dec 2, 2025 · 15:36

Lee Robinson explains how Cursor built Composer, its first agent model for real-world software engineering, by focusing on being both fast and smart — achieving 4x more efficient token generation than similarly intelligent models while matching open-source performance initially and approaching frontier models after reinforcement learning. The model's training posed infrastructure challenges: matching training and inference environments across thousands of GPUs, handling complex rollouts with hundreds of tool calls and up to millions of tokens, and ensuring consistency by using the same tool format and responses as production. Cursor solved these with custom kernels that sped up training by 3.5x on NVIDIA Blackwell chips for mixture-of-experts layers, load balancing across threads to avoid idle time, and co-designing RL infrastructure with its Cloud Agents product using virtual machines that mirror the production Cursor environment. This allowed the model to become a power user of tools like semantic search, which improved all models but especially Composer. RL also taught the model to parallelize tool calls (e.g., reading 10 files simultaneously) and to search more before editing,…

The Next Unicorns: 7 Top AI startups from the HF0 Residency

Aug 21, 2025 · 22:16

Diego Rodriguez (Krea) presents an AI creative suite that generated 1M images/day for a Fox ad, while OpenHome debuts the first AI smart speaker with 10K developers and 500 free dev kits. Josh’s Coframe made $20M for a travel firm by making websites adaptive, and Eugene’s Featherless AI built QWERTY 72B without transformer attention, claiming scale is dead in favor of reliability. Jonas Bauer’s Upside uses LLMs to structure enterprise data, Lengyue’s OpenAudio introduces S1, the first instructable voice model beating ElevenLabs, and Alex Atallah’s OpenRouter provides a single API for all LLMs, growing 10–100% monthly.

Real World Development with GitHub Copilot and VS Code — Harald Kirschner, Christopher Harrison

Aug 3, 2025 · 1:19:33

Harald Kirschner demonstrates three stages of Vibe Coding with GitHub Copilot and VS Code—YOLO, Structured, and Spectrum—arguing that developers can move from rapid prototyping to maintainable, scalable AI-assisted development. He shows how to use Agent mode, auto-approve settings, and new workspace scaffolding to build a hydration tracking app with React Vite and Material Design without looking at code. The talk covers custom instructions, MCP servers (Playwright, Jest Pad, Perplexity), and custom modes like TDD that enforce test-driven development by writing failing tests first. Kirschner explains how to constrain tools for deterministic behavior using tool sets and recommends committing often, pausing AI when needed, and iterating on instructions. He also discusses spec-driven development, critiquing specs with AI, and the importance of well-structured codebases for AI productivity.

Stateful environments for vertical agents — Josh Purtell, Synth Labs

Jul 22, 2025 · 6:51

Josh Purtell of Synth Labs argues that stateful environments—containerized, network-bounded workspaces that capture external state—make it easier to build effective agents for vertical applications like finance and health. By keeping the application logic separate from the agent, developers can revamp their agent when new models come out without rewriting everything. The environment exposes a tailored representation (e.g., just the terminal, not the whole OS) and supports resets and rollbacks, enabling techniques like tree search that improve long-horizon tasks. Purtell also notes that network boundaries allow reliable multi-agent setups and asynchronous work. The concepts are implemented in the open-source Synth AI Environments repository.

The emerging skillset of wielding coding agents — Beyang Liu, Sourcegraph / Amp

Jun 30, 2025 · 35:06

Beyang Liu, CTO and co-founder of Sourcegraph, argues that coding agents are a real and high-ceiling skill, contrary to skeptics like Jonathan Blow and Eric S. Raymond. He presents design decisions for the agentic era: agents should directly edit files instead of asking permission, UIs should be minimal (like Amp's bare-bones VS Code extension and CLI), and fixed pricing should give way to usage-based models. In a live demo, Liu uses Amp to implement a custom icon for the linear connector in Amp's own codebase, demonstrating agentic search and sub-agents. He shares power-user patterns, including writing long prompts, constructing feedback loops with Playwright and Storybook, and running multiple agents in parallel—as exemplified by Jeff Huntley using Amp to build a compiler while sleeping. Liu emphasizes that agents should enable more thorough code reviews, not replace human understanding.

How to build world-class AI products — Sarah Sachs (AI lead @ Notion) & Carlos Esteban (Braintrust)

Jun 27, 2025 · 1:43:46

Sarah Sachs (AI lead at Notion) and Carlos Esteban (Braintrust) explain that building great AI products requires 10% prompting and 90% evals and observability, with Notion AI using Braintrust to iterate on prompts and models. Sachs details their cycle: curate small datasets, tie them to scoring functions (LLM-as-a-judge with per-sample prompts and heuristic checks), run evals before shipping, and use production logs to catch regressions. She notes Notion AI supports 100M+ users, switches models in under a day, and 60% of enterprise users are non-English—requiring multilingual eval rigor. Carlos and Doug then walk through Braintrust's framework: tasks (prompts, tools, agents), datasets, and scores (0-1). They demonstrate offline evals via playground and SDK, online scoring in production, user feedback capture, human review setup, and remote evals to bridge complex code with the playground. The workshop covers moving from pre-prod evals to production monitoring, closing the feedback loop by adding underperforming spans to datasets.

[Full Workshop] Vibe Coding at Scale: Customizing AI Assistants for Enterprise Environments

Jun 27, 2025 · 1:20:38

Harold from the VS Code team demonstrates "Vibe Coding at Scale" at the AI Engineer World's Fair, presenting three stages—YOLO, structured, and spectrum vibes—for customizing AI assistants in enterprise environments. He live-codes a hydration tracking app using GitHub Copilot's agent mode, auto-approve, and new workspace scaffolding, then shows how to enforce design principles via Copilot instructions and reusable prompts. Harold introduces custom modes (e.g., TDD mode), tool sets, and MCP servers like Playwright and Perplexity for research and browser testing. He emphasizes iterating on instructions, committing often, and using spec-driven development to balance speed with reliability, arguing these techniques enable true flow-state collaboration even on complex codebases.

Building AI Agents that actually automate Knowledge Work - Jerry Liu, LlamaIndex

Jun 24, 2025 · 17:57

Jerry Liu, CEO of LlamaIndex, argues that building AI agents which actually automate knowledge work requires a combination of really good tools and carefully tailored agent reasoning, moving beyond naive RAG to a 'document toolbox' that includes parsing, extraction, indexing, and manipulation of complex unstructured data like PDFs, Excel, and PowerPoints. He introduces two main agent categories: assistive agents (chat-based, human-in-the-loop, unconstrained reasoning) and automation agents (batch-processing, constrained workflows, less human oversight). A key new capability is an Excel agent that uses reinforcement learning to learn a semantic map of unnormalized spreadsheets, achieving 95% accuracy on data transformation, surpassing the 75% baseline of LLMs with code interpreter and even human baselines of 90%. Real-world examples include financial due diligence combining automation (inhaling financial data) and assistant interfaces (analyst co-pilot), enterprise search with specialized agentic RAG, and technical data sheet injection for a global electronics company that reduced weeks of manual work to an automated extraction pipeline. Liu emphasizes that the right preprocessing…

The Web Browser Is All You Need - Paul Klein IV, Browserbase

Jun 20, 2025 · 17:31

Paul Klein IV, founder of Browserbase, argues that the web browser is the default MCP server for the rest of the internet, serving as the essential bridge between AI agents and legacy websites that lack APIs. He distinguishes between web agents (one prompt to many actions, e.g., OpenAI's Operator) and browser tools (one action per prompt), noting that both rely on vision-driven or text-based approaches to parse pages. Klein explains that Browserbase offers a horizontal MCP server for browsing, enabling automation of any website, and stresses the need for custom evals and observability to track agent behavior. In a live demo, he shows an agent navigating sfpca.org to find a dog for adoption, handling unexpected modals. He also addresses captchas, advising good citizenship and hinting at future agent authentication solutions.

MCPs are Boring (or: Why we are losing the Sparkle of LLMs) - Manuel Odendahl

Jun 10, 2025 · 28:32

Manuel Odendahl argues that MCPs (Model Context Protocols) are boring because they constrain LLMs to rigid, predefined tool calls, while LLMs truly shine as code generators that can create dynamic tools on the fly. He critiques the inefficiency of tool calling—repeating context, wasting tokens, and struggling with many tools—and advocates for an 'Eval' approach where the LLM writes and executes code (e.g., SQL, JavaScript) to solve tasks directly. Demonstrating with a JavaScript sandbox, he shows how a single Eval tool can introspect databases, write queries, create REST APIs, and even build a full CRM interface in one call, saving time and tokens. Odendahl urges engineers to think recursively: ask the LLM to write code that writes more code, unlocking infinite loops of creation and restoring the sparkle of LLMs beyond boring function calls.

Agents reported thousands of bugs, how many were real? - Ian Butler and Nick Gregory

Jun 3, 2025 · 18:39

Ian Butler and Nick Gregory introduce SM-100, a benchmark of 100 real-world bugs across 84 repositories, and present their agent Bismuth as outperforming existing agents in detecting complex bugs—finding 10 needles vs 7 for the next best. However, even top performers struggle: Bismuth's true positive rate is only 25%, and Codex leads at 45%, while most agents (Cursor, Debian) report thousands of false positives (97% for basic loops). They argue that agents universally exhibit narrow thinking, missing simple bugs like a form state issue that only Bismuth and Codex caught, and that high SWE-bench scores do not translate to maintenance tasks. The episode concludes that software maintenance remains an unsolved problem requiring deeper reasoning and targeted search.

Stateful Agents — Full Workshop with Charles Packer of Letta and MemGPT

Apr 19, 2025 · 1:19:34

Charles Packer, lead author of the MemGPT paper and co-founder of Letta, argues that statefulness (memory) is the most important problem to solve for building useful AI agents, since LLMs are inherently stateless transformers. He presents MemGPT's LMOS (Language Model Operating System) approach, which treats memory management as a context compilation problem solved by the LLM itself using tool calling to read and write structured memory blocks. The workshop demonstrates Letta's open-source stack (FastAPI, Postgres, Python) where agents persist state on a server, enabling long-running, learning interactions without context overflow—e.g., an agent can update its core memory (e.g., correcting a user's name) or search archival memory (e.g., recalling user preferences after a reset). Packer also shows multi-agent communication via async message passing between agents running as independent services, and highlights that tools are sandboxed by default with support for Composio integrations. The session includes a live notebook exercise and a low-code UI (ADE) showing context window management and memory editing, emphasizing that true stateful agents improve continuously over time rather…

Keynote: Why people think "agent" is a buzzword but it isn't

Feb 22, 2025 · 28:07

Chip Huyen argues that agents are not a buzzword but a practical yet hard technology, facing three core challenges: the curve of complexity, tool use translation, and context management. Even simple queries require multiple steps, and models' success rates drop rapidly past 5 steps—newer reasoning models like DeepSeek R1 are pushing the boundary, but most still fail after 10. Tool use requires translating ambiguous natural language to precise API calls, worsened by poor documentation; she advises narrow functions and asking for clarifications. Context is another bottleneck: agents must juggle instructions, tool docs, and outputs, often exceeding a model's efficient context (many hallucinate beyond 30K tokens), necessitating external memory like RAG. Her benchmark shows planning-specialized models struggle with long context and vice versa, and she recommends breaking tasks into subtasks and using test-time compute scaling.

Realtime Data Connectivity for AI: Tanmai Gopal

Oct 11, 2024 · 7:14

Tanmai Gopal from Hasura introduces Pacha DDN, an AI-powered data access layer that lets LLMs securely query live data from multiple sources, arguing that the key is treating all data—structured, unstructured, and APIs—with a unified SQL-based query language. He demonstrates with a Blockbuster example: writing an email to a top customer by querying database transactions and recent rentals via natural language. The system uses an object model for authorization that applies rules based on data schema and session properties, regardless of data origin. To overcome LLM reasoning limitations, Pacha DDN asks the LLM to write Python code to retrieve data instead of reasoning directly. The talk concludes that for AI to be useful, it needs realtime data access, and Pacha DDN provides a secure, explainable query planner to achieve that.

Building Reliable Agentic Systems: Eno Reyes

Aug 20, 2024 · 18:14

Eno Reyes, CTO of Factory.ai, explains how his team builds reliable agentic systems—called Droids—for automating software development tasks by applying techniques from robotics and control systems to handle planning, decision making, and environmental grounding. Reyes describes a pseudo-Kalman filter that passes intermediate reasoning through plan steps, converging reasoning but risking error propagation. He advocates for explicit plan criteria and hard-coded logic to improve reliability, despite reducing generalizability. On decision making, he recommends consensus mechanisms like self-consistency, explicit reasoning with checklists, fine-tuning for out-of-distribution decisions, and simulation via Monte Carlo tree search. For environmental grounding, he emphasizes building custom AI computer interfaces for domain-specific workflows, designing explicit feedback processing (e.g., parsing CI/CD logs), balancing bounded exploration with long-context models, and incorporating human guidance to boost reliability from 30–40% to 90–100%.

Lessons From A Year Building With LLMs

Jul 19, 2024 · 35:21

The six authors of the O'Reilly article "Lessons From A Year Building With LLMs" — Bryan Bischof, Jason Liu, Hamel Husain, Eugene Yan, Shreya Shankar, and Swix — argue that the model itself is not a moat and that success comes from continuous improvement centered on evals and data. They stress that AI engineers should treat models as SaaS, quickly swapping for better ones, and focus on product and user interactions. The talk warns against toxic practices like prematurely hiring ML engineers without data or blindly adopting tools, advocating instead for deliberate eval practice and data literacy. On tactics, they compare LLM-as-judge (quick to prototype) vs fine-tuned evaluators (more precise and faster), and emphasize looking at real user data regularly with automated guardrails. Ultimately, they conclude that going from demo to production requires sustained investment in infrastructure and evaluation, echoing MLOps lessons from a decade ago.