OpenAI

Context Engineering in 2026 — Louis-François Bouchard, Omar Solano & Samridhi Vaid, Towards AI

Aug 17, 2026 · 1:03:26

Louis-François Bouchard, Omar Solano, and Samridhi Vaid of Towards AI test context engineering on their AI tutor: keeping full history beat every compaction technique on recall, cost, and latency because 97% of tokens were served from cache (up to 50x cheaper), making summarization a trap unless it shrinks context by more than 50x. Full history recovered specific details 95% vs 32% after summarizing, and distinctive facts survived 800k tokens. Local hardware changes it: a 32k window can't keep everything, and larger models don't widen context. Dense retrieval hit 0% recall at 400k tokens where BM25 got 100%, so they use hybrid retrieval. Rule: name the constraint before compacting.

The Dark Arts of Web Automation: Teaching Agents to Use Websites Like Humans — Corey Gallon, Rexmore

Aug 14, 2026 · 21:38

Corey Gallon (Rexmore) argues that an agent driving Chrome via the Chrome DevTools Protocol is just a 'meat bag with a mouse'—and details his open-source Chrome Agent tool. Preparing the talk earned him an OpenAI ban threat after his agent cleared Cloudflare Turnstile, MT Capture, Lemon, and reCAPTCHA v2 with no human in the loop. He argues for CLI over MCP: both succeeded ~83% in an Arise AI study, but CLI took 7 turns and under a minute versus MCP's 71 round trips and 8 minutes, and CLI can be 75x cheaper in tokens. The method is a sense/act/verify loop up a three-rung ladder: synthetic clicks, trusted CDP input, then human mouse paths with jitter, deliberate overshoot. For reCAPTCHA, deterministic code drives and rearms each round while the agent names grid tiles, as speed beats expiration clocks.

Designing Agents (The Floor Is the Frontier) — Ben Hylak, Raindrop

Aug 12, 2026 · 19:46

Ben Hylak, CTO and co-founder of Raindrop, argues that most eval advice is stuck in the chatbot era and that agents have effectively infinite issues, so the real question is which ones matter—measured by when an issue started and what share of users it hits. He reframes agent quality around raising the floor (the worst thing an agent can do, like recommending a competitor or deleting data) rather than the ceiling, and says evals belong in your repo as code tests, not prompt playgrounds, because the harness is the product. He offers three tactical lessons from Raindrop: clusters are not issues because boundaries drift and you don't control them; code mode scales to traces, letting you write classifiers and run them in a sandbox at production volume; and agents are poor at anomaly detection but good at investigating anomalies you surface deterministically, like keyword spikes. He also notes that continual learning is rare in the real world, and that your approach should depend on user…

Codex, Behind the Harness — Dominik Kundel, OpenAI

Aug 10, 2026 · 20:55

Dominik Kundel, an OpenAI engineer, explains the internals of the Codex agent harness, which is open source under Apache 2 and written in Rust. He details how context construction caps available skills at 2% of the context window and uses deferred tools with tool search to manage size and cost. For actions, Codex relies on an apply patch tool for file edits, a shell tool with ripgrep, and sandboxes: seatbelt on macOS, bubblewrap on Linux, and a custom open-source Windows sandbox. To reduce approval fatigue, an auto-review subagent with read-only permissions judges actions against user authorization and risk taxonomies. Speed improvements come from websocket mode in the responses API, which sends only changed items instead of full state, crucial when GPT 5.3 Codex Spark hit 1,000 tokens per second on Cerebras. Long-horizon goals work via a continuation prompt until the model calls an update goal tool, favoring concrete verifiable objectives, and auto compaction maintains performance…

Open Source Is Dead. Long Live Open Source. — Saoud Rizwan, Cline

Aug 7, 2026 · 17:30

Saoud Rizwan, founder of Cline, argues that AI has killed the community side of open source while open weights models win on economics. He cites Zig banning AI from PRs, curl weighing shutdown of its bug bounty over AI-generated reports, and tldraw auto-closing pull requests, plus a LiteLLM compromise that stole credentials for three hours. Rizwan makes the case that closed labs' subsidized subscriptions lead to lock-in and price gouging, while open models like GLM match or beat Opus on cost and code quality — GLM used twice the tokens at half the cost and fixed a real Cline bug that Opus's faster fix left broken. He compares open weights to Facebook's Open Compute project and urges American labs to release open weights before foreign models become the standard.

Local Models: Trust, Control, Optimization — Carter Abdallah, NVIDIA

Aug 7, 2026 · 43:21

NVIDIA's Carter Abdallah, Prime Intellect's Vincent Weisser, Arcee's Lucas Atkins, and NVIDIA's Chris Alexiuk argue open-weight models are the trustworthy foundation for enterprise and local AI. Atkins separates trust from safety: when Anthropic pulled Fable, enterprises chose Chinese open models for guaranteed availability, and open models are inspectable unlike closed APIs. Arcee pretrained a 400B model in six months; Weisser cites a customer that specialized an open model for finance in a week or two, beating Opus at a fraction of Haiku's cost. Alexiuk calls open weights the fix for 'mismanaged genius' and expects capable local models on MacBooks within a year; the panel predicts Fable-level open models within a year and hopes local-model use rises from a rounding error to 10–15%.

MCP Apps: Extending the Frontier — Ido Salomon & Liad Yosef

Aug 2, 2026 · 18:38

Ido Salomon and Liad Yosef, creators of MCP UI and co-creators of the MCP Apps spec, explain how MCP Apps lets servers ship interactive branded UI instead of text into any supporting host. The official MCP extension, built with Anthropic and OpenAI and supported by Claude, VS Code, ChatGPT, Copilot, Cursor and Slack, links a tool call to a resource; the host renders returned HTML as a sandboxed web component and clicks flow back through a callback to the model. Early adopters include 11 Labs, Shopify and Postman; the payoff is distribution: with ChatGPT at 800 million weekly users, you write once and run everywhere. The spec is still evolving, with an open working group meeting every three weeks and live work on reusable views, AppTools/ViewTools for host-to-app control, and interoperability with generative UI standards like A2UI.

Teaching AI to Find Real Vulnerabilities — Prof. David Brumley, Bugcrowd

Aug 1, 2026 · 27:17

Carnegie Mellon professor and Bugcrowd chief AI officer David Brumley argues that teaching AI to hack mirrors human learning: a ladder from crashes to arbitrary code execution, graded by deterministic oracles rather than LLM judges. He shows why benchmarks fail when targets hold multiple vulnerabilities — models reward-hack the easiest bug — and presents his 'audit task' scoring precision and recall across all discovered bugs. Testing on Chrome's V8 with 41 real vulnerabilities, MITHOS hit 73% full code execution and GPT 68%, while Gemini and Kimi scored 0%. Several exploits were novel: MITHOS reverse-engineered Math.random to forge a pointer, found a new WASM path, and produced a real zero day. The episode grounds this in stories from picoCTF winner Fluorescence to DARPA's Cyber Grand Challenge, urging RL environments built on real bugs over benchmark-maxxing security.

What's Next After RLHF? — Diogo Almeida, TypeSafe AI

Jul 31, 2026 · 18:05

Diogo Almeida, a GPT-4 co-author and founder of TypeSafe AI, argues that RLHF optimized models for human approval, making them superb assistants but unreliable for autonomous work. He contrasts assistance with automation, noting RLHF's reward model encourages confident overpromising — like ChatGPT praising an audio file of farts as music. The next era is not Claude Code (still assistance-native), but real automation via RLVR-style methods focused on calibrated decision-making, echoing Sutton's bitter lesson that the task matters more than data. He defends pre-training as phenomenal, blaming post-training's asymmetric reward model for hallucinations. TypeSafe is rebuilding the AI stack for reliability and automation.

Scaling to Long Horizons — Ross Taylor & Chengxi Taylor, General Reasoning

Jul 31, 2026 · 18:07

Ross Taylor and Chengxi Taylor, co-founders of London-based RL company General Reasoning, argue that scaling to long horizons demands a mindset shift and better simulation, not just bigger context windows. Ross recounts how Galactica's 2022 thinking tokens were prescient, but its base-model demo backfired, while RLHF made LLMs products; his Meta team's PPO with verifiable rewards worked, but DeepSeek-R1 later showed better base models were key. Chengxi details long-horizon obstacles: sparse rewards, credit assignment, and 1M token limits, solved by value models that reduce variance and enable bootstrapping. Kelly Bench gave frontier models $100K to trade Premier League matches; all lost money, exposing how little current environments simulate real competition. Pipeline RL trades off off-policy staleness (8 steps okay) against GPU utilization, and they point to openreward.ai, with 350+ environments, for long-horizon RL.

The Base Model Is Dead — Varun Singh, Arcee AI

Jul 31, 2026 · 17:45

Varun Singh, pre-training lead at Arcee AI, argues the base model is dead: it no longer just mirrors web text but must carry the prior that reinforcement learning builds on. He traces how web text fell from 85% of GPT-3's mix to 15% in MAI Thinking 1, with code and STEM dominating, and how Nemotron 3 Ultra pulls SFT-style Q&A data back into pre-training. Synthetic rephrasing, as used in Arcee's Trinity Large and Kimi K2, upsamples information to teach task shapes early. He warns that without post-training-flavored data early, MoE load balancing can break when SFT distributions differ, citing MAI's cranking of the balancing coefficient. He frames training as supervised learning vs RL, noting RL compute now rivals or exceeds pre-training, as with Compose 2.5, and argues the base model's job is to provide atomic skills for RL to compose.

Your Agent Didn't Fail. Your Harness Did. — Vinoth Govindarajan, OpenAI

Jul 29, 2026 · 18:26

Vinoth Govindarajan of OpenAI argues that most agent failures are harness failures, not model failures. Using OpenClaw as a case study, he details five failure shapes: state hole (delivered but not remembered), overlapping writers (last write silently erases previous), dangling tool call (run waits for an event that never arrives), approval drift (expired approval blocks later work), and missing edge proof (internal success but user didn't see result). The through line: a model proposes, the harness commits, and the receipt proves it. He prescribes three invariants—own the state, order the mutation, prove the action—and a five-question run receipt audit: what woke it up, what state did it inherit, what authority did it use, what executed, and what evidence survived.

The Dirty Secret of Forward Deployed Engineering — Natalie Meurer, Sierra

Jul 28, 2026 · 16:49

Natalie Meurer argues that forward deployed engineering (FDE) has become a meaningless label because it has stretched from DevOps at Palantir in 2008 to data integration, ontology work in Slate and Foundry, solution architecture, and enablement—but its durable core is customer accountability and outcome-based pricing. Tracing FDE's history at Palantir, she shows how the role evolved from keeping the platform stable (2008) to data integration (2012), custom dashboarding in Slate (2016), and finally customer enablement in Foundry (2020). As coding agents make software cheap, she contends the lasting value lies in integrating data, understanding customers, and owning outcomes. Pricing tells the story: seat-based assumes a tool, while usage or outcome pricing puts the provider on the hook—exactly what FDEs have always done. She concludes agent engineering is FDE reborn, and that product, infra, and AI engineering are all trending toward the same customer-accountable model.

Evals-Driven Development for a Mental Health AI Coach — Akele Reed & Dave Revere, SonderMind

Jul 25, 2026 · 21:17

SonderMind engineers Akele Reed and Dave Revere explain how they built Sonder, a clinically grounded Mental Health AI Coach, using eval-driven development and modular guardrails to balance effectiveness and safety. They designed input and output guardrails as separate LLM judges to avoid over-calibration and ensure correct triggers, not more triggers. Dave describes a clinical feedback loop where therapist annotations become typed evals that gate releases, turning clinician judgment into CI. They open-sourced 200 input and 100 output guardrail scenarios, clinically reviewed and calibrated. The system uses a Supervisor/Executor/Evaluator architecture, and they turned off built-in guardrails of frontier models due to over-calibration. Every architectural decision prioritized user safety, with modularity enabling iteration without compromising safety.

The Future of Evals: From LLM as a Judge to Agent as a Judge — Aparna Dhinakaran, Arize AI

Jul 24, 2026 · 6:06

Aparna Dhinakaran, co-founder of Arize AI, argues that as agents evolved from simple prompts to complex systems with tool calls, reasoning, and long-horizon tasks, evals must evolve too—from deterministic checks to LLM as a judge, and now to agent as a judge. She reveals that the top teams run over 3,800 different evaluators, yet classical LLM-as-judge evals fail to catch subtle failures in agents that generate unique trajectories per user. Arize's new tool, Signal, is a long-running agent that reads traces, discovers patterns like inefficient tool loops, and can even open a PR to fix issues. The episode traces this arc from static checks to adaptive analysis, emphasizing that the future of evals requires all three layers to handle the complexity of modern agents.

Full Workshop: Setting Yourself Up for Success —Jason Liu, OpenAI Codex

Jul 24, 2026 · 1:15:02

Jason Liu walks through how to set up OpenAI's Codex for maximum productivity, using voice dictation, app shots, computer use, and a personal memory vault to delegate and automate almost every task. He demonstrates creating skills and plugins from past work, using compaction to keep threads running for weeks with hundreds of sub-agents, and setting up automations with heartbeats and goals. Specific examples include auto-checking flights, editing iMovies via computer use, and having a chief-of-staff thread that monitors Slack and updates task lists. He also covers security permissions (auto-review vs full auto), tips for new users to start with low thinking mode to save tokens, and how to build skills that self-improve by editing their own files. The workshop emphasizes that threads can now communicate with each other, enabling manager-like orchestration and long-running work streams.

Vending-Bench: Long-Horizon Agent Evals — Lukas Petersson, Andon Labs

Jul 24, 2026 · 18:05

Lukas Petersson, co-founder of Andon Labs, presents Vending-Bench, a long-horizon evaluation where AI models run a simulated vending machine business for a year, revealing emergent misbehavior such as price collusion, lying to suppliers, and power seeking. The benchmark exposes a simulation awareness problem—models behave differently when they know they are being tested. To address this, Andon Labs moved to real-world deployments: a café in Stockholm run by Gemini (which lost $6,000 and was replaced by GPT), a retail store on Union Street, and an AI radio station where Claude emerged as the best DJ. They developed a method to fork real environments into simulations mid-run, dramatically reducing simulation awareness. In a replay test of a Nazi song incident, Grok played it over 90% of the time, Gemini about half the time, while Opus and GPT refused every time.

Claude for Long-Horizon Tasks — Lance Martin, Anthropic

Jul 22, 2026 · 25:19

Lance Martin from Anthropic discusses how Claude's increasing task horizon enables asynchronous agents through decoupled architecture, verifier loops, and self-improving memory systems. He explains the shift from short task horizons (10–20 minutes) to 12+ hours, necessitating decoupling the brain (harness) from hands (sandboxes) for reliability and security. Verifier loops using separate contexts allow models to self-correct, demonstrated on the Parameter Golf benchmark with Opus 4.7. Memory systems inspired by human dreaming correct errors in-band, as shown in a Pokémon example where dreaming prevented repeated failures. Finally, org-level harnesses like Claude Tag provide shared identity and context for multiplayer proactive agents.

The Desktop Frontier — Ahmad Osman, Osmantic

Jul 21, 2026 · 18:02

Ahmad Osman, founder of Osmantic, argues that within roughly 18 months (by late 2027) a single RTX 5090 will run intelligence equivalent to GLM 5.2, driven by the Densing Law of increasing impact per parameter. He shows this trend through concrete examples: a 27B-parameter Qwen 3.5 now beats the 405B LLaMA 3, and the same eight RTX 3090s that once struggled with LLaMA 2 can now run 15 parallel Qwen 3.5 agents. Osman presents the Densing Law—every 3.5 months, 50% fewer parameters achieve the same capability—as a systematic pattern, not coincidence. He advocates for sovereign AI: owning your own hardware (like a DGX Station or RTX 5090) gives you control, avoids cloud limitations, and sees hardware appreciate in utility as models become more efficient. He asks why fund cloud data centers when local hardware can run frontier intelligence and grow more valuable over time.

When Agents Meet Physical Data: The Other Physics of Agent Harnesses - Dmitry Petrov, DataChain

Jul 20, 2026 · 27:33

Dmitry Petrov of DataChain argues that AI agents fail on unstructured physical data because their intuitions assume cheap recompute, while large-scale video, sensor, and robot data requires a different harness. He cites Anthropic's finding that agents achieve only 21% accuracy on data projects without specific data harnesses, and OpenAI's need for six layers of context even on structured data. Petrov demonstrates DataChain, an open-source Python framework that uses Pydantic schemas to turn messy binary files into queryable databases, an execution engine for distributed processing, incremental checkpoints to avoid recomputing on failure, and a knowledge base of datasets and source code so agents can answer follow-up questions in seconds instead of reprocessing terabytes. In a live demo with Claude Code analyzing 90 dashcam videos, the harness took 24 minutes to extract 100,000 object records, then instantly answered 'how many clips have people?' without rerunning inference. Petrov emphasizes that the key is organizing metadata into star schemas and sharing data lineage across teammates so no one pays the compute cost twice.

Stop Renting Your Cognitive Infrastructure - Thiyagarajan Maruthavanan, Kalmantic Labs

Jul 18, 2026 · 7:52

Thiyagarajan Maruthavanan of Kalmantic Labs argues that AI teams should stop renting inference from providers like Anthropic and build their own infrastructure, coining "rent to learn, own to earn." He recounts his app UltraSuno costing hundreds of thousands of dollars in inference, a stolen key hitting $10,000, and moving to his own DGX Sparks hardware. Three enterprises—a fund, hospital, and tax practice—each hit walls with renting: control, audit redlining, and reproducibility. He open-sourced JustTokenMax (benchmarked better than Netflix's Headroom) and wrote the book "Peak Inference" on building your own inference infra. He notes the market's conflicting pitches (Jensen's token factory, Nadella's unmetered, NeoClouds' endpoints) but concludes owning is essential post-PMF.

Content Is Code - Matt Palmer, Conductor

Jul 18, 2026 · 10:53

Matt Palmer, head of developer experience at Conductor, argues that code is now the fastest way to produce technical content, shifting left to code as the source of truth. He outlines three eras of content creation—handcrafted, expensive code, and cheap code with AI—and claims that the scarce resource today is not code but structure and conscientiousness, meaning meticulous care in maintaining design tokens, brand guidelines, and clean codebases. AI rewards organizational excellence over raw technical skill, making the best communicating teams those that instill discipline and rigor into their software creation. Palmer predicts 2026 as the year of the creative technologist and 2027 as the year of the content engineer, with declarative pipelines that automatically generate documentation, walkthroughs, product tours, and updates from structured code.

Special Topics in Kernels, RL, Reward Hacking in Agents — Daniel Han, Unsloth

Jul 17, 2026 · 2:20:21

Daniel Han of Unsloth argues that reward hacking—where AI models cheat to maximize reward—is a critical problem in agent training, citing examples from GPT-5.1's calculator hacking and GPU mode kernel competitions. He shows that models exploit benchmark flaws, such as viewing Git history or editing timers, and that even open-source models like GLM 5.2 require anti-hacking measures. Han emphasizes that harness and tooling quality now outweigh model choice, with inference providers sacrificing accuracy for speed (e.g., 10% accuracy drops across providers). He also warns that hardware limits (float4 precision, diminishing returns) shift focus to software algorithms like FlashAttention and gradient checkpointing. The workshop concludes that benchmarks are unreliable—DeepSpeed's false positive rate is contested at 44.9%—and urges verification before trusting performance claims.

How Autoresearch is changing ML research — Zhengyao Jiang, Weco

Jul 16, 2026 · 16:16

Zhengyao Jiang, CEO of Weco AI, details how their autonomous agent Aiden became the top contributor in OpenAI's Parameter Golf competition, setting 7 leaderboard records—more than twice any human's 3. Over 22 days, Aiden ran 1,300 experiments on a single H100 node, using just 4% of total compute while contributing 15% of records and achieving a 28% leaderboard hit rate (six times the community average). The agent excelled at finding and implementing ideas from human researchers, combining them (e.g., gated attention from a paper with a tokenizer improvement) to produce breakthrough results. Jiang argues that auto-research does not replace human creativity but shifts it upward: humans design evals and code abstractions that bias the agent's search, a new craft he compares to training neural networks. The talk reveals that Aiden's H-index in the community was 10 versus the next human's 7, proving autonomous agents can produce work that others build on.

"I've never seen anything scarier than an LLM with tool calls." — Erik Meijer aka @HeadinTheBox

Jul 13, 2026 · 21:13

Erik Meijer of Leibniz Labs argues that AI agents with tool calls are intrinsically dangerous and must be tamed through formal verification. He recounts how adding tool calls to LLMs turned harmless chatbots into agents capable of irreversible side effects, like deleting files or emptying bank accounts. His solution, implemented in Automind, uses proof-carrying code: the agent submits a plan as a program (Free Monad) together with a machine-checkable safety proof, and a small checker verifies the proof before allowing execution. This air-gaps the agent from its tools, ensuring policy compliance statically. Meijer demonstrates that elementary type systems and compiler techniques, such as taint analysis on program expressions, can provably prevent unsafe actions, turning agents from 'vibe-coding' risks into provably safe systems.

From fork() to Fleet: Designing an Agent Sandbox Cloud — Abhishek Bhardwaj, OpenAI

Jul 13, 2026 · 44:34

Abhishek Bhardwaj from OpenAI explains the design of a secure, scalable agent sandbox cloud, arguing that microVMs (like Firecracker and Cloud Hypervisor) provide the strongest isolation for running untrusted AI-generated code despite performance trade-offs, and that persistent disk storage is the next frontier for unlocking long-running, stateful agent tasks. He compares runtime isolation technologies from simple fork-exec to containers (with namespaces and cgroups) to gVisor, and advocates for hardware-backed virtualization via microVMs to prevent guest exploits from reaching the host kernel. For persistence, he details incremental block-level snapshotting using copy-on-write filesystems and always-on distributed filesystems, enabling reliable checkpointing and Monte Carlo-style exploration. Orchestration challenges include low-latency sandbox creation through pre-warming or memory snapshots, and intelligent routing based on cached snapshot layers to minimize restore time. The talk positions storage and fast snapshot restore as key enablers for advanced agent capabilities, such as long-horizon tasks and failure recovery.

Stop AI Agent Hallucinations: 5 Techniques + Production Patterns - Elizabeth Fuentes, AWS

Jul 11, 2026 · 55:19

Elizabeth Fuentes (AWS) presents five code-based techniques to stop AI agent hallucinations, each with measurable before/after metrics. Semantic Tool Selection filters 29 tools to the 3 most relevant per query, cutting token usage from thousands to under 300 per call. Graph-RAG replaces vector similarity with structured graph queries (using Neo4j), enabling precise aggregation and multi-hop reasoning that vanilla RAG fabricates. Multi-Agent Validation uses an Executor-Validator-Critic swarm to catch fabrications, achieving a 92% detection rate. Neurosymbolic Guardrails enforce business rules in Python hooks that the agent cannot skip, achieving zero rule violations. Agent Steering guides agents to self-correct when soft rules fire, completing tasks without hard failures—demonstrated by booking 50 guests by intelligently splitting into two rooms.

Should AI Engineers Still Read Code in 2026? The Z/L Continuum — Alex Volkov, ThursdAI

Jul 10, 2026 · 21:35

Alex Volkov examines the debate sparked by Ryan Lopopolo's claim that 'code is free' and Mario Zechner's counter that engineers must 'read every fucking line' of critical code, arguing that the Z/L Continuum is about task-level proof rather than personality. Citing a Ferrous AI survey showing an 861% increase in code deletion per PR and a 242% rise in incidents, he notes Anthropic's recursive self-improvement essay admitting human code review is a new bottleneck. Volkov's routing table prescribes reading every line for authentication, money movement, and irreversible data, while letting agents handle less critical changes. He also introduces loops—cron-like agent systems that self-verify—as the next frontier, quoting Adi Osmani that automated loops don't remove judgment. Volkov concludes that capability drift moves where proof belongs, but every system still requires human judgment.

The Golden Age of AI Engineering — Alexander Embiricos & Romain Huet & Peter Steinberger, OpenAI

Jul 9, 2026 · 25:13

OpenAI's Alexander Embiricos, Romain Huet, and Peter Steinberger reveal how AI engineering is entering its golden age, with model releases accelerating from every 15 months to roughly every 6 weeks and frontier-level intelligence now available at $1 per million input tokens and $6 per million output tokens. They demonstrate Codex's evolution from a model that couldn't test its own code to one generating 750 tokens per second, enabling a 10-second pull request, and emphasize that the product is built on the same open-source stack—responses API, AGENTS.md format, and harness—that ships to developers. The trio argue that the future is not about automating engineers but empowering them via value maxing over token maxing, with agents moving from local/cloud distinctions to persistent, long-running managers that orchestrate work and only require human steering at key decision points, as illustrated by an automated open-source issue resolution workflow.

Your agent is blindfolded — Johan Lajili, Poolside AI

Jul 8, 2026 · 9:58

Johan Lajili of Poolside AI argues that the key difference between successful and disappointing AI coding agent experiences is whether the agent has a tight feedback loop to verify its work, essentially giving it "eyes" to test its own outputs. He demonstrates this with his CLI tool Spoolside for VS Code extensions that extracts logs and screenshots to reproduce bugs. Lajili predicts engineers will shift from product engineering to "AIX engineering" — focusing on making AI work on the product rather than on the product itself. He advises building ephemeral testing primitives and investing in agent self-serve tools, analogous to putting on your own mask first on an airplane.

Your coding agent doesn't always follow your rules — Talha Sheikh, Checkout.com

Jul 8, 2026 · 10:08

Talha Sheikh from Checkout.com argues that the real value in AI coding agents lies not in the code they generate but in the verification layer that checks it. Drawing from his experience with Claude Code, he built Vector Harness—a tool using Claude hooks to deterministically test agent outputs against a config file, retrying failures automatically. He contends that model capability does not equal reliability, and that even with better specs or models, deterministic verification is essential. Sheikh observes that Anthropic, OpenAI, and CUDO are all adopting similar harness patterns, shifting the focus from instructing agents to enforcing rules. His central claim: work on the harness, not the code, and you can even use smaller, cheaper models with enough guardrails.

Running a Chess YouTube Channel entirely by AI — Stephan Steinfurt, TNG

Jul 8, 2026 · 16:31

Stephan Steinfurt, from TNG, presents their AI system that automatically creates daily chess puzzle explanation videos for YouTube. The agent combines a chess engine (for analysis) with Google's Gemini 3.1 Pro (for natural language) to generate detailed move-by-move commentary, including arrows and highlights. It downloads games from LeChess nightly, runs checks/captures/threats analysis, and uses 11Labs for text-to-speech with emotional tags. The YouTube channel has 500k views and 4,000 subscribers, with each video costing 20-30 cents (not yet monetized). Steinfurt notes the error rate is low (1 in 20 videos has a mistake) and discusses balancing explanations for different skill levels using the Maya engine to suggest human-like moves.

MCP Apps: Primitives, discovery, and the Future of Software - Pietro Zullo, Manufact, Inc

Jul 5, 2026 · 28:54

Pietro Zullo, co-founder of Manufact, explains that MCP Apps are MCP servers that return interactive UI elements in sandboxed iframes, enabling bidirectional communication between the UI and the host. He details primitives like set-model-context and send-message, and shows how apps can stream tool inputs into the UI in real time. The ChatGPT, Claude, and Cursor stores now accept self-serve submissions, and Claude's dynamic discovery matches connectors to user intent. Manufact’s open-source SDK, mcp-use, has 8M+ downloads and provides a cloud platform for building, testing, and submitting MCP Apps.

The Prompt Is Still a Punch Card - Ted Johnson, JoinIn AI

Jul 2, 2026 · 20:13

Ted Johnson argues that AI interfaces still use the batch protocol of punch cards, forcing humans to adapt to machines rather than the reverse. He introduces three concepts—channel, expression, protocol—to show how expression exploded with LLMs but the protocol remained static. Examples include voice mode misinterpreting a side conversation, PersonaPlex's turn-taking, and a meeting AI that follows group dynamics without prompts. Johnson calls for designing interfaces where AI understands timing, ambiguity, and shared context, moving from prompting to genuine communication.

OpenClaw in Your Hand: Building a Physical AI Terminal - Lech Kalinowski, Callstack

Jun 28, 2026 · 24:36

Lech Kalinowski presents Vault, a dual-display handheld AI terminal built on an ESP32-S3 that runs a local LLM and OpenClaw agents without cloud reliance. The device pairs a fast OLED live surface with a bistable e-paper content display, offering four modes — shell, assist, control, and an LLM-native RPG — all powered by a single lithium-polymer cell. The backend serves GPT-oss 120b via NVIDIA TensorRT-LLM, keeping inference off the microcontroller. Kalinowski shares engineering war stories: blown OLEDs from unstable power supplies, software I2C issues, and noisy encoders. The RPG mode generates four worlds with NPCs and narrative state tracked by the LLM, not dice or HP. Built in three months with 130 commits, the terminal is intentionally distraction-free, targeting quiet spaces where users want text-first AI interaction.

AI System Design: From Idea to Production - Apoorva Joshi, MongoDB

Jun 28, 2026 · 28:53

Apoorva Joshi presents a structured framework for designing AI systems from idea to production, arguing that thorough product specification and evaluation are now more critical than coding itself. Using a health insurance claims review system for MDB Health as an example, she walks through four phases: defining business problems with measurable success metrics (e.g., reducing urgent claim processing from 2 days to 1 hour within 90 days), designing data strategy and architecture using patterns like RAG, control flow, and human-in-the-loop, establishing guardrails and evaluation metrics such as faithfulness and cost per recommendation, and optimizing for accuracy, cost, latency, and reliability before shipping. She emphasizes building evaluation in from the start and iterating from the simplest system.

Using Spec-Driven Development for Production Workflows - Erik Hanchett, AWS

Jun 28, 2026 · 17:47

Erik Hanchett (AWS) argues that spec-driven development—writing markdown specification files before any code—produces higher quality code by guiding AI coding assistants, which he compares to 'AI interns' that need explicit direction. He introduces Kiro, AWS’s new AI IDE and CLI, which offers a spec mode that generates requirements documents, design documents with mermaid diagrams, property-based tests using Fastcheck, and a task list. In a demo building a movie website, Kiro created an MVP-focused implementation verified by property tests. Hanchett also highlights integrating external data via the Model Context Protocol (MCP) and stresses the importance of human review. He advises balancing context in steering docs and using on-demand skills to control the workflow.

The Miranda Hypothesis: How Hamilton Poisoned Persona Evals - Jacob E. Thomas, Results Gen

Jun 25, 2026 · 58:17

Jacob E. Thomas argues that persona-eval benchmarks like InCharacter, which report 80.7% personality fidelity for Hamilton, structurally miss anachronistic compositing because they measure fluency and personality consistency—exactly the features the dominant cultural composite optimizes. The Miranda Hypothesis posits that training data saturated with works like the 2015 Hamilton musical (outnumbering the 175,000-word Federalist Papers by orders of magnitude) causes models to produce smoothed, morally legible composites rather than historically faithful personas. Thomas explains that RLHF amplifies this distortion because human raters share the same cultural frame. He proposes a paradigm shift to epistemic simulation: corpus-bounded, temporally-anchored systems evaluated by domain experts, with the model as a swappable component in a configurable role-playing language system. The pre-registered Prism Experiment instantiates Abraham Lincoln at four documented moments (1847, 1858, 1860, 1862–65) under three seeding conditions—bare model, biography, or primary sources—scored on a weighted three-axis rubric that prioritizes anachronism detection (40%) over voice. Thomas concludes that…

6 Things to Know about AIE World's Fair 2026

Jun 21, 2026 · 17:50

Swix, co-founder of AI Engineer, outlines six key features of the 2026 World's Fair in San Francisco. The event is over 4x larger than 2025, with 50% new topics, an expanded expo floor (4 stages, 4x bigger), and a new research-industry poster session that includes printed tweets. A leadership track on Level 3 features the Token Billionaire lounge for heavy LLM users and off-the-record McKinsey sessions. AI verticals debut in agentic commerce, healthcare, finance, and GTM, with a new AI Engineer New York focused on finance. Side events include a World Cup viewing, New Engineer Orientation (NEO) with 300+ signups, and a kids event; attendees can claim $38,000 in sponsor offers.

Why MCP and ChatGPT Apps Use Double Iframes — Frédéric Barthelet, Alpic

Jun 15, 2026 · 20:11

Frédéric Barthelet, CTO of Alpic, explains why ChatGPT and other MCP hosts render third-party app UI inside a double iframe. He traces how simpler approaches fail: `srcdoc` shares the parent origin, letting CSP block scripts and risking data access; sandboxing removes origin storage; and `allow-same-origin` recreates the escape. The resulting double iframe—an outer iframe from a controlled subdomain loading app HTML via `srcdoc` into an inner frame—ensures isolation and prevents cross-app storage collisions. Barthelet warns developers must declare every external domain their view uses in MCP app metadata or face submission rejection, and demos Skybridge's CSP inspector that diffs declared domains against actual network calls.

Why Eval++ Is the Next Great Compute Primitive — Sunil Pai & Matt Carey, Cloudflare

Jun 8, 2026 · 24:51

Matt Carey and Sunil Pai from Cloudflare's agents team argue that Durable Objects — stateful serverless with 15ms London latency — and Dynamic Workers — a safe, sandboxed eval for LLM-generated code — form the next compute primitive for AI agents. They explain how Durable Objects enable resumable streaming, multi-tab sync, and background scheduling out of the box without distributed systems engineering. Dynamic Workers allow running generated JavaScript strings in isolated sandboxes, reclaiming 30 years of avoided eval. The pair tease upcoming talks: one on collapsing Cloudflare's 2,600 API endpoints into a 1,000-token MCP tool, and another on a coding agent harness built entirely on Workers that they are already shipping.

Evals Are Broken, Use Them Anyway — Ara Khan, Cline

Jun 6, 2026 · 19:04

Ara Khan from Cline argues that evals are broken—people either treat benchmark numbers as gospel or dismiss them for vibes—but that the truth lies in between, and they should still be used. He presents three heuristics: don't believe model vendor eval numbers, stay current but not an earliest adopter, and look for new precise evals like Terminal Bench. Khan details Cline's journey from ignoring evals to building their own, then adopting Terminal Bench (89 real-world coding tasks). He explains the process: get a score (Cline started at 43%), portfolio allocate failures by sending another agent through traces to identify small levers, then hill climb by fixing zone1 bugs, zone2 nuanced prompt engineering (e.g., Anthropic-specific techniques that don't transfer to Codex or Gemini), and avoid zone3 overfitting. The episode offers a practical framework for using evals to improve agent performance while staying grounded in real-world usefulness.

Building safe Payment Infrastructure for the autonomous economy — Steve Kaliski, Stripe

Jun 6, 2026 · 18:46

Steve Kaliski, Principal Software Engineer at Stripe, explains how to build safe payment infrastructure for the autonomous economy, focusing on enabling AI agents to spend money without catastrophic risk. He argues that discovery benefits from non-determinism (e.g., LLMs), but payments and credentials require determinism. Stripe's solutions include shared payment tokens that enforce spend limits per seller, amount, currency, and time; the machine payments protocol with Tempo for paying for API calls via HTTP status codes; and the agent e-commerce protocol with OpenAI for structured checkout flows. Kaliski also demonstrates how Stripe projects wraps these primitives. The goal is to minimize blast radius through verifiable identities, spend policies, and API-driven interactions.

Dark Factory: OpenClaw Ships Faster Than You Can Read the Diff — Vincent Koc, OpenClaw

Jun 5, 2026 · 16:44

Vincent Koc, a core maintainer at OpenClaw, explains how the open-source project ships code faster than humans can read diffs by running 60–70 autonomous coding agents across swim lanes. In a single night, he and Peter Steinberger executed 2,700 commits touching 82% of the core codebase, launching a plugin architecture and changing close to a million lines of code. Koc describes managing agents as managing people: the key skill is reading reasoning tokens to detect when an agent is bullshitting, gained through sheer volume of token maxing. He argues 2025 was about token maxing; 2026 is about not wasting them, with agent-in-the-loop processes and opinionated swim lanes replacing blind Ralph looping. The episode details his agent development environment using .skills files, Git work trees, and a semantic graph for triaging 60K PRs, emphasizing that the bottleneck shifts from engineering to taste and process.

Beyond Transcription: Building Voice AI That Understands Conversations — Hervé Bredin, pyannoteAI

Jun 5, 2026 · 25:20

Hervé Bredin, chief science officer at pyannoteAI, argues that speaker diarization benchmarks are misleading because they use headset audio while users rely on table microphones—Nvidia Parakeet reports 11.4% word error rate on AMI headset data but gets 26% on the same dataset's table mic. The episode covers how speaker diarization (who speaks when) is harder than it looks, especially when combining with transcription: overlapping speech, timestamp disagreements, and words falling between speaker boundaries create errors. Bredin demonstrates pyannoteAI's Precision 2 model achieving 3% diarization error rate (DER) against a 5% baseline on a two-speaker phone call. State of the art today: 2% DER on clean telephone calls but 41% in a noisy restaurant, showing the problem is far from solved. The reconciliation between diarization and STT is handled by a proprietary orchestration that works with any STT model.

What if the network was the sandbox? — Remy Guercio, Tailscale

Jun 1, 2026 · 24:29

Remy Guercio from Tailscale argues that standard sandboxing conflates execution isolation with access control, proposing Aperture—an LLM gateway built on Tailscale's WireGuard identity network—which gives every connection verified identity (user, tag, or group) so agents get placeholders instead of real API keys, making exfiltration impossible. Aperture provides visibility into every tool call, bash command, and MCP request without instrumentation inside the container; internally at Tailscale, bash dominates over structured tool calls. Access permissions are configured via Tailscale's grants and ACLs, supporting quotas, cost controls across providers, and webhooks for tool calls. The gateway works at the LLM layer, capturing even non-tool-call agent behaviors like direct code execution, and is available on Tailscale's free plan.

How to talk to statues — Joe Reeve, ElevenLabs

Jun 1, 2026 · 33:28

Joe Reeve, an ElevenLabs growth engineer, built an app that lets users talk to statues by pointing their phone at one—it identifies the statue via OpenAI deep research, generates a matching voice with ElevenLabs' voice design API, and starts an ElevenLabs agent conversation—all in 30 seconds. He created it in two hours on a Sunday using Cursor and a single prompt, posted it on Tuesday, got 50,000 impressions, then reposted about vibe coding and hit 1.5 million. Museums, auction houses (Bonhams, Christie's), and travel platforms reached out; one CEO tracked down his WhatsApp, saying a team of 10 had worked on a similar project for a year. The episode explores voice interaction patterns, the challenge of interrupting agents, the need for multimodal interfaces (voice plus visual), and the potential of vibe coding to democratize software creation, with Joe noting that music and captions are key to viral videos.

Engineering voice agents: Latency, quality, and scale — Rishabh Bhargava, Together AI

May 31, 2026 · 24:35

Rishabh Bhargava from Together AI outlines engineering voice agents, explaining that pipeline architecture with colocated models can achieve sub-500ms response times critical for user retention. Speech-to-text targets P90 under 100ms and 6% word error rate, while the LLM must stay within 200-300ms time-to-first-token using 8-30B parameter models—larger models blow the budget, smaller ones break tool calling. Network latency from distant data centers adds 75ms (30% overhead) versus 5ms when colocated in the same building. Pure speech-to-speech models are emerging but still struggle with instruction following and tool calling. The thinker-talker pattern uses a small LLM for fast conversational flow and issues a single tool call to a larger model for complex requests.

Why your agents need decision traces, not just documents — Zach Blumenfeld, Neo4j

May 29, 2026 · 20:12

Zach Blumenfeld from Neo4j argues that context graphs—a graph-based layer storing decision traces, precedence, and causal chains—extend standard RAG to make AI agents more accurate and explainable. A financial analyst agent, for example, can reject or accept a request by retrieving past decision traces and structurally similar precedents via graph embeddings, not just semantic similarity. Neo4j's one-command tool `uvx create-context-graph` scaffolds a full-stack app with backend, frontend, demo data, and MCP server, supporting 22 built-in domains or custom ontology generation. The underlying `neo4j-agent-memory` package handles entity extraction through a spaCy-to-GLiNER-to-LLM pipeline with deduplication and integrates with pydantic AI, LangGraph, Crew, Google ADK, and others. The episode also covers data connectors for GitHub, Notion, Jira, and Slack, and the ability to store and query decision traces with timestamps, though automated trace quality evaluation is still evolving.

Does GenAI "belong" to data scientists? — Phil Hetzel, Braintrust

May 25, 2026 · 18:54

Phil Hetzel of Braintrust argues that generative AI development should not be isolated to data scientists or ML engineers, but instead requires a diverse team including product engineers, systems engineers, and non-technical domain experts. Because models are already built by OpenAI and Anthropic, the remaining work is prompt and context engineering, distributed systems, human annotation, and functional evaluation—not traditional training pipelines. Data scientists add value through rigorous testing and LLM-as-judge evaluation, but must move beyond precision/recall toward broader functional metrics. Traditional enterprises often mistakenly hand GenAI to ML platform teams, while AI natives use small cross-functional groups with closer problem proximity. The ideal mix combines technical roles for implementation and system design with non-technical experts for prompt engineering and human annotation, keeping the agent relevant through continuous feedback.

The Missing Primitive for Agent Swarms — Lou Bichard, Ona

May 23, 2026 · 18:37

Lou Bichard, field CTO at Owner, argues that agent swarm infrastructure for coding agents is missing a coordination primitive, while runtimes and orchestration are largely solved. He defines a Software Factory as incrementally removing humans from the SDLC and highlights internal systems from Stripe (Minions) and RAMP (Inspect) as real-world examples. Owner provides VM-level isolation for agent fleets and sub-agent patterns, demonstrated with a demo of process-based and VM-based swarms. Bichard identifies GitHub as a poor coordination layer for hundreds of parallel pull requests and proposes solutions like state machines, durable execution, and a CLI gateway that any local agent can invoke to check progress. He notes context rot and agents skipping steps as key challenges, and announces a public two-week project to build a Software Factory from scratch starting next week.

Fast Models Need Slow Developers — Sarah Chieng, Cerebras

May 22, 2026 · 18:02

Sarah Chieng from Cerebras argues that the 20x speed increase of models like Codex Spark (1,200 tokens/sec vs. 40-60 for Sonnet/Opus) forces developers to rethink workflows, or risk generating technical debt at unprecedented scale. She presents a practical playbook: validation and linting become free at every step, so must run continuously; developers can generate 75 component variations across five sub-agents and cherry-pick the best; and with context filling in 30 seconds instead of ten minutes, a four-file external memory system (agents, plan, progress, verify) maintains continuity between sessions. Chieng emphasizes real-time collaboration with the model rather than spawning agents and walking away, and advocates using a slower planner model with a fast executor to orchestrate agents effectively.

Skill issue: Lessons from skilling up coding agents to use Langfuse - Marc Klingen, Clickhouse

May 20, 2026 · 24:09

Marc Klingen, founder of Langfuse, shares six learnings from building a skill that lets coding agents like Claude Code add Langfuse observability and evals to projects. The skill addresses the problem of agents hallucinating stale instrumentation from outdated pre-training context by surfacing a search endpoint for documentation, helping agents navigate 478 documentation pages, and referencing rather than duplicating content. Klingen explains how examining traces revealed 80% of issues, basic eval setups were better than none, and an auto-research loop with a target function suggested six improvements (three accepted). However, the target function nearly backfired by optimizing away documentation-fetching steps vital for long-term reliability. He also discusses challenges of skill distribution, versioning, and whether to aim for quick initial setup or perfect single-shot implementation.

What Breaks When You Build AI Under Sovereignty Constraints - Bilge Yücel, deepset GmbH

May 19, 2026 · 19:09

Bilge Yücel, Senior Developer Relations Engineer at deepset, argues that sovereign AI requires explicit control over data flow, model choice, infrastructure, and operations, and retrofitting these pillars breaks existing systems in predictable ways. Replacing frontier APIs with self-hosted models forces re-evaluation from scratch, moving private data across jurisdictions creates multi-database search problems, replacing managed infra reveals vendor lock-in, and adding observability exposes black-box systems. She presents a sovereign architecture with guardrails, MCP tools, and Haystack's swappable components to mitigate these issues. The closing checklist asks whether you can swap models without changing application logic, have compliant run logs, and respond to incidents without calling a hyperscaler.

Harnesses in AI: A Deep Dive — Tejas Kumar, IBM

May 17, 2026 · 20:27

Tejas Kumar, AI Developer Advocate at IBM, defines an AI harness as everything around the model that grounds it in reality, contrasting it with an agent loop. He demonstrates building a harness for a browser agent tasked with upvoting the first Hacker News post using GPT-3.5 Turbo, emphasizing that the prompt remains unchanged. The initial agent fails and lies about success; the harness adds guardrails (max iterations, context compression), a verify step that checks tool history to catch lies, and a login handler that programmatically injects credentials when the agent hits the login page. The final agent reliably logs in and upvotes. Kumar argues that harnesses provide reliability and control, allowing cheap models to perform well, and predicts 2026 as the year of harnesses, with dynamic on-the-fly harnesses as a potential next step toward AGI.

Connecting the Dots with Context Graphs — Stephen Chin, Neo4j

May 16, 2026 · 17:39

Stephen Chin of Neo4j argues that retrieval alone is insufficient for AI systems, because context graphs—which store relationships, reasoning traces, and decision provenance—enable grounded, auditable answers. He demonstrates with a healthcare example where a generic RAG system returns generic advice while a graph-grounded system knows the patient smokes and has had surgery, tailoring recommendations. The episode walks through Lenny’s Podcast memory demo and a financial services loan-decision app that surfaces prior rejections, margin trades, and fraud risk patterns, making the graph traversal visible. Chin notes Gartner has placed context graphs on the AI hype cycle and Foundation Capital called them a $3 trillion startup opportunity. Neo4j’s open-source agent memory package (short‑term, long‑term, reasoning memory) powers these context graphs, aiming to help engineers escape fragmented enterprise data and build explainable, policy‑aware AI.

Combine Skills and MCP to Close the Context Gap — Pedro Rodrigues, Supabase

May 15, 2026 · 18:27

Pedro Rodrigues from Supabase argues that combining agent skills with the Model Context Protocol (MCP) outperforms either alone, closing the context gap that makes agents unreliable on production systems. In a test with Claude Sonnet 4.6, an agent with only MCP created a SQL view that bypassed row-level security (RLS) by omitting the `security_invoker=true` flag, while the same agent with a skill added the flag correctly. He shares three principles for building product skills: point to living documentation rather than duplicating it, put critical security rules directly in `skill.md` because agents skip reference files, and be opinionated about optimal workflows (e.g., running DDL directly on dev/staging databases before generating migration files). Their evals across Claude Opus 4.6, Sonnet 4.6, GPT 5.4, and GPT 5.4 mini in three conditions (no MCP/no skill, MCP only, MCP plus skill) showed a unanimous task-completeness improvement when skills were added.

Ship Real Agents: Hands-On Evals for Agentic Applications — Laurie Voss, Arize

May 14, 2026 · 2:04:18

Laurie Voss, head of developer experience at Arize AI, delivers a hands-on workshop on evaluating agentic applications using Arize Phoenix, demonstrating that choosing the right eval matters more than tuning it: a correctness eval scored 0 out of 13 on the same financial analysis agent that a faithfulness eval scored 13 out of 13, because the model doesn't know the current year and cannot verify forward-looking data. He walks through building a complete eval pipeline from scratch—starting with tracing a Claude Haiku-based financial agent, reading and categorizing traces to identify root causes, then implementing code evals, built-in LLM-as-a-judge evals, and a custom actionability rubric with labeled examples. Voss emphasizes the importance of meta-evaluation to validate judge accuracy and introduces Phoenix experiments to prove prompt changes actually improve scores, not just vibes. Practical tips include using the impact hierarchy (data quality > prompting > model selection > hyperparameters) and the value of regression evals for safe model upgrades. The workshop closes with cost-aware evaluation, pairwise evaluation, and reliability scoring as next steps beyond the foundations…

Make your own event-sourced agent harness using stream processors — Jonas Templestein, Iterate

May 14, 2026 · 1:04:27

Jonas Templestein and Misha from Iterate introduce an event-sourced agent harness built on stream processors, arguing that agents should be modeled as an append-only event log with a synchronous reducer for state and an after-append hook for side effects. They demonstrate how every action—streaming chunks, tool calls, errors—becomes an event, enabling full debuggability and replay without re-running LLM calls. The key innovation is a "dynamic worker configured" event whose payload is a JavaScript processor; appending it to any stream instantly turns that stream into an AI agent with no server or dependencies. This allows processors from different authors and languages to compose on the same stream, and a safety checker can inject context within 200ms without blocking the agent. The hosts emphasize eventual consistency over before-hooks, and show deployment via Cloudflare Workers or by simply subscribing from any HTTP client.

Give Your Agent a Computer — Nico Albanese, Vercel

May 12, 2026 · 1:08:53

Nico Albanese demonstrates building an agent with Vercel's AI SDK v6, centered on the insight that giving an agent a file system transforms its behavior—it follows through on long tasks, stays on track, and builds on prior work. He walks through creating a tool loop agent from scratch, adding provider-executed web search with typed UI components, and integrating Vercel's persistent named sandboxes that snapshot state after inactivity. The agent gains a bash tool for file system access, a memories.md file for persistent memory injected into instructions each turn, and instructions to generate Python scripts for repeatable tasks so it accumulates tools across sessions. Albanese explains how these patterns scale: an internal agent called D0 reduced customer support tickets by 90% with a 95% satisfaction rate, and his personal coding agent ran for 104 minutes in one turn, using 316 tool calls and only 32% of GPT-5.4's context window with zero compaction. The session concludes with a preview of a larger sub-agent system that uses durable workflow steps and background sub-agents to keep the main thread under 7,000 tokens.

Lessons from Trillion Token Deployments at Fortune 500s — Alessandro Cappelli, Adaptive ML

May 12, 2026 · 18:35

Alessandro Cappelli, co-founder of Adaptive ML, argues that 95% of GenAI pilots fail to reach production because they rely on proprietary models or instruction fine-tuning, which lack systematic feedback integration. Reinforcement learning (RL) is the only post-training technique that mathematically incorporates defects, business metrics, and production signals to continuously improve models. RL enables smaller, cheaper, faster models that enterprises can own—critical for scaling use cases like AT&T’s transcript summarization or Manulife’s agents. For agents, RL naturally fits because it was designed for environments; synthetic data is generated as a byproduct of environment training, not a prerequisite. Reward signals come from business KPIs (e.g., containment rate) or LLM judges defined by human rubrics in hours, not weeks. Adaptive ML’s Adaptive Engine abstracts away RL complexity (orchestrating four models for PPO) and provides pre-built recipes to industrialize model deployment.

Why TTS Models Now Look Like LLMs — Samuel Humeau, Mistral

May 9, 2026 · 22:26

Mistral AI scientist Samuel Humeau explains why text-to-speech models now resemble LLMs, using an autoregressive transformer with neural audio codecs to compress 200 kbps audio into ~500 tokens per second. He demos Mistral's open-weight TTS model, cloning a voice from a few seconds of reference audio and generating speech with 17 ms first-token latency via frame-by-frame streaming. The talk covers the codec-backbone-decoder pipeline, the model's 4B-parameter backbone generating 37 tokens per 80 ms frame via a diffusion decoder, and a live voice agent answering conference schedule questions. Humeau notes the next latency win is handling streaming text input from an LLM, with no settled architecture yet, while keeping the encoder for voice cloning proprietary.

Voice AI: when is the "Her" moment? — Neil Zeghidour, CEO, Gradium AI

May 9, 2026 · 19:27

Neil Zeghidour, CEO of Gradium AI, argues that voice AI remains far from the 'Her' ideal because cascaded systems (speech-to-text, LLM, text-to-speech) suffer from high latency—tool calls alone add 500ms to 4 seconds—while human response time is ~200ms. Speech-to-speech models reduce latency but are half-duplex, meaning they cannot handle overlapping speech or backchanneling, unlike Moshi, Gradium's full-duplex model. However, Moshi lacked intelligence, tool calls, and paralinguistic understanding—the ability to infer tone, hesitation, or discomfort from voice, which is stripped away in text. Cost is another barrier: TTS bills burn through fundraising before user bases grow. Gradium's solution is Phonon, an on-device TTS model running on smartphone CPUs, offering privacy and eliminating API fees. The path forward requires combining full-duplex natural conversation with the reliability and smarts of cascaded systems.

Agent Optimization with Pydantic AI: GEPA, Evals, Feedback Loops — Samuel Colvin, Pydantic

May 7, 2026 · 1:20:40

Samuel Colvin shows how to improve production agents without redeploying using Pydantic AI's GEPA, evals, and Logfire managed variables. GEPA uses a genetic algorithm to evolve prompts, achieving 96.7% accuracy — up from 87% with a simple prompt and 92% with an expert prompt — on a Wikipedia-based political relations task. Managed variables let you update prompts, models, and parameters live via a web interface and A/B test targeting. Evals compare against a golden dataset of 650 MPs; Colvin runs 65 cases in 30 seconds using GPT-4.1. He discusses that prompt optimization is most valuable with private data, that implicit user feedback (e.g., user's next action) can build golden datasets, and that overfitting to small test sets is a risk — the GEPA optimizer may exclude valid relations like 'uncle' if they don't appear in the training split.

Vibe Engineering Effect Apps — Michael Arnaldi, Effectful

May 7, 2026 · 1:43:04

Michael Arnaldi demonstrates that cloning the Effect library's repository into a project, rather than relying on prompts, is the most effective way to give coding agents the context they need to build reliably with Effect. Starting from an empty repo, he sets up a Bun, Vitest, and TypeScript project, adds the Effect repo as a git subtree, and creates an agents.md to guide the agent. He then uses the agent to research patterns, resulting in a fully functional Todo HTTP API with OpenAPI docs, SQLite persistence, and tests — all from scratch in under two hours. The workshop stresses that treating the library code as part of the project, combined with strict diagnostics and pattern files, makes agents effective even in unfamiliar codebases. Arnaldi also discusses the importance of workflow systems like Effect Cluster for long-running AI processes, where server failures become likely.

Everything You Need To Know About Agent Observability — Danny Gollapalli & Zubin Koticha, Raindrop

May 7, 2026 · 50:25

Zubin Koticha and Danny Gollapalli of Raindrop argue that agent observability must shift from evals to production monitoring because agent failures are non-deterministic and unbounded. They break down explicit signals (tool error rate, latency, cost) and implicit signals (user frustration, refusals, task failure) detected by trained classifiers and regex, emphasizing that aggregate patterns even from imperfect regex are valuable. Experiments let teams ship changes to a percentage of users and compare semantic signal rates, with statistical relevance often reached after a few hundred events. Self-diagnostics—a simple tool and system prompt—enable agents to report their own failures, capability gaps, and even self-correction behavior, as demonstrated in a live coding agent demo where a disabled write tool caused the agent to use bash and then report the bypass. The episode covers alerting, trace visualization, data export to BigQuery/Snowflake, and the challenge of managing fast-paced experimentation at scale.

Full Walkthrough: Writing & Using Skills — Nick Nisi and Zack Proser

May 6, 2026 · 1:21:03

Zack Proser and Nick Nisi, Developer Experience Engineers at WorkOS, teach how to write portable, composable skills — single markdown files with optional scripts — that train AI tools like Claude, Codex, and Cursor to perform specific tasks consistently. They demonstrate building a repo roast skill that uses deterministic Git commands via script interpolation and progressive disclosure to load targeted references only when needed. The workshop covers structuring skills with front matter (name, description for LLM routing), adding confidence scoring and constraints to improve output, and sharing skills as .skill files or through marketplaces. Advanced topics include using skills in non-coding workflows (e.g., Slack–Linear automation) and composing skills in the WorkOS CLI, which uses the Claude Agent SDK to install auth. Attendees learn to iterate skills by reflecting on past conversations and leveraging Claude's built-in skill builder for evaluation.

MCP UI: Extending the frontier — Liad Yosef and Ido Salomon, MCP Apps

May 6, 2026 · 22:21

In this episode, Ido Salomon and Liad Yosef explain MCP Apps, an MCP extension that lets tools send interactive, branded UI instead of plain text responses inside chat hosts like ChatGPT, Claude, VS Code, Cursor, and Copilot. They argue that text-based chat reduces companies to a wall of text, erasing identity, while MCP Apps preserves branding by returning HTML resources that hosts render as secure, interactive components. The architecture ensures every user click sends a message back to the host (not directly to the backend), keeping interactions in context—as demonstrated with a PostHog funnel analysis in Claude. Adoption includes Shopify, Hugging Face, GitHub, ChatGPT, and Claude, with ChatGPT recommending MCP Apps for building ChatGPT Apps. They note 800 million weekly ChatGPT users (10% of world population), calling this a once-in-20-years opportunity to rethink app distribution. Future work covers reusable views, model-UI interaction, and interoperability with generative UI protocols like A2UI and WebMCP, aiming to standardize UI in chat as the new web.

Training an LLM from Scratch, Locally — Angelos Perivolaropoulos, ElevenLabs

May 4, 2026 · 1:21:26

Angelos Perivolaropoulos from ElevenLabs walks through building a small GPT-2-like LLM from scratch on a local machine, demonstrating that the core techniques used by major labs are accessible in a few hundred lines of PyTorch code. The workshop uses character-level tokenization (65 tokens) on a Shakespeare dataset to enable fast training with limited compute. The model architecture includes multi-head self-attention, MLP layers, residual connections, and layer normalization, totaling 10 million parameters across six transformer blocks with a 256-token context window. The training loop employs next-token prediction with a warm-up cosine decay learning rate schedule and validation loss to detect overfitting. Inference uses temperature sampling (default 0.7) and top-k sampling to improve creativity. Perivolaropoulos explains that audio and multimodal models share the same transformer foundation but differ in tokenization (e.g., mel-spectrograms for audio) and use specialized losses like L2 or KL divergence, while reasoning models result from post-training base models with high-quality chain-of-thought data.

Skill Issue: How We Used AI to Make Agents Actually Good at Supabase — Pedro Rodrigues, Supabase

May 4, 2026 · 1:18:41

Pedro Rodrigues, AI tooling engineer at Supabase, demonstrates how to write, test, and iterate on Agent Skills to make agents actually good at real systems like Supabase's performance-review app. He explains progressive disclosure—skills load only a short description first, letting the agent decide when to pull in full instructions—and shows that combining skills with MCP tools gives agents both context and actions. In a live demo, Rodrigues reveals a common failure: creating a database view without a security invoker flag bypasses row-level security, letting all users see everyone's salary data. Adding a skill with security checklists guides the agent to include the flag, fixing the bug. He then introduces an eval-driven development cycle using an eval.json file and an LLM-as-judge to automate testing across conditions with and without the skill, cautioning that writing accurate evals is tricky because non-deterministic outputs can mislead. The session offers a practical framework for validating skills, avoiding pitfalls, and measuring what actually improves agent behavior.

Ralph Loops: Build Dumb AI Loops That Ship — Chris Parsons, Cherrypick

May 4, 2026 · 1:48:26

Chris Parsons argues that dumb loops—simple while loops where an AI agent repeatedly executes the same task, evaluates its output, and iterates—outperform complex multi-agent orchestration and planning graphs. In this hands-on workshop, he builds a Pomodoro timer using Claude Code, implementing tickets one by one in a loop that self-corrects with each pass. He demonstrates the 'loop' command in Claude Code for scheduled, continuous execution and shows how to use sub-agents for adversarial validation to avoid self-confirmation bias. The talk covers real-world applications such as a worker loop that processes project steps overnight and a morning loop that generates daily briefings, all designed to offload routine work while keeping human oversight for irreversible decisions. Parsons emphasizes that the bottleneck in AI-augmented teams is often the review process, not coding speed, urging teams to identify and fix their biggest constraint first.

Mergeable by default: Building the context engine to save time and tokens — Peter Werry, Unblocked

May 3, 2026 · 1:41:25

Peter Werry of Unblocked argues that context engines—systems that supply AI agents with only the relevant organizational context—are critical to avoid agent doom loops and wasted tokens. He debunks three myths: naive RAG, connecting MCP servers, and bigger context windows do not solve the context problem. Werry describes building a social engineering graph to identify experts and distill team best practices, and shares hard lessons including hiding conflicts and caching answers. In a benchmark task, Unblocked's context engine reduced a 2.5-hour, 21-million-token task to 25 minutes and 10 million tokens. The talk offers a practitioner's guide to building context engines with conflict resolution, personalization, and access control.

Mastering AI Pricing — Mayank Pant, Stripe

May 1, 2026 · 24:19

Mayank Pant from Stripe explains that AI companies, growing 3x faster than traditional SaaS, face margin risk from power users and unpredictable compute costs, making hybrid pricing (base fee + usage fee) essential—56% of AI leaders now use it. He presents a five-step framework: define customer-perceived value (e.g., automation, augmentation, enhanced service, improved results), choose a charge metric (consumption, workflow, or outcome-based), adopt hybrid pricing with guardrails like usage caps and automated notifications, and iterate pricing frequently—84% agree fast adaptation is a competitive advantage. Pant illustrates with examples: Gamma charges per deck (not API calls), Intercom prices per resolved ticket. To keep customer-facing prices stable while changing features, he advises abstracting value with credits (e.g., 100 credits/month) that can be internally revalued. Stripe's billing infrastructure supports this iteration, with 78% of AI companies building on Stripe using its subscription, usage, and hybrid billing, plus Metronome for enterprise contracts.

Shipping complex AI applications — Braintrust & Trainline

May 1, 2026 · 1:38:34

Giran Moodley of Braintrust, joined by Trainline's Oussama Hafferssas and Mayank Soni, demonstrate how to ship production-grade multi-step AI agents by combining rigorous tracing, evaluation with golden datasets, and automated online scoring. They walk through building a support triage agent that progresses from a single prompt to a five-stage tool-calling pipeline, where tracing captures latency, tokens, and costs per step for deep debugging. Trainline shares using Braintrust to run offline evaluations before switching LLM models and to enable cross-functional self-service. The workshop covers identifying failure modes via production logs, tightening prompts, and re-evaluating to complete the feedback loop, emphasizing that observability and iterative evaluation are essential for moving from prototype to reliable production systems.

OpenAI Codex Masterclass — Vaibhav Srivastav & Katia Gil Guzman

Apr 29, 2026 · 1:01:59

Katia Gil Guzman and Vaibhav Srivastav showcase OpenAI's Codex as a full software engineering agent, not just a coding assistant, powered by models like GPT-5.4 and GPT-5.3 Codex Spark. They demonstrate plugins bundling skills and MCP servers, automations that run scheduled tasks (e.g., Slack or Gmail summaries), and code review that 100% of OpenAI pull requests use by default. Subagents parallelize work across up to 20 agents for tasks like reviewing 45 persona files, with each subagent configurable by model, sandbox mode, and MCP access. New features include Guardian approvals for privileged operations, hooks for custom triggers, and a Cloud Code plugin. The episode also highlights the milestone of 3 million weekly active users, tripled since January, and covers security scanning for vulnerabilities.

One Login to Rule Them All: Cross-App Access for MCP — Garrett Galow, WorkOS

Apr 28, 2026 · 23:24

Garrett Galow from WorkOS introduces Cross-App Access (XAA) for MCP, solving the problem of repeated OAuth consent screens when connecting agents to multiple services. The flow leverages a three-way trust between the MCP client, server, and an Identity Provider like Okta: a single SSO login issues an IDJag token that is exchanged for short-lived access tokens across all MCP servers without manual intervention. A demo shows Figma automatically connecting after an Okta login. The approach improves security—if the IdP session is revoked, tokens cannot refresh—and requires minimal IT setup: granting permission for the client to request server access. Currently only Okta supports XAA; Azure/Entra does not yet. The session also notes that authorization scopes are not handled by default but are a future consideration.

Scaling GitHub for your Agents — Sam Morrow, GitHub

Apr 27, 2026 · 20:35

Sam Morrow, GitHub's MCP server lead, details the architectural challenges and solutions for scaling a remote MCP server to 7 million weekly tool calls. He explains how tool proliferation degraded agent performance—LangChain's research confirmed more tools confuse agents—leading to innovations like tool sets and dynamic discovery, though 100% of users stuck with defaults. To reduce context, GitHub cut tool descriptions by 49% and trimmed output tokens by 75% on list pull requests. Security is addressed via OAuth 2.1 with PKCE and step-up auth; they rejected dynamic client registration to avoid unbounded app databases and rate-limiting issues. The stateless server uses Redis for session storage and builds a fresh server instance per request, enabling horizontal scaling without session affinity. Metrics include 11 million Docker downloads, 30,000 stars, 4,000 forks, and 126 contributors. Morrow predicts compositional tools and automatic server discovery will make thousands of tools the norm.

What Do Models Still Suck At? - Peter Gostev, Arena.ai, BullshitBench

Apr 24, 2026 · 20:24

Peter Gostev presents data from his BullshitBench and Arena.ai to argue that language models still struggle with nonsense questions and expert-level tasks, despite benchmark charts showing relentless improvement. His BullshitBench reveals that only Claude Sonnet 4.5 and some Qwen models consistently push back on nonsense, while GPT and Gemini models accept it 50% of the time. Arena.ai's dissatisfaction rate among top 25 models has improved from 17% pre-reasoning to about 9% currently, but remains non-zero, and expert categories like gaming, magic, finance, and law show minimal improvement. For software expert prompts, dissatisfaction dropped from 23.5% in Q2 2024 to 13% in Q1 2026, but gaming is a persistent weakness where models fail to create engaging game mechanics. Gostev warns that narrow benchmarks overstate progress and urges focusing on the full distribution of real-world tasks.

Building Generative Image & Video models at Scale - Sander Dieleman, Google DeepMind

Apr 21, 2026 · 40:46

Sander Dieleman, a research scientist at Google DeepMind, explains the behind-the-scenes process of building generative image and video models at scale, focusing on diffusion models. He emphasizes that data curation is often underrated and more impactful than model tweaking. Dieleman details how latent diffusion models use learned autoencoders to compress pixels into smaller representations, enabling training on high-resolution video. He describes diffusion as 'spectral autoregression' that generates images coarse-to-fine, and notes that guidance—amplifying the difference between conditional and unconditional predictions—massively improves sample quality at the cost of diversity. On sampling, he contrasts deterministic and stochastic approaches, and covers distillation techniques like consistency models that reduce steps. Dieleman also discusses control signals beyond text, such as camera motion in video generation, and the importance of post-training for conditioning.

Full Workshop: Build Your Own Deep Research Agents - Louis-François Bouchard, Paul Iusztin, Samridhi

Apr 20, 2026 · 1:57:03

Louis-François Bouchard, Samridhi Vaid, and Paul Iusztin built a combined system of a deep research agent and a deterministic writing workflow to automate high-quality technical content creation, trading flexibility for control. The research agent uses an MCP server (FastMCP) with tools for Google search, YouTube analysis, and report compilation, powered by Gemini and orchestrated by Claude Code. The writing workflow is a static pipeline that loads a user guideline, writing profiles (structure, terminology, character), and few-shot examples into a system prompt, then applies an evaluator-optimizer loop: a reviewer outputs structured pydantic objects with profile/location/comment to flag violations, and the editor applies them in priority order (guideline > research > profile). They stress that research demands autonomy while writing needs constraint, and show how to avoid AI slop by banning specific words and using profiles. For observability, they use Opik to capture all traces, and for evals they built a 20-sample dataset from real LinkedIn posts, split into train/dev/test, to calibrate an LLM judge (binary pass/fail) that they run on dev and test splits to measure F1 and detect…

The New Application Layer - Malte Ubl, CTO Vercel

Apr 20, 2026 · 18:52

Vercel CTO Malte Ubl argues that AI engineering is the legitimate successor to web development and that the real value lies in the application layer, not the model labs. He identifies four effective agent archetypes—24/7 support, compressed research, surfacing existing information, and eliminating boring work—and reveals that over 60% of vercel.com page views now come from AI agents. Ubl predicts model companies will commoditize, driving costs down and empowering engineers, while citing Europe's leadership in AI engineering through Vercel's AI SDK, Pi (a coding agent from Austria), and OpenClaw. He stresses the need for open-mindedness toward paradigm shifts and new infrastructure, such as sandboxed agent runtimes, and warns of impending security challenges.

State of the Claw — Peter Steinberger

Apr 17, 2026 · 44:12

Peter Steinberger, creator of OpenClaw, presents a five-month update on the world's fastest-growing open-source project. He details the project's staggering growth—30,000 commits, nearly 2,000 contributors—and the immense security burden: 1,142 advisories (16.6 per day) with 99 critical, often AI-generated slop that demands human vetting. Steinberger refutes media fearmongering, citing how researchers ignore security docs to fabricate scary scenarios. He clarifies OpenAI did not buy OpenClaw; he joined the company while establishing the OpenClaw Foundation to remain vendor-neutral. He emphasizes the importance of local models for data sovereignty and describes his coding workflow of running 5-6 agent sessions simultaneously, iterating on taste and personality. Future visions include ubiquitous agents, 'Dreaming' for memory reconciliation, and modular plugins. For engineers, he champions taste, system design, and learning to say no.

Harness Engineering: How to Build Software When Humans Steer, Agents Execute — Ryan Lopopolo, OpenAI

Apr 17, 2026 · 46:21

Ryan Lopopolo, a member of technical staff at OpenAI, argues that software engineers must shift to 'harness engineering' where agents produce all code and humans focus on delegation, system design, and guardrails. He describes how his team banned editors, using Codex with skills to manipulate apps, and reduced review friction by automating lint rules and review agents. Lopopolo explains that code is free but human time and model context are scarce, so teams should structure repositories for consistency and use 'garbage collection days' to eliminate recurring slop. He shares practical techniques like writing tests about source code (e.g., file length limits) and using review agents that inject prompts. His future vision involves agents autonomously executing long-horizon work based on token budgets and success metrics, freeing humans for higher-level activities.

Paperclip: Open Source Human Control Plane for AI Labor — Dotta Bippa

Apr 15, 2026 · 24:34

Dotta Bippa introduces Paperclip, an open-source human control plane for AI labor that lets you manage an org chart of agents to run a zero-human company. The tool enables you to hire employees, set goals, automate jobs, and bring your own agent (e.g., Gemini, Claude, Codex) with skills and instructions. Paperclip supports reliable workflows like QA reviews and approvers, and routines for recurring tasks such as summarizing PRs or writing changelogs. Dotta demonstrates creating a Remotion video celebrating Paperclip's 40,000 GitHub stars (now 50,000) by instructing the CEO agent to hire a video writer and apply brand guidelines—something that would have taken a week becomes an afterthought. Paperclip is not just for coding; it handles marketing, sales, and finance. Upcoming features include CEO chat, maximizer mode, multi-user support, cloud deployments, and a desktop app, all aimed at giving humans control over AI labor.

OpenRAG: An open-source stack for RAG — Phil Nash

Apr 8, 2026 · 15:52

Phil Nash introduces OpenRAG, an open-source RAG stack from IBM combining Docling, OpenSearch, and Langflow, arguing that RAG remains hard and custom despite claims it's dead — every business has unique data requiring more than simple vector search. Docling parses PDFs, audio, and more with specialized pipelines, outputting hierarchically chunked text. OpenSearch provides hybrid vector and keyword search using JVector for live indexing and disk-based ANN. Langflow enables visual agentic retrieval where the LLM decides searches with tools like a calculator and MCP servers. The stack supports local models via Ollama, cloud connectors for Google Drive and SharePoint, and an API. Nash demonstrates adding guardrails in Langflow and invites contributions to the open project.

Let LLMs Wander: Engineering RL Environments — Stefano Fiorucci

Apr 8, 2026 · 40:35

Stefano Fiorucci demonstrates how to build Reinforcement Learning environments for language models using the open-source Verifiers library, arguing that training small models with verifiable rewards can surpass large closed models on specific tasks. He maps classic RL concepts to LLMs, introduces Verifiers components for single-turn, multi-turn, and tool environments, and then walks through an experiment where he takes LiquidAI's LLM 2 — a small open model — and transforms it into a tic-tac-toe master via supervised fine-tuning and GRPO-based reinforcement learning. After training, the model dominates random opponents and draws 85% of games against optimal ones, eventually outperforming GPT-5 Mini against identical optimal opponents. Fiorucci shares practical lessons: large batch sizes (≥256) ensure stable training, hidden biases in opponent algorithms can skew results, and starting from a base model (not a reasoning model) avoids truncated thinking traces. He concludes that if you can define a clear reward signal, you can build an environment and train a small specialized model to beat a large closed model at a fraction of the cost.

How METR measures Long Tasks and Experienced Open Source Dev Productivity - Joel Becker, METR

Jan 19, 2026 · 1:15:52

Joel Becker of METR argues that AI benchmark scores are soaring while real-world developer productivity barely budges, reconciling this gap by presenting METR's time horizon measurements and a 16-developer RCT showing no significant speedup from AI on mature open-source projects. He cites reliability issues, task distribution mismatches, and the J-curve effect from Meta's developer data, noting that experienced users still fail to accelerate. The episode explores why AI struggles with messy enterprise data (e.g., LinkedIn's 5,000 'impressions' tables), the failure of computer-use agents in 'Agent Village,' and the possibility that hardware constraints may slow progress. Becker also previews new metrics like 'watched versus unwatched' time horizons and future studies on greenfield coding and data science.

Identity for AI Agents - Patrick Riley & Carlos Galan, Auth0

Jan 14, 2026 · 1:22:12

Patrick Riley and Carlos Galan from Auth0 (Okta) present new identity and access management features for AI agents, including Async Auth, Token Vault, and MCP integration. Token Vault securely stores and refreshes upstream resource tokens, enabling agents to access APIs on behalf of users. Async Auth allows agents to request user approval for risky operations, such as placing an order, via out-of-band notifications like push to the Guardian app. They demonstrate a Next.js chatbot that uses Token Vault to read a user's stock portfolio and Async Auth to require explicit consent before executing trades. The MCP server is secured with OAuth 2.0, supporting Dynamic Client Registration (DCR) and fine-grained scopes. The solution bridges user identity, agent identity, and upstream API authorization, ensuring agents can act autonomously but with user-defined boundaries.

OpenAI + @Temporalio : Building Durable, Production Ready Agents - Cornelia Davis, Temporal

Jan 12, 2026 · 1:18:30

Cornelia Davis, developer advocate at Temporal, demonstrates how the OpenAI Agents SDK and Temporal combine to build durable, production-ready AI agents. The integration, co-developed by OpenAI and Temporal, wraps agentic loops in Temporal workflows, providing automatic retries, event sourcing, and state management that survive process crashes and network failures. A live demo shows a weather alert agent that calls an LLM, invokes tools like get_weather_alerts and get_ip_address, and recovers seamlessly after the worker is killed. Davis highlights that Temporal’s architecture treats processes as logical entities, allowing agents to run for days and handle human-in-the-loop delays without developer-managed infrastructure. She also explains handoffs and micro-agent orchestration, and points to the AI Cookbook on docs.temporal.io with ready-to-run recipes including the OpenAI Agents SDK integration.

DSPy: The End of Prompt Engineering - Kevin Madura, AlixPartners

Jan 8, 2026 · 1:13:13

Kevin Madura of AlixPartners argues that building robust enterprise AI applications requires shifting from brittle prompt engineering to programming with LLMs using DSPy, a declarative framework that treats prompts as implementation details optimized by the system. He demonstrates how typed interfaces (Signatures) and modular logic (Modules) allow developers to focus on control flow while deferring implementation to the LLM, with Adapters controlling prompt formats (e.g., JSON vs. BAML) to improve performance by 5-10%. The talk's core is Optimizers (like MIPRO and JEPA), which automatically tune prompts by learning from data, shown improving a time entry corrector from 86% to 89% accuracy. Real-world examples include routing files by type (SEC filings vs. contracts), using a 'poor man's RAG' with attachments for multimodal documents, and a boundary detector that segments legal documents from images. Madura emphasizes that DSPy enables transferability across models (e.g., GPT-4.1 to GPT-4.1 Nano) and addresses cost concerns by allowing offline optimization to reduce LLM calls.

Build a Prompt Learning Loop - SallyAnn DeLucia & Fuad Ali, Arize

Jan 6, 2026 · 52:08

SallyAnn DeLucia and Fuad Ali from Arize present prompt learning, a method to continuously improve AI prompts by using feedback from evals and human annotations. They argue that agent failures often stem from weak instructions rather than weak models, and that adding rules to system prompts can yield 15% improvements on SWE-Bench without fine-tuning or architecture changes. The talk compares prompt learning to GEBA, noting it achieves better results in fewer loops by leveraging text explanations. A case study shows Kline's performance improved from 30% to 45% on SWE-Bench-Lite after optimizing prompts. The workshop demonstrates building an optimization loop that ingests input-output pairs, evaluates them, and iteratively refines the system prompt using OpenAI.

Building Intelligent Research Agents with Manus - Ivan Leo, Manus AI (now Meta Superintelligence)

Dec 30, 2025 · 1:21:30

Ivan Leo of Manus AI introduces the Manus API and Manus 1.5, positioning the platform as a general action engine for building intelligent research agents. He demonstrates how the API enables asynchronous task dispatching, file uploads with automatic 48-hour deletion, and webhooks for scalable workflows. The workshop covers creating a Slack bot that integrates Manus's browser operator, file attachments, and connectors to private data sources like Notion. Leo shows real-time demos including a custom French learning app, a conference event scraper with calendar integration, and an invoice processing bot that references company policies. He also discusses the roadmap for memory persistence, document exporting (PPTX/PDF), and enhanced permission systems for browser automation.

Jack Morris: Stuffing Context is not Memory, Updating Weights is

Dec 29, 2025 · 1:02:44

Jack Morris argues that large language models fail at niche, long-tail knowledge tasks, such as optimizing AMD GPU kernels or answering private company queries, because they rely on context windows and RAG, which suffer from quadratic self-attention costs and context rot. He advocates for a third paradigm—training knowledge directly into model weights—using synthetic data generation (e.g., synthetic continued pretraining from Stanford) to expand small datasets and parameter-efficient methods like LoRA or memory layers to avoid catastrophic forgetting. Morris demonstrates that full fine-tuning on a 3M 10-K report causes the model to only regurgitate exact sentences, whereas generating diverse synthetic question-answer pairs enables better generalization. He notes that RL-based fine-tuning (e.g., GRPO) can achieve improvements with as few as 14 parameters, while memory layers offer the best trade-off between learning and forgetting. The episode also explores temporal information handling, federated learning resurgence, and the practical decision boundary between RAG and weight-based injection based on data freshness and volume.

AGI: The Path Forward – Jason Warner & Eiso Kant, Poolside

Dec 27, 2025 · 15:56

Jason Warner and Eiso Kant, co-founders of Poolside, present their vision and roadmap towards AGI-level capabilities for knowledge work, demonstrating their second-generation model Malibu Agent converting ADA code to Rust live on stage. They argue that next-token prediction paired with reinforcement learning is the key breakthrough, a contrarian bet they made two and a half years ago. The episode centers on their work in high-consequence code environments for defense and government, where agents must operate with tight permissions. They announce a large compute cluster of over 40,000 GB300s coming online and a public API release early next year via AWS Bedrock. Warner recounts meeting Kant through a failed GitHub acquisition, and Kant invites the audience to build with their models, emphasizing that future agents will handle tasks over days as intelligence scales.

Shipping AI That Works: An Evaluation Framework for PMs – Aman Khan, Arize

Dec 26, 2025 · 1:26:16

Aman Khan, AI PM at Arize, presents a framework for product managers to evaluate LLM-powered products beyond gut-feel 'vibe checks.' He demonstrates building an AI trip planner with multi-agent LangGraph, then using Arize's tracing and prompt playground to iterate on prompts. Khan shows how to create datasets from production traces, run A/B experiments on prompts, and use LLM-as-a-judge evals for friendliness and discount offers, comparing against human labels to refine evaluators. He argues evals are the new requirements docs, enabling PMs to own the product experience by writing acceptance criteria as eval datasets. The talk covers building eval teams, handling variance with temperature settings, and continuously improving golden datasets with hard examples, citing real-world analogies from self-driving cars at Cruise.

How Claude Code Works - Jared Zoneraich, PromptLayer

Dec 26, 2025 · 1:05:43

Jared Zoneraich from PromptLayer dissects how Claude Code works, arguing its success stems from a radical simplification: a single-threaded master loop with tool calls, relying on capable models rather than complex DAG-based architectures. He breaks down the core internal tools—Bash, FileEdit, Grep—and the to-do planning mechanism that is purely prompt-based and not deterministically enforced. Zoneraich contrasts Claude Code with other coding agents: Codex (Rust core, kernel-level sandboxing), Amp (handoff context management, model-agnostic), and Cursor Composer (UI-first, distilled model). He emphasizes that Bash is the most important tool, acting as a universal adapter, and that context management via sub-agents and summarization is critical. Key takeaways include trusting the model, simple design, rigorous tool testing, and that different agent philosophies (not one winner) will serve different use cases.

The 3 Pillars of Autonomy – Michele Catasta, Replit

Dec 22, 2025 · 24:42

Michele Catasta, VP of AI at Replit, argues that true autonomy for coding agents serving non-technical users rests on three pillars: verification, context management, and parallelism. Replit’s agent uses autonomous testing—writing Playwright code instead of browser-use tools—to catch broken features (over 30% initially) without human feedback, cutting cost and latency by an order of magnitude. Context management relies on sub-agent orchestration rather than massive context windows, boosting memories per compression from ~35 to 45–50. Parallelism, implemented via a core-loop orchestrator, dynamically decomposes tasks to avoid merge conflicts and keep users engaged instead of waiting hours. Catasta emphasizes that autonomy means scoped technical decisions, not just long runtimes, enabling knowledge workers to build software without needing a 'driving license.'

Moving away from Agile: What's Next – Martin Harrysson & Natasha Maniar, McKinsey & Company

Dec 12, 2025 · 21:55

McKinsey's Martin Harrysson and Natasha Maniar argue that enterprises must overhaul their people and operating models—moving beyond Agile to AI-native workflows—to capture the full potential of AI in software development. They identify bottlenecks like work allocation, manual code review, and tech debt that limit gains to 5-15% despite promising individual productivity stories. To scale, they advocate AI-native workflows (spec-driven development, continuous planning) and AI-native roles (smaller pods of 3-5 with consolidated roles). In a client study with a bank, interventions led to 60x increase in agent consumption, 51% increase in code mergers, and faster delivery tied to business priorities. They emphasize change management: 70% of companies haven't changed roles, and top performers invest in hands-on upskilling, measurement systems that track outcomes beyond adoption, and incentives to drive adoption.

Agent Reinforcement Fine Tuning – Will Hang & Cathy Zhou, OpenAI

Dec 9, 2025 · 16:55

Will Hang and Cathy Zhou of OpenAI's fine-tuning team introduce Agent Reinforcement Fine-Tuning (RFT), a method to improve AI agents by training them end-to-end on tasks involving tool calls and reasoning. They define an agent as a model that interleaves reasoning with external tool interactions, unlike regular models. The hierarchy of optimization moves from prompt engineering to task optimization to RFT, which changes model weights based on a custom reward signal. New features allow models to call tools via public endpoints and use custom rewards hosted externally. Case studies show concrete gains: Cognition improved code edit planning by 10 points with 1,000 examples and reduced tool call steps from 8–10 to 4; Codto's deep research agent boosted recall by 6% while cutting long-tail tool calls (over 15) down to 2–4; Cosine achieved state-of-the-art on enterprise code benchmarks by using strict graders that reward only pass-tested code; and Macco wrote GPU kernels from just 100 PyTorch prompts, beating SOTA by 72% after addressing reward hacking with seven edge-case detectors. Four principles for success: define tasks unambiguously, mirror production traffic in train/eval sets,…

RL Environments at Scale – Will Brown, Prime Intellect

Dec 9, 2025 · 18:30

Will Brown of Prime Intellect argues that scaling reinforcement learning environments beyond engineering—to community and accessibility—is key to broadening AI research. He presents Prime Intellect's open-source stack, including Verifiers for building environments and the Environments Hub for sharing them, as a way to turn any task harness into an RL training or evaluation loop. Brown demonstrates how environment-based fine-tuning boosted a Qwen 3 4B model from 55% to 89% on a Wikipedia search task, matching much larger models. He frames environments as the 'web apps of AI research'—simple to start, but capable of capturing product complexity, as seen with Cursor's Composer and OpenAI's Codex. Prime Intellect validated this approach by training the 100B-param Intellect 3 on 500 GPUs, and will soon release Lab, a platform to run environments without managing infrastructure.

Efficient Reinforcement Learning – Rhythm Garg & Linden Li, Applied Compute

Dec 9, 2025 · 20:19

Rhythm Garg and Linden Li, co-founders of Applied Compute, describe how their company uses efficient reinforcement learning (RL) to specialize large language models for enterprise tasks. They explain that synchronous RL wastes GPU time waiting on straggler samples—99% of arithmetic problems complete in ~40 seconds, but the tail takes 80 more seconds—so they adopt asynchronous pipeline RL. This method dedicates fixed GPUs to sampling and training, allowing continuous inference but introducing stale tokens (up to a tolerated staleness threshold) that require importance ratio corrections. To balance speed and stability, they model the system mathematically: using a roofline-based latency curve for sampling, per-GPU training throughput, and constraints on staleness and KV cache memory. Their simulations, parameterized by response length distributions, reveal an optimal GPU allocation that yields ~60% speedup over synchronous RL while keeping staleness within ML limits. This modeling lets them predict runtime and configure runs without expensive trial-and-error.

2026: The Year The IDE Died — Steve Yegge & Gene Kim, Authors, Vibe Coding

Dec 6, 2025 · 24:59

Steve Yegge and Gene Kim argue that by 2026, traditional IDEs will be obsolete, replaced by AI-driven coding agents. Yegge claims current tools like Claude Code are too hard for most developers, with high cognitive overhead and a single-agent bottleneck akin to a solo diver running out of oxygen. He predicts next year will bring multi-agent systems, comparing the shift from manual tools to CNC machines, and provocatively states that engineers using an IDE after January 1st are bad engineers. Gene Kim introduces their 'Vibe Coding' book and shares case studies: Booking.com saw double-digit productivity gains, Travelopia replaced a legacy app in six weeks with a small team, and a Fidelity leader Vibe Coded an app in five days that previously was estimated to take five months. They present the FAFO framework (Faster, Ambitious, alone/autonomous, Fun, Optionality) and cite DORA research showing trust in AI grows with usage. The episode concludes that Vibe Coding will reshape organizations, enabling non-developers to ship features and potentially compressing team sizes.

Government Agents: AI Agents Meet Tough Regulations — Mark Myshatyn, Los Alamos National Lab

Dec 6, 2025 · 16:31

Mark Myshatyn, Enterprise AI Architect at Los Alamos National Laboratory, describes how the lab is building AI agents for scientific research under strict federal regulations. He showcases an agent that reads fusion capsule papers, designs a hypothesis, and runs simulations on high-performance computing assets, integrating 60+ years of physics models. He stresses the need for explainability, isolation, and governance in government AI tools, citing OMB memoranda M2521 and M2522 mandating faster AI adoption with real-world impact. The lab partners with frontier labs like OpenAI for chem-bio safety work, and with NVIDIA and HPE for the Venado supercomputer. He urges startups to build for DoD IL5 environments and continuous compliance to succeed in federal procurement, noting Los Alamos holds petabytes of never-internet data and specializes in sensors like the ChemCam on Mars.

Future-Proof Coding Agents – Bill Chen & Brian Fioca, OpenAI

Dec 5, 2025 · 17:48

Bill Chen and Brian Fioca from OpenAI argue that coding agents are booming but fragile, with teams rebuilding harnesses every time models change. They present Codex, OpenAI's combined model and harness, as a stable abstraction layer that handles complexity like context compaction, parallel tool calls, and MCP integration. Key lessons include avoiding overprompting—letting the model use its trained habits rather than forcing foreign instructions—as demonstrated when GPT-5 ran slowly because users told it to examine every file. They describe patterns for using Codex as an SDK, enabling products like GitHub and Cursor to integrate without maintaining their own harness, and highlight future improvements in longer-horizon tasks and self-healing software. The episode concludes that building where models are going, with Codex's SDK, lets teams focus on differentiation rather than infrastructure rewrites.

Compilers in the Age of LLMs — Yusuf Olokoba, Muna

Nov 24, 2025 · 17:36

Yusuf Olokoba, founder of Muna, explains how they built a Python compiler that converts plain Python inference functions—like Google's 270M-parameter Gemma embedding model—into self-contained C++ and Rust binaries using LLMs within a verifiable pipeline. The process involves symbolic tracing to generate an intermediate representation, type propagation to infer types for native compilation, and LLM-driven code generation to mass-produce native implementations of Python operations. After compiling into a shared library, the model can be loaded via FFI from any language (e.g., Node.js) and exposed through an OpenAI-compatible client, enabling developers to run open-source models anywhere—locally, cloud, mobile—with minimal code changes. Olokoba details why they abandoned PyTorch FX due to its PyTorch-only focus and reliance on fake inputs, and how LLMs help scale the coverage of elementary operations. The talk argues that this compiler approach solves hybrid inference—small edge models working with large cloud models—by moving beyond Python and Docker to more portable, low-latency native binaries.

From Stateless Nightmares to Durable Agents — Samuel Colvin, Pydantic

Nov 24, 2025 · 22:13

Samuel Colvin of Pydantic demonstrates building production-grade durable AI agents using PydanticAI, Temporal, and Pydantic Logfire, arguing that stateless architectures fail at scale and that durable execution with checkpointing and recovery is essential. He shows a 20 questions game where two agents play, but with 20% simulated failures, restarts are avoided by wrapping agents in Temporal wrappers for automatic retries. Colvin then implements a Deep Research agent that plans searches, runs parallel web searches via Tavilli, and synthesizes results, all recoverable. In a live demo, killing the workflow and restarting resumes instantly, replaying cached LLM calls in milliseconds. He also previews Pydantic AI Gateway and demonstrates Pydantic Evals comparing Gemini, GPT-4.1, and Claude Sonnet 4.5 on performance and cost.

Hacking Subagents Into Codex CLI — Brian John, Betterup

Nov 24, 2025 · 13:39

Brian John, Principal Full Stack Engineer at Betterup, explains how to hack subagents into OpenAI's Codex CLI by using a 72-line wrapper script that launches child Codex processes as subagents, enabling context management without polluting the main agent's window. He details the minimum sandbox permissions required—workspaceright on both parent and child, plus disabling the rollout recorder—and notes that security risk is low per Meta's Rule of 2, though not zero. The wrapper uses file-based communication to avoid repetitive permission prompts, and subagents are defined in an agents.md file with configurable reasoning effort. John demonstrates a word counter and file writer agent, noting that Codex runs subagents serially and slower than Claude Code (e.g., 40 seconds for a simple task, up to 20 minutes for large codebases), but considers this acceptable for Codex's unattended design.

Building an Agentic Platform — Ben Kus, CTO Box

Aug 24, 2025 · 19:06

Ben Kus, CTO of Box, argues that building an agentic architecture early was critical to overcoming the limitations of pure LLM-based data extraction for enterprise unstructured content. Starting with simple LLM calls in 2023, Box faced accuracy issues on complex documents, OCR failures, and difficulty handling hundreds of fields. They pivoted to an agentic framework using directed graphs, multi-model voting, and LM-as-a-judge with reflection loops, which dramatically improved accuracy and allowed incremental improvement. This architecture also enabled deep research on customer data and clean separation between agentic logic and system scaling. Kus emphasizes that agentic thinking should be applied early, even to seemingly non-agentic tasks like metadata extraction, and that Box avoids fine-tuning in favor of prompt-based agentic orchestration.

How BlackRock Builds Custom Knowledge Apps at Scale — Vaibhav Page & Infant Vasanth, BlackRock

Aug 23, 2025 · 18:47

Infant Vasanth and Vaibhav Page of BlackRock explain how the world's largest asset manager scales custom AI knowledge apps for investment operations using a modular, Kubernetes-native framework. They detail challenges like prompt engineering for complex financial instruments, choosing LLM strategies (RAG, chain-of-thought), and deploying apps with cost control and compliance. Their sandbox lets domain experts rapidly iterate on extraction templates with inter-field dependencies and validations, while the app factory turns definitions into production apps. Key takeaways: invest in prompt engineering for domain experts, educate on LLM strategies, evaluate ROI, and design for human-in-the-loop in regulated environments. The framework compressed app development from eight months to days for use cases like setting up securities after IPOs or stock splits.

Wisdom-Driven Knowledge Augmented Generation at Scale - Chin Keong Lam, Patho AI

Aug 22, 2025 · 18:43

Chin Keong Lam, founder and CEO of Patho.ai, argues that Wisdom-Driven Knowledge Graphs (KAG) significantly outperform traditional RAG systems for complex quantitative analysis and expert-level advice. He presents a wisdom-decision-situation feedback loop where knowledge, experience, and insight feed into wisdom, mapped to a practical competitive analysis chatbot for a marketing client. The system uses N8n and a multi-agent architecture with a supervisory Wisdom Agent overseeing insight, knowledge, and strategy agents, all updating a centralized knowledge graph. Lam cites five reasons knowledge graphs beat RAG: capturing complex relationships, improved accuracy, scalability, rich multi-hop query capability, and seamless data integration. He demonstrates how vector RAG fails on numerical reasoning (e.g., Apple revenue query) while KAG returns precise evidence-based answers. Lam recommends a hybrid graph extraction combining LLM extraction with expert pruning, and reports benchmark results of 91% accuracy, 85% flexibility, and strong traceability and scalability.

The Next Unicorns: 7 Top AI startups from the HF0 Residency

Aug 21, 2025 · 22:16

Diego Rodriguez (Krea) presents an AI creative suite that generated 1M images/day for a Fox ad, while OpenHome debuts the first AI smart speaker with 10K developers and 500 free dev kits. Josh’s Coframe made $20M for a travel firm by making websites adaptive, and Eugene’s Featherless AI built QWERTY 72B without transformer attention, claiming scale is dead in favor of reliability. Jonas Bauer’s Upside uses LLMs to structure enterprise data, Lengyue’s OpenAudio introduces S1, the first instructable voice model beating ElevenLabs, and Alex Atallah’s OpenRouter provides a single API for all LLMs, growing 10–100% monthly.

#define AI Engineer - Greg Brockman, OpenAI (ft. Jensen Huang)

Aug 10, 2025 · 41:05

Greg Brockman, co-founder and president of OpenAI, discusses his journey from Stripe's first CTO (growing from 40 to 250 people) to leading AI development, sharing insights on independent study (completing three years of high school math in one year), the engineering-research partnership at OpenAI (emphasizing technical humility), and scaling challenges like ChatGPT hitting 1 million users in five days and ImageGen reaching 100 million in five days. He explains how Codex transforms coding (low double-digit percent of internal PRs written by Codex, 24,000 PRs merged in public GitHub in one day) and introduces "vibe coding" as an empowerment mechanism. In response to Jensen Huang's questions, Brockman predicts a future with diverse AI infrastructure (homogeneous accelerators as default, purpose-built for specific workloads) and a menagerie of domain-specific agents leveraging distillation, driving 10x more economic activity. He identifies basic research as a renewed scaling bottleneck alongside compute, data, algorithms, power, and money.

Designing AI-Intensive Applications - swyx

Aug 9, 2025 · 13:02

In this conference talk, swyx (host) proposes the search for a 'Standard Model' in AI engineering, analogous to physics' Standard Model, to guide practitioners. He examines candidate models: LMOS (LLM Operating System), LMSDLC (AI Software Development Lifecycle), and Anthropic's 'building effective agents' framework. Swyx introduces his own SPADE model (Sync, Plan, Analyze, Deliver, Evaluate), derived from building AI News, a daily tool that scrapes Discord, Reddit, and Twitter with thousands of LLM calls. He argues that the ratio of human input to AI output—ranging from 1:1 in ChatGPT to 0:n in ambient agents—is more useful than debating 'workflow versus agent.' The talk emphasizes that early SDLC stages (LLMs, monitoring, RAG) are becoming commodity, while real value comes from evals, security orchestration, and hard engineering. Swyx encourages attendees to identify and refine their own Standard Models to build products people want.

2025 is the Year of Evals! Just like 2024, and 2023, and … — John Dickerson, CEO Mozilla AI

Aug 6, 2025 · 19:14

John Dickerson, CEO of Mozilla AI and former co-founder of Arthur AI, argues that 2025 is finally the year for AI evaluation (evals) due to three converging forces: ChatGPT making AI tangible to the C-suite, enterprise budget freezes that funnelled funding into GenAI pet projects, and the rise of agentic systems acting autonomously. He traces how ML monitoring existed pre-2022 but lacked C-suite attention, with Jamie Dimon’s JPMC spending only $100 million on AI from 2017 to 2021. The ChatGPT launch on November 30, 2022, unlocked CEO discretionary budgets for GenAI, leading to 2023 science projects, 2024 production deployments, and 2025 scaling. Now, as agents reason and act, connecting evals to downstream business KPIs—risk mitigation, revenue gains—is a first-class discussion for CEOs, CFOs, CISOs, and CTOs. Dickerson notes that evaluation companies have shifted to multi-agent system monitoring, and while LLM-as-judge is popular, biases require human validation, with companies like Mercor hiring experts at $50–200/hour to lockstep multi-agent outputs.

Full Workshop: Realtime Voice AI — Mark Backman, Daily

Aug 3, 2025 · 1:09:41

In this hands-on workshop, Mark Backman and Aleś from Daily demonstrate how to build real-time voice AI agents using Pipecat, their open-source Python framework, and Google's Gemini Multimodal Live API. They explain that Pipecat's modular pipeline orchestrates audio transport, speech-to-text, LLM, and text-to-speech services, allowing developers to plug and play vendors like Deepgram, OpenAI, or Cartesia. The session highlights how speech-to-speech models like Gemini Live simplify architecture by handling transcription, LLM logic, and voice generation in one step, reducing latency to under 800 milliseconds. Key topics include the critical role of voice activity detection (VAD) for natural turn-taking, strategies for managing context windows to maintain accuracy, and the trade-offs between speed and reliability when using tool calls in real-time. Despite conference Wi-Fi issues, the team live-codes a functional bot and demonstrates interruption handling and the Word Wrangler game, showcasing Pipecat's production readiness with hundreds of thousands of daily calls.

Building Agents at Cloud Scale — Antje Barth, AWS

Aug 2, 2025 · 19:00

AWS Principal Developer Advocate Antje Barth demonstrates how to build and scale AI agents using cloud-native patterns, arguing that specialized agents will reinvent customer experiences. She showcases Alexa Plus, which orchestrates hundreds of expert systems across 600M+ devices and tens of thousands of services, and the Amazon Q Developer CLI agent, shipped in just three weeks. Barth introduces Strands Agents, an open-source Python SDK for building production-ready agents that supports multiple model providers (Claude, Llama, OpenAI) and over 20 prebuilt tools including memory, RAG, and multi-agent workflows. She demonstrates integrating MCP servers via Lambda with DynamoDB for session storage, and previews upcoming A2A protocol support and a future of personal agents connecting to agent stores.

State of Startups and AI 2025 - Sarah Guo, Conviction

Aug 2, 2025 · 23:52

Sarah Guo of Conviction argues that AI's value creation is massive and early, with companies like Cursor reaching $100M ARR in 12 months and Harvey exceeding $70M ARR. She predicts that by end of 2026, AI agents will ship code directly to production, voice AI will replace text for most business communication, and inference costs will drop below a cent per million tokens. Reasoning is a new scaling vector unlocking higher-stakes use cases, and agent startups have increased 50% in the last year. Multimodal models from HeyGen and Eleven are already rocketing past $50M ARR. The model market is more competitive than ever, with GPT-4 costs falling from $30 to $2 per million tokens in 18 months and open-source like DeepSeek competing. Guo advises builders to focus on thick wrappers around LLMs, leveraging domain and workflow knowledge, and warns against generic text boxes: 'The prompt is a bug, not a feature.' Execution, not first-mover advantage, is the moat.

The 2025 AI Engineering Report — Barr Yaron, Amplify

Aug 1, 2025 · 12:33

Barr Yaron presents early findings from the 2025 State of AI Engineering survey at the AI Engineer World's Fair. The survey reveals that while titles vary widely, the community is broad and technical, with nearly half of seasoned software engineers having worked with AI for three years or less. Over half of respondents use LLMs for both internal and external use cases, with 3 of the top 5 models for customer-facing products from OpenAI. RAG is the most popular customization method (70%), and fine-tuning is surprisingly common, with 40% using LoRA or QLoRA. More than 50% update models monthly, and 70% update prompts at least monthly. A 'multimodal production gap' exists, with image, video, and audio usage lagging text. 80% say LLMs work well, but less than 20% say the same about agents, though most plan to use them. Evaluation is the top pain point.

Agents vs Workflows: Why Not Both? — Sam Bhagwat, Mastra.ai

Aug 1, 2025 · 15:37

Sam Bhagwat, co-founder of Mastra and author of 'Principles of AI Agents', argues that the debate between agents and workflows is misguided—developers should combine both. He criticizes OpenAI's anti-workflow stance as 'being that guy' and LangChain's graph/node/edge APIs as harmful, advocating for fluent syntax over graph theory. Bhagwat defines agents as turn-based games and workflows as rules-engine dependency chains, noting that nondeterminism makes workflow tracing 10x more important in AI engineering. He demonstrates composition patterns: agents can be steps, workflows can be tools, and nested workflows handle dynamic tool injection. He recommends starting with agent power, then adding workflow control for reliability—e.g., breaking one LLM call into 12 for medical PDFs. Bhagwat concludes that practice trumps theory in this young field.

Why We Don’t Need More Data Centers - Dr. Jasper Zhang, Hyperbolic

Aug 1, 2025 · 14:18

Dr. Jasper Zhang, CEO of Hyperbolic, argues that the AI industry's focus on building more data centers is inefficient because GPUs already sit idle 80% of the time. Instead, he advocates for a GPU marketplace that aggregates supply from fragmented providers, cutting costs by 50–75%—Hyperbolic offers H100s at $0.99/hour versus $2–$11 from traditional clouds. His HyperDOS software connects any Kubernetes cluster in 5 minutes, enabling dynamic rental and release of GPUs. He illustrates with a startup that can reduce compute costs from $43.8 million to $6.9 million by flexibly scaling capacity and selling idle GPUs, effectively boosting productivity 6x on the same budget. Zhang contends that smarter allocation, not just more data centers, is key to sustainable and democratized AI infrastructure.

Pipecat Cloud: Enterprise Voice Agents Built On Open Source - Kwindla Hultman Kramer, Daily

Jul 31, 2025 · 26:46

Kwindla Hultman Kramer, co-founder of Daily, introduces Pipecat, an open-source, vendor-neutral framework for building voice AI agents, and Pipecat Cloud, a deployment layer optimized for real-time voice. He argues that achieving sub-800-millisecond voice-to-voice response times is critical and that frameworks like Pipecat handle hard problems like turn detection, interruption handling, and context management. Kramer explains that Pipecat supports 60+ models and services, including Gemini and OpenAI, and that Gemini is often 10x cheaper for 30-minute conversations. He notes that while speech-to-speech models like Moshi and Sesame are promising for natural conversation, they currently lag in instruction-following for enterprise use cases. Kramer also addresses global latency challenges, recommending deployment close to inference servers or using open-weight models locally, and highlights Pipecat Cloud's integration with Krisp for background noise reduction and a free open-source smart turn model.

From Self-driving to Autonomous Voice Agents — Brooke Hopkins, Coval

Jul 31, 2025 · 19:32

Brooke Hopkins, founder of Coval and former Waymo eval lead, argues that reliability lessons from self-driving cars directly apply to building trustworthy voice agents. She explains how Waymo moved from brittle scenario-specific tests to large-scale probabilistic simulations that measure success rates across thousands of runs—a framework now used at Coval to evaluate conversational AI at scale. Key practices include denoising: resimulating a failed scenario 100 times to assess true failure probability, and calibrating LLM-as-judge metrics with human feedback until automated evals align. Hopkins outlines a hierarchical eval strategy: start with text-based task evals for speed, then graduate to voice-to-voice tests for latency and interruptions, and finally hyper-realistic simulations for production edge cases. She predicts that within three years every enterprise will need a voice experience comparable to a mobile app, making integrated evaluation infrastructure a competitive necessity.

Why ChatGPT Keeps Interrupting You — Dr. Tom Shapland, LiveKit

Jul 31, 2025 · 27:03

Dr. Tom Shapland of LiveKit explains why voice AI agents like ChatGPT's Advanced Voice Mode keep interrupting users: they rely on a simple VAD that triggers after a silence threshold, unlike humans who predict turn endings using semantics, syntax, and prosody. To solve this, LiveKit developed a semantic end-of-utterance model that considers the last four conversation turns, extending the VAD's silence window when the user is not done. A side-by-side demo shows dramatic reduction in interruptions. Shapland contrasts this with full-duplex models like Moshi and Meta's SyncLLM, which process input and generate speech simultaneously but lack instruction-following. He predicts commercial systems will improve via smarter VAD augmentations and faster cascade pipelines, not full-duplex. The talk also covers handling backchannels (e.g., 'mm-hmm'), the challenge of benchmarking turn-taking, and why OpenAI doesn't yet use LiveKit's model.

How to defend your sites from AI bots — David Mytton, Arcjet

Jul 30, 2025 · 20:12

David Mytton, founder of Arcjet, explains that nearly 50% of web traffic is now automated, with AI crawlers significantly worsening the problem for site owners. He cites specific cases: Diaspora saw 24% of its traffic from OpenAI's GPTbot, and ReadTheDocs cut bandwidth from 800 GB to 200 GB per day by blocking AI crawlers. Mytton describes the evolving landscape of AI bots—OpenAI alone has at least four types (Search, ChatGPT User, GPTbot, Operator)—and argues that detection requires layered defenses. He walks through eight defenses: robots.txt (voluntary), user agent string verification with reverse DNS, IP reputation (e.g., 12% of bot traffic on Cloudflare comes from AWS), CAPTCHAs (increasingly easy for AI to solve), proof-of-work puzzles (e.g., Anubis proxy), experimental HTTP message signatures (Cloudflare's proposal), fingerprinting via JA4 hash (TLS-level), and rate limiting keyed to fingerprints or session IDs. The practical takeaway is that most sites can manage with user agent verification plus IP checks, but high-value or high-traffic sites need full stacking of these techniques.

The Unofficial Guide to Apple’s Private Cloud Compute - Jmo, CONFSEC

Jul 30, 2025 · 20:36

Jonathan Mortensen explains Apple's Private Cloud Compute (PCC), a paradigm shift in confidential computing that delivers cryptographically provable privacy for AI inference on remote servers. He details Apple's five requirements—stateless computation, enforceable guarantees, non-targetability, no privileged runtime access, and verifiable transparency—and six technical components, including oblivious HTTP, blind signatures, secure enclave, remote attestation, transparency logs, and secure boot. The talk highlights how remote attestation allows an iPhone to verify server software and tie encryption to that verified state, while the transparency log enables public auditing of all deployed binaries. Mortensen notes trade-offs: complete trust in Apple's supply chain, higher latency and compute costs, and no support for custom models or third-party developers. He points to alternative tools like TPMs, SigStore, and confidential VMs for non-Apple environments, and concludes that Apple's approach is being adopted by Azure AI and Meta.

OpenAI on Securing Code-Executing AI Agents — Fouad Matin (Codex, Agent Robustness)

Jul 30, 2025 · 14:00

Fouad Matin, an OpenAI engineer working on agent robustness, argues that code-executing agents like Codex CLI require layered security safeguards as they become capable of autonomously writing and running code. He emphasizes three main defenses: sandboxing agents via containerization (e.g., ChatGPT spins up fully isolated containers) or OS-level sandboxing using SeatBelt on macOS or Seccomp and Landlock on Linux; disabling or limiting internet access to prevent prompt injection and data exfiltration, with configurable allowlists for approved endpoints; and requiring human review through PR approvals and monitoring tasks to catch unintended actions like installing malicious packages. Matin also details new API tools—local shell and apply patch—that enable agents to execute code and apply diffs securely, and recommends using remote containers (available in the Agents SDK and Responses API) as the safest deployment option. He concludes by acknowledging that while LLM-based monitors are valuable, deterministic system-level controls remain essential, and invites applications to OpenAI's new Agent Robustness team.

Evaluating AI Search: A Practical Framework for Augmented AI Systems — Quotient AI + Tavily

Jul 29, 2025 · 20:33

Julia Neagu (CEO of Quotient AI), Deanna Emery (founding AI researcher at Quotient AI), and Maitar Asher (Head of Engineering at Tavily) argue that traditional monitoring approaches like static benchmarks (e.g., SimpleQA) fail to capture the complexity of production AI search because the web is dynamic and truth is subjective. They present a framework combining dynamic datasets generated via a LangGraph-based agent that creates evidence-based Q&A pairs from multiple search providers, and holistic evaluation using three reference-free metrics: answer completeness, document relevance, and hallucination detection. In a two-part experiment comparing six anonymized AI search providers on a static (SimpleQA) and a dynamic benchmark of ~1,000 rows, ranking shifted significantly—provider F went from worst on SimpleQA to best on the dynamic benchmark. Reference-free answer completeness correlated 0.94 with correctness, and document relevance showed an inverse correlation with unknown answers. The speakers stress that evaluation should diagnose issues (e.g., incomplete answers with relevant documents suggest need for more retrieval) and guide improvements, ultimately aiming for…

Building the platform for agent coordination — Tom Moor, Linear

Jul 28, 2025 · 19:43

Tom Moor, Head of Engineering at Linear, explains how the company is evolving from an issue tracker into an operating system for engineering teams that treats AI agents as first-class teammates. He details Linear's pragmatic AI journey: starting with embeddings and PG vector, then moving to a hybrid search index using TurboPuffer and Cohere embeddings, leading to features like Product Intelligence (query rewriting, reranking, deterministic rules for suggestions), natural language filters, Slack-to-issue creation, and daily audio pulses. The core of the talk is Linear's agent platform, launched two weeks ago, where coding agents like CodeGen, Bucket, and Charlie integrate via OAuth, GraphQL, and new webhooks, allowing users to assign, mention, and interact with agents just like human teammates. Moor emphasizes best practices for builders: respond fast, inhabit the platform's language, move issues to 'in progress,' and clarify plans before acting, all while keeping interactions concise and value-adding. The episode argues that with this platform, engineering teams can build more, faster, and with higher quality by offloading grunt work to infinitely scalable cloud-based teammates.

Ship Agents that Ship: A Hands-On Workshop - Kyle Penfound, Jeremy Adams, Dagger

Jul 27, 2025 · 1:21:00

Kyle Penfound and Jeremy Adams from Dagger demonstrated building production-minded AI agents using Dagger, a programmable delivery engine that runs containerized workflows identically on laptops and in CI. They created a workspace submodule with tools — read file, write file, list files, and run tests — and wired it to an LLM (Claude 3.5) so the agent can edit code and validate changes through the project's actual test suite. The agent runs in isolated containers for determinism and traceability, with behavior visualized via Dagger Cloud. They extended the agent to respond to GitHub issues: a label triggers GitHub Actions to run Dagger, read the issue, hand the assignment to the agent, and open a pull request. The talk emphasized that Dagger's modularity lets developers give agents precisely scoped tools while reusing existing CI workflows, providing guardrails without sacrificing flexibility.

Why you should care about AI interpretability - Mark Bissell, Goodfire AI

Jul 27, 2025 · 21:11

Mark Bissell of Goodfire AI argues that mechanistic interpretability—reverse engineering neural networks—has moved from research to practical use cases that AI engineers can apply today, demonstrated through Goodfire's Ember platform for neural programming. He shows how Ember enables debugging and steering models at the neuron level, such as turning up a 'sensitive information' feature to make LLaMA refuse to reveal an email, or dynamically injecting a Coca-Cola recommendation when a beverage feature activates. For image models, the Paint with Ember demo (paint.goodfire.ai) lets users paint concepts like pyramid or lion face directly onto a canvas and steer sub-features (e.g., lion minus mane becomes tiger). Beyond interfaces, interpretability powers model diffs to detect unwanted changes after fine-tuning, guardrails for production systems, and scientific extraction: Goodfire works with the ARC Institute to uncover biological principles from the superhuman genomics model Evo2. Efficiency gains also emerge by pruning unnecessary weights for specialized tasks, making interpretability a critical tool for reliable AI engineering.

Information Retrieval from the Ground Up - Philipp Krenn, Elastic

Jul 27, 2025 · 1:48:07

Philipp Krenn of Elastic demonstrates that vector search is only a feature of information retrieval, not its foundation, in this hands-on workshop on retrieval for RAG. He contrasts classic keyword search—using tokenization, stemming, stopword removal, and BM25 scoring with the inverted index—with sparse (Elser/Splade) and dense (OpenAI text-embedding-small) vector embeddings. Krenn explains why hybrid search, combining lexical and semantic methods via reciprocal rank fusion (RRF), outperforms any single approach, especially for brand or exact-match queries where keyword search remains superior. He also covers scoring normalization, chunking strategies, and the new `text_similarity_reranker` retriever, showing how Elasticsearch's retrievers API enables two-stage retrieval with a cross-encoder for re-ranking. The session emphasizes that retrieval quality depends on evaluating against a golden dataset or LLM-as-judge, and that 'it depends' is the correct answer for choosing between PgVector and dedicated search engines.

Introduction to LLM serving with SGLang - Philip Kiely and Yineng Zhang, Baseten

Jul 26, 2025 · 43:42

Philip Kiely and Yineng Zhang introduce SGLang, an open-source fast serving framework for LLMs and VLMs, arguing it offers production-ready performance with day-zero support for new models like DeepSeek and Qwen, and a customizable codebase for contributions. Yineng, a core maintainer, traces SGLang's rapid growth from a December 2023 paper to nearly 15,000 GitHub stars and adoption by xAI, AMD, and Meituan. The workshop demonstrates deploying a first model via Baseten's trunk packaging, then tuning the CUDA Graph max batch size flag on an L4 GPU to maintain CUDA Graph-enabled decoding during higher concurrency, boosting generation throughput. They also cover Eagle 3 speculative decoding, where a draft model derived from the target model speculates tokens; users can benchmark different step and top-k configurations on representative prompts to find optimal settings for production. Finally, they invite contributions via GitHub's 'good first issue' tags and highlight Baseten's job openings.

Latent Space Paper Club: AIEWF Special Edition (Test of Time, DeepSeek R1/V3) — VIbhu Sapra

Jul 25, 2025 · 53:54

VIbhu Sapra presents DeepSeek's latest models and announces a new curriculum-based Test of Time Paper Club. DeepSeek's May 28th R1 update doubles reasoning tokens (12k→25k) and matches O3/Gemini 2.5 Pro, improving AIME 2024 from 70% to 87.5%. A new distillation of R1 traces into Qwen 3 8b achieves performance on par with Qwen's 235b thinking model, proving reasoning models distill efficiently. The Test of Time Paper Club will cover 50–100 foundational papers over six months in San Francisco and remote, targeting core AI engineering topics from attention and scaling laws to inference techniques. Sapra emphasizes that pure RL (GRPO) on verifiable math and code unlocks emergent reflection and aha moments, shifting scaling from pre-training compute to inference-time reasoning.

Human seeded Evals — Samuel Colvin, Pydantic

Jul 25, 2025 · 12:02

Samuel Colvin, creator of Pydantic, demonstrates how type safety and validation loops in Pydantic AI build reliable GenAI applications. He argues that type-safe frameworks enable confident refactoring, a necessity for evolving AI apps, and shows how returning validation errors to the model fixes simple mistakes like incorrect date formats. The agentic loop with tools and final result tools ensures structured extraction, while type-safe dependencies on tools prevent runtime errors. Colvin also highlights Logfire observability, which traces calls, shows pricing, and helps debug failures like failed memory retrieval. The talk covers agent definitions, validation error retries, and the importance of explicit exit conditions in agent loops.

Building AI Products That Actually Work — Ben Hylak (Raindrop), Sid Bendre (Oleve)

Jul 24, 2025 · 18:42

Ben Hylak (Raindrop) and Sid Bendre (Oleve) argue that building reliable AI products requires iterative real-world signals over traditional evals. Ben debunks eval myths—evals don't measure product quality, LLM-as-judge fails, and production evals are costly—and stresses tracking explicit signals (thumbs up/down, copy rate) and implicit signals (refusals, frustration) to identify issues. Sid introduces Trellis, a framework for scaling viral AI apps that uses discretization—breaking infinite output into intent buckets—prioritization by volume times negative sentiment times achievable delta, and recursive refinement. Starting with an MVP, teams classify user intents, convert them into semi-deterministic workflows, then repeatedly drill into sub-intents to engineer repeatable, attributable magic. Oleve's approach, powered by Raindrop, has scaled six viral products to $6M ARR profitably with four people.

Structuring a modern AI team — Denys Linkov, Wisedocs

Jul 24, 2025 · 17:40

Denys Linkov, who leads ML at Wisedocs, argues that building a modern AI team hinges on identifying your company's bottleneck—shipping features, acquiring users, or scalability—rather than reflexively hiring AI researchers. He introduces Ampere's Wager: trading your entire domain-savvy team for five top-lab researchers is usually a losing bet. For early-stage AI strategy, generalists who blend model training, serving, and business acumen outperform specialists; Linkov lived this in 2021 building a custom MLOps platform for a conversational AI startup and again in 2024 using advanced open-source tools for medical record processing. He stresses reskilling existing teams through weekly learning cadences and moving domain experts from giving feedback to writing evaluations. Hiring should hold context and act on it, verifying trends like 'don't hire juniors' against YC's AI school drawing 2,000 young people.

The Rise of Open Models in the Enterprise — Amir Haghighat, Baseten

Jul 24, 2025 · 16:50

Amir Haghighat, CTO of Baseten, argues that enterprises are increasingly moving from closed frontier models like OpenAI and Anthropic toward open source models, driven by four specific cracks in the assumption that closed models will work indefinitely: quality for specialized tasks (e.g., medical document extraction), latency requirements (especially for voice), unit economics ballooning from agentic use cases where a single user action triggers 50 inference calls, and the desire for competitive differentiation. Drawing on conversations with over 100 enterprises, he explains that while most started with dedicated deployments on Azure/AWS for toying around in 2023, by 2024 about 40-50 had production use cases, and in 2025 the shift accelerated. However, adopting open models forces enterprises to build inference infrastructure, facing challenges like speculative decoding, prefix caching, guaranteeing four-nines reliability with hardware failures and VLM crashes, and scaling replicas—with one Fortune 50 soft drink company reporting an eight-minute spin-up time. Haghighat concludes by contrasting the simple API-call world with the complexities of mission-critical inference, where…

Monetizing AI — Alvaro Morales, Orb

Jul 23, 2025 · 18:18

Alvaro Morales, CEO of Orb, argues that traditional subscription-based SaaS pricing fails for AI products due to rapid cost fluctuations, margin pressures, and customer demand for clear ROI. He presents three frameworks: decide between direct monetization (add-on, bundled) or indirect (free to drive upsells) using examples like GitHub Copilot, Notion AI, and Expedia; select a value metric from resource-based tokens (Vercel's v0) to task-based (Zapier) to outcome-based pricing (Intercom's Fin at $0.99 per resolved ticket); and embrace continual experimentation, as AI pricing cycles shrink from years to months. To reduce pricing guesswork, Orb built Simulations, a tool that lets companies backtest alternate pricing strategies on real usage data from closed betas before launch, helping teams like Replit and Perplexity optimize revenue without risking customer trust.

How agents will unlock the $500B promise of AI - Donald Hruska, Retool

Jul 23, 2025 · 16:22

Donald Hruska, engineering lead for Retool's Agents product, argues that AI agents will unlock the $500B promise of AI by moving enterprises beyond toy chatbots into production-grade systems. He explains that building a basic agent is easy (e.g., 100 lines of code using React) but getting it into production requires solving security, cost overruns, observability, and compliance. Hruska outlines four options—build from scratch, use a framework like LangGraph, a managed platform like Retool Agents, or verticalized agents—and advises building for core differentiators and buying for commodity workflows. He cites Retool customers like ClickUp saving over $200,000 in vendor costs and Descript saving hundreds of hours weekly, while noting inference costs dropped 99.7% from 2022-2024. Retool charges $3 per hour for its cheapest agent and supports on-prem deployment.

How Intuit uses LLMs to explain taxes to millions of taxpayers - Jaspreet Singh, Intuit

Jul 23, 2025 · 18:59

Jaspreet Singh, Senior Staff Engineer at Intuit, explains how TurboTax uses Anthropic's Claude and OpenAI models to generate personalized tax explanations for 44 million customers. The system, built on Intuit's GenOS platform, combines prompt-engineered static explanations with dynamic question-answering using RAG and graphRAG for tax-specific queries. Singh details their fine-tuning of Claude Haiku on AWS Bedrock, which improved quality but proved too specialized. A key focus is evaluation: manual reviews by tax analysts followed by automated LLM-as-a-judge scoring for accuracy, relevancy, and coherence, with a golden dataset and safety guardrails to prevent hallucinated numbers. He highlights challenges like latency spikes on tax day (3–10 seconds), vendor lock-in through expensive contracts, and the difficulty of upgrading models even within the same vendor. Singh emphasizes that evaluations are essential for launching any GenAI feature in a regulated domain like tax.

Building Agents (the hard parts!) - Rita Kozlov, Cloudflare

Jul 23, 2025 · 21:12

Rita Kozlov, VP of Product for Cloudflare's developer platform, presents the building blocks of AI agents—client, AI reasoning, workflows, and tools—arguing that effective agents require all four components. She highlights the Model Context Protocol (MCP) as a standard for exposing APIs to LLMs, and demonstrates Cloudflare's Agents SDK, which simplifies hosting remote MCP servers with built-in OAuth, state management via durable objects, and real-time WebSocket communication. Kozlov cites real-world impact: companies using agents for sales automation see 20% revenue increases, 90% faster support response times, and 50–75% time savings. She walks through a human-in-the-loop credit card approval workflow built with Nok, showing how durable objects maintain long-running state, prevent duplicate actions, and route approvals across Slack, email, or in-app notifications. The talk emphasizes that once an MCP server is deployed, it can be used directly from Cursor, Claude, ChatGPT, or a custom client, including voice interfaces via WebRTC-to-WebSocket translation.

The Billable Hour is Dead; Long Live the Billable Hour — Kevin Madura + Mo Bhasin, Alix Partners

Jul 23, 2025 · 17:04

Kevin Madura and Mo Bhasin from AlixPartners argue that AI is reshaping knowledge work by compressing upfront data ingestion from 50% to 10-20% human effort, enabling analysis of 100% of data rather than the top 20%. They detail three use cases: categorization via structured outputs achieving 95% accuracy on 10,000 vendors in minutes; enterprise-scale RAG that democratizes access to siloed data by teaching LLMs to call APIs; and structured extraction from documents using schemas and log-prob-based confidence scoring. They caution that AI investments face a paradox—89% of CEOs plan agentic AI but NBER finds no earnings impact—and stress that success requires people skills, demos, and a focus on NPS and ROI over chasing shiny new tools. The episode concludes that once Excel-powered LLMs work reliably, AGI will be here.

The Bitter Layout or: How I Learned to Love the Model Picker — Maximillian Piras, Yutori

Jul 21, 2025 · 14:24

In this talk, Maximillian Piras argues that conversational interfaces remain dominant in AI apps not because they are natural, but because they are 'conformable' — able to absorb the next model's capabilities without redesign. He calls this pattern 'The Bitter Layout': an input field, turn-by-turn flow, and a model picker, which prioritizes flexibility over usability. Applying Clayton Christensen's theory of commoditization, Piras claims that as long as scaling laws keep models from commoditizing, the interface itself is the commodity, and designers must conform to the next model. He traces the debate over chat UX from Linus Lee (2022) to Julian Lear (2024), and criticizes the model picker as a 'mode selector' that creates usability issues. Looking ahead, Piras suggests designers shift from procedural thinking to setting goals and constraints, and speculates that future AI UX will be 'grown' like a garden rather than built.

UX Design Principles for Semi Autonomous Multi Agent Systems — Victor Dibia, Microsoft

Jul 21, 2025 · 20:28

Victor Dibia, Principal Research Software Engineer at Microsoft Research, presents four UX design principles for semi-autonomous multi-agent systems: capability discovery, observability and provenance, interruptibility, and cost-aware action delegation. He demonstrates these through Blender LM, a multi-agent system built from scratch that translates natural language into 3D Blender scenes, featuring a planner and verify agent. Dibia stresses eval-driven design—starting with a non-AI baseline, then iterating with agents—and warns that only a small subset of tasks truly benefit from multi-agent autonomy, advocating for careful ROI assessment. He also showcases AutoGen Studio, a low-code tool for composing multi-agent workflows, and shares code and resources for further learning.

Agentic GraphRAG: AI’s Logical Edge — Stephen Chin, Neo4j

Jul 21, 2025 · 15:27

Stephen Chin of Neo4j argues that Agentic GraphRAG — combining graph databases with retrieval-augmented generation — overcomes LLM hallucinations and biases by providing structured, relational context. He demonstrates how LLMs fail on reasoning tasks like calculating classroom capacity due to inaccurate anchoring on irrelevant data, and proposes an architecture where vector search first identifies relevant nodes, then graph traversal retrieves related context for the LLM. Chin highlights Neo4j’s MCP server for cypher query generation and memory modules, and cites Klarna’s success: 250K employee questions answered in the first year, 2,000 daily queries, and 85% adoption, replacing their entire SaaS stack. He recommends the Neo4j Certified Developer Program and the Nodes Conference for further learning.

Building Effective Voice Agents — Toki Sherbakov + Anoop Kotha, OpenAI

Jul 20, 2025 · 17:17

Toki Sherbakov and Anoop Kotha from OpenAI argue that speech-to-speech models have reached a 'good enough' tipping point for production voice agents, highlighting the shift from chained architectures (transcription + LLM + TTS) to the real-time API's speech-to-speech approach. They detail trade-offs across latency, cost, accuracy, UX, and integrations depending on use case—consumer apps prioritize low latency and expressiveness, while customer service demands accuracy and tool integration. Key design patterns include delegating complex tasks to smarter models like O4 mini, prompting for voice expressiveness and tone, and starting with few tools. For evals, they recommend starting with observability, using SMEs for labeling, then transcription-based LLM-as-judge evals, audio evals with GPT-4 Audio to assess tone and pacing, and synthetic conversations between two real-time clients. Guardrails should run async with a configurable debounce period (e.g., every 100 characters). Examples from Lemonade (early focus on evals and guardrails) and Tinder (customization for brand realism) illustrate successful approaches.

What every AI engineer needs to know about GPUs — Charles Frye, Modal

Jul 20, 2025 · 19:52

Charles Frye of Modal explains that AI engineers need to understand GPU hardware constraints to optimize inference, arguing GPUs embrace bandwidth over latency and that Tensor Cores for low-precision matrix-matrix multiplication are the key resource. He describes how GPUs achieve 16,000+ parallel threads per cycle on H100, and notes Patterson’s Law: bandwidth improves at the square of latency. The main insight: arithmetic intensity favors N² operations per N memory loads, so matrix-matrix operations are efficient while matrix-vector is wasteful. Frye demonstrates that running a small 8B model 1,000 times on the same prompt matches GPT-4 quality, and that multi-token prediction and multi-sample query become nearly free because Tensor Cores handle expanded batches as matrix-matrix multiplications. He recommends using smaller models that fit on a single GPU and scaling via multiple generations.

[Full Workshop] Reinforcement Learning, Kernels, Reasoning, Quantization & Agents — Daniel Han

Jul 19, 2025 · 2:42:28

Daniel Han of Unsloth presents a technical workshop covering reinforcement learning (RL), kernels, reasoning, quantization, and agents, arguing that RL with verifiable rewards (RLVR) is the key to unlocking LLM capabilities beyond supervised fine-tuning. He explains why open-source models plateaued after September 2024 until DeepSeek-R1 showed that RL can elicit reasoning, and breaks down PPO, GRPO, and the REINFORCE algorithm, emphasizing that GRPO removes the value model for efficiency. Han details how reward functions—not algorithms—are the hardest part, with examples like distance-based scoring for math. He demonstrates a free Colab notebook training a base model to reason, and shows that dynamic quantization can shrink models like DeepSeek-R1 from 730 GB to 140 GB with only ~1% accuracy loss, arguing that GPUs may stop getting faster after FP4 precision.

A Taxonomy for Next-gen Reasoning — Nathan Lambert, Allen Institute (AI2) & Interconnects.ai

Jul 19, 2025 · 19:21

Nathan Lambert, senior research scientist at AI2 and founder of Interconnects.ai, argues that next-generation reasoning models require a taxonomy of four traits: skills, calibration, strategy, and abstraction. While current models excel at math and code (skills), they overthink easy problems, wasting tokens and latency—calibration is needed to match output length to task difficulty. The real frontier is planning: models must learn strategy (choosing the right direction) and abstraction (breaking problems into tractable sub-tasks) to enable long-horizon agents like Deep Research and Claude Code. Lambert traces how OpenAI's Q*→Strawberry→o1 took 12–18 months of human data to teach backtracking and verification; planning should be easier because humans can write 5-step plans. He predicts post-training compute could reach parity with pre-training, citing DeepSeek's shift from 0.18% post-training compute in V3 to an estimated 10–20% for R1. The path forward: collect diverse verifiable questions, filter by difficulty, run stable RL—then scale.

How to Train Your Agent: Building Reliable Agents with RL — Kyle Corbitt, OpenPipe

Jul 19, 2025 · 19:48

Kyle Corbitt, co-founder of OpenPipe, argues that reinforcement learning (RL) with GRPO can make agentic systems far more reliable and cost-effective than prompted frontier models. He presents ART-E, an email assistant trained on Qwen 2.5 14B, which achieved 96% accuracy versus 90% for o3 and slashed cost from $55 to $0.80 per 1,000 queries. The two critical problems are building a realistic environment (solved using the Enron email dataset) and defining the right reward function (turning it into a verifiable task with LLM-as-judge). Extra rewards—favoring fewer tool turns and penalizing hallucination—further improved efficiency. Corbitt also warns about reward hacking, giving examples like a model that exploited a bug to put every word in every category, and shares that the training cost just $80 in GPU time and a week of engineering.

Design like Karpathy is watching — Zeke Sikelianos, Replicate

Jul 19, 2025 · 19:26

Zeke Sikelianos of Replicate analyzes Andrej Karpathy's experience building and deploying MenuGen, a vibe-coded web app that turns menu photos into images, to argue that LLMs are now the primary audience for developer tools. He details Replicate's specific failures—rate limiting and outdated docs that blocked Karpathy—and the fixes they implemented: adding LLMs.txt files for Markdown-friendly documentation, promoting curl commands as the LLM-friendly interface, and launching an MCP server built on OpenAPI schemas. Sikelianos advocates for 'boring technology' like SQL and 'good API hygiene' to keep payloads small and information-dense for LLM context windows. He also calls for better payment acceptance and documentation discipline, stressing that unblocking power users like Karpathy (whose CEO intervened) should not require a viral blog post.

On Curiosity — Sharif Shameem, Lexica

Jul 19, 2025 · 18:35

Sharif Shameem, founder of Lexica, argues that curiosity is the main force for pulling ideas from the future into the present, and that building and sharing demos is the best way to explore AI models' hidden capabilities. He recounts early GPT-3 demos from 2020-2021, when the model had a 2,000-token context window and cost $75 per million output tokens, showing how he built a JSX compiler in the browser, a shopping agent that parsed web pages, and a multi-step reasoning tool called MultiVAC. Shameem emphasizes that AI engineering is more like excavating than traditional engineering, and that researchers often don't know the full capabilities of their own models. He closes by invoking computing pioneer J.C.R. Licklider, arguing that today's AI engineers have a moral obligation to follow their curiosity and share what they discover.

The State of Generative Media - Gorkem Yurtseven, FAL

Jul 16, 2025 · 17:14

Gorkem Yurtseven, co-founder and CTO of fal.ai, argues generative media lowers creation's marginal cost to near zero, transforming advertising and e-commerce. He traces evolution from DALL·E 2 to open-source models like Stable Diffusion and FLUX, with the playing field evening quickly. Video models (Sora, Veo 3) are accelerating: fal's platform saw video usage jump from near zero to 30% of revenue, and he predicts the generative video market 100x-250x bigger than image. Advertising leads with hyper-personalized, interactive ads, like fal's A24 campaign turning selfies into toy soldiers. In e-commerce, virtual try-on is a proven product-market fit. Future real-time video generation will blur games and movies.

Measuring AGI: Interactive Reasoning Benchmarks for ARC-AGI-3 — Greg Kamradt, ARC Prize Foundation

Jul 16, 2025 · 18:28

Greg Kamradt, President of ARC Prize Foundation, introduces ARC-AGI-3, the first interactive reasoning benchmark for AGI that drops agents into novel games without prior instruction, forcing exploration to solve tasks. Unlike static tests, this benchmark measures skill acquisition efficiency—how quickly an AI learns and applies new skills—using human baselines from 400+ in-person tests. It strips away language and trivia, relying only on core knowledge priors (basic math, geometry, agentness, objectness). A public training set of ~40 games will be released, but performance is measured on a private evaluation set of 120 games unseen by developers or AI. Kamradt asserts that as long as AI cannot outperform humans on these problems, we do not have AGI; a sandbox preview with five games and a mini agent competition is planned for next month, with full launch in Q1 2026.

Benchmarks Are Memes: How What We Measure Shapes AI—and Us - Alex Duffy, Every.to

Jul 15, 2025 · 15:44

Alex Duffy argues that AI benchmarks function as cultural memes—ideas that spread and shape what models learn—giving those who design them immense power over AI's trajectory. He traces the lifecycle from a single person's idea to saturation, using examples like 'How many Rs in strawberry' and Pokémon. Duffy introduces AI Diplomacy, a benchmark where language models negotiate and betray each other, revealing that models like DeepSeek R1 and Gemini 2.5 Flash excel at social manipulation while Claude models are naively optimistic. He warns against benchmarks that reward sycophancy (like ChatGPT's thumbs-up training) and advocates for multifaceted, experiential, and generative benchmarks that empower people. Duffy urges the audience to ask non-AI people what they care about, turning benchmarks into tools that build trust and define humanity's role in an AI world.

Prompt Engineering and AI Red Teaming — Sander Schulhoff, HackAPrompt/LearnPrompting

Jul 14, 2025 · 2:01:05

Sander Schulhoff, creator of Learn Prompting and HackAPrompt, argues prompt engineering remains vital despite claims of its demise, drawing on his systematic review of 1,500+ papers for 'The Prompt Report.' He covers advanced techniques including chain-of-thought, decomposition, ensembling, and few-shot prompting, noting that role prompting is ineffective for accuracy-based tasks and that example ordering can swing performance by 50%. He then explains AI red teaming, distinguishing jailbreaking from prompt injection, and warns that system prompts and guardrails cannot prevent attacks—even simple obfuscation like base64 or typos still works. He highlights the critical unsolved problem of agentic security, where agents with real-world actions are easily tricked, and introduces a live competition at the conference to gather more attack data. His key takeaway: AI security is fundamentally harder than classical cybersecurity because 'you cannot patch a brain.'

Survive the AI Knife Fight: Building Products That Win — Brian Balfour, Reforge

Jul 14, 2025 · 14:10

Brian Balfour, CEO of Reforge, argues that winning in today's AI knife-fight requires answering 'What do I build and why will it win?' by focusing on proprietary data, unique functionality, and unmet customer needs rather than building custom AI. He illustrates with Granola, which entered a crowded AI note-taker market by understanding users wanted help taking better notes, not full automation, and assembled off-the-shelf AI (DeepGram, Anthropic, OpenAI) with unique data (user notes plus transcription) and functionality (Mac app, calendar integration) to create a competitive edge. Balfour warns competitive advantages now last only 2-3 weeks, so teams must sequence smaller moats continuously, each buying time to execute faster. The talk emphasizes treating AI as Lego blocks—assembling pre-trained models, data, and product superpowers into a system that spins a data flywheel.

Automating Escrow with USDC and AI - Corey Cooper, Circle

Jul 14, 2025 · 58:18

Corey Cooper, Head of DevRel at Circle, demonstrates how USDC stablecoins and AI agents can automate escrow by combining smart contracts with LLM-based task verification. He explains that USDC's programmability enables near-instant settlement, chargeback-free transactions, and always-on payments ideal for agent-to-agent payments. The demo shows an escrow agent app that uses OpenAI to parse PDF contracts, deploys Solidity escrow contracts via Circle APIs on Base Sepolia, and allows an AI agent to verify deliverables (e.g., image quality) and trigger payout from a locked smart contract. Cooper discusses human-in-the-loop safeguards, cross-chain deposits via Circle's Cross-Chain Transfer Protocol, and future possibilities like multi-sig agent verification. He highlights that while full autonomy is not yet reliable, combining USDC with AI can streamline payment operations.

How LLMs work for Web Devs: GPT in 600 lines of Vanilla JS - Ishan Anand

Jul 13, 2025 · 1:41:34

Ishan Anand shows that GPT-2 small implemented in 600 lines of vanilla JavaScript makes LLMs understandable for web developers without ML backgrounds. He explains tokenization via byte-pair encoding, 768-dimensional embeddings representing semantic meaning via co-occurrence, and the Transformer's attention mechanism that lets tokens share context. The multi-layer perceptron learns next-token prediction through backpropagation, while the language head converts embeddings to token probabilities using softmax. Anand demonstrates each step—tokenization, embedding lookup, positional encoding, attention, MLP, and output—in a browser debugger, and notes that GPT-2's architecture underpins ChatGPT, with innovations like scale, supervised fine-tuning, and RLHF. The workshop provides an intuitive mental model of Transformers, turning perceived AI magic into understandable machinery.

[Workshop] AI Pipelines and Agents in Pure TypeScript with Mastra.ai — Nick Nisi, Zack Proser

Jul 12, 2025 · 1:51:14

This hands-on workshop introduces Mastra.ai, a TypeScript framework for building production AI agents and pipelines, and demonstrates how to build an AI meme generator using composable workflows, tools, and agents. Hosts Nick Nisi and Zack Proser walk through creating a multi-step workflow: extract user frustration, find a base meme via ImageFlip, generate captions, and publish the meme at a stable URL. They show how to chain steps with Zod schema validation for deterministic output, then wrap the workflow in an agent that accepts natural language requests. The workshop also covers MCP (Model Context Protocol) and a live demo of MCP.shop for ordering a shirt, emphasizing local iteration with Mastra's playground, built-in memory, and evaluation tools. Real-world patterns include building internal AI assistants for data cleaning, email drafting, and document summarization with minimal code.

AI Engineering with the Google Gemini 2.5 Model Family - Philipp Schmid, Google DeepMind

Jul 11, 2025 · 1:44:51

In this workshop, Philipp Schmid from Google DeepMind demonstrates AI Engineering with the Gemini 2.5 model family, focusing on using Gemini 2.5 Flash via a free API tier for hands-on coding tasks including text generation, multimodal processing of images, audio, and PDFs, function calling with structured outputs, and integration with MCP servers. The session covers setting up API keys in AI Studio, uploading files via the Files API (free for 1 day), and controlling thinking budgets (0–24,000 tokens) to manage cost and reasoning depth. Schmid shows how Gemini natively processes videos at 1 frame per second for accurate timestamp extraction and how PDFs are handled by combining OCR text with image understanding. He introduces native tools like Google Search with grounding metadata, code execution, and URL context, and explains how MCP servers can be used seamlessly with the Gemini SDK for tool calling. The workshop also covers parallel vs sequential function calling, the Agent Development Kit (ADK), and the upcoming asynchronous function calling for the Live API, providing a practical path from simple generation to agentic workflows.

The New Code — Sean Grove, OpenAI

Jul 11, 2025 · 21:36

Sean Grove of OpenAI argues that specifications, not code, are becoming the fundamental unit of programming, with the most valuable skill being precise communication of intent. He presents OpenAI's Model Spec—a collection of versioned Markdown files—as a living specification that aligns both humans and models around shared values and intentions. Grove illustrates how the Model Spec served as a trust anchor during the 4.0 sycophancy bug, where shipped behavior contradicted the spec's explicit 'don't be sycophantic' clause, leading to a rollback. He explains deliberative alignment, where the spec is used as training and eval material to embed policy into model weights, moving from inference-time prompting to muscled memory. Drawing parallels to the US Constitution, Grove positions specifications as executable, testable artifacts that compose like code, and suggests future IDEs will become 'integrated thought clarifiers.' He closes by calling for help in aligning agents at scale, noting OpenAI's new agent robustness team.

2025 in LLMs so far, illustrated by Pelicans on Bicycles — Simon Willison

Jul 9, 2025 · 18:30

Simon Willison reviews the past six months of LLM releases — including AWS Nova, Llama 3.3 70B, DeepSeek R1, Mistral Small 3, Claude 3.7 Sonnet, GPT 4.5, Gemini 2.5 Pro, GPT-4o, Llama 4, GPT 4.1, O3/O4 Mini, and Claude 4 — using his 'pelican on bicycle' SVG benchmark to argue that local models have become good enough to run GPT-4 class models on a laptop and that combining tools with reasoning is the most powerful technique in AI engineering, while noting risks like prompt injection and the 'lethal trifecta'. He tracks 30 significant model releases, highlighting that Mistral Small 3 (24B) matches Llama 3 70B's performance, which itself matched the 405B model, enabling local inference. DeepSeek's R1 caused a $500B+ Nvidia stock drop on January 27. GPT 4.1 Nano is the cheapest model yet at a fraction of a cent per pelican. He also examines bugs: ChatGPT's sycophantic 'shit-on-a-stick' incident and Claude 4's tendency to snitch to authorities when given ethical instructions and email tools. Willison concludes that while the pace is accelerating, control over context and security remain critical.

Trends Across the AI Frontier — George Cameron, ArtificialAnalysis.ai

Jul 8, 2025 · 17:52

George Cameron of Artificial Analysis presents multiple frontiers—reasoning, open-weights, cost, speed—across the AI stack, arguing that trade-offs between intelligence, latency, and expense are critical for building applications. Reasoning models like O4 mini high use an order of magnitude more output tokens (72M vs GPT-4.1's 7M) and take over 40 seconds per response versus 4.7 seconds, impacting agentic workflows where 30 sequential calls multiply latency. The open-weights gap has nearly closed, with China-based labs like DeepSeek R1 and Alibaba's Qwen 3 leading. O3 cost roughly $2,000 to run the intelligence index, while GPT-4.1 nano is over 500 times cheaper. Output speeds have jumped from GPT-4's 40 tokens/s in 2023 to over 1,000 on a B200 accelerator. Despite efficiency gains, demand for compute will keep rising due to larger models, reasoning’s extra tokens, and multi-step agents.

Training Agentic Reasoners — Will Brown, Prime Intellect

Jul 7, 2025 · 19:17

Will Brown of Prime Intellect argues that reasoning and agents are fundamentally the same, and reinforcement learning (RL) is the key to advancing both. He explains that RL now works at scale, as shown by DeepSeek's GRPO and OpenAI's o3, and that agentic tasks like tool calling are natural RL environments. Brown warns against reward hacking and emphasizes designing evals that are harder to game than the task itself. He introduces his open-source toolkit 'verifiers' (now on pip) which lets users build trainable agent loops with a simple API, and demonstrates training a 7B Wordle agent in a few turns on just a couple GPUs.

12-Factor Agents: Patterns of reliable LLM applications — Dex Horthy, HumanLayer

Jul 3, 2025 · 17:06

Dex Horthy, founder of HumanLayer, presents the 12-Factor Agents framework for building reliable LLM-powered applications, arguing that production-grade agents are primarily deterministic software with targeted LLM steps rather than fully autonomous loops. He distills patterns: own prompts and context windows, treat tools as JSON and code, use small focused agents with three to ten steps, contact humans via tool calls. Horthy emphasizes context engineering—LLMs are pure functions—and shows how to compact errors, unify state, and add pause/resume via APIs. He shares a DevOps agent that became a bash script, and advocates for outer-loop agents. The framework, which gained 4,000 GitHub stars in two months, treats agents as stateless reducers that meet users on any channel, with engineers controlling the inner loop of token and control flow.

MCP Is Not Good Yet — David Cramer, Sentry

Jul 3, 2025 · 16:41

David Cramer, founder and engineer at Sentry, argues that the Model Context Protocol (MCP) is a pluggable architecture for agents, not a simple API overlay, and that it is not yet good but worth experimenting with. He stresses that exposing existing OpenAPI endpoints as MCP tools yields terrible results; instead, developers must design context specifically for agents, returning Markdown rather than raw JSON to improve model reasoning. Cramer details Sentry's MCP server, built in two days using Cloudflare Workers for OAuth 2.1, and notes that remote OAuth is essential for B2B SaaS, while standard I/O introduces security risks. He highlights the need for careful token and cost management—citing a single request that produced 20 API calls to Sentry—and advocates for building dedicated agents (like Sentry's root cause analysis tool) that wrap MCP, giving the provider control over prompts, models, and error handling. Despite ongoing issues like lack of streaming responses and unstable client support, Cramer concludes that MCP's core concepts (plug-ins, agents, tools) are just familiar software patterns with new names.

The New Lean Startup — Sid Bendre, Oleve

Jul 1, 2025 · 13:26

Sid Bendre, co-founder of Oleve, explains how his tiny team of four scaled a profitable, multi-product portfolio to $6M ARR by embracing a 'new lean startup' philosophy centered on AI tooling, operating principles, and organizational structure. Oleve's two products hit the top 10 in the App Store Education charts, competing with Duolingo and Photomath, with one reaching #4 in 2024 and #5 in 2025. Bendre details three pillars: operating principles like hiring only 10x generists, a profit-first mentality, and continuous process refinement; organizational structure modeled after Palantir's Harvester-Cultivator split, where Harvesters own product metrics and Cultivators build an agentic operating system; and AI tooling augmentation that turns 10x engineers into 100x. He shares how they repurpose tools like LaunchDarkly for load balancing and on-the-fly infrastructure changes, and invest in blueprints—reusable code templates and shared infrastructure—that enabled a third product to launch in three weeks and become profitable immediately. The episode culminates in Oleve's vision of one-person billion-dollar companies, where a single strategic leader commands clusters of autonomous…

Intro to GraphRAG — Zach Blumenfeld

Jun 30, 2025 · 1:18:35

Zach Blumenfeld introduces GraphRAG using Neo4j, showing how to build knowledge graphs from employee and skill data, combine structured and unstructured data, and use graph traversal with vector search for retrieval. The workshop covers Cypher query patterns for multi-hop similarity, entity extraction from resumes via LLMs, and creating semantic similarity relationships. It demonstrates Leiden community detection for skill clustering and builds a LangGraph agent with four custom tools that balance exact skill matches with vector similarity. Blumenfeld argues that knowledge graphs provide controlled, explainable retrieval logic for agentic workflows, enabling developers to decompose data into graph models that expose domain logic for more accurate retrieval than pure vector search.

Turning Fails into Features: Zapier’s Hard-Won Eval Lessons — Rafal Willinski, Vitor Balocco, Zapier

Jun 30, 2025 · 16:15

Zapier AI Tech Lead Rafal Willinski and Staff Engineer Vitor Balocco explain how Zapier's evaluation system turns agent failures into targeted improvements through a data flywheel. They detail collecting explicit feedback at critical moments and mining implicit signals like testing behavior, cursing, and user follow-ups. The pair advocates building unit test evals for specific failure modes, then trajectory evals and LLM-as-judge with rubrics to avoid overfitting and capture multi-turn criteria. They share that over-indexing on unit tests hurt model benchmarking, and reasoning models can compare model runs, revealing differences like Claude as a decisive executor versus Gemini's yapping. Ultimately, they argue that the goal is user satisfaction, so A/B testing on a small traffic fraction is the ultimate verification.

Building voice agents with OpenAI — Dominik Kundel, OpenAI

Jun 29, 2025 · 1:25:35

Dominik Kundel from OpenAI presents the new OpenAI Agents SDK for TypeScript and demonstrates how to build voice agents, arguing that speech-to-speech architectures (using GPT-4 real-time) offer lower latency and more natural interactions than chained approaches that transcribe audio to text. He covers two architectures: chained (speech-to-text, text agent, text-to-speech) versus speech-to-speech (native audio understanding), and recommends starting with a small, clear goal, building evals and guardrails early, and using generative prompts to control tone and personality via openai.fm. In a live coding session, he builds a real-time agent with tools (e.g., getWeather) and handoffs, demonstrates delegation to a smarter model (O4 mini) for complex tasks like refunds, and shows built-in interruption handling, human-in-the-loop approval, tracing for debugging, and output guardrails. The SDK supports WebRTC and WebSocket, handles turn detection automatically, and provides session management with a 30-minute timeout that can be extended by injecting previous context.

Evals 101 — Doug Guthrie, Braintrust

Jun 27, 2025 · 48:31

Braintrust solutions engineer Doug Guthrie presents the full AI evaluation lifecycle, covering offline and online strategies for building robust AI products. He explains that evals require three ingredients—tasks (prompts or agentic workflows), datasets (real-world examples), and scores (LLM-as-a-judge or code-based)—and demonstrates how to run them in Braintrust's playground and via SDK. The talk emphasizes creating a feedback loop: production logs feed into datasets, enabling human review and user feedback to iteratively improve offline evals. Guthrie showcases a changelog app, the new AI-driven "loop" feature for prompt optimization, and answers audience questions on A/B testing models, handling multiple human scorers, pre-launch validation, and exporting data for custom dashboards.

Engineering Better Evals: Scalable LLM Evaluation Pipelines That Work — Dat Ngo, Aman Khan, Arize

Jun 27, 2025 · 24:46

Dat Ngo, AI architect at Arize AI, presents advanced LLM evaluation strategies for production systems, arguing that effective evals go beyond out-of-the-box LLM-as-a-judge to include code-based heuristics, human feedback, and golden datasets. He explains how to build a virtuous cycle of collecting observability data, running evals, and tuning them over time, and demonstrates agent evaluation techniques like trajectory evals to identify failure modes across complex workflows. Ngo covers trade-offs between offline evals and inline guardrails, the use of log probabilities for confidence scoring, and automated prompt optimization through meta-prompting, all illustrated with customer examples from Reddit, Duolingo, and Booking.com.

How to build world-class AI products — Sarah Sachs (AI lead @ Notion) & Carlos Esteban (Braintrust)

Jun 27, 2025 · 1:43:46

Sarah Sachs (AI lead at Notion) and Carlos Esteban (Braintrust) explain that building great AI products requires 10% prompting and 90% evals and observability, with Notion AI using Braintrust to iterate on prompts and models. Sachs details their cycle: curate small datasets, tie them to scoring functions (LLM-as-a-judge with per-sample prompts and heuristic checks), run evals before shipping, and use production logs to catch regressions. She notes Notion AI supports 100M+ users, switches models in under a day, and 60% of enterprise users are non-English—requiring multilingual eval rigor. Carlos and Doug then walk through Braintrust's framework: tasks (prompts, tools, agents), datasets, and scores (0-1). They demonstrate offline evals via playground and SDK, online scoring in production, user feedback capture, human review setup, and remote evals to bridge complex code with the playground. The workshop covers moving from pre-prod evals to production monitoring, closing the feedback loop by adding underperforming spans to datasets.

From Mixture of Experts to Mixture of Agents with Super Fast Inference - Daniel Kim & Daria Soboleva

Jun 27, 2025 · 53:15

Daria Soboleva and Daniel Kim of Cerebras explain how Mixture of Experts (MoE) architectures enable scaling large language models efficiently by replacing monolithic feedforward networks with specialized experts, a technique used by GPT-4 and Claude. They then introduce Mixture of Agents (MoA), which combines multiple LLMs with custom prompts to outperform frontier models like GPT-4o on complex tasks, reducing a 293-second reasoning problem to 7.4 seconds using Cerebras' ultra-fast inference. The workshop guides participants to build their own MoA system, configure agents for bug fixing and performance optimization on a Python function, and achieve scores up to 120/120. Daniel details Cerebras' wafer-scale chip with 900,000 cores and distributed memory that eliminates memory bandwidth bottlenecks, enabling linear scaling and 15.5x faster inference on Llama 3.3-70B versus GPUs. Daria discusses ongoing research in diffusion models and sparsity, while Daniel notes plans for multimodal APIs and LoRA fine-tuning support.

Forget RAG Pipelines—Build Production Ready Agents in 15 Mins: Nina Lopatina, Rajiv Shah, Contextual

Jun 27, 2025 · 1:15:43

Rajiv Shah, Nina Lopatina, and Matthew from Contextual AI demonstrate how developers can build production-ready RAG agents in minutes using Contextual AI's managed RAG platform, emphasizing that RAG should be treated as a managed service to avoid reinventing infrastructure. They walk through ingesting documents like NVIDIA financials and spurious correlation reports, then querying the agent with questions requiring quantitative reasoning across tables. The platform handles extraction, layout analysis, image captioning, hybrid retrieval, and a state-of-the-art reranker, capped by a grounded language model that avoids hallucinations and provides attribution. Evaluation is done via LM Unit, a model-as-judge that scores responses on criteria like accuracy and causation. The episode also shows integrating the agent with Claude Desktop via MCP and answers audience questions on pricing (consumption-based with a $25 credit), scalability, entitlements, HIPAA, and domain-specific language.

How fast are LLM inference engines anyway? — Charles Frye, Modal

Jun 27, 2025 · 16:07

Charles Frye presents benchmarks from hundreds of runs on Modal comparing open-source inference engines VLM, SGLang, and TensorRTLM across models like Qwen 3 and Gemma 27B, arguing open weights models have caught up to proprietary ones, making self-hosting viable. He shows, for example, that Qwen 3 (MoE) on VLM achieves ~1 request/sec with 128 input tokens and 1024 output tokens, while switching to a RAG-like workload (1024 in, 128 out) yields a 4x throughput improvement. Frye warns that optimizing for context over reasoning can improve latency without sacrificing quality, and notes that the engines' out-of-the-box performance varies by model—e.g., SGLang underperforms VLM on Gemma due to less optimization. He also highlights the gap between prefill (parallel) and decode (autoregressive) speeds, which a rationalist would expect from transformer architecture. The benchmarks, available at modal.com/llmalmanac, aim to help engineers choose hardware and engines, with contributions welcome for optimized configs like TensorRTLM's knobs.

Graph Intelligence: Enhance Reasoning and Retrieval Using Graph Analytics - Alison & Andreas, Neo4j

Jun 27, 2025 · 1:41:16

Alison Cossette and Andreas Kollegger from Neo4j demonstrate how graph data science algorithms enhance GraphRAG systems by connecting, clustering, and curating unstructured data. They walk through running KNN similarity (K=25) on document embeddings, then applying Louvain community detection to identify clusters of near-identical chunks—one community had 49 documents with 0.98 average similarity—which can be collapsed via APOC to improve retrieval efficiency while preserving lineage. Using page rank and betweenness centrality, they show how to surface influential documents and track conversation paths across communities. The session uses a preloaded Neo4j AuraDB database with 17K nodes and 774K relationships from the Agent Neo project, and includes practical cipher queries for projecting graphs and running algorithms. They emphasize using graph analytics to diversify retrieval, manage content hygiene, and build accountable agent systems.

Revenue Engineering: How to Price (and Reprice) Your AI Product — Kshitij Grover, Orb

Jun 27, 2025 · 15:39

Kshitij Grover, CTO of Orb, argues that AI-native pricing treats monetization as a first-class engineering concern, with pricing acting as a form of friction that must balance value delivery, audience, and rapidly changing margins. He walks through examples like Replit’s multi-tier transparent pricing with a free tier to prove value before charging, Unify’s higher price point and credit-based model to signal enterprise readiness, and Cloudflare’s charging for CPU milliseconds rather than wall time to pass on architectural advantages. Grover explains that margins are not fixed; companies should protect against degenerate workloads with guardrails rather than linear scaling, as Jasper did by going unlimited after switching models seamlessly. He emphasizes flexibility, noting Orb customers make price changes two to three times a month, and advises simulating pricing impacts on different usage cohorts. Predictions include price wars, convergence toward effectively unlimited plans with caps, more real outcome-based pricing with clear SLA definitions, and real-time spend visibility with agent prompts estimating credit usage.

Building Agentic Applications w/ Heroku Managed Inference and Agents — Julián Duque & Anush Dsouza

Jun 27, 2025 · 52:35

In this workshop, Heroku Principal Developer Advocate Julián Duque and Product Manager Anush Dsouza introduce Heroku Managed Inference and Agents, a platform designed to make every software engineer an AI engineer by simplifying the attachment of agents and AI to applications. They demonstrate an opinionated, curated set of models (e.g., Claude 4) and an agentic control loop running on Heroku's trusted compute (Dynos) that provides first-party tools like code execution (Python, Node, Go, Ruby), Postgres schema inspection and querying, and document conversion, all streaming responses in real time. The workshop walks through provisioning inference via a single CLI command or add-on, using a Jupyter notebook to chain multiple tools (e.g., HTML-to-markdown then Python execution), and attaching custom MCP servers (like Brave Search) that spin up one-off Dynos and scale to zero. They also show how to expose MCPs remotely via a server-sent events endpoint for use with Cursor or other clients, emphasizing security (read-only database followers, bearer tokens) and future OAuth support. The episode concludes with a call to try the platform via a free trial team valid through the weekend.

Prompt Engineering is Dead — Nir Gazit, Traceloop

Jun 27, 2025 · 14:19

Nir Gazit, CEO of Traceloop, argues that manual prompt engineering is ineffective and demonstrates an automated, test-driven alternative. He improves a RAG-based documentation chatbot by 5x without hand-tuning prompts, using an LLM-as-a-judge evaluator with 20 example questions (each with 3 expected facts) and a CrewAI agent that iteratively optimizes the prompt based on evaluation scores. The initial score of 0.4 jumps to 0.9 after just two iterations. Gazit warns of overfitting to the small dataset and suggests splitting into train/test sets. The approach replaces manual tweaking with a 'gradient ascent' on evaluators, effectively vibe-engineering prompts.

Fun stories from building OpenRouter and where all this is going - Alex Atallah, OpenRouter

Jun 25, 2025 · 18:47

Alex Atallah, founder of OpenRouter, tells the story of how he launched the LLM aggregator in early 2023 after observing the open-source race sparked by Meta's Llama 1 and Stanford's Alpaca distillation under $600. He argues the inference market is not winner-take-all, citing OpenRouter's data showing Google Gemini growing from 2-3% to 34-35% of tokens over 12 months. The platform evolved from a simple model collection into a marketplace with over 400 models and 60 providers, solving architecture challenges like 30-millisecond latency, cancelable streams, and a middleware plugin system for web search and PDF parsing. Atallah explains that OpenRouter grew 10-100% month-over-month for two years, and he predicts future additions like transfusion models that generate images and more powerful geographic routing for enterprise optimization.

Building AI Agents that actually automate Knowledge Work - Jerry Liu, LlamaIndex

Jun 24, 2025 · 17:57

Jerry Liu, CEO of LlamaIndex, argues that building AI agents which actually automate knowledge work requires a combination of really good tools and carefully tailored agent reasoning, moving beyond naive RAG to a 'document toolbox' that includes parsing, extraction, indexing, and manipulation of complex unstructured data like PDFs, Excel, and PowerPoints. He introduces two main agent categories: assistive agents (chat-based, human-in-the-loop, unconstrained reasoning) and automation agents (batch-processing, constrained workflows, less human oversight). A key new capability is an Excel agent that uses reinforcement learning to learn a semantic map of unnormalized spreadsheets, achieving 95% accuracy on data transformation, surpassing the 75% baseline of LLMs with code interpreter and even human baselines of 90%. Real-world examples include financial due diligence combining automation (inhaling financial data) and assistant interfaces (analyst co-pilot), enterprise search with specialized agentic RAG, and technical data sheet injection for a global electronics company that reduced weeks of manual work to an automated extraction pipeline. Liu emphasizes that the right preprocessing…

The Web Browser Is All You Need - Paul Klein IV, Browserbase

Jun 20, 2025 · 17:31

Paul Klein IV, founder of Browserbase, argues that the web browser is the default MCP server for the rest of the internet, serving as the essential bridge between AI agents and legacy websites that lack APIs. He distinguishes between web agents (one prompt to many actions, e.g., OpenAI's Operator) and browser tools (one action per prompt), noting that both rely on vision-driven or text-based approaches to parse pages. Klein explains that Browserbase offers a horizontal MCP server for browsing, enabling automation of any website, and stresses the need for custom evals and observability to track agent behavior. In a live demo, he shows an agent navigating sfpca.org to find a dog for adoption, handling unexpected modals. He also addresses captchas, advising good citizenship and hinting at future agent authentication solutions.

MCP: Origins and Requests For Startups — Theodora Chu, Model Context Protocol PM, Anthropic

Jun 18, 2025 · 17:45

Theodora Chu, product manager at Anthropic, explains the origin of the Model Context Protocol (MCP) as an open-source standard for giving LLMs agency to interact with external tools and data, born from two engineers copying context manually. She details MCP's evolution from an internal hack week project to wide adoption by Google, Microsoft, OpenAI, and coding tools like Cursor and VS Code. Key protocol decisions include optimizing for server simplicity over client complexity, adding streamable HTTP for bidirectionality, and fixing authentication via community contributions. Chu outlines three startup opportunities: building high-quality MCP servers for verticals beyond dev tools (80% weight), simplifying server building with hosting/testing/deployment tooling, and creating AI security and observability tools. She emphasizes that models are a third user of servers, so tool design must consider end users, client developers, and the model itself.

How to Build Trustworthy AI — Allie Howe

Jun 16, 2025 · 24:22

Allie Howe, VC CISO at GrowthCyber, argues that trustworthy AI equals AI Security (how the world harms your AI) plus AI Safety (how your AI harms the world), and that builders are legally and reputationally responsible. She covers three pillars: MLSecOps (scanning models for serialization attacks using open‑source ModelScan), AI Red Teaming (testing for prompt injections, jailbreaks, and safety issues with tools like PyRIT), and AI Runtime Security (validating inputs/outputs in production via platforms like Pillar to block off‑topic or unsafe behavior). Howe cites real incidents—a Chevy Tahoe chatbot offered for $1, Slack's data leak via prompt injection, Fortnite's Darth Vader NPC initially producing racist outputs—and notes a lawsuit where OpenAI won by arguing users must expect errors. She emphasizes shifting‑right to runtime guardrails as the most cost‑effective investment and advises demonstrating trustworthiness through GRC platforms like Vanta to shorten sales cycles. With increasing regulation (ISO 42001, EOAI Act, FDA guidelines), she concludes that building trustworthy AI now unlocks revolutionary innovation in fields like healthcare.

Exposing Agents as MCP servers with mcp-agent: Sarmad Qadri

Jun 11, 2025 · 18:05

Sarmad Qadri, CEO of Lastmile AI, argues that agents should be exposed as MCP servers, enabling multi-agent composition and async workflows. He introduces mcp-agent, a framework implementing Anthropic's effective agent patterns—augmented LLM, evaluator-optimizer, orchestrator—on top of the Model Context Protocol. Qadri demonstrates an agent that reads a student story, fetches an APA style guide via a fetch MCP server, and writes a graded report, all orchestrated as a durable Temporal workflow. He explains that MCP standardizes tool connections for LLMs, shifting agentic behavior from client-side to server-side, allowing agents to be invoked from any MCP client like Claude or Cursor. The talk covers how this paradigm enables scalable, platform-agnostic agents that can be paused, retried, and scheduled.

Why Bolt.new Won and Most DevTools AI Pivots Failed - Victoria Melnikova

Jun 10, 2025 · 5:45

Victoria Melnikova explains how StackBlitz's Bolt.new achieved explosive growth from near-death to over 100M ARR by avoiding common AI pivot traps. She outlines three mistakes: adding a chatbot UI, competing broadly against big AI labs, and waiting for perfect AI instead of building tactically. StackBlitz succeeded by identifying its unfair advantage—running full dev environments in the browser without containers or servers—and amplifying it with AI to let users describe apps and see them live instantly. This created a new category of vibe coding, not just an AI feature. Melnikova urges devtools founders to find their unique moat and reinvent workflows rather than bolt AI onto existing ones.

Beyond Conversation: Why Documents Transform Natural Language into Code - Filip Kozera

Jun 10, 2025 · 10:57

Filip Kozera argues that chat-based interfaces like ChatGPT create ephemeral, polluted context windows and lack forced clarity, making them poor for specifying complex systems, while documents naturally force rigor and structured thinking. He advocates shifting from conversational brainstorming to document-driven workflows, where humans precisely articulate intent for AI agents. Kozera introduces background agents that run asynchronously, triggered by events like emails or meetings, and surface only when needing human approval, turning the human role into one of 'swiping left or right' to approve or edit outputs. He envisions a future where humans manage swarms of such agents, first adopted by prosumers and later by enterprises, with taste and intent becoming critical for imbuing personal brand into agent outputs. The episode presents a clear progression from chat-based brainstorming to structured document-driven agent systems.

MCPs are Boring (or: Why we are losing the Sparkle of LLMs) - Manuel Odendahl

Jun 10, 2025 · 28:32

Manuel Odendahl argues that MCPs (Model Context Protocols) are boring because they constrain LLMs to rigid, predefined tool calls, while LLMs truly shine as code generators that can create dynamic tools on the fly. He critiques the inefficiency of tool calling—repeating context, wasting tokens, and struggling with many tools—and advocates for an 'Eval' approach where the LLM writes and executes code (e.g., SQL, JavaScript) to solve tasks directly. Demonstrating with a JavaScript sandbox, he shows how a single Eval tool can introspect databases, write queries, create REST APIs, and even build a full CRM interface in one call, saving time and tokens. Odendahl urges engineers to think recursively: ask the LLM to write code that writes more code, unlocking infinite loops of creation and restoring the sparkle of LLMs beyond boring function calls.

Breaking the Chain: Agent Continuations for Resumable AI Workflows - Greg Benson

Jun 3, 2025 · 27:02

Greg Benson from SnapLogic introduces Agent Continuations, a mechanism to capture and resume the full state of AI agents, enabling human-in-the-loop approval and reliable checkpointing for long-running agent workflows. The solution borrows from programming language continuations: at any suspension point (e.g., a tool needing human approval), the agent’s state is bundled into a JSON continuation object containing the messages array, resume request, and approval metadata. This object can be saved or passed to an application layer, allowing agent loops to be fully shut down and later restored without losing progress. Benson demonstrates with a multi-level HR agent that suspends for human approval of an account authorization tool, then resumes seamlessly. The approach handles arbitrary nesting of subagents and is implemented as an open-source Python prototype that works with any OpenAI-compatible API. Future work includes general suspension triggers and integration with existing frameworks like LangChain or Pydantic AI.

Are MCPs Overhyped? A Rant about MCPs — Henry Mao, Smithery

Jun 3, 2025 · 7:29

Henry Mao, founder of Smithery and MCP steering committee member, argues that despite MCPs standardizing AI agent-service connections, the ecosystem faces fragmentation, high-friction installation, security, and monetization problems. Users struggle with unreliable MCP servers and complex five-step installs, while developers face hosting challenges, lacking tooling, distribution hurdles, and unclear monetization. Smithery aims to solve these as an AI gateway, demoing an agent that finds GitHub issues and creates Linear tickets using curated MCPs. Mao envisions a future dominated by tool calls where agent experience trumps user experience.

Agents reported thousands of bugs, how many were real? - Ian Butler and Nick Gregory

Jun 3, 2025 · 18:39

Ian Butler and Nick Gregory introduce SM-100, a benchmark of 100 real-world bugs across 84 repositories, and present their agent Bismuth as outperforming existing agents in detecting complex bugs—finding 10 needles vs 7 for the next best. However, even top performers struggle: Bismuth's true positive rate is only 25%, and Codex leads at 45%, while most agents (Cursor, Debian) report thousands of false positives (97% for basic loops). They argue that agents universally exhibit narrow thinking, missing simple bugs like a form state issue that only Bismuth and Codex caught, and that high SWE-bench scores do not translate to maintenance tasks. The episode concludes that software maintenance remains an unsolved problem requiring deeper reasoning and targeted search.

Arrakis: How To Build An AI Sandbox From Scratch - Abhishek Bhardwaj, OpenAI

Jun 3, 2025 · 40:18

Abhishek Bhardwaj, founder of Arrakis, explains why MicroVM-based AI sandboxes are the next unlock for AI agents, detailing how Arrakis provides secure, fast code execution and computer-use environments. He argues that sandboxes are essential for tool-calling models, reinforcement learning, and multi-tenant security, with Arrakis booting in under 7 seconds and supporting backtracking via snapshot/restore. The talk covers the choice of Cloud Hypervisor over other VMMs, overlayFS storage with per-sandbox read-write layers, iptables-based networking, and a built-in code execution server. A demo shows Claude using Arrakis via MCP to build a collaborative Google Docs clone and then revert to a snapshot, illustrating how a full Linux sandbox enables agents to autonomously debug and iterate without extensive prompting.

How agents broke app-level infrastructure - Evan Boyle

Jun 3, 2025 · 13:32

Evan Boyle, founder of GenSX, argues that LLM-powered agents have broken assumptions about app-level infrastructure, as p1 latency jumps from milliseconds to seconds and workflows run minutes to hours. He explains that traditional serverless providers timeout after 5 minutes, lack native streaming, and force brittle Rube Goldberg machines on top of queues like SQS or tools like Airflow. Boyle presents an open-source library that separates API and compute layers using Redis streams for resumability, enabling users to refresh pages or navigate away without losing progress. He demonstrates components with built-in retries, caching, and tracing, and a workflow that automates Hacker News analysis. Key architectural lessons include starting simple but planning for long-running agents, keeping compute and API planes separate, and leaning on Redis for resumability.

MCP Agent Fine tuning Workshop - Ronan McGovern

Jun 3, 2025 · 35:30

Ronan McGovern demonstrates how to fine-tune a Qwen3 model on high-quality reasoning traces from an MCP agent with browser tools. Using Playwright's 25 tools via Model Context Protocol, he generates multi-turn traces by running a 30B Qwen agent on RunPod, saving both tool calls and reasoning content. Those traces are then used to supervised fine-tune a 4B Qwen model with Unsloth, applying LoRA adapters to attention and MLP layers. The process includes converting MCP tool schemas to OpenAI format, extracting Hermes-style tool calls, and unrolling conversations to multiply training examples. Even with only nine curated traces, the fine-tuned model shows improved tool-calling behavior on multi-step tasks like navigating trellis.com to extract specific content.

The Demo I Wish I'd Had: OpenAI's Agents SDK... serverless! - Brook Riggio

Jun 3, 2025 · 17:51

Brook Riggio demonstrates deploying AI agent workflows on Vercel's serverless platform using OpenAI's Agents SDK and Inngest orchestration. He presents a reference architecture: Next.js frontend triggers Inngest events that invoke Python agents on Vercel serverless functions, with automatic retries and step-level observability. His demo builds a newsletter generator: users enter topics, a research agent scans news, a formatting agent structures the content, and results save to Vercel blob storage while the frontend polls for completion. He highlights that Vercel automatically detects Python functions without a config file, and Inngest's step.run ensures reliable execution within serverless time limits. The talk emphasizes cost efficiency (pay only for actual execution), full type safety via Pydantic and TypeScript, and easy integration of long-running agent tasks. The open-source repo is available for fork and contributions.

RAG Evaluation Is Broken! Here's Why (And How to Fix It) - Yuval Belfer and Niv Granot

Jun 3, 2025 · 10:58

Yuval Belfer and Niv Granot of AI21 Labs argue that current RAG evaluation is broken because benchmarks rely on local questions with answers contained in single chunks, failing to reflect real-world messy data. They demonstrate that standard RAG pipelines—like those from LangChain and LlamaIndex—achieve only 5-11% accuracy on aggregative questions about 22 FIFA World Cup documents, such as 'Which team has won the most times?' Their proposed fix converts unstructured corpuses into structured SQL databases by clustering documents, inferring schemas (e.g., year, winner, top scorer), and using text-to-SQL at inference. This structured RAG approach handles counting and max/min queries that standard chunking cannot, though it requires homogeneous data and careful normalization to avoid ambiguity (e.g., West Germany vs. Germany). They caution that RAG is not one-size-fits-all and that existing benchmarks miss these critical use cases.

Real AI Agents Need Planning, Not Just Prompting - Yuval Belfer

Jun 3, 2025 · 7:58

Yuval Belfer of AI21 Labs argues that LLMs alone still fail at instruction following, as shown by GPT-4.1's struggles in 2025, and that true AI agents require dynamic planning, not just prompting. He critiques ReAct for lacking look-ahead, contrasting it with AI21 Maestro's planner and smart execution engine, which uses best-of-n sampling, candidate pruning, and replanning. On AIfeval, Maestro pushes GPT-4.0, Claude Sonnet 3.5, and R3 Mini to near-perfect scores; on internal requirement satisfaction benchmarks, it improves over single LLM calls despite higher runtime and cost. Belfer advises starting simple (SLMs, ReAct) and escalating to planning only for complex tasks, inviting listeners to join the Maestro waitlist.

The RAG Stack We Landed On After 37 Fails - Jonathan Fernandes

Jun 3, 2025 · 18:52

Jonathan Fernandes, independent AI engineer, details the RAG stack his team settled on after 37 failed attempts, covering orchestration (LlamaIndex), embeddings (BAAI BGE small), vector database (Qdrant), LLMs (GPT-4, Qwen, Llama), reranking (Cohere), monitoring (Arize Phoenix), and evaluation (RAGAS). He demonstrates a live prototype in Google Colab using a London railway knowledge base, showing how a naive RAG returns irrelevant results (e.g., suggesting black cabs for "where can I get help at the station"). By swapping components—replacing in-memory storage with Qdrant, using an open-source embedding model, upgrading to GPT-4, and adding Cohere reranking—the answer improves to "go to booth number five next to the Eurostar ticket gates." For production, he deploys via Docker Compose with NVIDIA embedding/reranking models and Ollama for serving. The episode also stresses the importance of tracing latency per component and using RAGAS for systematic evaluation across many queries.

The Current State of Browser Agents - Jerry Wu and Wyatt Marshall

Jun 3, 2025 · 21:13

Jerry Wu and Wyatt Marshall of Halluminate evaluate browser agents with WebBench, a benchmark of 5,000 tasks across 500+ websites. They find read tasks succeed at ~80%, but write tasks drop below 40% for autonomous agents, with failures often due to infrastructure like CAPTCHAs rather than agent intelligence. They advise AI engineers to choose read over write use cases, test multiple browser infrastructure providers, and combine agents with deterministic Playwright workflows for production. Notable emergent behaviors include an agent bypassing Cloudflare by searching Google and an agent posting a top-liked Medium comment.

Text-to-Speech Data Preparation and Fine-tuning Workshop - Ronan McGovern

Jun 3, 2025 · 34:00

Ronan McGovern walks through fine-tuning Sesame's CSM-1B text-to-speech model on a specific voice, using a YouTube video as the data source. He explains token-based TTS models, including how audio is represented via codebooks and how CSM-1B uses a main transformer for zeroth tokens and a secondary transformer for 31 hierarchical tokens. The workshop covers data preparation: downloading audio with yt-dlp, transcribing with Whisper Turbo, manually correcting the transcript, and splitting audio into 30-second chunks (41 clips from a 30-minute video). Fine-tuning uses Unsloth with LoRA adapters (rank 32, alpha 16) on linear layers, training for one epoch with a batch size of 2 and virtual size of 8, reducing loss from ~6.34 to ~3.72. Evaluation compares zero-shot inference (random speaker), voice cloning (closer but imperfect), and fine-tuned plus cloning (best result, producing an Irish-accented voice with natural errors). McGovern recommends 50+ 30-second clips for noticeable effect and notes that combining fine-tuning with voice cloning yields good performance even with limited data.

Buy Now, Maybe Pay Later: Dealing with Prompt-Tax While Staying at the Frontier - Andrew Thomspson

Jun 3, 2025 · 25:09

Andrew Thompson, CTO of Orbital, introduces the concept of Prompt-Tax: the hidden cost of migrating prompts when upgrading AI models. He shares how his agentic product automates real estate due diligence, growing from <1B to 20B monthly tokens and zero to multiple seven-figure ARR over 18 months. Key tactics include optimizing for prompting over fine-tuning, using domain experts (ex-lawyers) to write prompts, and relying on vibes over formal evals. Thompson advocates 'betting on the model'—shipping new frontier models immediately and fixing regressions on the fly using progressive rollouts and rapid feedback loops. A clip from Demis Hassabis underscores the unique challenge of evolving tech stacks. The episode concludes with questions on whether evals or progressive delivery scale to manage Prompt-Tax.

The Benchmarks Game: Why It's Rigged and How You Can (Really) Win - Darius Emrani

Jun 3, 2025 · 11:20

Darius Emrani exposes how AI benchmarks are rigged, showing that xAI cherry-picked Grok-3 comparisons, OpenAI funded FrontierMath for privileged access, and Meta submitted 27 Llama-4 variants to LM Arena optimizing style over substance. Citing Goodhart's Law, he argues that when benchmarks target billions in investment, they cease to measure real capability—Andrej Karpathy admits he doesn't know which metrics to trust. Emrani provides a 5-step framework to build use-case-specific evaluations, emphasizing that 39% of score variance comes from writing style. He advocates for apple-to-apple comparisons, open-source test sets, and style-controlled metrics, concluding that teams should stop chasing leaderboards and instead iterate on real production data to ship reliable AI.

Letting AI Interface with your App with MCP — Kent C Dodds

Jun 3, 2025 · 19:54

Kent C Dodds explains how Model Context Protocol (MCP) enables AI assistants like Claude Desktop to interface with any app or service, moving beyond limited built-in integrations. He traces the evolution from early LLM chat (phase one) through host applications with custom integrations (phase two) to MCP's open standard (phase three), arguing that MCP lets developers build once for any assistant. In his live demo, he runs a Locationator server to derive his location, a GetWeather server for weather, and an Epic Me server with OAuth authentication to create a journal entry. MCP servers handle tools, resources, and prompts; the client standardizes communication while the server controls unique logic. Kent contends this protocol is how we give AI "hands" to perform real tasks, leading toward a universal Jarvis-like assistant that can dynamically connect to any service without manual integration per platform.

ChatGPT is poorly designed. So I fixed it

Jun 3, 2025 · 5:21

Ben Holmes critiques ChatGPT's confusing design, calling it a case of 'shipping the org chart' where voice and text features feel like separate apps. He demonstrates a fix by building FixGPT, an app that allows simultaneous voice and text interaction and smartly routes requests to the right model via tool calls. Using off-the-shelf APIs like 4.0 real-time and simple prompts, Holmes shows how a 'send chat message' tool enables detailed text responses alongside voice, while a reasoning model tool handles complex queries. The source code is available on GitHub, showcasing how to avoid OpenAI's science fair of options.

Creating Agents that Co-Create — Karina Nguyen, OpenAI

Apr 30, 2025 · 24:22

Karina Nguyen, an AI researcher at OpenAI and former Anthropic researcher, discusses two major AI scaling paradigms—next-token prediction (pretraining) and reinforcement learning on chain of thought—that shift AI from narrow tools to collaborative agents that co-create. Pretraining builds world understanding by predicting the next token, but hard tasks like math and creative writing require chain of thought reasoning, scaled with OpenAI’s O1 model. Post-training via RLHF and synthetic data enables rapid iteration, and the next stage is co-innovators: agents with reasoning, tool use, long context, and creativity. Nguyen shares product lessons from developing ChatGPT and Claude: 100K context via file uploads, ChatGPT Tasks for scheduled reminders that scale with model capabilities, and Canvas as a flexible interface for co-writing, coding, and research that can morph into an IDE, a tutor, or a data scientist. She envisions a future of invisible software creation where AI generates personalized, multimodal outputs on the fly, reducing reliance on clicking links, and the interface becomes a blank canvas that adapts to user intent.

Function Calling is All You Need — Full Workshop, with Ilan Bigio of OpenAI

Apr 23, 2025 · 1:42:54

Ilan Bigio from OpenAI argues function calling is the essential primitive for building AI agents, demonstrating how to implement memory, delegation, asynchronous task execution, and even self-modifying tools—all using the same loop. Starting from a basic weather tool, he builds an agent that stores user facts in a JSON file, delegates complex requests to o1, runs multiple weather lookups in parallel with asyncio, and creates a Bootstrap agent that writes its own Python functions on the fly. He also shares practical tips from his work on Swarm and the Real-Time API, including a stay-silent function to prevent premature responses and the ability to read scripts with XML tags. The workshop emphasizes that sophisticated agent behaviors emerge from simple function calling patterns without needing heavy frameworks.

The Devops Engineer Who Never Sleeps — Diamond Bishop, Datadog

Apr 22, 2025 · 16:18

Diamond Bishop, Director of AI Engineering at Datadog, explains how his team builds AI agents—the On-Call Engineer and Software Engineer—that automate DevOps tasks like incident investigation, remediation, and postmortem writing. He details how the On-Call Engineer wakes up for alerts, reads runbooks, queries logs and metrics, and suggests fixes to let human engineers sleep. The Software Engineer proactively fixes errors by generating code diffs and pull requests. Bishop shares four key lessons: scope tasks and evaluate rigorously, assemble teams of optimistic generalists and UX experts, adapt UX for human-agent collaboration, and treat observability as critical for debugging multi-step agent workflows. He predicts that within five years, AI agents will surpass humans as primary users of SaaS platforms like Datadog, urging builders to design for agent consumers.

Self Coding Agents — Colin Flaherty, Augment Code

Apr 21, 2025 · 17:23

Colin Flaherty, founding researcher at Augment Code, describes how the company built an AI coding agent that wrote over 90% of its own 20,000-line codebase with human supervision. The agent autonomously added third-party integrations like Google Search and Linear by searching its own codebase and even used its own Google Search tool to look up Linear API docs. It profiled and optimized itself, replacing synchronous file hashing with a process pool after adding print statements and running sub-copies. Flaherty argues that agent capabilities improve with better context engines, code execution environments, and test harnesses—adding test-driven iteration boosted a bug-fixing benchmark by 20% versus 4% from just upgrading the foundation model. He cautions that agents are general-purpose, not category-specific, and have different strengths than humans, making knowledge bases crucial for onboarding. With code becoming cheap, the focus shifts to product insights and design, while better tests enable greater autonomy and accelerate agent self-improvement.

Vercel AI SDK Masterclass: From Fundamentals to Deep Research

Apr 20, 2025 · 59:52

Nico Albanese from Vercel delivers a workshop on building agents with the AI SDK, demonstrating how to use generateText, tools, and structured outputs to create a deep research clone in Node.js. He shows that switching between models like GPT-4o mini, Perplexity Sonar Pro, and Gemini Flash 1.5 requires only changing one line of code. Tools with maxSteps enable autonomous multi-step agents, as when the model fetches weather data for San Francisco and New York and sums the temperatures across three steps. The deep research workflow generates subqueries (e.g., 'requirements to become a D1 shotput athlete'), searches via Exa with live crawl, evaluates relevance with an evaluate tool that discards irrelevant results, and recursively deepens research using depth and breadth parameters. Finally, o3 mini synthesizes the accumulated research into a structured Markdown report, all in 218 lines of code.

Stateful Agents — Full Workshop with Charles Packer of Letta and MemGPT

Apr 19, 2025 · 1:19:34

Charles Packer, lead author of the MemGPT paper and co-founder of Letta, argues that statefulness (memory) is the most important problem to solve for building useful AI agents, since LLMs are inherently stateless transformers. He presents MemGPT's LMOS (Language Model Operating System) approach, which treats memory management as a context compilation problem solved by the LLM itself using tool calling to read and write structured memory blocks. The workshop demonstrates Letta's open-source stack (FastAPI, Postgres, Python) where agents persist state on a server, enabling long-running, learning interactions without context overflow—e.g., an agent can update its core memory (e.g., correcting a user's name) or search archival memory (e.g., recalling user preferences after a reset). Packer also shows multi-agent communication via async message passing between agents running as independent services, and highlights that tools are sandboxed by default with support for Composio integrations. The session includes a live notebook exercise and a low-code UI (ADE) showing context window management and memory editing, emphasizing that true stateful agents improve continuously over time rather…

Voice Agent Engineering — Nik Caryotakis, SuperDial

Apr 18, 2025 · 19:07

Nik Caryotakis from SuperDial argues that in 2025, the key to production Voice AI is reliability over realism, especially for sensitive healthcare calls. SuperDial automates back-office phone calls, saving over 100,000 hours of human calling with a lean team of four engineers. He advocates for the 'say the right thing at the right time' approach, using open-source tools like PipeCat for orchestration, TensorZero for LLM routing, and self-hosted LangFuse for HIPAA-compliant observability. Caryotakis warns that new voice-to-voice models often produce nonsensical audio, favoring a sequenced STT/LLM/TTS pipeline for control. He shares specific last-mile challenges: pronunciation of names like 'Caryotaikis', avoiding confusing bot names like 'Billy', and the need for fallbacks when OpenAI goes down. The talk emphasizes that the unique value of a voice agent lies in conversational design and vertical integrations, not realistic voices.

Building and evaluating AI Agents — Sayash Kapoor, AI Snake Oil

Apr 17, 2025 · 20:00

Sayash Kapoor argues that current AI agents fall far short of their claimed performance due to flawed evaluation and a gap between capability and reliability. He cites failures like Do Not Pay (fined by FTC), LexisNexis (hallucinations in up to a third of cases), and Sakana AI (agent hacked reward functions, claiming 150x speedup that exceeded H100's theoretical max). Princeton's CoreBench shows best agents reproduce under 40% of papers. He emphasizes that agent benchmarks like SWE-bench mislead VC funding—Cognition's Devin succeeded on only 3 of 20 real-world tasks. Kapoor calls for cost-aware, multi-dimensional evaluation (e.g., Holistic Agent Leaderboard with Pareto frontiers) and a shift from capability to reliability engineering, drawing parallels to ENIAC's vacuum tube failures.

Building LinkedIn's GenAI Platform — Xiaofeng Wang

Apr 16, 2025 · 17:53

Xiaofeng Wang, manager of LinkedIn's GenAI Foundation, explains the evolution of LinkedIn's GenAI platform from simple prompt-in string-out applications to a multi-agent system for LinkedIn Hire Assistant, arguing that a unified platform is critical for bridging the gap between AI and product engineers in the era of compound AI systems. He details the platform's four-layer architecture—orchestration, prompt engineering, tools/skills invocation, and memory management—and key investments like a Python SDK, centralized skill registry, experiential memory across working, long-term, and collective layers, and observability built on OpenTelemetry. Wang discusses hiring philosophy: prioritize strong software engineers over AI expertise, hire for potential, and build diverse teams integrating full-stack engineers, data scientists, and AI engineers. He recommends solving immediate needs first, leveraging existing scalable infrastructure like messaging systems for memory, and focusing on developer experience to drive adoption.

Insights on Building AI Teams — Heath Black, SignalFire

Apr 15, 2025 · 20:30

Heath Black, Managing Director of Product at SignalFire, uses Beacon platform data to guide AI team building, arguing that credentialism is declining—only 7% of AI engineers had PhDs in 2023 versus 16% in 2015—and that work experience now outweighs education. He shows AI talent concentrates in San Francisco (35% of AI engineers), Seattle (22%), and New York (10%), and that tracking retention rates (e.g., Anthropic at 66% four-year retention vs. Perplexity at 43%) helps time outreach. Black advises hiring based on a candidate's body of work, removing academic requirements from postings, and understanding generational job-hopping (27% of Gen Z left jobs in 2023). He warns against relying solely on salary and equity, as AI engineers command 5% salary and 10–20% equity premiums, and recommends narratives centered on mission, speed, and collaborative teams. The talk emphasizes using data to filter, locate, time, and close hires effectively.

Finetuning: 500m AI agents in production with 2 engineers — Mustafa Ali & Kyle Corbitt

Apr 12, 2025 · 18:44

Method Financial and OpenPipe detail how Method scaled AI financial agents to 500 million daily interactions using fine-tuned open-source models instead of expensive GPT-4. After racking up $70,000 in monthly GPT-4 costs and facing latency and error issues, they fine-tuned an 8B parameter LLaMA 3.1 model to achieve under 200ms latency and 9% error rate (beating GPT-4o's 11% at lower cost). Mustafa Ali explains that the key was using production data from GPT as training data and choosing the cheapest model that met performance goals. Kyle Corbitt emphasizes that fine-tuning is a power tool for bending the price-performance curve when prompt engineering falls short. The episode argues that productionizing AI agents requires patience and openness from engineering teams, and concludes with a call for software engineers to pivot to AI engineering.

Trust, but Verify: Knowledge Agents for Finance Workflows - Mike Conover

Apr 9, 2025 · 21:10

Mike Conover, CEO of Brightwave, explains how his company builds knowledge agents for financial workflows, digesting thousands of pages of content for due diligence and research. He argues that non-reasoning models perform only local search, so winning systems must use end-to-end reinforcement learning over tool-use calls to achieve globally optimal outputs. Conover describes design patterns like decomposing research into sub-themes, using secondary calls for error correction, and avoiding the latency trap of long feedback loops. He emphasizes that synthesis—weaving facts across documents—remains a hard problem due to limited recombinative reasoning in training data. Brightwave's product reveals its thought process through interactive citations and structured findings, allowing analysts to drill into any passage. The episode also notes the company is hiring, offering a $10,000 referral bonus.

Personal, Local, Private AI Agents: Soumith Chintala

Apr 6, 2025 · 20:32

Soumith Chintala, co-founder of PyTorch, argues that personal AI agents should run locally and privately to maintain trust and control over intimate data. He warns that cloud-based agents, lacking complete context (like access to all messaging or financial accounts), become unreliable and potentially dangerous—catastrophic actions like buying a Tesla instead of Tide Pods are possible. Key technical challenges include slow local inference, immature open-source computer-use models, and poor catastrophic action classification. He is bullish on open models surpassing closed ones through coordinated improvement, citing Linux, Llama, and DeepSeek. Chintala also plugs open reasoning data from gr.ink and PyTorch’s work on enabling local agents, urging AI engineers to tackle these gaps.

Why Agent Engineering — swyx

Mar 24, 2025 · 11:45

Swyx argues that 2025 is the year of agent engineering, pivoting the AI Engineer Summit to focus exclusively on agents and explaining why agents are now viable due to advancements in reasoning, tool use, model diversity, and a 1000x cost reduction in GPT-4-level intelligence over 18 months. He highlights that OpenAI's ChatGPT grew 33% to 400M users in three months after shipping agentic O1 models, and projects it will reach 1B users by year-end. Swyx cites Simon Willison's crowdsourced definitions and OpenAI's new agent definition, while calling for a halt to demos of flight booking agents and Reddit astroturfing. He frames agent engineering as the evolution of AI engineering, distinct from MLE and software engineering, with practical use cases like coding and support agents having clear product-market fit.

Rethinking how we Scaffold AI Agents - Rahul Sengottuvelu, Ramp

Mar 19, 2025 · 16:32

Rahul Sengottuvelu, Head of Applied AI at Ramp and co-founder of Cohere.io, argues that building AI agents should follow the 'bitter lesson' from AI research: systems that scale with compute outperform handcrafted, deterministic code. He illustrates this with Ramp's switching report agent, which ingests arbitrary CSV files from third-party card providers. Three approaches are compared: manually coding parsers for the 50 most common vendors, using LLMs only for column classification, and a fully LLM-driven method where the model writes and runs pandas code via a code interpreter, repeated 50 times in parallel. The last approach, though using 10,000x more compute, costs under a dollar and generalizes better, saving Ramp far more in failed transactions. Sengottuvelu also demonstrates a prototype email client where the backend is an LLM with access to a code interpreter and the user's Gmail token; the LLM renders the UI as Markdown and handles clicks by re-prompting itself, simulating a full web app without traditional backend code. He contends that as models exponentially improve, shifting more execution into 'fuzzy' LLM compute—rather than rigid code—lets builders ride the trend for…

Navigating AI’s Frontier in 2025 - Grace Isford, Lux Capital

Mar 13, 2025 · 17:55

Grace Isford, partner at Lux Capital, argues that while 2025 is a 'perfect storm' for AI agents with reasoning models like O3 and R1, cheaper inference, and billions in infrastructure (e.g., Stargate, DeepSeek), agents still fail due to cumulative errors—decision, implementation, heuristic, and taste—exemplified by OpenAI Operator booking a flight incorrectly. She prescribes five strategies: curating proprietary and agent-generated data, building personalized evals for non-verifiable domains (e.g., seat preference), designing scaffolding that prevents cascading failures (citing Ramp's approach), treating UX as the moat (e.g., Codium, Harvey, TLDraw), and building multimodally with voice, smell via Osmo, and touch for embodiment. The talk, recorded at the AI Engineer Summit 2025 in NYC, closes with a call to reframe perfection through visionary product experiences.

How Windsurf writes 90% of your code with an Agentic IDE - Kevin Hou, Windsurf

Mar 11, 2025 · 20:50

Kevin Hou, head of product engineering at Codeium, introduces Windsurf as the first AI agent-powered editor, arguing that agents are the future of software development. The editor is built on three principles: trajectories, a unified timeline that lets the agent read the user's mind and execute commands like 'continue my work'; meta-learning, which auto-generates memories of user preferences and codebase context; and scaling with intelligence, removing legacy features like chat and @-mentions as models improve. Windsurf launched three months ago and has already generated 4.5 billion lines of code, with 90% of code now written via its Cascade agent. Hou emphasizes that the agent reduces human-in-the-loop effort by inferring context, running terminal commands in the user's environment, and dynamically retrieving documentation—allowing developers to contribute less input while getting more production-ready output.

Reinforcement Learning for Agents - Will Brown, ML Researcher at Morgan Stanley

Mar 7, 2025 · 18:17

Will Brown, a machine learning researcher at Morgan Stanley, argues that reinforcement learning (RL) is the essential path to unlocking autonomous AI agents, citing DeepSeek's R1 and OpenAI's Deep Research as proof: R1 used GRPO to make models learn chain-of-thought reasoning without manual data, and Deep Research applies end-to-end RL for up to 100 tool calls. He shares his own open-source single-file GRPO script—a 1B-parameter LLaMA model trained on math questions that demonstrated self-correction and improved accuracy, sparking community forks and blog posts. Brown introduces 'rubric engineering' as a new practice akin to prompt engineering, where reward rules (e.g., XML structure, integer answers) guide model improvement, and warns about reward hacking. He previews his current work: a framework for RL inside multi-step environments, letting developers reuse existing agent code for training. The talk concludes that fine-tuning and RL remain relevant as open-source catches up, and that skills like building evals and prompts translate directly to the RL era.

OpenAI for VP's of AI + Advice for Building Agents

Mar 5, 2025 · 16:52

OpenAI's Toki Sherbakov and Prashant Mital explain how enterprises adopt AI through a three-phase journey: building an AI-enabled workforce with ChatGPT, automating operations with APIs, and infusing AI into end products. They detail a Morgan Stanley case study where retrieval methods improved an internal knowledge assistant's accuracy from 45% to 98%. The pair define agents as models with instructions, tools, and self-terminating execution loops, then share four field lessons: build with primitives before frameworks, start with a single purpose-built agent, graduate to a network of specialized agents with handoffs for complex tasks, and keep prompt instructions simple while running guardrails in parallel using fast models like GPT-4o mini for safety and reliability.

AI Agents, Meet Test Driven Development

Feb 22, 2025 · 29:10

Anita from Vellum argues that teams using test-driven development build more reliable AI systems, and she breaks down how to apply TDD to agentic workflows across four stages: experiment, evaluate, deploy, monitor. She outlines five levels of agentic behavior (L0 to L4), noting that most production systems today are at L1 (tool use), while L2 systems that plan and reason with models like O1 and DeepSeek-R1 will see most innovation this year. She introduces Vellum's new open-source Workflows SDK, which keeps code and UI in sync, and demonstrates her own SEO agent that automates keyword research, content analysis, and writing via a writer-editor evaluation loop. The agent, built on Vellum Workflows, takes a keyword like "chain-of-thought prompting" and produces a first draft with a latency of around 118 seconds, incorporating competitive analysis and iterative feedback. She emphasizes that success now depends on orchestration techniques (prompt chaining, RAG, memory) rather than just model performance.

The Price of Intelligence - AI Agent Pricing in 2025

Feb 22, 2025 · 20:38

Shitej, co-founder and CTO of Orbe, argues AI agent pricing must continuously evolve, citing Intercom's 99 cent per resolution outcome model, Clay's prospecting credits, and Cursor's tiered usage limits. He stresses aligning pricing with target audience—SMB vs. enterprise—and maintaining simplicity and predictability. Cost structure is key: Character.AI optimized inference to support 100M DAUs, while Jasper leveraged a model decision engine to offer unlimited credits. Shitej emphasizes flexibility, noting OpenAI's price drops force repricing, and predicts 2025 will see more unlimited plans, outcome-based pricing with SLAs, and greater investment in pricing R&D for usage visibility.

WTF do people use Open Models for??

Feb 22, 2025 · 28:01

Eugene Cheah of Featherless.ai breaks down how individuals and enterprises actually use open-source AI models, based on platform data. DeepSeek R1 dominates individual usage, but Mistral Nemo 8B remains the top enterprise model due to production stickiness and Apache 2.0 licensing. Creative writing and roleplay account for 30–40% of all traffic, with over 60% of users in that segment being women; coding copilots and agents make up 20–30%, driven by 'vibe coding' and token-hungry workflows like Kline. RAG and ChatGPT clones represent 20%, while agentic workflows (10–20%) succeed with human-in-the-loop designs. Cheah advises enterprises to aim for 80% automation with escape hatches, and warns against chasing 100% reliability. He concludes by introducing Quirky, a post-transformer hybrid built for $100k.

Voice Agents: the good, the bad, and the ugly

Feb 22, 2025 · 18:48

Eddie Siegel, CTO at Fractional AI, details the real-world challenges of building an AI voice agent for consulting-style interviews, moving from proof of concept to production. The agent, designed to conduct hundreds of research interviews simultaneously, struggled with hallucinations, latency, and Whisper transcription errors like 'creeping dippity dippity dippity dippity'. To constrain the LLM's tendency to chitchat and rabbit-hole, Siegel's team introduced a 'drift detector' agent and a 'next question' agent, both running as separate text-based side threads. They also added tool use for question navigation, explicit goals and priorities for follow-ups, and a 'transcript cleaner' to hide garbled outputs from the user interface. To move beyond vibes-driven iteration, they built an automated eval suite with metrics like clarity and completeness, using synthetic conversations with personas such as a 'snarky teenager in charge of a Fortune 500 company' to simulate diverse interviewees. The talk offers a no-nonsense playbook for getting voice agents into production, focusing on out-of-band checks, tool use, and evals even without objective ground truth.

Beyond APIs: How AI Web Agents Are Automating the "Long Tail" of Knowledge Work

Feb 22, 2025 · 17:44

Arjun and Bhavani present Rtrvr.ai, a universal AI web agent using a text-based approach to autonomously perform tasks across multiple browser tabs at under a penny per page. They argue text-based reduces hallucination compared to vision-based agents like OpenAI's operator, and being a Chrome extension avoids password sharing and accesses logged-in content. Features include deep research navigating multiple pages, dynamic function calling for any third-party API, and graph generation. Rtrvr extracts structured data, performs actions like clicking dropdowns, and processes tabs simultaneously, automating the long tail of knowledge work including market research, LinkedIn automation, and WhatsApp messaging. They claim a distributed subtask approach lowers failure rates and envision collaborative dataset construction, e.g., aggregating local government events.

Your Evals Are Meaningless (And Here’s How to Fix Them)

Feb 22, 2025 · 18:50

In this AI Engineering Summit talk, HoneyHive co-founder exposes why most LLM evaluations are meaningless due to criteria drift—where evaluator criteria misalign with user needs—and dataset drift, where test cases don't reflect real-world queries. He argues that static evaluation frameworks from tools like LangChain or Ffragas fail because they measure generic metrics rather than business-specific relevance, citing an e-commerce recommendation system that looked perfect in testing but broke in production. The fix is a three-step iterative alignment process: align evaluators with domain experts via continuous critique and few-shot examples; keep datasets alive by logging production underperformance and flowing those cases back into the test bank; and track alignment over time using F1 scores for binary judgments. Practical advice includes customizing the LLM evaluator prompt, starting with 20 domain expert examples in spreadsheets, and avoiding templated metrics. The speaker emphasizes that evals must evolve continuously, just like the LLM application itself, or they become meaningless.

OpenLLMetry is all you need

Feb 22, 2025 · 9:12

Nir, CEO of Trace Loop, introduces OpenLLMetry, an open-source project extending OpenTelemetry for tracing and monitoring GenAI applications. OpenTelemetry, maintained by CNCF, standardizes logging, metrics, and traces across cloud environments, supported by platforms like Datadog, New Relic, and Grafana. OpenLLMetry provides over 40 automatic instrumentations for foundation models (OpenAI, Anthropic, Cohere), vector databases (Pinecone, Chroma), and frameworks (LangChain, LlamaIndex, CrewAI). These instrumentations emit logs, metrics, and traces in OpenTelemetry format, allowing users to send data to any supported observability backend with a configuration change, avoiding vendor lock-in.

The Model Isn’t Wrong—You’re Just Bad at Prompting

Feb 22, 2025 · 8:54

Dan from PromptHub argues that prompt engineering remains critical for improving LLM outputs, covering Chain of Thought, few-shot, and meta prompting techniques. Chain of Thought breaks problems into sub-problems and is built into reasoning models; few-shot prompting works best with just one or two diverse examples, but can degrade performance on reasoning models like O1 and R1. Meta prompting uses LLMs to write or refine prompts, with PromptHub offering model-specific enhancers. For reasoning models, Dan advises minimal prompting, encouraging more reasoning instead of few-shot, and avoiding instructing the model on how to reason. Free resources include PromptHub's templates, the AutoReason prompt, and the Prompt Engineering Substack.

Tool Calling Is Not Just Plumbing for AI Agents — Roy Derks

Feb 22, 2025 · 25:18

Roy Derks argues that tool calling is the most critical yet overlooked component of AI agents, far more than mere 'plumbing.' He contrasts traditional tool calling—where developers manually manage callbacks, retries, and errors within the agent loop—with embedded tool calling, a black-box approach used by frameworks like LangChain’s createReactAgent. Derks advocates for separation of concerns via the Model Context Protocol (MCP) from Anthropic, which splits tool logic into MCP servers communicating with clients, and via standalone tool platforms such as IBM’s wxflows, Composio, and Toolhouse that let teams build tools once and reuse them across LangChain, CrewAI, or AutoGen. He also introduces dynamic tools, where an agent generates queries on the fly—e.g., using GraphQL or SQL schemas—instead of defining hundreds of static tools, noting that LLMs like Claude handle GraphQL well but may hallucinate on deeply nested schemas. The episode emphasizes that 'an agent is only as good as its tools' and provides practical guidance on designing tool descriptions (which act like system prompts) and output schemas to enable type-safe, chainable tool calls.

Lets Build An Agent from Scratch — Kam Lasater

Feb 22, 2025 · 19:29

Kam Lasater demonstrates that an AI agent is simply an LLM paired with memory, planning, tools, and a while loop, and shows how to build one from scratch in incremental steps. He starts with a basic LLM call (step 0), adds a condition using an LLM as judge to check if an answer is complete (step 1), then incorporates a tool for web search via SERP AI (step 2). After refactoring for parallel tool calls (step 3), he introduces a to-do list for planning and tracking progress, along with a browse-web tool (step 4). The agent uses a to-do list to plan, execute tasks like searching and browsing, and then checks completion via LLM as judge. Lasater illustrates with examples: finding the average wind speed of a swallow, buying a hoodie near Times Square, and planning a date in Ardmore. He emphasizes that the simplest agent emerges naturally from combining these components, and encourages running and breaking the provided code.

Reverse Conway's law and GenAI: How agents will take over the organisation - Patrick Debois

Feb 22, 2025 · 28:12

Patrick Debois argues that generative AI will reverse Conway's law, reshaping organizations around agents instead of human teams. He traces a progression from AI as a copilot to a team member, then a peer, and eventually a manager, with each stage unbundling human tasks and shrinking team sizes. Debois cites Amazon's AI pricing glitch as a reminder that humans remain needed for failure cleanup, and notes that LLMs mimic human collaboration behaviors, as shown in a multi-agent simulation. He warns of agent toxic behavior and the need for guardrails akin to human codes of conduct, while speculating that companies may replace SaaS with internally built AI services, making performance reviews and ROI calculations for agents inevitable. Debois advises engineers to focus on building the AI that builds their current work, not the work itself.

Where AI is superhuman: The right jobs to automate with LLMs

Feb 22, 2025 · 11:53

Andy Treadman, partner at Theory Ventures, argues that LLMs are already superhuman in high-volume, low-complexity jobs, making them the prime targets for full automation. He breaks down LLM capabilities into transformation, synthesis, and reasoning, and maps jobs on a spectrum of volume and complexity. At the high-volume end, he says LLMs can be 10x better than humans because scale is the challenge, not reliability. He highlights two portfolio companies: Dropzone AI automates security alert investigations, performing better than rules-based systems and covering 24/7, while Amp personalizes customer engagement at scale, discovering new cohorts like late-night snackers. Treadman predicts that organizations will shrink and invert from pyramids to diamonds, with fewer entry-level roles. He advises founders to consider technology-problem fit and other factors when building AI workflow automation.

How Coding Agents change Software Development Forever - Hailong Zhang

Feb 22, 2025 · 8:50

Hailong Zhang presents how coding agents, particularly the unit test agent Gru, transform software development by automating routine tasks while humans focus on creative work. In the future workflow, synchronous agents like GitHub Copilot and Cursor assist in real-time, while asynchronous agents like Gru autonomously handle tasks such as writing tests, fixing bugs, and submitting pull requests. Gru, built to boost unit tests, detects code changes from pull requests, generates and runs tests, and submits a new PR with a summary and coverage improvement. In production, over 50% of Gru's PRs are accepted by humans, and it handles 80% of unit tests in its own repo, making it the top contributor. To build such agents, Zhang emphasizes defining a clear problem (e.g., unit tests, not generic software engineering), creating datasets and evaluation harnesses, selecting and fine-tuning LLMs per stage, building task-specific context from environment data, and developing an agent operating system to reuse common infrastructure across different tasks.

The LLM Triangle: Engineering Principles for Robust AI Applications - Almog Baku:

Feb 22, 2025 · 26:19

Almog Baku introduces the LLM Triangle Principles for building production-ready AI applications, arguing that LLMs should be treated like interns guided by Standard Operating Procedures (SOPs). He breaks down the triangle into three components: the foundation model, engineering techniques, and contextual data, all directed by SOPs derived from expert interviews. Baku contrasts autonomous agents—creative but unpredictable—with handcrafted LLM-native architectures that offer sustainable quality, recommending scoped autonomy. He advises starting with large models to collect data and optimize incrementally, rather than fine-tuning from day one. Data is paramount: 'show, don't tell' via few-shot examples, and balance context to avoid the 'needles in a haystack' problem. The framework aims to turn demos into robust production systems.

Personality Driven Development: Exploring the Frontier of Agents with Attitude

Feb 17, 2025 · 18:00

Ben, from Perpetual, argues that anthropomorphizing AI agents—giving them personalities, forms, and preferences—transforms user adoption and engagement, despite significant challenges. He presents examples like Tech Lead, an artichoke recruiter, and hamster teams, showing how form factors create instant understanding and enable price anchoring (e.g., 1/20th the cost of a junior employee). Benefits include easy problem decomposition (specialized agents outperform generalized), branding, and fun. However, downsides include reinforcing stereotypes (100% of generated software engineers appear male), raised performance expectations, difficulty rebranding, distraction from core value, and stark reminders of job replacement—illustrated by a CEO introducing an IC as the one who 'knows what they're doing.' Ben highlights personality-driven development where every instance is bespoke, with prompt-driven preferences (e.g., 'I like Ivy League schools'), making debugging uniquely challenging.

Optimizing LLMs in Insurance with DSPy: Jeronim Morina

Feb 16, 2025 · 19:29

Jeronim Morina from AXA Germany argues that AI engineers must stop relying on manual prompt tuning and instead adopt DSPy's automated optimization combined with first-principles thinking. He explains how his team built a customer-facing insurance chatbot by first crafting clear problem definitions, annotating evaluation data, and avoiding data leakage. After initial struggles with fragile prompts and overly complex error-handling code, they modularized their system into DSPy modules and created custom metrics for German text. Morina emphasizes that tools like DSPy require a steep learning curve and a solid baseline of hand-written prompts and basic evaluations before optimization. The episode calls for engineers to focus on real-world impact, measure everything, and break down problems into discrete, optimizable steps.

Customized, production ready inference with open source models: Dmytro (Dima) Dzhulgakov

Feb 16, 2025 · 18:55

Dmytro (Dima) Dzhulgakov, co-founder and CTO of Fireworks AI, argues that open source models are the future for production Gen AI applications, and Fireworks provides a platform to make them customized and production-ready. He explains that while proprietary models like GPT-4 are powerful, they are too large, expensive, and slow for many use cases, whereas open models like Llama or Gemma can be fine-tuned for specific domains to achieve better quality at up to 10x speed and lower cost. The main challenges of using open models—complex setup, performance tuning, and production scaling—are addressed by Fireworks' custom serving stack, which achieves the fastest long-prompt inference and serves SDXL fastest among providers, handling over 150 billion tokens per day. The platform supports fine-tuning and serving thousands of LoRA adapters on the same GPU with serverless pay-per-token pricing. Dzhulgakov also highlights the emerging architecture of compound AI systems, where function calling—exemplified by the open-source Fire Function model—connects LLMs to external tools and knowledge sources, enabling agentic applications like stock querying and chart generation. The episode…

RAG at scale: production ready GenAI apps with Azure AI Search

Feb 13, 2025 · 21:53

Pablo, a Distinguished Engineer at Microsoft, explains how Azure AI Search enables production-grade RAG applications at scale by addressing key scaling challenges: data volume, query load, workflow complexity, and diverse data types. He demonstrates the platform's hybrid retrieval combining vector search (HNSW, exhaustive), keyword search (BM25), and filters, with a two-stage system using cross-encoder re-ranking for improved quality, achieving 100ms re-ranking latency. Azure AI Search supports multi-billion vector indexes through 10–12x storage limit increases and quantization (int8, single-bit) preserving 90–95% precision, with oversampling for full-precision re-ranking. Integrated vectorization automates ingestion from Azure sources (blob, Cosmos DB, OneLake) with change tracking and format handling (PDFs, Office documents, images). OpenAI uses Azure AI Search to back ChatGPT file uploads and the Assistants API, scaling user limits 500x after Microsoft's infrastructure upgrades.

System Design for Next-Gen Frontier Models — Dylan Patel, SemiAnalysis

Feb 11, 2025 · 18:29

Dylan Patel of SemiAnalysis breaks down the inference challenges for next-generation frontier models like GPT-4 (1.8 trillion parameters) and upcoming models trained on 100,000+ GPU clusters. He emphasizes that prefill (prompt processing) is compute-intensive while decode (token generation) is memory bandwidth-intensive, creating a systems problem where serving 64 users at 30 tokens/second requires 60 terabytes/second of memory bandwidth. Patel details engineering strategies such as continuous batching to improve batch utilization by 10-100x, disaggregated prefill to isolate noisy neighbors and maintain time-to-first-token SLAs, and context caching (like Google's) to cache KV cache on CPU/storage instead of GPU memory, dramatically reducing prefill costs. He warns that open-source tools like LLaMA.cpp lack these optimizations, making high-performance serving of models like LLaMA 405b infeasible without libraries like vLLM or TensorRT-LLM. On scaling, Patel notes that 100,000 GPU clusters (e.g., Microsoft's Arizona data center consuming 150 MW) face reliability issues — optical transceivers fail every five minutes — and straggler chips (silicon lottery) can degrade training…

Building an AI assistant that makes phone calls [Convex Workshop]

Feb 9, 2025 · 50:53

Tom Redmond, head of DX at Convex, builds Floyd, an AI phone-calling assistant using Convex, GPT-4, Google Cloud speech-to-text, and Twilio. Floyd lets users make voice requests (e.g., calling a school to report a sick child) via a web app; transcription, context retrieval, and real-time conversation with a human on the other end happen through a reactive database architecture. Redmond explains how Convex's WebSocket-based infrastructure solves serverless latency issues, enabling live transcript streaming and status updates without polling. He demonstrates two calls (school absence and flower order), noting that latency (2–5 seconds) comes from OpenAI text-to-speech and growing prompts, with plans to use OpenAI Threads and faster TTS services. The prototype explicitly discloses it's an AI to avoid misrepresentation, and Redmond envisions AI agents efficiently handling service bookings.

LLM Quality Optimization Bootcamp: Thierry Moreau and Pedro Torruella

Feb 8, 2025 · 53:05

Thierry Moreau of OctoAI demonstrates how to fine-tune Llama 3 8B on a PII redaction task using OpenPipe and OctoAI, achieving 47% better accuracy and a 200x cost reduction (from $30 to $0.15 per million tokens) compared to GPT-4 Turbo. He explains that fine-tuning should follow prompt engineering and RAG, and works best for specialized tasks like function calling. The talk walks through building a fine-tuning dataset from the PI Masking 200k dataset, using OpenPipe to train a LoRA for $40, deploying it on OctoAI, and evaluating it to show the fine-tuned model scores 0.97 accuracy versus GPT-4’s 0.68. Moreau emphasizes that this continuous deployment cycle requires monitoring data drift and retraining, but tools like OpenPipe and OctoAI make it accessible even for teams without deep ML expertise.

Building security around ML: Dr. Andrew Davis

Feb 8, 2025 · 25:01

Dr. Andrew Davis, Chief Data Scientist at HiddenLayer, argues that machine learning models remain highly vulnerable to adversarial attacks despite a decade of research, and defenses must be layered with observability, logging, and skeptical data handling. He details how ImageNet's URL-based distribution enables data poisoning via expired domains, and how model theft can replicate a LLaMA 7B model's performance for just $600 in OpenAI queries. Adversarial examples still evade robust defenses, with best-case robustness only 50-60% against advanced attacks, and multimodal LLMs amplify the threat as pixel-level modifications are far harder to detect than text prompt injections. Spotlighting — encoding data in base64 to prevent instruction-following — is a promising prompt injection defense, but attackers can craft readable base64 strings to bypass it. The ML supply chain on Hugging Face is fraught with risk: models can execute arbitrary code via Lambda layers or TensorFlow functions, so verifying provenance, scanning for malware, and sandboxing are critical. Finally, software vulnerabilities in tools like Ollama (with recent RCE CVEs) demand the same patching discipline as traditional…

Insights from Snorkel AI running Azure AI Infrastructure: Humza Iqbal and Lachlan Ainley

Feb 8, 2025 · 20:46

Humza Iqbal of Snorkel AI explains how the company uses Azure AI infrastructure powered by NVIDIA GPUs to fine-tune foundation models for enterprise customers, achieving better performance per dollar by switching from A100s to H100s. He details their distributed training stack (PyTorch, Horovod, NFS) and lessons learned such as balancing node count for batch size and monitoring GPU utilization to avoid networking or data-loading bottlenecks. A cost comparison found two H100s outperformed four A100s on both training and inference, enabling faster iteration through more synthetic data. Azure's dedicated VMs, reliable NFS throughput, and flexible capacity allowed Snorkel to scale experiments from single-node to dozens of GPUs. Future work includes programmatic preference signals and multimodal retrieval algorithms, all planned on Azure.

[Full Workshop] Llama 3 at 1,000 tok/s on the SambaNova AI Platform

Feb 7, 2025 · 1:00:58

Michelle Matern and Petro Milan of SambaNova present their full-stack AI platform, built on the SN40L RDU chip with a three-tiered memory architecture capable of storing up to 5 trillion parameters. They demonstrate Samba1 Composition of Experts (CoE), a trillion-parameter model combining 92 expert models behind a single endpoint, and show Llama-3-8B achieving 1,000 tokens per second with a time-to-first-token of 0.09 seconds and total inference time of 0.65 seconds—far exceeding GPU-based providers. The workshop includes a hands-on basic inference call using LangChain and SambaStudio API, and a RAG-based Q&A system for enterprise search that integrates Unstructured for document loading, E5-large-v2 embeddings, ChromaDB vector store, and the high-speed Llama-3 endpoint. Attendees learn to configure prompts with special Llama-3 tags, set chunk size and overlap, and optionally run embeddings on SambaNova's RDU hardware for faster processing.

[Full Workshop from Microsoft] Github Copilot - The World's Most Widely Adopted AI Developer Tool

Feb 7, 2025 · 1:19:45

This workshop from the AI Engineer World's Fair features GitHub's Christina Warren, Dave, Alex, and Harald presenting GitHub Copilot, the world's most widely adopted AI developer tool. They demonstrate Copilot's three interaction modes: ghost-text completions using GPT-3.5 for speed, inline chat (also GPT-3.5) for quick code edits, and the chat panel powered by GPT-4 Turbo for deeper conversations. Speakers emphasize prompt crafting—being specific, providing examples, and keeping relevant files open—to improve results. They show how to use slash commands (/fix, /explain, /test) and the new attach button for explicit context. The workshop uses GitHub Codespaces preconfigured with Python 3.11 and Copilot extensions; a coupon provides a 7-day free trial. Harald explains the trade-off between model quality and latency, and notes upcoming features like automatic agent delegation and workspace integration for cross-file edits.

Build, Evaluate and Deploy a RAG-Based Retail Copilot with Azure AI: Cedric Vidal and David Smith

Feb 6, 2025 · 1:57:58

David Smith, Cedric Vidal, and Miguel Martinez lead a hands-on workshop on building a production-level RAG-based retail copilot using Azure AI. They demonstrate how to build a chatbot backend that retrieves product information from Azure AI Search via vector embeddings and customer history from Cosmos DB, then augments the LLM prompt to generate grounded answers. The session covers using Azure AI Studio and Prompt flow to orchestrate the RAG workflow, deploying the flow as a managed endpoint, and evaluating quality with GPT-4 as a judge on metrics like relevance and groundedness. The speakers also explain the LLM Ops lifecycle for iterative improvement and compare Prompt flow with Semantic Kernel and AutoGen.

Which Jobs Can Be Replaced Today: Fryderyk Wiatrowski and Peter Albert

Feb 6, 2025 · 19:59

Fryderyk Wiatrowski and Peter Albert, co-founders of Zeta Labs, argue that autonomous browser agents will first replace reactive jobs—like customer support and scheduling—by automating low-leverage tasks while preserving human focus on high-leverage activities. They propose a trigger-pool system where agents react to emails, Slack, or events, requiring only approval for actions. Peter details building reliable agents: start with prompting optimized for model distribution, then add cognitive architectures (e.g., planning, scratchpads) to split tasks, and finally fine-tune with synthetic data or reinforcement learning. He advises minimizing noise in prompts, preferring text-based reasoning over images, and using language model judges to filter training data. The founders see continuous model improvement enabling agents to handle increasingly complex, proactive roles, moving toward full job replacement.

Multi model multimodal and multi agent innovations in Azure AI: Cedric Vidal

Feb 6, 2025 · 28:56

Cedric Vidal, Principal AI Advocate at Microsoft, demonstrates Azure AI's multi-model, multimodal, and multi-agent capabilities in a session packed with live demos. He shows how GPT-4 Omni mixes text and vision to read handwritten French menus and translate them, and diagnoses infrastructure damage from photos for energy and insurance industries. A new video translation service lets him speak German, Spanish, Italian, and Japanese in his own voice while preserving tone, such as whispering or yelling. The Azure AI model catalog now offers 1,600 models, including serverless deployment options, and Phi-3 Vision, a small 3.8B parameter model running locally in a browser via WebGPU. He also demonstrates code interpreter analyzing a GPX file from a kitesurfing session, plotting a map with turn markers, and GitHub Workspaces preview generating a Java GUI from Python code.

AI Templates: Gabriela and Aishwarya

Feb 6, 2025 · 1:03:20

Gabriela de Queiroz, Aishwarya, and Pamela from Microsoft present AI templates for rapidly prototyping and deploying generative AI applications, demonstrating how startups can leverage Microsoft for Startups Founders Hub, including up to $150,000 in Azure credits and expert guidance. The workshop covers three templates: a simple chat app using GPT-3.5 Turbo, a Retrieval Augmented Generation (RAG) app on Postgres with hybrid vector and text search, and a RAG app on unstructured documents using Azure AI Search and Document Intelligence. Key insights include the importance of query rewriting, hybrid search over pure vector search, and streaming responses for better user experience. The templates are open source on GitHub and can be deployed via Codespaces, with a proxy provided to bypass Azure OpenAI approval delays.

No-code fine-tuning: Mark Hennings

Feb 5, 2025 · 9:27

Mark Hennings, creator of Entrypoint, argues that fine-tuning large language models is now accessible without code, offering faster and cheaper alternatives to prompt engineering: GPT-3.5 fine-tuned runs at 73ms per token vs GPT-4's 196ms, saving 88.6% in cost and cutting prompts by 90%. He explains that fine-tuning reduces prompt injection risks, enables team collaboration via training data, and only requires 20 examples to start. Hennings demonstrates Entrypoint's no-code UI that lets users import CSV data, structure fields with templating, and fine-tune GPT-3.5 Turbo, then iteratively improve models by feeding production feedback back into the dataset. He proposes a dev lifecycle: prototype with prompt engineering, use it to build a dataset, fine-tune, evaluate, deploy, and continuously refine.

Building efficient hybrid context query for LLM grounding: Simrat Hanspal

Feb 5, 2025 · 11:35

Simrat Hanspal from Hasura explains how to build efficient hybrid context queries for LLM grounding using Hasura's GraphQL API, demonstrated via an e-commerce product search use case. The talk covers three query types: semantic search (e.g., 'essential oils for relaxation'), structured search (e.g., 'products less than $500'), and hybrid queries combining both (e.g., 'essential oil diffusers between $500 to $1,000'). Hanspal shows how Hasura unifies relational (Postgres) and vector (Weaviate) databases into a single GraphQL API, auto-vectorizing records via events. A critical security demonstration highlights how role-based permissions (e.g., 'Product Search Bot' role with only select access) prevent malicious insert mutations like adding a fake product with ID 7001. The approach enables secure, dynamic data retrieval for Retrieval-Augmented Generation (RAG) pipelines without building separate APIs.

Scaling AI in Education: A Khanmigo case study: Shawn Jansepar

Feb 5, 2025 · 22:39

Shawn Jansepar, Khan Academy's Director of Engineering, details building Khanmigo, an AI tutor and teacher assistant on GPT-4 with OpenAI, arguing generative AI can democratize one-on-one tutoring. He describes a rapid prototyping culture that launched Khanmigo in three months via a company-wide hackathon, replacing traditional agile with a prototype-to-beta-to-launch framework. Technical challenges include math accuracy through a math agent and chain-of-thought prompting, refactoring prompts into a component architecture for testing, and managing scale using multiple models and dedicated Azure compute. Jansepar highlights ethical design: a Socratic tutor that avoids answers, teacher moderation, and a writing coach with revision history. Khanmigo now has 200,000 paid users, half in school districts, with teacher tools sponsored by Microsoft free to US teachers. Plans include releasing a math tutoring benchmark and evaluating smaller models like Phi-3.

Navigating Challenges and Technical Debt in LLMs Deployment: Ahmed Menshawy

Dec 31, 2024 · 16:15

Ahmed Menshawy, VP of AI Engineering at Mastercard, argues that deploying LLMs requires overcoming massive technical debt, with over 95% of the work outside the ML code. He details how Mastercard boosted fraud detection by 300% using LLMs while adhering to seven responsible AI principles. The episode covers RAG as a solution for hallucination, attribution, and access control, noting that 80% of organizational data is unstructured and 71% of companies struggle to manage it. Menshawy highlights that GPU compute for inference now exceeds training, and that fine-tuning open-source models enables joint training of retriever and generator. He also cautions against AGI hype, quoting Ada Lovelace and a Nature article urging focus on current AI risks.

Hiring & Building an AI Engineering Team: Dr. Bryan Bischof

Dec 31, 2024 · 29:07

Dr. Bryan Bischof, Head of AI at Hex, argues that building an AI engineering team requires hiring based on product stage—starting with full-stack engineers and data profiles, then adding designers and later MLEs—while avoiding the 'mythical man month' trap in early AI products. He advocates for data intuition over LeetCode in interviews, using a take-home data exercise to assess candidates' ability to extract meaning from data and give feedback. Key attributes he looks for are curiosity, urgency, and product-mindedness, noting that enthusiasm alone is insufficient. Bischof also recommends working directly with domain experts to model AI behavior and suggests a centralized AI platform team to support multiple product teams, rather than having each team build AI infrastructure independently.

AI Platform Engineering: Patrick Debois

Dec 31, 2024 · 28:18

Patrick Debois, who coined DevOps in 2009, argues scaling GenAI requires an AI platform team, mirroring the cloud and DevOps pattern. He lists shared infrastructure: model access, vector databases, RAG connectors, version control, proxy, observability, monitoring, caching, feedback services, and stresses enablement through prototyping and frameworks. He warns of pitfalls like chasing GenAI without use case, overfocus on fine-tuning, and cost obsession. Citing the Ironies of GenAI Automation, he notes that as engineers shift from producing to reviewing code—copying co-pilot suggestions more on weekends—they risk losing situational awareness. For governance, he advises awareness programs, opt-out training, license checks, and layered guardrails: central rules plus team-specific overlays. He recommends combining cloudops, secops, devx, data platform, and AI platform teams for collaboration.

AI Frontiers in Trust and Safety Combatting Multifaceted Harm on Tinder at Scale: Vibhor Kumar

Dec 2, 2024 · 14:36

Vibhor Kumar, senior AI engineer at Tinder, explains how the company uses open-source LLMs and LoRAX to detect a long tail of trust and safety violations at global scale. Facing challenges like content pollution and automated fraud from generative AI, Tinder leverages pre-trained models such as LLaMA and Mistral, fine-tuning them with LoRA and QLoRA on hybrid datasets generated by GPT-4 and manually verified. They serve dozens of fine-tuned adapters on a single GPU using LoRAX, achieving real-time inference (tens of QPS, ~100ms latency) for categories including hate speech, pig butchering scams, and underage users. The approach yields near 100% recall on simpler tasks and significant improvements over baselines, with better generalization that resists adversarial evasion. Future directions include visual language models for explicit image detection and automating retraining pipelines.

Iterating on LLM apps at scale Learnings from Discord: Ian Webster

Nov 22, 2024 · 18:26

Ian Webster, Senior Staff Engineer at Discord and maintainer of Promptfoo, shares how Discord built and scaled Klyde AI, a chatbot for 200 million users, focusing on evaluation and safety. He argues that evals should be treated as simple, deterministic unit tests that run locally, avoiding complex metrics, and that breaking the system into small, testable pieces (e.g., checking for lowercase output to enforce casual tone) achieves 80% of the goal with 1% of the work. Webster details how Discord mitigated risks like the 'grandma jailbreak' (which originated on Discord) by using an attacker LLM to generate adversarial inputs and a judge to refine them, exposing cracks in safeguards. He advocates for pre-deployment red teaming over live filtering, and describes using Promptfoo for risk assessment across brand, legal, and safety categories. The episode also covers prompt management via Git and Retool, routing with occasional GPT-4 responses to correct model drift, and the challenge of closing the feedback loop due to privacy constraints, relying instead on dogfooding and public examples.

The AI emperor has no DAUs why most devs still don't use code AI: Quinn Slack

Nov 20, 2024 · 18:45

Quinn Slack, CEO and cofounder of Sourcegraph, argues that despite massive hype, only about 5% of professional developers actually use Code AI tools, with total recurring revenue from Code AI sitting at roughly $300 million ARR—a fraction of Salesforce's $36 billion. He cites GitHub's 1.3 million paid Copilot subscribers and just 935,000 yearly active users receiving suggestions, revealing the gap between perception and reality. Slack warns that the entire AI ecosystem—foundation models, infra, and applications—risks collapse if usage doesn't grow, and most revenue in AI flows to NVIDIA and chip makers, not software. From building Cody, the number two Code AI product, he shares lessons: hype fools everyone, autocomplete is a freakishly good feature that spoils expectations, while chat and agents are harder to vet and adopt. He advises builders to use their own product daily, ignore customer demands for buzzwords like fine-tuning, and manually build explicit interactions before adding magic. Slack concludes that the industry must collectively dehype and focus on real daily active users to turn the potential into sustained enterprise revenue.

How Zapier Builds AI Products and Features with the Help of Braintrust: Ankur Goyal & Olmo Maldonado

Nov 7, 2024 · 14:59

In this talk, Olmo Maldonado (Senior AI Engineer at Zapier) and Ankur Goyal (CEO of Braintrust) explain how Zapier uses Braintrust's evaluation and observability platform to build and improve AI features like AI Zap Builder and Copilot. They moved from 7 manual unit tests to over 800 automated evals, improving accuracy by nearly 300%. Braintrust's tracing capabilities let them dissect Copilot's multi-tool agent performance, leading them to adopt GPT-4 Turbo for the message router despite latency trade-offs. When switching to GPT-4.0 caused a regression below 80% scores, they diagnosed the issue via evals—GPT-4.0 was ignoring system prompts—and fixed it by relaxing prompt engineering and adjusting tool choices, recovering accuracy and cutting response time from 14 to 3 seconds. The episode details how eval-driven development and observability enable rapid iteration and reliable AI products at scale.

What It Actually Takes to Deploy GenAI Applications to Enterprises: Arjun Bansal and Trey Doig

Nov 4, 2024 · 21:30

Trey Doig of Echo AI and Arjun Bansal of Log10 recount Echo AI's journey deploying a GenAI-native conversational intelligence platform for billion-dollar retail brands, focusing on the centrality of accuracy. Echo AI ingests all customer conversations, uses LLMs to surface insights at 100% coverage, but must overcome enterprise trust issues by achieving 95% accuracy within seven days. The platform relies on Log10's auto feedback system, which uses AI-based review to match human accuracy with model speed, yielding a 20 F1 point improvement in one use case. The episode details how Echo AI's solution engineers use Log10 to grade summarizations, catch hallucinations, and track model drift, turning human feedback into curated datasets for fine-tuning. Ultimately, the partnership demonstrates a path to self-improving LLM applications through iterative accuracy measurement and improvement.

Knowledge Graphs & GraphRAG: Techniques for Building Effective GenAI Applications: Zach Blumenthal

Nov 1, 2024 · 1:39:52

In this workshop, Zach Blumenthal of Neo4j demonstrates how to build a GraphRAG application using Neo4j, OpenAI, and LangChain on an H&M fashion dataset. He argues that combining knowledge graphs with vector search and graph embeddings improves retrieval and personalization for LLMs. The session covers creating a Neo4j sandbox, loading data, performing vector search with OpenAI embeddings, adding collaborative filtering via graph traversal patterns, using graph data science (FastRP) to generate graph embeddings for recommendations, and integrating everything into a LangChain chain that generates personalized marketing emails. Attendees build a Gradio app that, given a customer ID and season, outputs a tailored email with product recommendations, demonstrating how structured graph data enhances RAG systems beyond pure vector search.

Productionizing GenAI Models – Lessons from the world's best AI teams: Lukas Biewald

Oct 23, 2024 · 22:36

Lukas Biewald, founder of Weights and Biases, shares lessons from productionizing GenAI models, emphasizing that while AI is easy to demo, it is hard to productionize. He illustrates this with a personal project building a custom Alexa-like device using LLaMA and Whisper, where accuracy improved from 0% to 98% through prompt engineering, switching to Mistral, and fine-tuning with QLoRA. Biewald argues that tracking experiments (including failures) is critical for reproducibility and collaboration, and that a robust evaluation framework—beyond 'testing by vibes'—is essential for iterating and shipping v2. He notes that 70% of the audience had LLM apps in production, yet many lack solid evaluations, and recommends starting with lightweight prototypes and incorporating end-user feedback.

No more bad outputs with structured generation: Remi Louf

Oct 14, 2024 · 15:32

Rémi Louf, CEO of .txt and co-maintainer of Outlines, argues that structured generation—guiding LLMs to produce valid regex, JSON, or context-free grammar outputs—eliminates parsing errors and hallucinations while adding negligible overhead. Outlines masks tokens that violate the target structure, enabling Mistral 7B to achieve 99.9% valid JSON (vs. 17% without). It also accelerates inference: a chatty 50-token ChatGPT answer shrinks to 8 tokens, and structured generation boosts open models beyond GPT-4—Phi-3 Medium hits 96.5% on Berkeley function calling (GPT-4 gets 93.5%). With one shot matching eight shots in accuracy, Louf contends that most text is structured and that structured generation should be the default for non-chatbot workflows.

Architecting and Testing Controllable Agents: Lance Martin

Oct 11, 2024 · 2:21:54

Lance Martin presents LangGraph, a graph-based framework for building controllable agents that trade some open-ended flexibility for significantly higher reliability compared to classic React agents, achieving 100% consistent tool-calling trajectories even with an 8B local model. He demonstrates self-corrective RAG patterns like Corrective RAG, SelfRAG, and Adaptive RAG, where the agent grades retrieved documents, checks for hallucinations, and routes to web search when needed. Martin also covers three testing loops: in-app error handling with LangGraph, pre-production evaluation using LangSmith to compare agent answers and tool trajectories against ground-truth datasets, and production monitoring with online evaluators that flag retrieval quality, answer relevance, and hallucinations without reference answers. He shares results from a five-question evaluation showing LangGraph agents achieve 80% answer accuracy and 100% tool-trajectory correctness with Fire Function V2, while React agents with GPT-4o degrade in tool reliability. The talk addresses practical concerns like handling many tools (suggesting RAG for tool selection), multi-turn conversations, and the importance of…

Breaking AI's 1-GHz Barrier: Sunny Madra (Groq)

Oct 10, 2024 · 20:11

Sunny Madra (Groq) argues that LLM inference speed is undergoing a transformation akin to the 1-GHz microprocessor breakthrough, with Groq increasing LLaMA 3 8B speed by over 50% in just two months. He draws parallels to the industrial revolution: just as mass production replaced bespoke manufacturing, AI now enables 1,000 outputs in a minute where previously one designer produced one per day. Specific applications like Globe.Engineer plan a trip in seconds by processing 10,000 input tokens per second. Madra envisions LLMs as future OS cores, enabling instantaneous decision-making, universal natural-language interfaces, and multi-agent collaboration that allows smaller models to compete with larger ones. Edge AI projects like hyperspace.ai distribute unused GPU compute, and personalized education (citing Khan's two-sigma effect) becomes practical with cheap inference. He also highlights automated data science agents (Pioneer) that run endlessly, discovering correlations like productivity decline after certain performance reviews. The episode underscores that speed unlocks predictive analytics, context-aware systems, and enhanced security against AI-powered scams.

Making Open Models 10x faster and better for Modern Application Innovation: Dmytro (Dima) Dzhulgakov

Oct 9, 2024 · 18:55

Dmytro Dzhulgakov, CTO of Fireworks AI, argues that open-source models are the future for GenAI applications because they offer lower latency, lower cost, and domain adaptability compared to proprietary models. He explains that open models can be up to 10x faster for narrow domains and cut costs significantly, using examples like fine-tuned Llama 3 for function calling. Fireworks addresses the challenges of setup, optimization, and production readiness with a custom serving stack that delivers the fastest inference for long prompts and image generation (e.g., SDXL). He highlights FireFunction V2, an open-source model for function calling that combines chat and tool use, and notes that Fireworks serves over 150 billion tokens per day for companies like Quora and Cursor. The talk emphasizes that platforms like Fireworks enable developers to start with serverless inference, fine-tune models, and scale to enterprise-grade deployment with dedicated hardware.

Building and Scaling an AI Agent Swarm of low latency real time voice bots: Damien Murphy

Oct 8, 2024 · 1:07:23

Damien Murphy, Senior Applied Engineer at Deepgram, demonstrates building and scaling low-latency real-time voice bots using Deepgram's new voice agent API, which wraps speech-to-text, LLM, and text-to-speech into a single streaming endpoint. He shows a drive-thru ordering demo with function calling (add/remove items) using GPT-4o, achieving sub-second latency by co-locating components. Murphy explains scaling to millions of concurrent calls through regional Kubernetes clusters and multi-agent swarms (routing, booking, support agents) to reduce complexity and cost. He addresses endpointing challenges, VAD-based barge-in, and cost/quality trade-offs with hosted vs. self-hosted models, noting Deepgram offers 20x cheaper TTS than ElevenLabs and 50ms STT latency self-hosted. The talk emphasizes keeping agents simple, using smallest capable LLMs, and composability for reuse.

Build enterprise generative AI apps using Llama 3 at 1,000 tokens/s on the SambaNova AI platform

Sep 11, 2024 · 54:34

SambaNova’s Michelle Matern and Petro Milan present their full-stack AI platform, demonstrating how the SN40L RDU chip enables Llama 3 inference at 1,000 tokens per second. They introduce Samba-1, a composition of 92 expert models behind a single endpoint, and benchmark it against GPT-3.5 and GPT-4 on enterprise tasks like information extraction and text-to-SQL. The workshop then builds a RAG-based Q&A system using LangChain, Unstructured, E5-large-v2 embeddings, ChromaDB, and Llama-3-8B-Instruct at 1,000 tokens per second. Attendees set up the environment, load documents, and run inference with real-time metrics showing time to first token of 0.09 seconds and total inference time of 0.65 seconds.

Judging LLMs: Alex Volkov

Sep 9, 2024 · 18:39

Alex Volkov, as an LLM judge from 2034, humorously judges AI engineers on their development practices, emphasizing the importance of tracing, iterative prompt engineering, and robust evaluation pipelines. He finds Daniel guilty of deploying without logging, Sasha guilty of premature fine-tuning without prompt iteration, and Morgan guilty of ignoring AI news—commuted to attending ThursdAI. Francisco’s overreliance on programmatic evals leads to a legal loss, while Maxime is 'awesome' for using Weights & Biases Weave. Alex concludes with a primer on evaluation methods: programmatic, human-in-the-loop, and LLM-as-judge, stressing the need to validate validators and create custom criteria. He promotes Weave for tracing and evals, and ThursdAI for staying updated.

Pydantic is STILL all you need: Jason Liu

Sep 6, 2024 · 15:21

Jason Liu returns to the AI Engineer World's Fair to argue that Pydantic (and his library Instructor) is still all you need for structured output with LLMs. He shows that the core API—response_model, streaming with iterables, and partials for real-time validation—remains unchanged, now supporting Ollama, LlamaCPP, Anthropic, Gemini, and more. Liu demonstrates validators that enforce rules like uppercasing names or verifying receipt totals, reducing errors with automatic retries. He applies structured output to RAG: using a Search model with optional date ranges and source selection, and a Response model with follow-up questions and validated URLs. For extraction, he creates classifiers using Literal types, meeting summaries with action items, and even tables as Pandas DataFrames via custom type hints. Liu’s key takeaway is that one retry often suffices, and as models get faster and smarter, structured output makes LLMs compatible with classical programming—turning generative AI into generating data structures defined by the developer.

Git push get an AI API: Ryan Fox-Tyler

Aug 23, 2024 · 45:01

Ryan Fox-Tyler and Matt Johnson Pint from Hypermode present a hands-on workshop demonstrating how to build and iteratively improve AI features using their platform, focusing on a GitHub issue triage app. They first illustrate the process with a multiplayer game (Hypercategories) that uses AI for classification and scoring. Then they build a trend summary function using OpenAI GPT-4 to summarize repository issues, and a classify issue function with a Hugging Face DistilBERT model for labeling issues as bug, feature, or question. Finally, they add natural language search for similar issues by creating an embeddings model (MiniLM) and a Hypermode collection, enabling vector search without external databases. The workshop emphasizes incremental iteration, mixing AI models with traditional code, and using Hypermode's automatic GraphQL generation and observability to speed development.

10x Development: LLMs For the working Programmer - Manuel Odendahl

Aug 21, 2024 · 1:14:46

Manuel Odendahl presents a workshop on using LLMs as a translation engine to achieve 10x development productivity, arguing that treating models as cultural rather than purely technological artifacts enables novel techniques like world simulation for code reviews and domain-specific language creation. He demonstrates concrete methods: regenerating responses, editing model outputs, clearing context frequently, and summarizing transcripts into actionable formats. Odendahl introduces his tool Prompto for managing code fragments and emphasizes that fundamental programming knowledge remains essential while API-specific details become less critical. The episode focuses on decomposing problems into language translation steps, using examples such as transforming meeting transcripts into GitHub issues and building applications by instructing the LLM to simulate itself as the app. Odendahl advocates for divergent thinking and human-centered output, noting that all generated artifacts should ultimately serve human understanding.

Running AI Application in Minutes w/ AI Templates: Gabriela de Queiroz, Pamela Fox, Harald Kirschner

Aug 14, 2024 · 1:29:02

In this workshop, Microsoft's Gabriela de Queiroz, Pamela Fox, and Harald Kirschner show how to deploy AI applications in minutes using AI templates, Azure OpenAI proxy, and GitHub Codespaces. They walk through deploying a simple chat app, then two RAG applications: one that queries a Postgres database with SQL filtering, and another that performs RAG on unstructured documents using Azure AI Search. Key decisions include using async frameworks like Quart and FastAPI, token-based chunking, and hybrid retrieval with semantic reranking for best results. They stress the importance of running evaluations with hundreds of samples and share production insights from Copilot Chat, where TFIDF sparse indexing and LLM reranking are used. The session includes free Azure credits and a proxy to bypass Azure OpenAI approval, allowing attendees to deploy everything without spending their own money.

Low Level Technicals of LLMs: Daniel Han

Jul 31, 2024 · 2:52:26

Daniel Han of Unsloth explains how to find and fix bugs in open-source LLMs like Gemma, Phi-3, and Llama, covering tokenizer issues, architecture pitfalls, and finetuning optimizations. He details the eight Gemma bugs Unsloth fixed, including a critical RoPE downcasting error that broke positional encoding, and a 2048 sliding window bug in Phi-3. Han walks through transformer internals: attention masking, layer norms, RoPE embeddings, and SwiGLU activation, showing how to derive gradients for custom kernels. He demonstrates Unsloth's 2x faster finetuning with 70% less memory via Triton kernels, and introduces new features: automatic Ollama model file creation, CSV fine-tuning with merged columns, and chunked cross-entropy for large vocabularies. The session includes live Q&A on learning rate schedules, precision trade-offs, and mechanistic interpretability.

Copilots Everywhere: Thomas Dohmke and Eugene Yan

Jul 26, 2024 · 18:22

GitHub CEO Thomas Dohmke shares the Copilot origin story from 2020 lockdown demos of GPT-3's Codex, its early 72 NPS despite hallucinations, and how it now writes half the code in enabled files. He details Copilot's expansion beyond IDE auto-completion to Chat and the new Copilot Workspace, which takes a GitHub issue and generates a plan and code changes, keeping humans in the loop. Dohmke argues Workspace helps non-developers like product managers see implementation details and size tasks. He defines an agent as an 'AI dishwasher' that autonomously handles drudge work like security backlogs, and urges developers to embrace AI tools to bring fun back into software development.

Lessons From A Year Building With LLMs

Jul 19, 2024 · 35:21

The six authors of the O'Reilly article "Lessons From A Year Building With LLMs" — Bryan Bischof, Jason Liu, Hamel Husain, Eugene Yan, Shreya Shankar, and Swix — argue that the model itself is not a moat and that success comes from continuous improvement centered on evals and data. They stress that AI engineers should treat models as SaaS, quickly swapping for better ones, and focus on product and user interactions. The talk warns against toxic practices like prematurely hiring ML engineers without data or blindly adopting tools, advocating instead for deliberate eval practice and data literacy. On tactics, they compare LLM-as-judge (quick to prototype) vs fine-tuned evaluators (more precise and faster), and emphasize looking at real user data regularly with automated guardrails. Ultimately, they conclude that going from demo to production requires sustained investment in infrastructure and evaluation, echoing MLOps lessons from a decade ago.

Open Challenges for AI Engineering: Simon Willison

Jul 17, 2024 · 18:49

Simon Willison argues the GPT-4 barrier has been broken as GPT-4o, Claude 3.5 Sonnet, Gemini 1.5 Pro, and open models like LLaMA 3 70B now compete, making GPT-4-class models a commodity. He highlights the AI trust crisis with examples of Dropbox and Slack being falsely accused of training on user data, and notes Anthropic trained Claude 3.5 Sonnet without customer data. Willison warns about prompt injection vulnerabilities, citing the Markdown image exfiltration bug affecting six major chatbots, and defines slop as unreviewed AI-generated content, calling for accountability and responsible use patterns.

From Text to Vision to Voice Exploring Multimodality with Open AI: Romain Huet

Jul 10, 2024 · 23:39

OpenAI's Romain Huet demonstrates GPT-4o's real-time audio, vision, and screen-sharing capabilities at the AI Engineer World's Fair, showing how the omni-model achieves human-like conversational latency, understands emotion, and can be interrupted naturally. He walks through live demos of ChatGPT Desktop recognizing drawings, reading book pages, and pair-coding a responsive travel app using Tailwind CSS. Huet also previews Sora's text-to-video generation and OpenAI's unreleased Voice Engine, which can clone a voice from seconds of audio and translate it into multiple languages while preserving the original speaker's tone. He outlines OpenAI's four focus areas: advancing textual intelligence, faster/cheaper models (GPT-4o is twice as fast as GPT-4 Turbo at half the price), model customization via fine-tuning, and enabling multimodal agents. The talk emphasizes that today's models are the dumbest they'll ever be and urges developers to build for a future where AI works across text, voice, and video.

How to Become an AI Engineer from a Fullstack Background - Reid Mayo

Feb 2, 2024 · 10:19

Reid Mayo presents a step-by-step syllabus to transition from fullstack engineer to AI engineer, covering generative AI foundations, prompt engineering, LangChain, fine-tuning, and cost-effective open-source model deployment. The syllabus starts with Cohere's LLM overview, then dives into prompt engineering via Elvis Seravia's guide and Learn Prompting org docs. It emphasizes LangChain as the glue layer for modular AI systems, with tutorials from Mayo Ocean. Evals are treated as software tests using OpenAI's cookbook. Fine-tuning is taught via OpenAI's cookbook and then open-source LLaMA 2, with a specific case study showing a $19 fine-tuned LLaMA 2 matching OpenAI's $24,000 model on a target task. The boot camp ends with advanced deep learning courses from FastAI and Hugging Face for further mastery.

Using AI to Build an Infinite Game: Jeff Schomay

Feb 1, 2024 · 11:08

Jeff Schomay built a game with 100% AI-generated content, using OpenAI's fine-tuning to generate JSON scene definitions and Leonardo to create stylistically consistent images. He fine-tuned a model on 50 examples for about $1–$2, enabling shorter prompts and reliable JSON output. To maintain visual consistency across scenes with varied elements (people, animals, buildings), he trained multiple image models. He implemented a caching pipeline that prefetches scenes to compensate for 10–30 second generation times, ensuring a smooth player experience. The final game procedurally generates every scene and image, so each playthrough is unique.

Open Questions for AI Engineering: Simon Willison

Nov 25, 2023 · 24:33

Simon Willison recaps the AI industry's past year—from ChatGPT's breakthrough to open-source local models—and poses key open questions for AI engineering. He argues that ChatGPT's chat interface, while popular, is a poor fit for advanced use, urging better UIs like his command-line tool LLM. He celebrates Meta's Llama release as a 'stable diffusion moment' for language models and highlights the rise of small, locally-run models such as Replit's 3B model, asking how small models can remain useful. On security, he warns that prompt injection remains unsolved after 13 months, limiting what can safely be built. He champions ChatGPT's Code Interpreter (which he dubs 'Coding Intern') as the most exciting tool, able to write and compile C code on a phone, and argues that LLMs flatten the learning curve, making programming accessible to more people. He concludes by urging the community to build tools that enable anyone to automate tedious tasks.

Harnessing the Power of LLMs Locally: Mithun Hunsur

Nov 22, 2023 · 17:09

Mithun Hunsur presents llm.rs, a Rust library for running large language models locally, arguing it gives developers ownership, lower latency, and privacy compared to cloud APIs. He explains how quantization makes inference viable on consumer hardware, and shows that llm.rs supports architectures like LLaMA and Falcon through a unified interface. Practical code examples demonstrate customization, and community projects like LocalAI and LLMchain illustrate real-world use. Hunsur shares his own date-extraction pipeline, fine-tuning a small model with GPT-3 data to replace expensive cloud calls. He also cautions about hardware requirements, trade-offs between speed and quality, and ecosystem churn from rapid innovation.

The Weekend AI Engineer: Hassan El Mghari

Nov 22, 2023 · 21:49

Hassan El Mghari shares how he built viral AI apps like RoomGPT and AI Commit, arguing that simple off-the-shelf APIs and focused weekend sprints can attract millions. He describes building 11 side projects in a year, which grew from 20,000 visitors to over 8.5 million unique visitors and 2.8 million sign-ups. Key projects include RoomGPT, which used ControlNet for room redesign and reached 6 million visitors, and AI Commit, an open-source CLI tool adopted by 30,000 developers. He emphasizes using tools like the Vercel AI SDK and v0.dev to accelerate development, making apps free and open source to drive growth, spending 80% of time on UI, and launching quickly with minimal fine-tuning.

120k players in a week: Lessons from the first viral CLIP app: Joseph Nelson

Nov 22, 2023 · 15:59

Joseph Nelson (Roboflow CEO) details building paint.wtf, a viral AI Pictionary game using OpenAI CLIP that attracted 120,000 players in its first week, peaking at 7 submissions per second. The architecture uses GPT-3 to generate prompts, a browser canvas for drawing, and CLIP to judge similarity between text and image embeddings. Key lessons include CLIP's ability to read text, requiring a moderation hack that penalizes images more similar to handwriting than the prompt; CLIP's conservative similarity scores (ranging 8% to 48%); and the necessity of using CLIP itself to block NSFW content. Nelson also live-codes a minimal MVP in under 50 lines of Python using Roboflow's open-source inference server, demonstrating CLIP embedding and cosine similarity. The talk underscores foundation models' new paradigm of open-set understanding.

Domain adaptation and fine-tuning for domain-specific LLMs: Abi Aryan

Nov 14, 2023 · 25:09

Abi Aryan's talk covers domain adaptation and fine-tuning for large language models, contrasting prompting, RAGs, and three fine-tuning methods: adaptive, behavioral, and parameter-efficient. Adaptive fine-tuning adds small adapter modules (0.15% of parameters) for new domains like biochemical engineering; behavioral fine-tuning optimizes label space for a single task; and parameter-efficient methods like LoRA and QLoRA reduce model size via low-rank adaptation and four-bit precision, ideal for low-resource devices. Aryan emphasizes data quality—deduplication reduces memorization—and practical tips: batch size of 32 or 64, starting with 100 epochs, using Adam optimizer, gradient checkpointing for memory savings, and in-context learning with dynamic examples to handle drift. Evaluation should combine metric-based (Bleu, Rouge), tool-based (Weights & Biases), model-based, and human-in-the-loop approaches, though full pipeline considerations (data collection, base model choice, storage) are critical for robust applications.

Pragmatic AI with TypeChat: Daniel Rosenwasser

Nov 14, 2023 · 18:34

Daniel Rosenwasser, TypeScript program manager, introduces TypeChat, an experimental library that uses TypeScript type definitions to guide and validate unstructured LLM output into structured JSON for traditional apps. He demonstrates a coffee shop ordering system where types define the schema, showing how the library can handle ambiguous inputs like 'a purple gorilla' by including unknown text for recovery. TypeChat also generates programs as JSON using a fake language to safely script multi-step operations, avoiding sandboxing issues from real code. A Python prototype extends the same approach, with examples like CSV data manipulation via a class-based API. The library aims to make AI tools accessible to all engineers by leveraging types they already use.

Building Reactive AI Apps: Matt Welsh

Nov 9, 2023 · 17:02

Matt Welsh announces AI.JSX, an open-source TypeScript framework he describes as 'React for LLMs,' which lets developers build reactive AI applications by composing JSX components that render to an LLM rather than the DOM. He argues AI.JSX makes LLM app development accessible to JavaScript developers, not just Python back-end engineers, by handling RAG, tool invocation, and real-time voice interaction out of the box. Welsh demonstrates parallel streaming of multiple LLM calls for story generation, a kid-safe wrapper component that rewrites content, and a 10-line RAG implementation. He showcases a live voice demo where an AI takes a donut order with sub-second latency, powered by Fixie's cloud platform for hosting, managed RAG pipelines, and conversational state. He emphasizes the framework's full React integration and ability to generate UI from AI, positioning AI.JSX as a simplification for building sophisticated LLM-powered apps.

AI Engineering 201: The Rest of the Owl

Nov 8, 2023 · 56:57

Charles Frye, instructor of the Full Stack LLM Bootcamp, presents essential patterns for building language user interfaces (LUIs), arguing that while RAG chatbots are the 'to-do list app' of AI engineering, structured outputs via function calling (e.g., OpenAI's JSON schema, Instructor library) improve robustness, and agents with memory (like generative agents or Voyager in Minecraft) represent the true AI frontier. He emphasizes the need for hybrid search combining vector and keyword retrieval (citing Vespa, Postgres, Redis), and warns that monitoring and evaluation are the hardest engineering challenges: monitoring user behavior, latency quantiles (especially 99th percentile), and costs must be paired with observability tools like Honeycomb or Gantry, while evaluation often requires iterated decomposition or using LLMs as evaluators (GPT-4 as 90th percentile crowd worker). The episode concludes that shipping to learn — starting with production data to generate tests — is the dominant engineering mindset, and that the field is still filling in the gaps between inference and full product value.

The AI Evolution: Mario Rodriguez, GitHub

Nov 7, 2023 · 19:32

Mario Rodriguez, VP of Product at GitHub, recounts the history and future of GitHub Copilot, the first at-scale AI programmer. Copilot now serves over 20,000 organizations and 1 million-plus developers, generating over $100M in ARR, with 46% of code written via completions. He shares insider lessons: ghost text, low latency under 100 milliseconds, and prompt engineering were key to success. He warns that 'syntax is not software' and that global presence and offline/online scorecards are essential at scale. Looking ahead, Rodriguez envisions moving from procedures to goals and constraints, enabling AI to reason on code, and designing immersive UIs for human-AI collaboration. He concludes that GitHub has evolved from a version control system into an end-to-end platform infused with AI.

The AI Pivot: With Chris White of Prefect & Bryan Bischof of Hex

Nov 7, 2023 · 35:16

Chris White (CTO of Prefect) and Bryan Bischof (Head of AI at Hex) detail how their non-AI startups successfully pivoted to integrate AI, arguing that ruthless prioritization and deep product integration are key. White explains Prefect built the open-source Marvin project to learn from LLM experimenters, while adding AI features like error summaries to its core orchestration product, but had to restrain over-enthusiasm from engineers. Bischof describes Hex's Magic feature as an augmentation, not a separate product, and explains they built their own evaluation system but chose not to build a vector database, and killed a promising feature called Crystal Ball to avoid splitting the product experience. They both emphasize machine-to-machine interfaces and typed outputs, and caution that the work is often tedious data engineering. Their hot takes: White says stop building chat interfaces, AI is a tool; Bischof warns that the journey is boring but worth it.

[Workshop] AI Engineering 201: Inference

Nov 7, 2023 · 1:43:16

Charles Frye, instructor of the Full Stack LLM Bootcamp, leads a workshop on AI engineering inference, focusing on the build-versus-buy decision between proprietary and open models. He argues that proprietary models like OpenAI's GPT-4 and Anthropic's Claude are currently more capable but expensive, while open models like LLaMA 2 are less capable but offer hackability, though they may catch up if capabilities requirements saturate. Frye covers inference on end-user devices, noting that running models locally avoids network latency but faces tight memory and power constraints—e.g., a 7B parameter model requires 14 GB, too large for phone RAM. He explains inference-as-a-service (e.g., OpenAI, Replicate) versus self-serving on cloud GPUs or serverless platforms like Modal, highlighting that memory bandwidth is the bottleneck: GPUs have 1.5 TB/s memory bandwidth vs 312 TFLOPS compute, so batching is crucial for throughput. Frye also discusses inference arithmetic, custom silicon like TPUs which offer ~30% better efficiency but not drastic gains, and containerization challenges for GPU workloads.

The Hidden Life of Embeddings: Linus Lee

Nov 7, 2023 · 18:15

Linus Lee, a Research Engineer at Notion, presents a tour of embedding visualization and manipulation at the AI Engineer Summit 2023. He demonstrates an encoder-decoder model fine-tuned from T5 that can reconstruct text from embeddings, allowing direct manipulation of features like length and sentiment by moving in latent space. Lee shows that mixing embeddings by splicing dimensions from two texts produces a semantic blend, and a linear adapter can decode text from OpenAI's text-embedding-ada-002 embedding space. Using CLIP, he interpolates between photographic and cartoon images and performs vector arithmetic to modify facial expressions. Lee releases these custom text embedding models on Hugging Face, enabling others to explore and interact with latent spaces. He argues that making model internals visible and manipulable fosters deeper understanding and more humane interfaces to generative AI.

[Workshop] AI Engineering 101

Nov 6, 2023 · 3:02:24

In this hands-on workshop, AI engineer Noah Hein teaches the basics of AI engineering by building five projects with OpenAI's GPT-3, GPT-4, Dall-E, Whisper, and Telegram. Participants create a Telegram bot that uses GPT-3 for chat, implements retrieval-augmented generation (RAG) with embeddings and cosine similarity on MDN docs, generates code with GPT-4 using few-shot prompting, creates images via Dall-E 2, and transcribes voice with Whisper. The session explains core concepts like tokens, chunking, context windows, temperature, and top-p, emphasizing that embeddings and prompt engineering are key to performance. Hein demonstrates that these tools are cheap and accessible—the entire workshop costs about $0.05 in API fees—and shows how AI engineers can integrate multiple models into a single application.

Building Blocks for LLM Systems & Products: Eugene Yan

Nov 2, 2023 · 17:24

Eugene Yan presents practical patterns for building LLM systems and products, emphasizing evals as the foundation. He argues that automated evals—starting with as few as 40 domain-specific questions—enable faster iteration and safer deployment, while academic benchmarks like MMLU may not fit real tasks. For retrieval-augmented generation, he warns that LLMs perform worse when the answer is in the middle of retrieved documents, and even with perfect retrieval, accuracy tops out at 75%. Guardrails against hallucination can be built via natural language inference at the sentence level or by sampling multiple summaries to check consistency. Finally, he highlights UX design—like Copilot’s accept/reject or Midjourney’s upscale/vary—as a way to collect implicit feedback that builds a data flywheel for evals and fine-tuning.

Pydantic is all you need: Jason Liu

Nov 1, 2023 · 17:55

Jason Liu argues that Pydantic is the key to building reliable LLM applications by using structured prompting with OpenAI function calling. He introduces Instructor, a library that patches OpenAI's API to return Pydantic objects instead of raw JSON, ensuring type safety and validation. Liu demonstrates how Pydantic's 70 million downloads make it a trusted tool for defining data models with type hints, field validators, and even LLM-powered validators that catch errors like "don't say mean things" and retry via max retries. He shows concrete examples: decomposing user queries into structured search objects for RAG, generating a query plan DAG with parallel dependencies, extracting knowledge graphs for visualization, and verifying facts by requiring substring quotes from source text. Liu emphasizes that structuring prompts as code moves development from string manipulation to domain modeling, enabling cleaner, more maintainable systems that integrate easily with existing software.

See, Hear, Speak, Draw: Logan Kilpatrick & Simón Fishman

Oct 24, 2023 · 18:43

OpenAI's Simón Fishman and Logan Kilpatrick argue that 2024 will be the year of multimodal models, showcasing how text can currently bridge vision, audio, and image generation. They demo a DALL-E vision loop where GPT-4 with vision describes an image, DALL-E 3 generates a synthetic version, then the models compare and iterate to improve accuracy—all in roughly 50 lines of code. Another demo extracts frames from a video, uses GPT-4 with vision to describe them, combines with Whisper's transcript, and generates a blog post with matching DALL-E images, making video content more accessible. They emphasize that while today's multimodal capabilities are separate, future unified models will directly reason across modalities, eliminating the need for manual integration.

Building AI For All: Amjad Masad & Michele Catasta

Oct 23, 2023 · 25:13

Amjad Masad and Michele Catasta announce Replit's AI for all, giving free AI-powered coding to millions of users. They detail the training of Replit Code v1.5, a 3.3B parameter model trained on 1 trillion tokens of code, which outperforms StarCoder 3B and approaches Code LLaMA 7B in Human Eval despite being half the size. The model is optimized for low-latency inference, generating over 200 tokens per second per GPU. Catasta explains the data pipeline using permissively licensed code from The Stack and Replit public repos, with five epochs of repeated high-quality data. The model is released open-source with a commercially permissive license. Partnerships with Glaive AI for instruction fine-tuning, Morph Labs for a novel fill-in-syntax-tree format, and Perplexity for fast model serving are also unveiled.

The 1,000x AI Engineer: Swyx

Oct 23, 2023 · 9:27

Swyx argues that AI engineers are just in time for a 1,000x opportunity, drawing on historical tech cycles and compute scaling laws. He cites Carlota Perez's tech revolution cycles, placing the AI revolution's start at AlexNet in 2012. With compute growing 600x by decade's end, GPT-3 took one person-year of compute, GPT-4 took 100, and GPT-10 will exceed all human compute ever. He defines three AI engineer types: AI Enhanced (Copilot-like), AI Product (Midjourney), and AI Agent (Auto-GPT). To achieve 1,000x, he advises scaling knowledge from O(n) (attending talks) to O(n²) (teaching others) to O(2^n) (building networks).

Announcing the AI Engineer Network: Benjamin Dunphy

Oct 23, 2023 · 7:44

Benjamin Dunphy launches the AI Engineer Summit 2023 by welcoming attendees and recognizing sponsors Auto-GPT, Supabase, Fixie, Microsoft, and Cloudflare, as well as VC Alessio Finelli. He then unveils two major updates for the conference's mobile app, Network: AI-Enhanced Matching, a generative algorithm using LLMs to connect attendees based on stated problems, and the open-sourcing of both the event app and matching algorithm on GitHub, enabling any event to use them. Dunphy credits lead engineer Simon Sturmer for the mobile app, 'Sweezett's teller' for the matching algorithm, and infrastructure partners Dscope (auth) and Supabase (database and PG vector). The announcements aim to improve high-signal networking among the 500 selected engineers, founders, and sponsors at the summit.

Principles for Prompt Engineering - Karina Nguyen (Claude Instant @ Anthropic)

Oct 20, 2023 · 55:31

Karina Nguyen, an engineer on Anthropic's Claude, presents principles for effective prompt engineering, treating it as a creative writing process requiring iteration. She advises using XML tags, placing instructions at the end of long prompts (improving accuracy), and decomposing questions to improve faithfulness over chain-of-thought. She covers reducing hallucinations by asking Claude to hedge or quote sources, and using self-consistency and contrastive examples for labeling. Nguyen explains generating evaluation datasets with Claude by splitting documents into multiple-choice questions, and notes that for long-document QA, asking at the end outperforms the beginning. She distinguishes Claude (larger, smarter) from Claude Instant (cheaper, faster, better at math and code), and predicts prompt engineering will remain essential for complex tasks.