All episodes

MCPs are Boring (or: Why we are losing the Sparkle of LLMs) - Manuel Odendahl

Jun 10, 2025 · 28:32

Manuel Odendahl argues that MCPs (Model Context Protocols) are boring because they constrain LLMs to rigid, predefined tool calls, while LLMs truly shine as code generators that can create dynamic tools on the fly. He critiques the inefficiency of tool calling—repeating context, wasting tokens, and struggling with many tools—and advocates for an 'Eval' approach where the LLM writes and executes code (e.g., SQL, JavaScript) to solve tasks directly. Demonstrating with a JavaScript sandbox, he shows how a single Eval tool can introspect databases, write queries, create REST APIs, and even build a full CRM interface in one call, saving time and tokens. Odendahl urges engineers to think recursively: ask the LLM to write code that writes more code, unlocking infinite loops of creation and restoring the sparkle of LLMs beyond boring function calls.

Break It 'Til You Make It: Building the Self-Improving Stack for AI Agents - Aparna Dhinakaran

Jun 10, 2025 · 14:25

Aparna Dhinakaran, co-founder of Arise, details how to build a self-improving evaluation stack for AI agents using her company's open-source tools Arise Phoenix and RiseX. She argues that production agents require three evaluation layers: tool-call correctness (right function and arguments), trajectory accuracy (correct order of steps), and multi-turn session consistency (context retention). Dhinakaran demonstrates with real traces from Arise's own Copilot, showing how a bottleneck in search Q&A correctness (only 50% accuracy) was traced to incorrect argument passing in a tool call. She stresses a dual iteration loop: improving agent prompts and simultaneously refining eval prompts (LLM-as-judge) to avoid static evaluation criteria. The talk concludes that continuous eval improvement is essential for agents to move from working once to working reliably in production.

The 4 Patterns of AI Native Development — Patrick Debois

Jun 4, 2025 · 14:11

Patrick Debois presents four patterns of AI-native development that redefine software engineering roles: from producer to manager (reviewing AI-generated code, managing agents), from implementation to intent (specifying requirements in markdown files), from delivery to discovery (rapid prototyping with tools like Lovable and Bolt), and from content creation to knowledge (capturing learnings for reuse). He argues these shifts mirror senior developer activities and are supported by new tools, such as the AI Native Dev landscape featuring roughly 300 tools. The talk emphasizes that AI changes not just coding speed but the entire workflow, reducing cognitive load through smarter reviews and intent-based coding.

Breaking the Chain: Agent Continuations for Resumable AI Workflows - Greg Benson

Jun 3, 2025 · 27:02

Greg Benson from SnapLogic introduces Agent Continuations, a mechanism to capture and resume the full state of AI agents, enabling human-in-the-loop approval and reliable checkpointing for long-running agent workflows. The solution borrows from programming language continuations: at any suspension point (e.g., a tool needing human approval), the agent’s state is bundled into a JSON continuation object containing the messages array, resume request, and approval metadata. This object can be saved or passed to an application layer, allowing agent loops to be fully shut down and later restored without losing progress. Benson demonstrates with a multi-level HR agent that suspends for human approval of an account authorization tool, then resumes seamlessly. The approach handles arbitrary nesting of subagents and is implemented as an open-source Python prototype that works with any OpenAI-compatible API. Future work includes general suspension triggers and integration with existing frameworks like LangChain or Pydantic AI.

Are MCPs Overhyped? A Rant about MCPs — Henry Mao, Smithery

Jun 3, 2025 · 7:29

Henry Mao, founder of Smithery and MCP steering committee member, argues that despite MCPs standardizing AI agent-service connections, the ecosystem faces fragmentation, high-friction installation, security, and monetization problems. Users struggle with unreliable MCP servers and complex five-step installs, while developers face hosting challenges, lacking tooling, distribution hurdles, and unclear monetization. Smithery aims to solve these as an AI gateway, demoing an agent that finds GitHub issues and creates Linear tickets using curated MCPs. Mao envisions a future dominated by tool calls where agent experience trumps user experience.

Why the Best AI Agents Are Built Without Frameworks (Primitives over Frameworks) — Ahmad Awais, CHAI

Jun 3, 2025 · 27:06

Ahmad Awais, founder of LangBase/CHAI, argues that production AI agents like Perplexity, Cursor, v0, Lovable, Bolt, and CHAI itself should be built using AI primitives rather than frameworks. He demonstrates live coding a 'Chat with PDF' agent using LangBase primitives (memory, parser, chunker, threads). Awais presents eight agent architectures—augmented LLM, prompt chaining, agent router, parallel agents, orchestrator-worker, evaluator-optimizer, tool-calling, and memory-based—all built with plain code and no framework. He claims frameworks are bloated and slow, while primitives offer flexibility and automatic scaling (e.g., memory storing terabytes). Awais shows how to compose multiple LLMs (Gemini, DeepSeek Llama, Claude) in a router agent, and notes that his team has built agents for OCR, image analysis, and deep research using primitives and external tools like Mistral OCR.

GPU-less, Trust-less, Limit-less: Reimagining the Confidential AI Cloud - Mike Bursell

Jun 3, 2025 · 43:41

Mike Bursell of Super Protocol argues that Confidential AI, built on hardware Trusted Execution Environments (TEEs) like Intel TDX, AMD SEV-SNP, and NVIDIA GPU TEEs, solves the trust problem in AI by enabling secure processing of sensitive data and proprietary models without exposure. Super Protocol, a decentralized confidential AI cloud and marketplace, allows users to deploy models in TEEs, verify execution via cryptographic attestation, and collaborate across organizations without blind trust. Demos show deploying DeepSeek on H100 GPUs, running n8n healthcare workflows, distributing vLLM inference across four GPU nodes, and provably training a medical model on datasets from Alice's lab and Bob's clinic. Case studies include Realize achieving 75% accuracy and 3-5% sales increase for Mars, and BEL reducing FDA audit time from weeks to 1-2 hours. The protocol replaces trust with on-chain proofs, enabling GPU-less, trustless, limitless AI.

Grounded Reasoning Systems for Cloud Architecture - Iman Makaremi

Jun 3, 2025 · 26:05

Iman Makaremi, co-founding head of AI at Catayaw, details how his team built a grounded reasoning system for cloud architecture using a knowledge graph and multi-agent orchestration. The system uses LangGraph for workflows, Flyte for distributed execution, and AWS Bedrock with Claude 3. A domain-specific knowledge graph serves as long-term memory, enabling persistent representation of architectural state. The multi-agent system comprises a chief architect and ten staff architects specialized in domains like infrastructure and API, plus retrievers for requirements and architecture. Key findings include that semantic grounding improves but doesn't always work, graph memory supports continuity, structured outputs improve control, and human evaluation is essential early on—Catayaw built an internal tool called Eagle Eye. The workflow proceeds through list generation, conflict resolution, and design proposal with agent cloning for parallel tasks.

The Agent Native Company — Rick Blalock, Agentuity

Jun 3, 2025 · 20:58

Rick Blalock of Agentuity argues that an agent-native company, built from the ground up with AI agents at the core of product, operations, and culture, is fundamentally different from an AI-enhanced one that merely uses AI as a tool. He contrasts the two: removing agents from an agent-native company would halt productivity, while an AI-enhanced business would just become less efficient. Blalock describes the agent-native workday, where humans oversee agents that handle routine tasks, and notes the rise of roles like 'Agent Manager' and the importance of AI fluency in hiring. He shares how his 7-person team built an entire agentic cloud infrastructure in weeks using agents like Devin, arguing that this paradigm shift requires founders to rethink org charts, roles, and skills. The episode concludes that businesses must decide whether they are just using AI or ready to be built around it.

Building Reliable Support Agents Using the Effect Typescript Library - Michael Fester

Jun 3, 2025 · 7:22

Michael Fester, co-founder and CTO of Fourteen.ai, explains how his team built production-ready AI support agents using the Effect TypeScript library, arguing it provides strong type safety, composability, and reliability for systems that rely on LLMs. The architecture uses Effect across the entire stack, including Effect RPC, Effect HTTP, Effect SQL, and a custom DSL for agent workflows. Reliability features include fallback between LLM providers (e.g., GPT-4.0 mini to Gemini Flash 2.0), retry policies with state tracking, and duplicated token streams for analytics. Dependency injection allows easy mocking of LLM providers for testing. However, Fester warns of pitfalls like silently losing errors from upstream catches and the steep learning curve. He recommends incremental adoption, starting with a single service.

7 Habits of Highly Effective Generative AI Evaluations - Justin Muller

Jun 3, 2025 · 25:39

Principal Applied AI Architect Justin Muller argues that generative AI evaluations are the missing piece to scaling, offering seven habits from over 100 projects. He recounts a customer whose document processing workload had 22% accuracy with no evals; after building an evaluation framework, accuracy reached 92% and the system became the largest such workload on AWS in North America. The habits include fast 30-second eval cycles using AI-as-judge, quantifiable scores averaged across numerous test cases, explainable reasoning for both generation and scoring, segmented evaluation through prompt decomposition, diverse test sets covering all use cases, and traditional techniques for numeric outputs or cost/latency. He emphasizes that evaluations should primarily discover errors, not just measure quality, and that prompt decomposition into chained steps often boosts accuracy by removing dead space.

Blender MCP and The Future Of Creative Tools - Siddharth Ahuja

Jun 3, 2025 · 16:50

Siddharth Ahuja introduces Blender MCP, an open-source project that lets LLMs like Claude control Blender via the Model Context Protocol, dramatically lowering the barrier to 3D creation. The tool, which has garnered 11,500 GitHub stars and over 160,000 downloads, enables users to generate complex scenes—like a dragon guarding a pot of gold or a full terrain with nodes—in minutes through natural language prompts. Ahuja explains that Blender’s scripting capabilities are key, and notes that keeping MCP tools lean avoids confusion for the LLM. He demonstrates integrations with AI asset generators and even built an Ableton MCP to create soundtracks, arguing that MCPs will become the glue for creative toolchains, allowing LLMs to orchestrate Blender, Unity, and Ableton together. This shifts creators from tool experts to orchestra conductors, unlocking a new wave of accessible content creation.

The Knowledge Graph Mullet: Trimming GraphRAG Complexity - William Lyon

Jun 3, 2025 · 32:52

William Lyon presents the 'Knowledge Graph Mullet' concept, combining property graph interfaces with RDF triple storage in Dgraph to optimize GraphRAG systems and AI agent development. Dgraph uses a property graph model for data modeling and DQL (inspired by GraphQL) for queries, while leveraging RDF triples for scalable storage and data interchange. Features include vector similarity search (e.g., finding articles about 'money laundering'), geospatial indexes, and a Model Context Protocol (MCP) server that exposes tools to models like Claude Desktop for auto-generating schema and querying. Lyon demonstrates building a news knowledge graph from the New York Times, chunking articles into nodes, and using graph traversal to surface related topics after vector search. He also showcases Hypermode agents, created from a prompt with access to GitHub and Notion via MCP servers, which can analyze repos like hyper-news and generate social media posts with code snippets, then save them to a Notion workspace. Hypermode agents are now in early access.

The Coherence Trap: Why LLMs Feel Smart (But Aren’t Thinking) - Travis Frisinger

Jun 3, 2025 · 20:47

Travis Frisinger argues that large language models are not intelligent but coherent, introducing 'coherence reconstruction' as a mental model for understanding their utility. He explains that hallucinations are a feature, not a bug, as models fill gaps predictably to maintain pattern completion. Prompts act as force vectors navigating latent space, and the key to reliable outputs is a 'frame, generate, judge, iterate' loop. Frisinger presents a three-layer model: latent space, execution layer (tools and RAG), and conversational interface. He advises engineers to design for emergence rather than control, using dense context as anchors to steer generation, and to watch for breakdowns in tone as early signs of lost coherence.

Agents reported thousands of bugs, how many were real? - Ian Butler and Nick Gregory

Jun 3, 2025 · 18:39

Ian Butler and Nick Gregory introduce SM-100, a benchmark of 100 real-world bugs across 84 repositories, and present their agent Bismuth as outperforming existing agents in detecting complex bugs—finding 10 needles vs 7 for the next best. However, even top performers struggle: Bismuth's true positive rate is only 25%, and Codex leads at 45%, while most agents (Cursor, Debian) report thousands of false positives (97% for basic loops). They argue that agents universally exhibit narrow thinking, missing simple bugs like a form state issue that only Bismuth and Codex caught, and that high SWE-bench scores do not translate to maintenance tasks. The episode concludes that software maintenance remains an unsolved problem requiring deeper reasoning and targeted search.

The Voice-First AI Overlay: Designing Conversational Co-Pilots - Gregory Bruss

Jun 3, 2025 · 16:44

Gregory Bruss introduces the Voice-First AI Overlay, an agent that assists live human conversations without becoming a third speaker. He argues that as AI grows more powerful, this paradigm can keep humans in the loop through voice, the most natural interface. The overlay passively listens and surfaces relevant help—like language suggestions in a demo—addressing timing, relevance, attention, and latency. Engineering challenges include jitterbug input, context repair, premature interrupts, and glanceable ghost. Bruss emphasizes design principles: transparency, minimal cognitive load, and progressive autonomy. He also explores future directions like full-duplex speech models and on-device inference for privacy.

Arrakis: How To Build An AI Sandbox From Scratch - Abhishek Bhardwaj, OpenAI

Jun 3, 2025 · 40:18

Abhishek Bhardwaj, founder of Arrakis, explains why MicroVM-based AI sandboxes are the next unlock for AI agents, detailing how Arrakis provides secure, fast code execution and computer-use environments. He argues that sandboxes are essential for tool-calling models, reinforcement learning, and multi-tenant security, with Arrakis booting in under 7 seconds and supporting backtracking via snapshot/restore. The talk covers the choice of Cloud Hypervisor over other VMMs, overlayFS storage with per-sandbox read-write layers, iptables-based networking, and a built-in code execution server. A demo shows Claude using Arrakis via MCP to build a collaborative Google Docs clone and then revert to a snapshot, illustrating how a full Linux sandbox enables agents to autonomously debug and iterate without extensive prompting.

The Robots are coming for your job, and that's okay - Elmer Thomas and Maria Bermudez

Jun 3, 2025 · 8:57

Elmer Thomas and Maria Bermudez of Twilio's developer education team explain how they built six single-purpose AI agents to automate repetitive documentation tasks like editing, alt text generation, and jargon simplification, rather than replacing human writers. The agents, powered by GPT-4/01 and validated with tools like Veil Lint and CI tests, reduce hallucinations through layered human review. Maria demos the automated editor, SEO metadata generator, alt text generator, and jargon simplifier. The team’s playbook: identify a pain point, pick a repeatable task, and iterate with weekly user feedback.

Agentic Enterprise - What your CEO must know about AI - Hubert Misztela

Jun 3, 2025 · 28:04

Hubert Misztela, an AI research lead at Novartis, argues that organizations may be run by AI agents within three years and must pivot from traditional roles to persona-based workflows to harness agentic automation. He explains that AI agents glue multiple cognitive steps, automating workflows where only humans could operate. Understanding deep context around each workflow is crucial, as this knowledge is often undocumented. Misztela introduces five employee personas (silent achiever, individual contributor, connector, multiplier, knowledge hub) to project how agents merge tasks, magnify impact, and replace roles. He warns that intelligence and domain knowledge become cheap commodities, so companies need multidisciplinary or deeper specialization, and new roles like workflow miner will emerge. Employees must build their own agents using no-code tools, and ethical questions around value alignment remain.

How agents broke app-level infrastructure - Evan Boyle

Jun 3, 2025 · 13:32

Evan Boyle, founder of GenSX, argues that LLM-powered agents have broken assumptions about app-level infrastructure, as p1 latency jumps from milliseconds to seconds and workflows run minutes to hours. He explains that traditional serverless providers timeout after 5 minutes, lack native streaming, and force brittle Rube Goldberg machines on top of queues like SQS or tools like Airflow. Boyle presents an open-source library that separates API and compute layers using Redis streams for resumability, enabling users to refresh pages or navigate away without losing progress. He demonstrates components with built-in retries, caching, and tracing, and a workflow that automates Hacker News analysis. Key architectural lessons include starting simple but planning for long-running agents, keeping compute and API planes separate, and leaning on Redis for resumability.

The End of Awkward AI Transcriptions - Travis Bartley and Myungjong Kim

Jun 3, 2025 · 16:24

NVIDIA speech AI researchers Travis Bartley, Myungjong Kim, and Jae-hun detail how their Fast Conformer backbone powers both streaming (Parakeet) and multitask (Canary) models, achieving top-5 Hugging Face ASR leaderboard rankings by prioritizing customization over one-size-fits-all. They explain CTC/TDT for low-latency streaming and attention encoder-decoder for high-accuracy multitask models, with Subformer enabling unified speaker diarization and target-speaker ASR. Training uses open-source and proprietary data with pseudo-labeling via the Nemo toolkit. Deployment on NVIDIA Riva NIM with TensorRT optimization supports low-latency streaming and offline processing. Customization includes fine-tuning acoustic models, language models, punctuation, and inverse text normalization for domain-specific terms like medical and food ordering.

MCP Agent Fine tuning Workshop - Ronan McGovern

Jun 3, 2025 · 35:30

Ronan McGovern demonstrates how to fine-tune a Qwen3 model on high-quality reasoning traces from an MCP agent with browser tools. Using Playwright's 25 tools via Model Context Protocol, he generates multi-turn traces by running a 30B Qwen agent on RunPod, saving both tool calls and reasoning content. Those traces are then used to supervised fine-tune a 4B Qwen model with Unsloth, applying LoRA adapters to attention and MLP layers. The process includes converting MCP tool schemas to OpenAI format, extracting Hermes-style tool calls, and unrolling conversations to multiply training examples. Even with only nine curated traces, the fine-tuned model shows improved tool-calling behavior on multi-step tasks like navigating trellis.com to extract specific content.

The Demo I Wish I'd Had: OpenAI's Agents SDK... serverless! - Brook Riggio

Jun 3, 2025 · 17:51

Brook Riggio demonstrates deploying AI agent workflows on Vercel's serverless platform using OpenAI's Agents SDK and Inngest orchestration. He presents a reference architecture: Next.js frontend triggers Inngest events that invoke Python agents on Vercel serverless functions, with automatic retries and step-level observability. His demo builds a newsletter generator: users enter topics, a research agent scans news, a formatting agent structures the content, and results save to Vercel blob storage while the frontend polls for completion. He highlights that Vercel automatically detects Python functions without a config file, and Inngest's step.run ensures reliable execution within serverless time limits. The talk emphasizes cost efficiency (pay only for actual execution), full type safety via Pydantic and TypeScript, and easy integration of long-running agent tasks. The open-source repo is available for fork and contributions.

RAG Evaluation Is Broken! Here's Why (And How to Fix It) - Yuval Belfer and Niv Granot

Jun 3, 2025 · 10:58

Yuval Belfer and Niv Granot of AI21 Labs argue that current RAG evaluation is broken because benchmarks rely on local questions with answers contained in single chunks, failing to reflect real-world messy data. They demonstrate that standard RAG pipelines—like those from LangChain and LlamaIndex—achieve only 5-11% accuracy on aggregative questions about 22 FIFA World Cup documents, such as 'Which team has won the most times?' Their proposed fix converts unstructured corpuses into structured SQL databases by clustering documents, inferring schemas (e.g., year, winner, top scorer), and using text-to-SQL at inference. This structured RAG approach handles counting and max/min queries that standard chunking cannot, though it requires homogeneous data and careful normalization to avoid ambiguity (e.g., West Germany vs. Germany). They caution that RAG is not one-size-fits-all and that existing benchmarks miss these critical use cases.

Will Agent evaluation via MCP Stabilize Agent Networks? - Ari Heljakka

Jun 3, 2025 · 14:11

Ari Heljakka, CEO of Root Signals, argues that agent evaluation via the Model Context Protocol (MCP) can stabilize agent networks by enabling self-improvement loops. Using a hotel reservation agent example, he shows that without MCP the agent wrongly recommended a competitor hotel, but with MCP it invoked policy evaluators and corrected its behavior automatically. He also demonstrates marketing message optimization via Cursor, where the agent scored and improved text using an MCP-connected evaluation charge. Heljakka emphasizes that success requires a powerful evaluation library with systematic lifecycle maintenance, and introduces Root Signals' free MCP server. The approach promises more controllable, transparent, and dynamically self-correcting agents across different frameworks.

From PM at Stripe to Building an AI startup, a recent founder's journey - Mounir Mouawad

Jun 3, 2025 · 11:59

Mounir Mouawad, CEO and co-founder of Porsche AI, explains how building an AI startup differs from product roles at Stripe, Google, and Amazon, using video game analogies. He argues user problems are an 'emergent property' requiring hypothesis-driven iteration rather than conventional roadmaps. Product development is gratifying with releases in hours or days, but velocity is a 'stable stick' as opportunities like MCP come and go quickly. The hardest part is outreach without big brand support—like playing Crash Bandicoot without boosters—so he finds people followers, advocates, and partnerships (e.g., with Browserbase) essential. He asks listeners to star Porsche AI's GitHub repo.

My AI Thinks I'm Eating My Feelings (and Other Nutritional Insights) - Rami Alhamad

Jun 3, 2025 · 9:59

Rami Alhamad, founder of Alma, presents the AI nutrition companion that simplifies food tracking through voice, text, and photo inputs, using LLMs broken into sequential steps to deliver near-instant feedback. AlmaScore, developed with Harvard’s Dr. Eric Krim, rates meals out of 100 based on nutritional quality. The system learns user preferences continuously via a knowledge dataset and proactively surfaces insights like pairing blueberries with dark chocolate to boost vitamin C absorption. Rami emphasizes that real-time user feedback (a 'How did Alma do?' prompt) outperforms evaluation benchmarks, and that brand design, trusted partnerships, and community-generated data are key differentiators as foundation models commoditize code.

Real AI Agents Need Planning, Not Just Prompting - Yuval Belfer

Jun 3, 2025 · 7:58

Yuval Belfer of AI21 Labs argues that LLMs alone still fail at instruction following, as shown by GPT-4.1's struggles in 2025, and that true AI agents require dynamic planning, not just prompting. He critiques ReAct for lacking look-ahead, contrasting it with AI21 Maestro's planner and smart execution engine, which uses best-of-n sampling, candidate pruning, and replanning. On AIfeval, Maestro pushes GPT-4.0, Claude Sonnet 3.5, and R3 Mini to near-perfect scores; on internal requirement satisfaction benchmarks, it improves over single LLM calls despite higher runtime and cost. Belfer advises starting simple (SLMs, ReAct) and escalating to planning only for complex tasks, inviting listeners to join the Maestro waitlist.

Rust is the language of the AGI - Michael Yuan

Jun 3, 2025 · 29:14

Michael Yuan argues that Rust, with its strong type system and compiler feedback, is the ideal language for AI code generation, unlike human-friendly Python or JavaScript. He presents Rust Coder, an open-source project supported by two Linux Foundation Mentorship grants, which uses MCP tools to generate, compile, and fix Rust projects. The system integrates a coding LLM (e.g., Qwen Coder) with a self-improving knowledge base of compiler errors, enabling it to generate correct code and automatically fix bugs. Yuan demonstrates its use in a Rust camp of 1000+ students and envisions future AI agents generating and deploying Rust code autonomously for tasks like drone control. He invites contributions to expand the knowledge base and enhance the tools for AGI.

The RAG Stack We Landed On After 37 Fails - Jonathan Fernandes

Jun 3, 2025 · 18:52

Jonathan Fernandes, independent AI engineer, details the RAG stack his team settled on after 37 failed attempts, covering orchestration (LlamaIndex), embeddings (BAAI BGE small), vector database (Qdrant), LLMs (GPT-4, Qwen, Llama), reranking (Cohere), monitoring (Arize Phoenix), and evaluation (RAGAS). He demonstrates a live prototype in Google Colab using a London railway knowledge base, showing how a naive RAG returns irrelevant results (e.g., suggesting black cabs for "where can I get help at the station"). By swapping components—replacing in-memory storage with Qdrant, using an open-source embedding model, upgrading to GPT-4, and adding Cohere reranking—the answer improves to "go to booth number five next to the Eurostar ticket gates." For production, he deploys via Docker Compose with NVIDIA embedding/reranking models and Ollama for serving. The episode also stresses the importance of tracing latency per component and using RAGAS for systematic evaluation across many queries.

Invisible Users, Invisible Interfaces: Accelerating Design Iteration with AI Simulation - Alex Liss

Jun 3, 2025 · 12:37

Alex Liss, VP of Data Science and AI at Huge, argues that AI should be used as a tool for design itself rather than as a feature, proposing Intelligent Twins—AI simulations of user personas—to accelerate needfinding and interface testing. Citing a trust gap where only 32% of US adults trust AI, he advocates for repairing it by designing better interfaces, not adding chatbots. He demonstrates a global audit of sports websites (NBA, Olympics, Premier League) where Intelligent Twins simulated casual and super fans across 72 tasks, revealing strong navigation but declining performance in information architecture and engagement. The methodology uses computer use agents for visual evaluation and human-in-the-loop review to generate focused design briefs. Limitations include reproducibility and the need for test-and-control validation, with future potential in integrating MCP protocol for turning Figma components into code. Liss concludes that AI simulation can help teams gather insights faster and create simpler, trustworthy interfaces.

Luminal - Search-Based Deep Learning Compilers - Joe Fioti

Jun 3, 2025 · 24:35

Joe Fioti presents Luminal, a search-based deep learning compiler that simplifies ML libraries to 12 primitive operations and uses search to automatically discover optimized kernels like flash attention. By representing models as directed acyclic graphs of these simple ops, Luminal keeps its codebase under 5,000 lines yet can run all major models. Its compiler applies 20-25 rewrite rules to search through equivalent GPU kernels, profiling to find the fastest—automatically rediscovering flash attention, an algorithm that took five years for the industry to develop. Data movement accounts for 99% of runtime, so kernel fusion merges many ops into one, dramatically speeding execution. An external auto-grad crate adds training support without altering the core. Future plans include supporting AMD, TPUs, and a serverless cloud that exports optimized graphs for inference.

The Current State of Browser Agents - Jerry Wu and Wyatt Marshall

Jun 3, 2025 · 21:13

Jerry Wu and Wyatt Marshall of Halluminate evaluate browser agents with WebBench, a benchmark of 5,000 tasks across 500+ websites. They find read tasks succeed at ~80%, but write tasks drop below 40% for autonomous agents, with failures often due to infrastructure like CAPTCHAs rather than agent intelligence. They advise AI engineers to choose read over write use cases, test multiple browser infrastructure providers, and combine agents with deterministic Playwright workflows for production. Notable emergent behaviors include an agent bypassing Cloudflare by searching Google and an agent posting a top-liked Medium comment.

Text-to-Speech Data Preparation and Fine-tuning Workshop - Ronan McGovern

Jun 3, 2025 · 34:00

Ronan McGovern walks through fine-tuning Sesame's CSM-1B text-to-speech model on a specific voice, using a YouTube video as the data source. He explains token-based TTS models, including how audio is represented via codebooks and how CSM-1B uses a main transformer for zeroth tokens and a secondary transformer for 31 hierarchical tokens. The workshop covers data preparation: downloading audio with yt-dlp, transcribing with Whisper Turbo, manually correcting the transcript, and splitting audio into 30-second chunks (41 clips from a 30-minute video). Fine-tuning uses Unsloth with LoRA adapters (rank 32, alpha 16) on linear layers, training for one epoch with a batch size of 2 and virtual size of 8, reducing loss from ~6.34 to ~3.72. Evaluation compares zero-shot inference (random speaker), voice cloning (closer but imperfect), and fine-tuned plus cloning (best result, producing an Irish-accented voice with natural errors). McGovern recommends 50+ 30-second clips for noticeable effect and notes that combining fine-tuning with voice cloning yields good performance even with limited data.

Stop Ordering AI Takeout A Cookbook for Winning When You Build In House - Jan Siml

Jun 3, 2025 · 10:45

Jan Siml argues that small in-house teams can generate millions in revenue by focusing on one job-to-be-done, tracking dollar outcomes, and pushing proactive insights instead of chasing multi-agent systems and expensive models. Over 10 sprint weeks with two developers, his team built a sales alert system driving several million dollars ARR. He shares five lessons: go deep on one value event, trace everything to revenue (offline evals never sign contracts), push insights proactively (daily digests had 20-point higher NPS than chat UI), convert time saved into guided action, and invest in data and UX over bigger models (changing models only affected costs and evals, not user outcomes). Owning data and tight feedback loops create a revenue flywheel.

Buy Now, Maybe Pay Later: Dealing with Prompt-Tax While Staying at the Frontier - Andrew Thomspson

Jun 3, 2025 · 25:09

Andrew Thompson, CTO of Orbital, introduces the concept of Prompt-Tax: the hidden cost of migrating prompts when upgrading AI models. He shares how his agentic product automates real estate due diligence, growing from <1B to 20B monthly tokens and zero to multiple seven-figure ARR over 18 months. Key tactics include optimizing for prompting over fine-tuning, using domain experts (ex-lawyers) to write prompts, and relying on vibes over formal evals. Thompson advocates 'betting on the model'—shipping new frontier models immediately and fixing regressions on the fly using progressive rollouts and rapid feedback loops. A clip from Demis Hassabis underscores the unique challenge of evolving tech stacks. The episode concludes with questions on whether evals or progressive delivery scale to manage Prompt-Tax.

The Benchmarks Game: Why It's Rigged and How You Can (Really) Win - Darius Emrani

Jun 3, 2025 · 11:20

Darius Emrani exposes how AI benchmarks are rigged, showing that xAI cherry-picked Grok-3 comparisons, OpenAI funded FrontierMath for privileged access, and Meta submitted 27 Llama-4 variants to LM Arena optimizing style over substance. Citing Goodhart's Law, he argues that when benchmarks target billions in investment, they cease to measure real capability—Andrej Karpathy admits he doesn't know which metrics to trust. Emrani provides a 5-step framework to build use-case-specific evaluations, emphasizing that 39% of score variance comes from writing style. He advocates for apple-to-apple comparisons, open-source test sets, and style-controlled metrics, concluding that teams should stop chasing leaderboards and instead iterate on real production data to ship reliable AI.

Unlocking Africa's Potential with AI — Thabang Ledwaba

Jun 3, 2025 · 26:17

Thabang Ledwaba argues that Africa, often seen as a latecomer to AI, actually has immense potential to lead in AI-driven innovation by leveraging its unique challenges and creativity. He points to Kenya's third-highest daily ChatGPT usage and fintech successes like M-Pesa as evidence of immersion in technology. Criticizing over-engineered solutions, he contrasts ticket systems for home office queues with his idea to auto-initiate ID applications at age of eligibility. He highlights Africa's 30% of earth minerals, noting the irony of exporting raw materials only to import finished goods, and calls for a mindset shift akin to China's 'serve yourself first' strategy. Ledwaba showcases African innovations like Nigerian pharma wings and Moroccan Project Cumulus, urging Africans to see themselves as producers, not just consumers, and to harness AI for sustainable development.

Analyzing 10,000 Sales Calls With AI In 2 Weeks — Charlie Guo

Jun 3, 2025 · 9:49

Charlie Guo explains how his team at Pully used Claude 3.5 Sonnet to analyze 10,000 sales call transcripts in two weeks, turning a task that would take 625 days manually into a $500 project. They chose Claude over smaller models due to unacceptable hallucination rates, and reduced costs by up to 90% using prompt caching and extended outputs. The system combined RAG enrichment, chain-of-thought prompting, and structured JSON outputs with citations to ensure accuracy. The analysis, originally for the executive team, became a company-wide resource, enabling marketing to pull customer quotes and sales to automate transcript downloads, saving dozens of hours weekly. Guo emphasizes that good engineering—JSON outputs, database schemas, and thoughtful integration—matters as much as AI capability, and challenges listeners to mine their own untapped customer data.

Letting AI Interface with your App with MCP — Kent C Dodds

Jun 3, 2025 · 19:54

Kent C Dodds explains how Model Context Protocol (MCP) enables AI assistants like Claude Desktop to interface with any app or service, moving beyond limited built-in integrations. He traces the evolution from early LLM chat (phase one) through host applications with custom integrations (phase two) to MCP's open standard (phase three), arguing that MCP lets developers build once for any assistant. In his live demo, he runs a Locationator server to derive his location, a GetWeather server for weather, and an Epic Me server with OAuth authentication to create a journal entry. MCP servers handle tools, resources, and prompts; the client standardizes communication while the server controls unique logic. Kent contends this protocol is how we give AI "hands" to perform real tasks, leading toward a universal Jarvis-like assistant that can dynamically connect to any service without manual integration per platform.

Effective AI Agents Need Data Flywheels, Not The Next Biggest LLM – Sylendran Arunagiri, NVIDIA

Jun 3, 2025 · 16:41

Sylendran Arunagiri of NVIDIA argues that effective AI agents rely on data flywheels, not the largest LLMs, enabling smaller models to achieve top accuracy at a fraction of cost. He details how NVIDIA's NeMo microservices power a continuous cycle of data curation, fine-tuning, evaluation, and guardrailing. Using an internal employee support agent (NVINFO), they achieved 96% accuracy with a 70B model but found that fine-tuning a smaller 8B model matched that accuracy, while a 1B model reached 94% with 98% lower inference cost and 70x latency reduction. The process involved curating 685 ground truth data points from user feedback and error analysis. He provides a framework: monitor user feedback, attribute errors, plan model experimentation, and execute regular retraining cycles.

ChatGPT is poorly designed. So I fixed it

Jun 3, 2025 · 5:21

Ben Holmes critiques ChatGPT's confusing design, calling it a case of 'shipping the org chart' where voice and text features feel like separate apps. He demonstrates a fix by building FixGPT, an app that allows simultaneous voice and text interaction and smartly routes requests to the right model via tool calls. Using off-the-shelf APIs like 4.0 real-time and simple prompts, Holmes shows how a 'send chat message' tool enables detailed text responses alongside voice, while a reasoning model tool handles complex queries. The source code is available on GitHub, showcasing how to avoid OpenAI's science fair of options.

Designing AI To Scale Human Thought — Jun Yu Tan, Tusk

Jun 3, 2025 · 12:24

Jun Yu Tan, founding engineer at Tusk, argues that AI should augment human thinking rather than automate tasks, introducing three interaction patterns — blind spot detection, cognitive partnership, and proactive guidance — to help users reveal blind spots, spark creativity, and amplify decision-making. At Tusk, this approach caught verified bugs in 43% of pull requests and added nearly 1,000 tests in two months by systematically surfacing edge cases and assumptions. Tan emphasizes progressive, contextual, and bidirectional trust, and advocates for product metrics that track user skill growth, not just engagement. The episode concludes that the most profound technologies make us more thoughtful, not just productive, and that AI should act as a mirror for our minds, showing us blind spots we habitually miss.

open-rag-eval: RAG Evaluation without "golden" answers — Ofer Mendelevitch, Vectara

Jun 3, 2025 · 5:03

Ofer Mendelevitch from Vectara presents Open-RAG-Eval, an open-source framework that enables RAG evaluation without requiring golden answers or golden chunks, solving a major scalability problem. Backed by research with the University of Waterloo's Jimmy Lin Lab, it uses UMBRELA for retrieval scoring on a 0–3 scale that correlates well with human judgment, and AutoNuggetizer for generation via nugget creation, Vital/OK ratings, and an LLM judge analyzing the top 20 nuggets for support. Additional metrics include citation faithfulness and Vectara's HHEM hallucination detection model. Connectors are available for Vectara, LangChain, and LlamaIndex, with results viewable through an intuitive UI at OpenEvaluation.ai.

The Future of Qwen: A Generalist Agent Model — Junyang Lin, Alibaba Qwen

Jun 3, 2025 · 25:14

Junyang Lin from Alibaba Qwen presents the latest developments in the Qwen model series, including Qwen3's hybrid thinking mode that combines thinking and non-thinking behaviors in a single model, supporting over 119 languages and dialects. The flagship 235B MOE model activates only 22B parameters and competes with top-tier models like O3-mini, while a 4B model deployable on mobile devices rivals Qwen 2.5 72B. Qwen3 introduces dynamic thinking budgets for inference-time scaling, achieving over 80% on AIME 2024 with 32K thinking tokens, and enhances agent capabilities with MCP support. The vision-language model Qwen2.5-VL excels in benchmarks like MMU and MathVista, and the omni model Qwen2.5-Omni accepts text, vision, and audio inputs while generating text and audio, achieving competitive vision-language performance. Future directions include scaling reinforcement learning with environment feedback, extending context to at least 1M tokens, and unifying understanding and generation across modalities, marking a shift from training models to training agents.

Creating Agents that Co-Create — Karina Nguyen, OpenAI

Apr 30, 2025 · 24:22

Karina Nguyen, an AI researcher at OpenAI and former Anthropic researcher, discusses two major AI scaling paradigms—next-token prediction (pretraining) and reinforcement learning on chain of thought—that shift AI from narrow tools to collaborative agents that co-create. Pretraining builds world understanding by predicting the next token, but hard tasks like math and creative writing require chain of thought reasoning, scaled with OpenAI’s O1 model. Post-training via RLHF and synthetic data enables rapid iteration, and the next stage is co-innovators: agents with reasoning, tool use, long context, and creativity. Nguyen shares product lessons from developing ChatGPT and Claude: 100K context via file uploads, ChatGPT Tasks for scheduled reminders that scale with model capabilities, and Canvas as a flexible interface for co-writing, coding, and research that can morph into an IDE, a tutor, or a data scientist. She envisions a future of invisible software creation where AI generates personalized, multimodal outputs on the fly, reducing reliance on clicking links, and the interface becomes a blank canvas that adapts to user intent.

How to Build Your Own AI Data Center in 2025 — Paul Gilbert, Arista Networks

Apr 27, 2025 · 23:00

Paul Gilbert, tech lead at Arista Networks, explains the key considerations for building AI data center networks in 2025, emphasizing the stark differences from traditional enterprise networks. He details the backend GPU network (eight 400G ports per H100 server, no over-subscription), the front-end storage network (calmer, 100-200G), and the need for lossless Ethernet with ECN and PFC flow control to prevent packet drops during synchronized GPU bursts. Gilbert highlights power challenges (10.2kW per GPU server, requiring 100-200kW water-cooled racks) and the importance of telemetry like RDMA error monitoring and an AI agent that correlates GPU and network issues. He also covers advanced load balancing (cluster-aware, up to 93% utilization) and smart system upgrades without downtime, while noting the upcoming Ultra Ethernet Consortium (v1.0 in 2025) that shifts congestion control to NICs.

Function Calling is All You Need — Full Workshop, with Ilan Bigio of OpenAI

Apr 23, 2025 · 1:42:54

Ilan Bigio from OpenAI argues function calling is the essential primitive for building AI agents, demonstrating how to implement memory, delegation, asynchronous task execution, and even self-modifying tools—all using the same loop. Starting from a basic weather tool, he builds an agent that stores user facts in a JSON file, delegates complex requests to o1, runs multiple weather lookups in parallel with asyncio, and creates a Bootstrap agent that writes its own Python functions on the fly. He also shares practical tips from his work on Swarm and the Real-Time API, including a stay-silent function to prevent premature responses and the ability to read scripts with XML tags. The workshop emphasizes that sophisticated agent behaviors emerge from simple function calling patterns without needing heavy frameworks.

Ensure AI Agents Work: Evaluation Frameworks for Scaling Success — Aparna Dhinkaran, CEO Arize

Apr 23, 2025 · 15:28

Aparna Dhinakaran, CEO of Arize AI, explains that evaluating AI agents requires testing three core components—routers, skills, and memory—each at different trace levels. Routers must be checked for correct skill selection and parameter passing; skills need LLM-as-judge or code-based evals for chunk relevance and answer correctness; and convergence measures whether the agent takes a consistent number of steps to complete a task. For voice agents, additional evaluations on audio chunks—sentiment, speech-to-text accuracy, and tone consistency—are necessary. Dhinakaran demonstrates Arize's own Copilot, where evals run at every trace step (router choice, argument passing, task completion) to isolate failures. She argues that observability-driven evaluation frameworks, with multiple eval layers, transform experimental agents into production-ready enterprise tools.

The missing pieces of workflow automation — Shirsha Chaudhuri, Thomson Reuters Labs

Apr 23, 2025 · 14:37

Shirsha Chaudhuri, head of co-innovation at Thomson Reuters Labs, identifies eight missing pieces preventing true AI workflow automation in the enterprise. She argues that while 71% of Fortune 500 companies still run mainframes and 68% of IT production workloads remain on mainframe, current agentic efforts lack connectors to bridge legacy systems, standardized agent architectures, and reliable ROI metrics. She highlights the need for domain experts to reimagine processes alongside AI practitioners, collaborative UX design, and AI governance that translates ethics into agent architecture. Control balance between deterministic and agent-driven steps remains unresolved, and the fast-evolving agent lifecycle lacks a clear update strategy. Drawing from Thomson Reuters' own journey—from a 2023 open AI arena through RAG and prompt engineering to 2024's agentic experiments—she calls for a reimagined workflow design rather than just task-level automation.