Google

Tethered: Our Agents Are Us — Shu Fang, Two Sigma

Sep 3, 2026 · 21:10

Two Sigma's Shu Fang explains why every employee at the 25-year-old quant fund runs a cloud agent as themselves, not a service account — echoing the doubles in the film Us. A separate agent identity collapses fast: permissions drift, licensing doubles, and systems like Google Workspace refuse two identities on the same data. Agents instead run in per-person Kubernetes namespaces built for automated jobs, with a sidecar mounting the identity into the pod. Since agent and human share one identity, a header propagated like a trace ID recovers not just the actor but the replayable chain behind a result. For web access, they use Google's web grounding for enterprise — an index inside their network boundary — and deny native search and fetch tools, trading freshness (about 24 hours) for removing exfiltration and prompt injection risk.

Teaching agents to pay — Anna Spysz, Stripe

Sep 1, 2026 · 19:10

Anna Spysz of Stripe walks through how agentic commerce works in a talk based on her own experience building a shopping agent. She demonstrates how agents discover and buy products: reading structured catalog data rather than rendered pages, speaking protocols like UCP, and operating within guardrails like disclosed fees and logged decisions. Using her headphones purchase as the running example, she shows how a merchant capabilities manifest makes a store visible to agents, and how a persona config change can flip an agent from pushy to trustworthy. The episode's core claim is that agentic commerce demands deliberate design from both sides of a transaction, not just a checkout API.

Agentic Sites: Building Hyper Personalized Websites — Carlos Sanchez, Adobe

Aug 29, 2026 · 20:43

Carlos Sanchez, principal scientist at Adobe, argues that agentic sites can deliver hyper-personalized web pages in real time by treating the whole site as a corpus and generating only specific blocks. He demonstrates a coffee-machine site that builds a personalized camping page in 1.64 seconds using Google's Gemma 4 on Cerebras at 2,300 tokens per second. Model choice must be evaluated per site for accuracy and speed, he says, showing 1.1 seconds against 4.6 for the runner-up, and this doesn't need a frontier model because work is choosing and arranging blocks. Marketers define personas in natural language, browsing signals feed an 'audience of one' loop for pre-generated recommendations, and a tool builds an agentic site for any URL in less than an hour.

How Anthropic Builds: Lessons from Labs — Mike Krieger, Anthropic

Aug 27, 2026 · 26:11

Instagram co-founder Mike Krieger, now an individual contributor at Anthropic's Labs, argues for being 'unreasonable' with Claude—he had it port a few hundred thousand lines of Python to TypeScript over a weekend. He attributes timid asks to first-gen AI's boxed tools; cites Instagram-era habits (pre-measure everything, knobs); and explains tagging Claude in Slack as multiplayer delegation. Labs runs two-week 'persevere or pivot' reviews, bet leads manage nobody, and code review is bottlenecked by comprehension, so 2,000-line PRs travel with artifacts explaining intent and tradeoffs. He also sketches Claude Design's future, urges unship of product complexity, and advises startups to bet on verticals like finance. On burnout: no job is so important you cannot be offline for two days.

KV Cache-Aware Routing and P/D Disaggregation on Kubernetes — Yuchen Fama & Ashish Kamra, Red Hat

Aug 27, 2026 · 21:48

Yuchen Fama and Ashish Kamra of Red Hat argue that public inference benchmarks hide the chaotic reality of agentic workloads, and show how KV cache-aware routing plus prefill/decode disaggregation in the open source llm-d framework tackles it. Red Hat's traces show agentic sessions running from a few turns to 3,000, cache hit rates over 90%, and input-output token ratios past 100:1, making a 10x cost gap between cached and uncached tokens. A live demo shows routing reusing cache on the same pod cutting time from 3 seconds to 1, while a fresh system prompt pays the full 3 again. For prefill/decode disaggregation, they report P99 inter-token latency dropping from roughly 900 milliseconds to about 100 across 16 H100s serving gpt-oss, but note it only wins in the middle concurrency band and requires RDMA or RoCE to move KV caches. Their closing case study runs GLM 5.2 on H200s with three prefill workers to one decode, achieving 4x faster time to first token and 60% more requests.

Building Blocks for Uber’s Software Factory— Uday Kiran Medisetty & Adam Huda, Uber

Aug 21, 2026 · 18:26

Uber's Uday Kiran Medisetty and Adam Huda explain the six building blocks of their agentic software factory, which now generates more than 70% of pull requests and has doubled lines of code per engineer year over year. A model gateway handles 100 million requests a day across 800 projects with redaction of 20+ PII types and safety models under 100 milliseconds; an MCP gateway cuts token use over 40%, a skills marketplace runs 20,000 executions daily across 2,500 skills, and a context graph of 40 million entries replaces 20-30 systems. Adam shows one feature end-to-end, from Slack idea to draft PR that stops before CI, validating via simulator screenshots against Figma in an inner loop. The bottleneck, they argue, has moved to deciding whether a thing should be built at all.

The Rise of CaaS: Context-as-a-Service for Agentic AI — Omer Primor, Bright Data

Aug 14, 2026 · 22:20

Omer Primor of Bright Data argues that rented AI search and context-as-a-service (CaaS) lose to a self-built pipeline once query volume passes a tipping point. The web is context, not data, and it decays: social content goes stale within a day, news/finance/retail within 30 days, so context is never a one-time snapshot. His test enriching 100 sponsor companies across 25 fields found general search beat dedicated CaaS vendors on coverage, because CaaS only answers from data it already holds; costs were similar, but frequency is the real killer — every repeated query costs the same when nothing changed. He built scrapers for LinkedIn, jobs, and Crunchbase in a day, priced setup at $5,000, and put the crossover just over 15,000 entities. Owned context compounds while rented decays.

From RL to IRL — Gaurav Mishra, Amazon AGI Lab

Aug 14, 2026 · 17:46

Gaurav Mishra of the Amazon AGI Lab explains why RL for computer-use agents works in games but breaks in real life, and how his team turns failures into training data. He shows early browser-training runs where an agent guesses its password and locks the account, and clicks a sponsored button styled like the submit button, landing elsewhere. The talk catalogs partial observability, irreversibility, expiring credentials, and ambiguous success, then proposes flight-school sandboxes, process reward models, calibrated confidence, and adversarial tasks. A later trajectory shows the agent recognizing the sponsored button, refusing to guess the password, and handing off to the user, echoing the talk's point that the difference between a demo and a product is what happens after the first failed click.

Open Source Is Dead. Long Live Open Source. — Saoud Rizwan, Cline

Aug 7, 2026 · 17:30

Saoud Rizwan, founder of Cline, argues that AI has killed the community side of open source while open weights models win on economics. He cites Zig banning AI from PRs, curl weighing shutdown of its bug bounty over AI-generated reports, and tldraw auto-closing pull requests, plus a LiteLLM compromise that stole credentials for three hours. Rizwan makes the case that closed labs' subsidized subscriptions lead to lock-in and price gouging, while open models like GLM match or beat Opus on cost and code quality — GLM used twice the tokens at half the cost and fixed a real Cline bug that Opus's faster fix left broken. He compares open weights to Facebook's Open Compute project and urges American labs to release open weights before foreign models become the standard.

MCP Apps: Extending the Frontier — Ido Salomon & Liad Yosef

Aug 2, 2026 · 18:38

Ido Salomon and Liad Yosef, creators of MCP UI and co-creators of the MCP Apps spec, explain how MCP Apps lets servers ship interactive branded UI instead of text into any supporting host. The official MCP extension, built with Anthropic and OpenAI and supported by Claude, VS Code, ChatGPT, Copilot, Cursor and Slack, links a tool call to a resource; the host renders returned HTML as a sandboxed web component and clicks flow back through a callback to the model. Early adopters include 11 Labs, Shopify and Postman; the payoff is distribution: with ChatGPT at 800 million weekly users, you write once and run everywhere. The spec is still evolving, with an open working group meeting every three weeks and live work on reusable views, AppTools/ViewTools for host-to-app control, and interoperability with generative UI standards like A2UI.

Why Large? Tiny LMs & Agents on Edge/Robotics — Cormac Brick, Google

Jul 25, 2026 · 21:45

Cormac Brick of Google AI Edge argues that the real constraint on edge AI is DRAM cost, not compute, making tiny models essential for widespread deployment. Small models (1–4 billion parameters) require 4–8 GB of RAM, but their quantized Gemma 2B model—at just 2.9 bits per weight—runs on a Raspberry Pi at 7.6 tokens per second and on a Qualcomm NPU at 31 tokens per second. For even lower-end devices, tiny models (50–500 million parameters) need under 2 GB of RAM and can be fine-tuned for specific tasks like voice-to-function calling, reaching over 86% reliability on ten actions. A shipped example is an offline voice dictation app that uses two fine-tuned sub-billion Gemma models to clean up ums and ahs without a subscription. The talk covers the Gemini-optimized toolchain (Lighter TLM, MediaPipe), synthetic data generation for fine-tuning, and the trade-off between zero-shot prompting in small models versus fine-tuning tiny models for reach and speed.

Agentic Security: Permissions, Provenance, and the Agent Supply Chain — Steve Yegge, Gas Town

Jul 20, 2026 · 22:32

Steve Yegge argues that AI-written code will dramatically increase security vulnerabilities unless developers adopt a separate security pass using tools like Snyk and Chainguard. He shares a bank architect's insight that shipping 10x faster with the same defect rate produces a 10x vulnerability surface, made worse by models writing code. Yegge demonstrates the gap by noting Fable's security hardening missed 241 vulnerabilities that Snyk found in his 30-year-old game. He warns of new attack surfaces like slop squatting, where models hallucinate package names that attackers then backfill with malicious versions. Yegge advocates for multiple passes—correctness, then security—and urges incorporating tools into agent workflows. He also cautions that Five Eyes predicts open-source models will autonomously hack production systems within months, and that personal scams using AI-generated voice and video are imminent.

It's 10pm. Do You Know Where Your Agents Are? — Kim Maida, Keycard

Jul 20, 2026 · 23:02

Kim Maida of Keycard argues that standard API keys dangerously overprivilege AI agents, enabling incidents like a night-shift agent dropping a production Postgres database because its kitchen-sink credential allowed it. Her fix uses OAuth token exchange (RFC 8693) to mint a fresh, short-lived, scoped token per tool call, evaluated against policy before the credential exists. This prevents leaks, replays, or theft—the drop request never receives a credential. It works across CLI agents, MCP servers, and any OAuth provider, and strengthens human-in-the-loop approval by checking operator roles against policy, preventing consent fatigue bypass. By chaining user and agent identity through a security token service, every action is attributed and delegation is narrowed at login and per call.

You Didn't Ship a Bug. You Just Wrote It for a Human. - Ravi Madabhushi, Scalekit

Jul 19, 2026 · 12:50

Ravi Madabhushi, co-founder of Scalekit, explains why infrastructure built for humans breaks when AI agents act 60 times faster than users, citing a production database spike caused by a 'last seen' timestamp updating every tool call. He argues that OAuth scopes, designed for deterministic human-written programs, cannot constrain non-deterministic agents, which need attribute-level, time-bound, and principle-scoped permissions. Madabhushi warns that 60% of LLM errors stem from rate limits designed for humans, not agents, and advocates for just-in-time authorization and absolute visibility into every agent action—who took it, on behalf of whom, and when authorized. He concludes that without deterministic guardrails, teams are merely 'praying' agents won't delete production data.

"Software engineering is not about writing code" — Benoit Schillings, Google DeepMind VP of Research

Jul 17, 2026 · 20:26

Benoit Schillings, VP of Technology at Google DeepMind, argues that the era of syntax generation is over, with AI now capable of superhuman code writing, shifting the bottleneck to architecture and validation. He explains that 80% of new code on GitHub is machine-generated, leading to a saturation of human training data and the need for self-play techniques inspired by AlphaZero. Schillings highlights DeepMind's use of self-play to generate and verify coding challenges, enabling models to reach superhuman performance. He emphasizes the shift to inductive architecture, where models must plan, decompose complex problems, and transfer knowledge across domains. Additionally, he calls for new programming languages designed for models rather than humans, and explores how AI's rapid experimentation will transform fields like chemistry and biology by uncovering patterns invisible to humans.

The AI bugpocalypse is here. Now what? - Jack Cable, Corridor

Jul 12, 2026 · 19:44

Jack Cable, Co-Founder and CEO of Corridor, argues that frontier AI models can now find and exploit software vulnerabilities at scale, creating an 'AI bugpocalypse' that demands fundamental security changes. He cites 84% of developers using AI coding tools, with models like Mythos outperforming humans in finding vulnerabilities. He notes that basic vulnerability classes like buffer overflows remain prevalent despite decades of known fixes, and that memory-safe languages like Rust can eliminate 60-70% of such bugs, as shown by Android's drop from 75% memory safety vulnerabilities in 2019 to 30% in 2022. He warns that even advanced models introduce vulnerabilities 20-40% of the time (backsbench.com) and points to an Opus 4.6 incident that lost $2M. His recommendations to Congress include preventing vulnerabilities in new code, hardening open-source software, and fostering American open-weight models, arguing that defenders need access to these models to stay ahead.

The Factory That Dreams: 39 AI Agents, No Framework - Rushabh Doshi, Machinecraft

Jul 11, 2026 · 9:58

Rushabh Doshi, CEO of Machinecraft, explains how his 100-person factory built Ira, a 36-agent AI OS running its go-to-market and memory without a framework. The system uses 36 specialist agents, each with one job, orchestrated by Athena, handling outbound emails, quoting, leads, and replies from one Cursor tab. Built for $30,000 versus $230,000 agency quote, it runs on a few thousand monthly with no training. Doshi says the secret is well-organized memory, vectors, graphs, and biological-inspired architecture including senses, digestion, memory layers, a nightly dream cycle, and immune system for fact-checking. The system is grounded in a SOUL.md constitution based on Jain philosophy, and ForkMyBrain.org helps others build their own forkable brain by mapping their business from the inside.

The Log Is The Agent - Ishaan Sehgal, Omnara

Jun 25, 2026 · 15:11

Ishaan Sehgal, CEO of Omnara, argues that an AI agent's true identity is not the model, runtime, or tools but its append-only event log—the durable record of every user input, model output, tool call, and permission. Drawing an analogy to a video game save file that persists beyond hardware failure, he explains that treating the log as the agent enables reliability, scalability, forking, migration, multiplayer, and ownership. He warns that deepest lock-in is log lock-in: if a provider owns the log, it effectively owns the agent. Omnara's open-source managed agents platform is built around the session log as the system, not a side effect, allowing agents to survive crashes, resume across machines, and be fully owned and inspected by users.

The Miranda Hypothesis: How Hamilton Poisoned Persona Evals - Jacob E. Thomas, Results Gen

Jun 25, 2026 · 58:17

Jacob E. Thomas argues that persona-eval benchmarks like InCharacter, which report 80.7% personality fidelity for Hamilton, structurally miss anachronistic compositing because they measure fluency and personality consistency—exactly the features the dominant cultural composite optimizes. The Miranda Hypothesis posits that training data saturated with works like the 2015 Hamilton musical (outnumbering the 175,000-word Federalist Papers by orders of magnitude) causes models to produce smoothed, morally legible composites rather than historically faithful personas. Thomas explains that RLHF amplifies this distortion because human raters share the same cultural frame. He proposes a paradigm shift to epistemic simulation: corpus-bounded, temporally-anchored systems evaluated by domain experts, with the model as a swappable component in a configurable role-playing language system. The pre-registered Prism Experiment instantiates Abraham Lincoln at four documented moments (1847, 1858, 1860, 1862–65) under three seeding conditions—bare model, biography, or primary sources—scored on a weighted three-axis rubric that prioritizes anachronism detection (40%) over voice. Thomas concludes that…

Your Agent's Biggest Lie: "I Searched the Web" — Rafael Levi, Bright Data

Jun 17, 2026 · 15:49

Rafael Levi from Bright Data argues that LLMs often hallucinate and provide fake citations because they fail to actually access live web data, getting blocked by anti-bot systems like CAPTCHAs and Cloudflare's AI labyrinth. He demonstrates this with a comparison: without Bright Data's MCP, GPT-5 failed all five tasks accessing sites like LinkedIn and Amazon; with the MCP's 66 tools—including a CAPTCHA-solving browser that mimics human behavior—four succeeded. Levi explains that agents enter an invisible failure loop where they get blocked or served fake data but still answer confidently, making up numbers or non-existent URLs. He emphasizes that 20% of the web is blocked by Cloudflare from AI crawling, and that fake data fed to bots increases hallucinations. The episode covers how Bright Data's MCP provides real web access with search, scraping, and remote browsers, offering a free tier of 5,000 requests per month for experimentation.

Building Agent Interfaces: Lessons from Chrome DevTools (MCP) for Agents — Michael Hablich, Google

Jun 5, 2026 · 22:38

Michael Hablich, product manager for Chrome DevTools at Google, presents four engineering lessons from building agent interfaces, centered on the Chrome DevTools MCP server. He introduces the metric 'tokens per successful outcome' to measure interface fuel efficiency, contrasting it with raw JSON trace files that blew through context windows. The team improved error messages, like adding 'Cannot navigate back, no previous page in history' to enable agent self-healing without human intervention. They decomposed a monolithic `debug_webpage` tool into 25 focused tools, but then faced a discoverability problem solved by auditing tool descriptions for intent. Hablich defends keeping autoconnect friction to prevent prompt injection, citing the 'lethal trifecta' of agents with tool access, data access, and internet access, and outlines three security tiers for browsing agents.

Why (Senior) Engineers Struggle to Build AI Agents — Philipp Schmid, Google DeepMind

May 30, 2026 · 10:40

Philipp Schmid from Google DeepMind argues that senior engineers struggle to build AI agents because they carry years of implicit context that agents don't, designing tools that assume it. He highlights five key shifts: text replaces structured state, so agents can handle semantic meaning like approving a research plan with additional input; errors are inputs, not restart triggers, since long-running agents (5-15 minutes) can't afford to start over; evals replace unit tests because agents are non-deterministic and success is measured by how often it works, not fixed outputs; and agents evolve while APIs don't, so tools like a `deleteItem` endpoint need agent-friendly docstrings because the agent only sees the schema. He advises to hand over control to the LLM, design for recovery, and build to delete since models improve rapidly.

How Google DeepMind Runs Agents at Scale — KP Sawhney & Ian Ballantyne, Google DeepMind

May 24, 2026 · 25:13

Ian Ballantyne and KP Sawhney from Google DeepMind explain how the company scales its agentic platform anti-gravity, featuring a Darwinian skills library and quota management prioritizing paying customers. KP explains deep research currently passes huge context blobs, but his focus is replacing them with a shared file system, enabling artifact generation like infographics. Token-hungry agents require brute-force quota limits; 24/7 monitoring stops spikes, and internal teams face worse quotas than customers. Observability uses a custom web app tracing agent trajectories to raw predict requests. The skills library relies on contributions from domain experts, with survival through evaluation in sandboxed environments. For code review, per-language auto-review models fine-tuned on style guides automatically comment on PRs.

Prompt to Pipeline: Building with Google's Gen Media Stack — Paige & Guillaume, Google DeepMind

May 23, 2026 · 1:54:35

Paige Bailey, Guillaume Vernade, and Ian Valentine from Google DeepMind demonstrate the company's full generative media stack—from Gemini 3.1 Flash Light's video analysis at $0.25 per million tokens to Genie 3's playable world models and Gemma 4's on-device agentic coding—showing developers how to build multimodal apps without cloud APIs. Paige shows AI Studio's Build feature creating a bookshelf scanning app with Firestore and OAuth. Guillaume walks through a workshop using Nano Banana 2 for character portraits, VO 3.1 Lite for video generation at $0.05 per image, LIA 3 for chapter scores, and text-to-speech with distinct voices. Ian runs Gemma 4's 26B mixture-of-experts model on a MacBook to generate 10 SVGs in parallel and build a game from a spec, all locally. The episode argues that Google's models absorb common agent patterns, making custom fine-tunes less necessary.

From 46% to 90%: Fine-Tuning Tiny LLMs for On-Device Agents — Cormac Brick, Google

May 20, 2026 · 21:01

Google's Cormac Brick explains how developers can build on-device AI agents using either system-level Gemini Nano via AI Core or app-level custom models via LiteRT-LM. He demonstrates a skill harness built on Gemma 4 that enables agentic tasks like restaurant roulette, running fully on-device with JavaScript UI. For fine-tuned tiny models, Function Gemma at 270M parameters improved from 46% to over 90% accuracy on eight of ten app-intent functions after synthetic data fine-tuning. The talk also covers the Eloquent transcription app, which chains two Gemma 3-based models (ASR and text polishing) under a few hundred million parameters for offline use. Key trade-offs are latency, privacy, and customization versus system integration effort.

What Breaks When You Build AI Under Sovereignty Constraints - Bilge Yücel, deepset GmbH

May 19, 2026 · 19:09

Bilge Yücel, Senior Developer Relations Engineer at deepset, argues that sovereign AI requires explicit control over data flow, model choice, infrastructure, and operations, and retrofitting these pillars breaks existing systems in predictable ways. Replacing frontier APIs with self-hosted models forces re-evaluation from scratch, moving private data across jurisdictions creates multi-database search problems, replacing managed infra reveals vendor lock-in, and adding observability exposes black-box systems. She presents a sovereign architecture with guardrails, MCP tools, and Haystack's swappable components to mitigate these issues. The closing checklist asks whether you can swap models without changing application logic, have compliant run logs, and respond to incidents without calling a hyperscaler.

Why TTS Models Now Look Like LLMs — Samuel Humeau, Mistral

May 9, 2026 · 22:26

Mistral AI scientist Samuel Humeau explains why text-to-speech models now resemble LLMs, using an autoregressive transformer with neural audio codecs to compress 200 kbps audio into ~500 tokens per second. He demos Mistral's open-weight TTS model, cloning a voice from a few seconds of reference audio and generating speech with 17 ms first-token latency via frame-by-frame streaming. The talk covers the codec-backbone-decoder pipeline, the model's 4B-parameter backbone generating 37 tokens per 80 ms frame via a diffusion decoder, and a live voice agent answering conference schedule questions. Humeau notes the next latency win is handling streaming text input from an LLM, with no settled architecture yet, while keeping the encoder for voice cloning proprietary.

MCP UI: Extending the frontier — Liad Yosef and Ido Salomon, MCP Apps

May 6, 2026 · 22:21

In this episode, Ido Salomon and Liad Yosef explain MCP Apps, an MCP extension that lets tools send interactive, branded UI instead of plain text responses inside chat hosts like ChatGPT, Claude, VS Code, Cursor, and Copilot. They argue that text-based chat reduces companies to a wall of text, erasing identity, while MCP Apps preserves branding by returning HTML resources that hosts render as secure, interactive components. The architecture ensures every user click sends a message back to the host (not directly to the backend), keeping interactions in context—as demonstrated with a PostHog funnel analysis in Claude. Adoption includes Shopify, Hugging Face, GitHub, ChatGPT, and Claude, with ChatGPT recommending MCP Apps for building ChatGPT Apps. They note 800 million weekly ChatGPT users (10% of world population), calling this a once-in-20-years opportunity to rethink app distribution. Future work covers reusable views, model-UI interaction, and interoperability with generative UI protocols like A2UI and WebMCP, aiming to standardize UI in chat as the new web.

Accelerating AI on Edge — Chintan Parikh and Weiyi Wang, Google DeepMind

May 5, 2026 · 23:58

Chintan Parikh and Weiyi Wang from Google DeepMind present Gemma 4E edge models (2B and 4B) and the LiteRT framework for on-device AI, arguing that edge AI delivers latency, privacy, offline capability, and cost savings. The Gemma 4E models introduce agentic capabilities including built-in function calling, structured JSON output, and chain-of-thought reasoning, all optimized for hardware-native support across CPUs, GPUs, and NPUs. LiteRT supports cross-platform deployment on Android, iOS, macOS, Linux, Windows, and IoT devices like Raspberry Pi, with a CLI tool and AI Edge portal for benchmarking. Performance benchmarks show up to 56 tokens per second on iOS, 30x speedups with NPU acceleration, and 35x faster than Llama on mobile. The Gallery app demonstrates on-device skills such as Wikipedia querying, mood tracking, photo-to-music generation, and voice agents, with open-source code and a Hugging Face repository. Q&A addresses use cases like local security camera face recognition, LiteRT vs TensorRT on Orin, multi-agent architectures, and audio model support.

TLMs: Tiny LLMs and Agents on Edge Devices with LiteRT-LM — Cormac Brick, Google

May 3, 2026 · 1:20:58

Cormac Brick from Google AI Edge explains how Tiny LLMs (sub-1B parameters) and on-device agent skills are making edge AI practical. He details LiteRT-LM, an open-source runtime that runs Gemma models on Android, iOS, and embedded systems, achieving over 1,000 tokens/s on high-end phones. Agent skills use progressive disclosure—loading skill details on demand—enabling reliable tool calling on 2B-4B models. For app deployment, fine-tuning boosts tiny model reliability by 20-40 points (e.g., Function Gemma 270M hit 86% accuracy on 10 functions). Synthetic data workflows and modular design (ASR + text polishing) power real apps like AI Edge Eloquent, which runs entirely offline. Safety is managed through system-level checkers and narrow functional scope for tiny models.

What Do Models Still Suck At? - Peter Gostev, Arena.ai, BullshitBench

Apr 24, 2026 · 20:24

Peter Gostev presents data from his BullshitBench and Arena.ai to argue that language models still struggle with nonsense questions and expert-level tasks, despite benchmark charts showing relentless improvement. His BullshitBench reveals that only Claude Sonnet 4.5 and some Qwen models consistently push back on nonsense, while GPT and Gemini models accept it 50% of the time. Arena.ai's dissatisfaction rate among top 25 models has improved from 17% pre-reasoning to about 9% currently, but remains non-zero, and expert categories like gaming, magic, finance, and law show minimal improvement. For software expert prompts, dissatisfaction dropped from 23.5% in Q2 2024 to 13% in Q1 2026, but gaming is a persistent weakness where models fail to create engaging game mechanics. Gostev warns that narrow benchmarks overstate progress and urges focusing on the full distribution of real-world tasks.

The New Application Layer - Malte Ubl, CTO Vercel

Apr 20, 2026 · 18:52

Vercel CTO Malte Ubl argues that AI engineering is the legitimate successor to web development and that the real value lies in the application layer, not the model labs. He identifies four effective agent archetypes—24/7 support, compressed research, surfacing existing information, and eliminating boring work—and reveals that over 60% of vercel.com page views now come from AI agents. Ubl predicts model companies will commoditize, driving costs down and empowering engineers, while citing Europe's leadership in AI engineering through Vercel's AI SDK, Pi (a coding agent from Austria), and OpenClaw. He stresses the need for open-mindedness toward paradigm shifts and new infrastructure, such as sandboxed agent runtimes, and warns of impending security challenges.

The Future of MCP — David Soria Parra, Anthropic

Apr 19, 2026 · 18:46

David Soria Parra from Anthropic argues that MCP (Model Context Protocol) is the key to connecting agents to tools and data in production, with 110 million monthly downloads—outpacing React's growth at the same stage. He lays out a 2026 connectivity stack combining Skills, MCP, and CLI/Computer Use, each suited for different needs, and emphasizes that best agents will use all three seamlessly. To improve client harnesses, he introduces Progressive Discovery—deferring tool loading via Tool Search to reduce context usage—and Programmatic Tool Calling, where models write scripts to compose tool outputs efficiently. Upcoming MCP protocol improvements include stateless transport (with Google) for easier scaling, async agent-to-agent tasks, enterprise features like Cross App Access and Server Discovery via well-known URLs, and a Skills-over-MCP extension for shipping usage instructions with servers. He calls for community feedback on these directions.

DSPy: The End of Prompt Engineering - Kevin Madura, AlixPartners

Jan 8, 2026 · 1:13:13

Kevin Madura of AlixPartners argues that building robust enterprise AI applications requires shifting from brittle prompt engineering to programming with LLMs using DSPy, a declarative framework that treats prompts as implementation details optimized by the system. He demonstrates how typed interfaces (Signatures) and modular logic (Modules) allow developers to focus on control flow while deferring implementation to the LLM, with Adapters controlling prompt formats (e.g., JSON vs. BAML) to improve performance by 5-10%. The talk's core is Optimizers (like MIPRO and JEPA), which automatically tune prompts by learning from data, shown improving a time entry corrector from 86% to 89% accuracy. Real-world examples include routing files by type (SEC filings vs. contracts), using a 'poor man's RAG' with attachments for multimodal documents, and a boundary detector that segments legal documents from images. Madura emphasizes that DSPy enables transferability across models (e.g., GPT-4.1 to GPT-4.1 Nano) and addresses cost concerns by allowing offline optimization to reduce LLM calls.

Welcome to AIE CODE - Jed Borovik, Google DeepMind

Jan 5, 2026 · 4:10

Jed Borovik, Google DeepMind's Gemini assistant and lead of the Jewels coding agent product engineering team, opens the 2025 AI Engineering Code Summit in New York by declaring that the most important problem in applied AI is code. He frames the event as a single-track summit designed to push the entire AI coding industry forward, not any single company. Borovik highlights the previous leadership track, which covered how AI transforms software organizations, and sets the day's focus on patterns, systems, and products enabling that transformation. He thanks DeepMind as presenting sponsor, noting the timely release of Gemini 3 and Nano Banana Pro, along with Anthropic as platinum sponsor and other gold and silver sponsors. Borovik encourages attendees to visit sponsor booths in the expo area and expresses excitement for the keynotes to come.

Leadership in AI Assisted Engineering – Justin Reock, DX (acq. Atlassian)

Dec 19, 2025 · 18:11

Justin Reock, Deputy CTO at DX (acquired by Atlassian), argues that AI's impact on engineering productivity varies wildly and that leaders must move beyond top-down mandates to focus on psychological safety, measurement of actual outcomes, and targeted integration across the SDLC. He presents data showing a 2.6% average increase in change confidence but extreme variability across companies, with some seeing 20% drops. Emphasizes that writing code is rarely the bottleneck; instead, leaders should identify and fix bottlenecks like context switching, citing Morgan Stanley's DevGenAI saving 300,000 hours annually by converting legacy code specs and Zapier reducing engineer onboarding to two weeks via AI agents. Introduces DX's AI Measurement Framework covering utilization, impact, and cost, and stresses trust-building through system prompt feedback loops and temperature settings. The episode delivers actionable guidance on measuring AI's true impact and enabling engineers through education, time to learn, and creative unblocking of usage.

Paying Engineers like Salespeople – Arman Hezarkhani, Tenex

Dec 19, 2025 · 14:53

Arman Hezarkhani, CTO of Tenex, describes his company's outcome-based compensation model where engineers are paid per story point completed, tying incentives directly to shipped value. He argues that hourly billing and salary with equity fail to motivate engineers to use AI tools effectively, leading to misaligned incentives. Tenex's system involves strategists scoping tickets and engineers receiving a flat base plus quarterly story-point bonuses. Risks like inflated story points, quality drops, and sharp elbows are mitigated by strategist counterbalance, multi-round QA, and a rigorous hiring process. He cites examples: a billboard company's image moderation AI built in two weeks at 96% accuracy, and a retail heat-mapping project with five parallel models. Arman asserts that AI amplifies existing talent, and current compensation must change to unlock potential.

Building in the Gemini Era – Kat Kampf & Ammaar Reshi, Google DeepMind

Dec 15, 2025 · 17:57

Kat Kampf and Ammaar Reshi from Google DeepMind present Gemini 3 and Nano Banana Pro, arguing that these models enable anyone to build complex, aesthetic applications through natural language alone. Gemini 3 achieves state-of-the-art results in one-shot UI design and agentic tool calling, with SweBench outperformance. Nano Banana Pro integrates Google Search for world knowledge and renders text accurately, handling up to 14 consistent people per image. They demonstrate vibe coding in AI Studio, building a personalized comic book with precise text, laptop stickers grounded in search, and a 3D racing game that scaled to 23 players live. Upcoming full-stack runtime adds backend support and automatic database integration, further democratizing software creation.

Five hard earned lessons about Evals — Ankur Goyal, Braintrust

Aug 23, 2025 · 19:46

Ankur Goyal of Braintrust argues that successful AI applications depend on deliberately engineered evaluations (evals) to reflect real user feedback and drive product improvements. He details three signs of effective evals: launching updates within 24 hours (citing Notion), converting complaints into evals, and using evals offensively to assess use cases before shipping. Evals require custom scorers as specs—not off-the-shelf—and context engineering (optimizing tool definitions and outputs) is the new frontier; shifting outputs from JSON to YAML improves token efficiency. New models can upend everything, as shown by a benchmark that jumped from 10% to viable with Claude 4 Sonnet, so a model-agnostic architecture with proxies is key. Optimize the whole system (data, task, scoring)—Braintrust's Loop auto-optimizes prompts, data, and scorers. In Q&A, he advises human judgment when adding user feedback to evals to avoid overfitting.

State of Startups and AI 2025 - Sarah Guo, Conviction

Aug 2, 2025 · 23:52

Sarah Guo of Conviction argues that AI's value creation is massive and early, with companies like Cursor reaching $100M ARR in 12 months and Harvey exceeding $70M ARR. She predicts that by end of 2026, AI agents will ship code directly to production, voice AI will replace text for most business communication, and inference costs will drop below a cent per million tokens. Reasoning is a new scaling vector unlocking higher-stakes use cases, and agent startups have increased 50% in the last year. Multimodal models from HeyGen and Eleven are already rocketing past $50M ARR. The model market is more competitive than ever, with GPT-4 costs falling from $30 to $2 per million tokens in 18 months and open-source like DeepSeek competing. Guo advises builders to focus on thick wrappers around LLMs, leveraging domain and workflow knowledge, and warns against generic text boxes: 'The prompt is a bug, not a feature.' Execution, not first-mover advantage, is the moat.

Useful General Intelligence — Danielle Perszyk, Amazon AGI

Aug 2, 2025 · 19:58

Danielle Perszyk, a cognitive scientist at Amazon's AGI SF Lab, argues that general-purpose AI agents like Nova Act must be built to augment human intelligence rather than replace it, drawing parallels to how human cognition relies on controlled hallucinations and social co-evolution. She explains that current agents struggle with basic computer tasks like clicking, typing, and scrolling, so Nova Act combines a specialized version of Amazon Nova trained via RL for UI reliability with an SDK that allows developers to string together atomic actions (act calls) into complex workflows. The lab's vision centers on grounding agents in a shared environment and eventually giving them models of human minds, which requires a common language for humans and agents to reciprocally level up intelligence. Demos with teammates Carolyn (searching apartments) and Fjord (extracting data with Python and parallel browsers) show how Nova Act enables three-line code starts and granular control, positioning the product as a primitive for a cognitive technology that aligns representations and stabilizes human thinking.

Why We Don’t Need More Data Centers - Dr. Jasper Zhang, Hyperbolic

Aug 1, 2025 · 14:18

Dr. Jasper Zhang, CEO of Hyperbolic, argues that the AI industry's focus on building more data centers is inefficient because GPUs already sit idle 80% of the time. Instead, he advocates for a GPU marketplace that aggregates supply from fragmented providers, cutting costs by 50–75%—Hyperbolic offers H100s at $0.99/hour versus $2–$11 from traditional clouds. His HyperDOS software connects any Kubernetes cluster in 5 minutes, enabling dynamic rental and release of GPUs. He illustrates with a startup that can reduce compute costs from $43.8 million to $6.9 million by flexibly scaling capacity and selling idle GPUs, effectively boosting productivity 6x on the same budget. Zhang contends that smarter allocation, not just more data centers, is key to sustainable and democratized AI infrastructure.

How to defend your sites from AI bots — David Mytton, Arcjet

Jul 30, 2025 · 20:12

David Mytton, founder of Arcjet, explains that nearly 50% of web traffic is now automated, with AI crawlers significantly worsening the problem for site owners. He cites specific cases: Diaspora saw 24% of its traffic from OpenAI's GPTbot, and ReadTheDocs cut bandwidth from 800 GB to 200 GB per day by blocking AI crawlers. Mytton describes the evolving landscape of AI bots—OpenAI alone has at least four types (Search, ChatGPT User, GPTbot, Operator)—and argues that detection requires layered defenses. He walks through eight defenses: robots.txt (voluntary), user agent string verification with reverse DNS, IP reputation (e.g., 12% of bot traffic on Cloudflare comes from AWS), CAPTCHAs (increasingly easy for AI to solve), proof-of-work puzzles (e.g., Anubis proxy), experimental HTTP message signatures (Cloudflare's proposal), fingerprinting via JA4 hash (TLS-level), and rate limiting keyed to fingerprints or session IDs. The practical takeaway is that most sites can manage with user agent verification plus IP checks, but high-value or high-traffic sites need full stacking of these techniques.

Layering every technique in RAG, one query at a time - David Karam, Pi Labs (fmr. Google Search)

Jul 29, 2025 · 20:22

David Karam of Pi Labs (formerly Google Search) walks through layering every technique in RAG, from in-memory retrieval to planet-scale search with 70+ corpus mix of token, embeddings, and knowledge graphs, jointly retrieved and re-ranked at 160,000 queries per second in under 200msec. He advocates a Quality Engineering Loop: baseline with simplest methods, analyze losses, then apply incremental techniques based on complexity-adjusted impact. Karam explains why queries like "falafel" are notoriously hard due to ambiguous intent, highlights failures of chunking documents, and shows when BM25 suffices, when relevance embeddings are needed, and when custom embeddings or domain-specific signals (price, popularity, user clicks) become essential. He covers query orchestration via fan-out, supplementary retrieval across back ends, and distillation for cost optimization, concluding that at sufficient complexity, problems must be punted to LLM or UX for graceful degradation.

[Full Workshop] Building Metrics that actually work — David Karam, Pi Labs (fmr Google Search)

Jul 29, 2025 · 40:28

Achim and David Karam, former Google Search leaders now at Pi Labs, argue that building reliable AI evaluations requires a multi-signal scoring system rather than single LLM-as-judge metrics. They walk through designing custom metrics starting with simple correlated signals, using a copilot to generate and iterate on dimensions like 'includes insights' or 'title length,' then calibrating them against ground truth thumbs up/down data from users. The workshop demonstrates applying this to a meeting summarizer, integrating the scoring system into Google Sheets to score 120 examples and produce a confusion matrix showing alignment with human feedback. They also show how to use the same scoring system for model comparison (e.g., GPT-1.5 vs 2.5) and online best-of-N sampling, which boosts response quality by generating multiple responses and selecting the highest-scored one.

Everything is ugly, so go build something that isn't — Raiza Martin, Huxe (ex NotebookLM)

Jul 28, 2025 · 25:15

Raiza Martin, former lead of Google's NotebookLM and founder of Huxe, argues that the current chaotic phase of AI product design is a once-in-a-career opportunity to rebuild from first principles, calling everything we use 'the ugliest that it will ever be.' Drawing from her experience forcing NotebookLM into existence against skepticism, she emphasizes that personal clarity of vision fuels product building, and purpose must be relentlessly focused on a single outcome—for NotebookLM, enabling users to upload 50 files and interact with them. She stresses earning trust by nailing deterministic behavior first, noting that 90% of first queries were summarization and failures drove users away forever, then layering on delightful probabilistic features like podcast generation. Finally, she warns against the 'kitchen sink' approach, citing her own Huxe app that did everything but users only used one feature, advocating restraint as an innovation multiplier and focus on one excellent outcome to avoid building ugly products.

Waymo's EMMA: Teaching Cars to Think - Jyh Jing Hwang, Waymo

Jul 26, 2025 · 17:28

Jyh-Jing Hwang of Waymo presents EMMA, an end-to-end multimodal model built on Gemini that directly processes camera inputs and route text into driving waypoints, achieving state-of-the-art open-loop planning on the NuScenes benchmark without LiDAR or HD maps. Adding chain-of-thought reasoning—identifying critical objects and meta-decisions—further surpasses specialized models like MotionLM on Waymo's 100K dataset. Co-training on tasks such as 3D detection and road graph estimation maintains competitive performance across domains. For evaluation, Waymo leverages generative video models (e.g., Veo2) to simulate diverse conditions like rain and night, testing EMMA's robustness. The work demonstrates how Gemini's generalization can help scale Waymo's autonomous driving to new cities by handling long-tail scenarios.

Your Coding Agent Just Got Cloned And Your Brain Isn't Ready - Rustin Banks, Google Jules

Jul 25, 2025 · 13:40

Rustin Banks, an AI product manager at Google Labs, shares how Jules, Google's asynchronous coding agent powered by Gemini 2.5 Pro, enables developers to shift from sequential coding to orchestrating many parallel agents. He demonstrates real-world use on a conference schedule website, adding tests with Jest and Playwright simultaneously, implementing features like Google Calendar integration and AI summaries, and running accessibility audits. Banks argues that success requires clear verification criteria, creating 'disagreement' with agents to ensure correctness, and a robust merge/test framework. He notes that in its first two weeks, Jules generated 40,000 public commits, and emphasizes that agents-as-a-service in the cloud allow infinite scalability and development from any device.

Latent Space Paper Club: AIEWF Special Edition (Test of Time, DeepSeek R1/V3) — VIbhu Sapra

Jul 25, 2025 · 53:54

VIbhu Sapra presents DeepSeek's latest models and announces a new curriculum-based Test of Time Paper Club. DeepSeek's May 28th R1 update doubles reasoning tokens (12k→25k) and matches O3/Gemini 2.5 Pro, improving AIME 2024 from 70% to 87.5%. A new distillation of R1 traces into Qwen 3 8b achieves performance on par with Qwen's 235b thinking model, proving reasoning models distill efficiently. The Test of Time Paper Club will cover 50–100 foundational papers over six months in San Francisco and remote, targeting core AI engineering topics from attention and scaling laws to inference techniques. Sapra emphasizes that pure RL (GRPO) on verifiable math and code unlocks emergent reflection and aha moments, shifting scaling from pre-training compute to inference-time reasoning.

Human seeded Evals — Samuel Colvin, Pydantic

Jul 25, 2025 · 12:02

Samuel Colvin, creator of Pydantic, demonstrates how type safety and validation loops in Pydantic AI build reliable GenAI applications. He argues that type-safe frameworks enable confident refactoring, a necessity for evolving AI apps, and shows how returning validation errors to the model fixes simple mistakes like incorrect date formats. The agentic loop with tools and final result tools ensures structured extraction, while type-safe dependencies on tools prevent runtime errors. Colvin also highlights Logfire observability, which traces calls, shows pricing, and helps debug failures like failed memory retrieval. The talk covers agent definitions, validation error retries, and the importance of explicit exit conditions in agent loops.

AI That Pays: Lessons from Revenue Cycle — Nathan Wan, Ensemble Health

Jul 24, 2025 · 18:19

Nathan Wan, head of AI at Ensemble Health Partners, argues that Revenue Cycle Management (RCM) is a critical yet overlooked domain for AI disruption in healthcare, where 40% of hospitals operate at negative margins due to broken manual financial processes. He explains that RCM's vast structured and unstructured data, rule-based workflows, and direct financial impact make it ripe for AI, contrasting it with more publicized clinical AI applications. Wan details how Ensemble uses AI to predict and prevent claim denials—often caused by technical errors like missing data—and to accelerate clinical denial appeals via GenAI, achieving a 40% reduction in processing time and improved overturn rates. He emphasizes that Ensemble's end-to-end view of the revenue cycle, supported by their EIQ data platform, enables longitudinal error correction and agent-driven automation, turning upstream fixes into measurable ROI for providers facing increasing payer denial rates.

Structuring a modern AI team — Denys Linkov, Wisedocs

Jul 24, 2025 · 17:40

Denys Linkov, who leads ML at Wisedocs, argues that building a modern AI team hinges on identifying your company's bottleneck—shipping features, acquiring users, or scalability—rather than reflexively hiring AI researchers. He introduces Ampere's Wager: trading your entire domain-savvy team for five top-lab researchers is usually a losing bet. For early-stage AI strategy, generalists who blend model training, serving, and business acumen outperform specialists; Linkov lived this in 2021 building a custom MLOps platform for a conversational AI startup and again in 2024 using advanced open-source tools for medical record processing. He stresses reskilling existing teams through weekly learning cadences and moving domain experts from giving feedback to writing evaluations. Hiring should hold context and act on it, verifying trends like 'don't hire juniors' against YC's AI school drawing 2,000 young people.

How to Hire AI Engineers when EVERYONE is cheating with AI — Beth Glenfield, DevDay

Jul 22, 2025 · 6:45

Beth Glenfield, from DevDay, argues that AI has broken technical hiring because candidates cheat with AI tools, making LeetCode puzzles obsolete, and small companies are crushed in the talent war with big tech. She cites the AI cheating service CLUI, which raised $5.3M and is heading toward $1M ARR, and notes 93% LeetCode wizard success rates for Google/Meta interviews, with 1 in 3 interviews now using AI assistants. Instead of puzzles, she proposes real-world workplace simulations where candidates collaborate with AI agents (perfectionist, pragmatist, etc.) to measure skills like handling ambiguity, mentoring, and business judgment. She highlights that big tech can brute-force hiring (100 candidates for 5 hires), but startups cannot afford bad hires costing $20k-$60k. Referencing Mark Zuckerberg's prediction that AI will handle mid-level engineering and Marc Benioff's announcement that Salesforce won't hire software engineers due to a 30% productivity boost from AI, Glenfield argues engineering jobs now require creativity, collaboration, and working with AI rather than being replaced by it.

Books reimagined: AI to create new experiences for things you know — Lukasz Gandecki, TheBrain.pro

Jul 22, 2025 · 9:44

Łukasz Gandecki of TheBrain.pro presents Books Reimagined, an AI-powered reading companion that adds music, graphics, and conversational AI to books like '1984' and 'The Snow Queen', arguing that AI can create magical experiences by hiding itself behind a polished user interface. He started by vibe-coding a companion for a Trump reelection book to track characters, then iterated into a full multimedia reader. He demonstrates features including 100-millisecond voice response, natural language search via embeddings, and deep research that scans the entire book. Gandecki emphasizes that human curation of AI-generated music and graphics is essential for quality, and he is open-sourcing the player at bookgenius.net.

tldraw.computer - Steve Ruiz, tldraw

Jul 21, 2025 · 18:45

Steve Ruiz, founder and CEO of tldraw, demonstrates the company's AI experiments on their infinite canvas, including Make Real—which turns hand-drawn wireframes into working web apps using vision models—and tldraw computer, a visual programming environment where arrows and LLMs power a graph of connected nodes that can execute multi-step prompts, generate images and speech, and even run loops indefinitely. He also shows Draw Fast for real-time image generation and Teach, where Claude can draw and edit shapes on the canvas. The episode explains how tldraw's SDK (tldraw.dev) enables others to build custom canvas applications, and highlights the company's philosophy of 'shitty but amazing' rapid prototyping.

CIAM for AI: Authn/Authz for Agents — Michael Grinich, CEO of WorkOS

Jul 21, 2025 · 20:13

Michael Grinich, CEO of WorkOS, argues that AI agents need first-class identity and access management, distinct from human or machine-to-machine auth. He identifies key challenges: headless login, least privilege for non-deterministic systems, and compliance tracking. Grinich presents four emerging patterns—persona shadowing, delegation chains, capability tokens, and human-in-the-loop escalation—and references standards like OAuth, UMA, GNAP, OIDC for agents, and verifiable credentials. He predicts a shift from 95% human traffic to 95% agent traffic, calling for middleware trust boundaries and urgent collaboration on agent identity standards.

[Full Workshop] Reinforcement Learning, Kernels, Reasoning, Quantization & Agents — Daniel Han

Jul 19, 2025 · 2:42:28

Daniel Han of Unsloth presents a technical workshop covering reinforcement learning (RL), kernels, reasoning, quantization, and agents, arguing that RL with verifiable rewards (RLVR) is the key to unlocking LLM capabilities beyond supervised fine-tuning. He explains why open-source models plateaued after September 2024 until DeepSeek-R1 showed that RL can elicit reasoning, and breaks down PPO, GRPO, and the REINFORCE algorithm, emphasizing that GRPO removes the value model for efficiency. Han details how reward functions—not algorithms—are the hardest part, with examples like distance-based scoring for math. He demonstrates a free Colab notebook training a base model to reason, and shows that dynamic quantization can shrink models like DeepSeek-R1 from 730 GB to 140 GB with only ~1% accuracy loss, arguing that GPUs may stop getting faster after FP4 precision.

The rise of the agentic economy on the shoulders of MCP — Jan Curn, Apify

Jul 18, 2025 · 18:08

Jan Curn, founder of Apify, argues that MCP (Model Context Protocol) enables a future agentic economy where AI agents autonomously discover and purchase tools from other agents or businesses (B2A/A2A). He explains that Apify's marketplace of 5,000 Actors (Docker-based tools) now integrates with MCP, allowing agents to dynamically discover and call any Actor via tool discovery — a key MCP feature. Curn demonstrates this with Claude Desktop, where an agent uses Apify's MCP server to find a venue, scrape Twitter, and even fill a form via a nested MCP server from Browserbase, all without prior configuration. He notes that Apify pays creators over $250,000 monthly, with total Actor revenue exceeding $1.5M/month, and that any developer can publish an Actor to monetize their tools instantly across the ecosystem. The talk closes with open questions about reliability, trust, and whether autonomous agent interaction can lead to AGI.

The State of Generative Media - Gorkem Yurtseven, FAL

Jul 16, 2025 · 17:14

Gorkem Yurtseven, co-founder and CTO of fal.ai, argues generative media lowers creation's marginal cost to near zero, transforming advertising and e-commerce. He traces evolution from DALL·E 2 to open-source models like Stable Diffusion and FLUX, with the playing field evening quickly. Video models (Sora, Veo 3) are accelerating: fal's platform saw video usage jump from near zero to 30% of revenue, and he predicts the generative video market 100x-250x bigger than image. Advertising leads with hyper-personalized, interactive ads, like fal's A24 campaign turning selfies into toy soldiers. In e-commerce, virtual try-on is a proven product-market fit. Future real-time video generation will blur games and movies.

RL for Autonomous Coding — Aakanksha Chowdhery, Reflection.ai

Jul 16, 2025 · 19:27

Aakanksha Chowdhery, CEO of Reflection AI and former lead researcher on PaLM and Gemini at Google, argues that reinforcement learning (RL) scaling is the next frontier for autonomous coding agents. She explains that inference-time techniques like majority voting and self-revision improve accuracy but require many samples (e.g., 10,000 for rare correct generations). RL training can learn to generate correct outputs directly, especially in verifiable domains like code with unit tests. Reflection AI aims to build superintelligence starting with autonomous coding, leveraging automated verification to design better reward functions. She notes challenges in scaling RL, including system complexity and reward hacking, but sees coding as ideal due to execution feedback.

Benchmarks Are Memes: How What We Measure Shapes AI—and Us - Alex Duffy, Every.to

Jul 15, 2025 · 15:44

Alex Duffy argues that AI benchmarks function as cultural memes—ideas that spread and shape what models learn—giving those who design them immense power over AI's trajectory. He traces the lifecycle from a single person's idea to saturation, using examples like 'How many Rs in strawberry' and Pokémon. Duffy introduces AI Diplomacy, a benchmark where language models negotiate and betray each other, revealing that models like DeepSeek R1 and Gemini 2.5 Flash excel at social manipulation while Claude models are naively optimistic. He warns against benchmarks that reward sycophancy (like ChatGPT's thumbs-up training) and advocates for multifaceted, experiential, and generative benchmarks that empower people. Duffy urges the audience to ask non-AI people what they care about, turning benchmarks into tools that build trust and define humanity's role in an AI world.

AI Engineering with the Google Gemini 2.5 Model Family - Philipp Schmid, Google DeepMind

Jul 11, 2025 · 1:44:51

In this workshop, Philipp Schmid from Google DeepMind demonstrates AI Engineering with the Gemini 2.5 model family, focusing on using Gemini 2.5 Flash via a free API tier for hands-on coding tasks including text generation, multimodal processing of images, audio, and PDFs, function calling with structured outputs, and integration with MCP servers. The session covers setting up API keys in AI Studio, uploading files via the Files API (free for 1 day), and controlling thinking budgets (0–24,000 tokens) to manage cost and reasoning depth. Schmid shows how Gemini natively processes videos at 1 frame per second for accurate timestamp extraction and how PDFs are handled by combining OCR text with image understanding. He introduces native tools like Google Search with grounding metadata, code execution, and URL context, and explains how MCP servers can be used seamlessly with the Gemini SDK for tool calling. The workshop also covers parallel vs sequential function calling, the Agent Development Kit (ADK), and the upcoming asynchronous function calling for the Live API, providing a practical path from simple generation to agentic workflows.

Thinking Deeper in Gemini — Jack Rae, Google DeepMind

Jul 10, 2025 · 18:13

Jack Rae, lead of Gemini Thinking at Google DeepMind, presents thinking as a solution to the fixed test-time compute bottleneck in large language models. He explains that Gemini inserts a thinking stage where the model iterates via reinforcement learning, learning to self-correct and explore multiple strategies. This enables a continuous cost-performance tradeoff via thinking budgets, improving reasoning on math and code. Future work includes Deep Think, which raises USA Math Olympiad performance from the 50th to the 65th percentile by scaling inference compute further. Rae envisions models that, like mathematician Ramanujan, achieve deep, data-efficient reasoning from limited knowledge.

A year of Gemini progress + what comes next — Logan Kilpatrick, Google DeepMind

Jul 10, 2025 · 11:58

Logan Kilpatrick, head of product for Google AI Studio at DeepMind, announces the final update to Gemini 2.5 Pro, which achieves state-of-the-art results on Aider and HLE benchmarks. He details Google's 50x increase in AI inference over the past year, driven by merging research and product teams into DeepMind. The episode outlines Gemini's evolution toward a universal assistant that unifies Google products, with upcoming features including proactivity, native audio and video capabilities (Veo), and smaller models. Kilpatrick also previews developer-focused updates: a SOTA embeddings model, a deep research API, and Veo 3 and Imagine 4 in the API, alongside repositioning AI Studio as a dedicated developer platform.

2025 in LLMs so far, illustrated by Pelicans on Bicycles — Simon Willison

Jul 9, 2025 · 18:30

Simon Willison reviews the past six months of LLM releases — including AWS Nova, Llama 3.3 70B, DeepSeek R1, Mistral Small 3, Claude 3.7 Sonnet, GPT 4.5, Gemini 2.5 Pro, GPT-4o, Llama 4, GPT 4.1, O3/O4 Mini, and Claude 4 — using his 'pelican on bicycle' SVG benchmark to argue that local models have become good enough to run GPT-4 class models on a laptop and that combining tools with reasoning is the most powerful technique in AI engineering, while noting risks like prompt injection and the 'lethal trifecta'. He tracks 30 significant model releases, highlighting that Mistral Small 3 (24B) matches Llama 3 70B's performance, which itself matched the 405B model, enabling local inference. DeepSeek's R1 caused a $500B+ Nvidia stock drop on January 27. GPT 4.1 Nano is the cheapest model yet at a fraction of a cent per pelican. He also examines bugs: ChatGPT's sycophantic 'shit-on-a-stick' incident and Claude 4's tendency to snitch to authorities when given ethical instructions and email tools. Willison concludes that while the pace is accelerating, control over context and security remain critical.

Milliseconds to Magic: Real‑Time Workflows using the Gemini Live API and Pipecat

Jun 27, 2025 · 21:43

Kwindla Kramer (Daily, Pipecat) and Shrestha Basu Mallick (Google DeepMind, Gemini API) argue that voice is the most natural interface and that the Gemini Live API combined with the Pipecat framework enables developers to build real-time multimodal voice agents covering the full stack from models to application code. They demo a voice-driven task management app that handles grocery, reading, and work lists, showing impressive context-aware tool use (e.g., consolidating lists, web searching for 'Dream Count' author) but also jagged edge limitations like turn detection errors and persistent misspelling of 'Kwin.' They discuss how capabilities like turn detection migrate down the stack over time. The episode also covers proactivity, multilinguality, telephony integration, and experimental native audio models for emotive, steerable dialogue.

Agentic GraphRAG: Simplifying Retrieval Across Structured & Unstructured Data — Zach Blumenfeld

Jun 27, 2025 · 15:25

Zach Blumenfeld (Neo4j) demonstrates how a knowledge graph simplifies agentic retrieval across structured and unstructured data, using an employee skill graph as an example. He shows that a plain document search fails to answer questions like "how many Python developers" (returns 5 instead of 28) or "who is most similar to Lucas Martinez" — but after extracting entities into a Neo4j graph, the agent can run precise Cypher queries via an MCP server, correctly reporting 28 Python devs and identifying Sarah as the closest match via skill overlap. He also shows how to flexibly add new relationships (e.g., project collaborators from an HR system) without schema changes, enabling questions like "which individuals collaborated most on AI projects" to return exact pairings like Sarah and Amanda. The talk emphasizes accuracy, explainability, and the ability to start with a simple data model and expand as agents ingest more data.

MCP: Origins and Requests For Startups — Theodora Chu, Model Context Protocol PM, Anthropic

Jun 18, 2025 · 17:45

Theodora Chu, product manager at Anthropic, explains the origin of the Model Context Protocol (MCP) as an open-source standard for giving LLMs agency to interact with external tools and data, born from two engineers copying context manually. She details MCP's evolution from an internal hack week project to wide adoption by Google, Microsoft, OpenAI, and coding tools like Cursor and VS Code. Key protocol decisions include optimizing for server simplicity over client complexity, adding streamable HTTP for bidirectionality, and fixing authentication via community contributions. Chu outlines three startup opportunities: building high-quality MCP servers for verticals beyond dev tools (80% weight), simplifying server building with hosting/testing/deployment tooling, and creating AI security and observability tools. She emphasizes that models are a third user of servers, so tool design must consider end users, client developers, and the model itself.

Exposing Agents as MCP servers with mcp-agent: Sarmad Qadri

Jun 11, 2025 · 18:05

Sarmad Qadri, CEO of Lastmile AI, argues that agents should be exposed as MCP servers, enabling multi-agent composition and async workflows. He introduces mcp-agent, a framework implementing Anthropic's effective agent patterns—augmented LLM, evaluator-optimizer, orchestrator—on top of the Model Context Protocol. Qadri demonstrates an agent that reads a student story, fetches an APA style guide via a fetch MCP server, and writes a graded report, all orchestrated as a durable Temporal workflow. He explains that MCP standardizes tool connections for LLMs, shifting agentic behavior from client-side to server-side, allowing agents to be invoked from any MCP client like Claude or Cursor. The talk covers how this paradigm enables scalable, platform-agnostic agents that can be paused, retried, and scheduled.

RAG Evaluation Is Broken! Here's Why (And How to Fix It) - Yuval Belfer and Niv Granot

Jun 3, 2025 · 10:58

Yuval Belfer and Niv Granot of AI21 Labs argue that current RAG evaluation is broken because benchmarks rely on local questions with answers contained in single chunks, failing to reflect real-world messy data. They demonstrate that standard RAG pipelines—like those from LangChain and LlamaIndex—achieve only 5-11% accuracy on aggregative questions about 22 FIFA World Cup documents, such as 'Which team has won the most times?' Their proposed fix converts unstructured corpuses into structured SQL databases by clustering documents, inferring schemas (e.g., year, winner, top scorer), and using text-to-SQL at inference. This structured RAG approach handles counting and max/min queries that standard chunking cannot, though it requires homogeneous data and careful normalization to avoid ambiguity (e.g., West Germany vs. Germany). They caution that RAG is not one-size-fits-all and that existing benchmarks miss these critical use cases.

From PM at Stripe to Building an AI startup, a recent founder's journey - Mounir Mouawad

Jun 3, 2025 · 11:59

Mounir Mouawad, CEO and co-founder of Porsche AI, explains how building an AI startup differs from product roles at Stripe, Google, and Amazon, using video game analogies. He argues user problems are an 'emergent property' requiring hypothesis-driven iteration rather than conventional roadmaps. Product development is gratifying with releases in hours or days, but velocity is a 'stable stick' as opportunities like MCP come and go quickly. The hardest part is outreach without big brand support—like playing Crash Bandicoot without boosters—so he finds people followers, advocates, and partnerships (e.g., with Browserbase) essential. He asks listeners to star Porsche AI's GitHub repo.

Letting AI Interface with your App with MCP — Kent C Dodds

Jun 3, 2025 · 19:54

Kent C Dodds explains how Model Context Protocol (MCP) enables AI assistants like Claude Desktop to interface with any app or service, moving beyond limited built-in integrations. He traces the evolution from early LLM chat (phase one) through host applications with custom integrations (phase two) to MCP's open standard (phase three), arguing that MCP lets developers build once for any assistant. In his live demo, he runs a Locationator server to derive his location, a GetWeather server for weather, and an Epic Me server with OAuth authentication to create a journal entry. MCP servers handle tools, resources, and prompts; the client standardizes communication while the server controls unique logic. Kent contends this protocol is how we give AI "hands" to perform real tasks, leading toward a universal Jarvis-like assistant that can dynamically connect to any service without manual integration per platform.

Vercel AI SDK Masterclass: From Fundamentals to Deep Research

Apr 20, 2025 · 59:52

Nico Albanese from Vercel delivers a workshop on building agents with the AI SDK, demonstrating how to use generateText, tools, and structured outputs to create a deep research clone in Node.js. He shows that switching between models like GPT-4o mini, Perplexity Sonar Pro, and Gemini Flash 1.5 requires only changing one line of code. Tools with maxSteps enable autonomous multi-step agents, as when the model fetches weather data for San Francisco and New York and sums the temperatures across three steps. The deep research workflow generates subqueries (e.g., 'requirements to become a D1 shotput athlete'), searches via Exa with live crawl, evaluates relevance with an evaluate tool that discards irrelevant results, and recursively deepens research using depth and breadth parameters. Finally, o3 mini synthesizes the accumulated research into a structured Markdown report, all in 218 lines of code.

Insights on Building AI Teams — Heath Black, SignalFire

Apr 15, 2025 · 20:30

Heath Black, Managing Director of Product at SignalFire, uses Beacon platform data to guide AI team building, arguing that credentialism is declining—only 7% of AI engineers had PhDs in 2023 versus 16% in 2015—and that work experience now outweighs education. He shows AI talent concentrates in San Francisco (35% of AI engineers), Seattle (22%), and New York (10%), and that tracking retention rates (e.g., Anthropic at 66% four-year retention vs. Perplexity at 43%) helps time outreach. Black advises hiring based on a candidate's body of work, removing academic requirements from postings, and understanding generational job-hopping (27% of Gen Z left jobs in 2023). He warns against relying solely on salary and equity, as AI engineers command 5% salary and 10–20% equity premiums, and recommends narratives centered on mission, speed, and collaborative teams. The talk emphasizes using data to filter, locate, time, and close hires effectively.

AI Engineers: The Next Generation — Stefania Druga, Google Gemini

Apr 13, 2025 · 21:47

Stefania Druga, a research scientist on Google Gemini, presents Cognimates and its new AI copilot that teaches children to become AI engineers. Built on Scratch’s visual programming, Cognimates lets kids train custom models, program robots, and build games, shifting their perception from ‘AI is magic’ to understanding data and confidence levels. Druga’s design studies with 18 kids from 11 countries showed the copilot doubled programming time by supporting ideation, debugging, and creative agency—never giving answers unless stuck three times. The tool now integrates screenshot analysis and asset generation, with plans to embed the agent across the OS. Druga ties this to the EU AI Act’s mandate for AI literacy, arguing early, hands-on creation is the path to informed users.

How to Fail at AI Strategy: Hamel Husain & Greg Ceccarelli

Apr 13, 2025 · 17:03

Greg Ceccarelli and Hamel Husain argue that the most reliable path to AI failure is to follow a set of inverted worst practices, including cultivating disconnect between executives and builders, promising unrealistic AI capabilities, drowning communication in jargon, and avoiding data analysis. They describe how to divide your company by incentivizing secrecy and using jargon like 'agents' to exclude domain experts, ensuring that AI projects are disconnected from real needs. The speakers advocate faking strategy by highlighting random paragraphs from last year's report, announcing vague goals like 'become the global AI leader in everything,' and creating a massive backlog with no timeline. They recommend throwing tools at problems—buying expensive vector databases or switching frameworks—without understanding root causes, and blindly trusting off-the-shelf evaluation metrics like BLEU and ROUGE. Crucially, they insist on never looking at data, using complex systems inaccessible to domain experts, and trusting gut feelings over evidence. This inverted guide guarantees wasted resources, alienated teams, and spectacular failure.

The Agent Development Life Cycle — Zack Reneau-Wedeen, Sierra

Apr 11, 2025 · 18:40

Zack Reneau-Wedeen from Sierra explains the company's Agent Development Lifecycle for building reliable, testable AI agents at scale for brands like Chubbys and SiriusXM. He contrasts LLMs' nondeterministic, slow nature with traditional software, calling them a 'foundation of Jello.' Sierra treats every agent as a product, using an experience manager to review conversations, file issues, create tests, and release improvements—growing from hundreds of thousands of requests for Chubbys to tens of millions for larger customers. The lifecycle spans quality assurance, testing, and deployment, with reasoning models acting as a force multiplier. Voice agents launched generally in October 2024, handling calls with the same underlying platform, enabling responsive design across channels.

How Deep Research Works - Mukund Sridhar & Aarush Selvan, Google DeepMind

Mar 26, 2025 · 15:15

Aarush Selvan and Mukund Sridhar from Google DeepMind explain how Gemini Deep Research works, a personal research agent that trades latency for comprehensiveness by taking up to five minutes to browse the web and synthesize reports. They discuss product challenges like building an asynchronous experience in a synchronous chatbot, setting user expectations, and presenting thousand-word outputs. Technical challenges covered include iterative planning with partial information, handling the fragmented web with entity resolution, managing growing context via recency-biased retrieval, and ensuring robustness to intermediate failures. The episode also explores future directions: going from aggregating information to providing expert-level insights, personalizing research to user roles, and combining web research with coding, data science, or video generation.

Beyond APIs: How AI Web Agents Are Automating the "Long Tail" of Knowledge Work

Feb 22, 2025 · 17:44

Arjun and Bhavani present Rtrvr.ai, a universal AI web agent using a text-based approach to autonomously perform tasks across multiple browser tabs at under a penny per page. They argue text-based reduces hallucination compared to vision-based agents like OpenAI's operator, and being a Chrome extension avoids password sharing and accesses logged-in content. Features include deep research navigating multiple pages, dynamic function calling for any third-party API, and graph generation. Rtrvr extracts structured data, performs actions like clicking dropdowns, and processes tabs simultaneously, automating the long tail of knowledge work including market research, LinkedIn automation, and WhatsApp messaging. They claim a distributed subtask approach lowers failure rates and envision collaborative dataset construction, e.g., aggregating local government events.

Lets Build An Agent from Scratch — Kam Lasater

Feb 22, 2025 · 19:29

Kam Lasater demonstrates that an AI agent is simply an LLM paired with memory, planning, tools, and a while loop, and shows how to build one from scratch in incremental steps. He starts with a basic LLM call (step 0), adds a condition using an LLM as judge to check if an answer is complete (step 1), then incorporates a tool for web search via SERP AI (step 2). After refactoring for parallel tool calls (step 3), he introduces a to-do list for planning and tracking progress, along with a browse-web tool (step 4). The agent uses a to-do list to plan, execute tasks like searching and browsing, and then checks completion via LLM as judge. Lasater illustrates with examples: finding the average wind speed of a swallow, buying a hoodie near Times Square, and planning a date in Ardmore. He emphasizes that the simplest agent emerges naturally from combining these components, and encourages running and breaking the provided code.

How Coding Agents change Software Development Forever - Hailong Zhang

Feb 22, 2025 · 8:50

Hailong Zhang presents how coding agents, particularly the unit test agent Gru, transform software development by automating routine tasks while humans focus on creative work. In the future workflow, synchronous agents like GitHub Copilot and Cursor assist in real-time, while asynchronous agents like Gru autonomously handle tasks such as writing tests, fixing bugs, and submitting pull requests. Gru, built to boost unit tests, detects code changes from pull requests, generates and runs tests, and submits a new PR with a summary and coverage improvement. In production, over 50% of Gru's PRs are accepted by humans, and it handles 80% of unit tests in its own repo, making it the top contributor. To build such agents, Zhang emphasizes defining a clear problem (e.g., unit tests, not generic software engineering), creating datasets and evaluation harnesses, selecting and fine-tuning LLMs per stage, building task-specific context from environment data, and developing an agent operating system to reuse common infrastructure across different tasks.

The LLM Triangle: Engineering Principles for Robust AI Applications - Almog Baku:

Feb 22, 2025 · 26:19

Almog Baku introduces the LLM Triangle Principles for building production-ready AI applications, arguing that LLMs should be treated like interns guided by Standard Operating Procedures (SOPs). He breaks down the triangle into three components: the foundation model, engineering techniques, and contextual data, all directed by SOPs derived from expert interviews. Baku contrasts autonomous agents—creative but unpredictable—with handcrafted LLM-native architectures that offer sustainable quality, recommending scoped autonomy. He advises starting with large models to collect data and optimize incrementally, rather than fine-tuning from day one. Data is paramount: 'show, don't tell' via few-shot examples, and balance context to avoid the 'needles in a haystack' problem. The framework aims to turn demos into robust production systems.

Customized, production ready inference with open source models: Dmytro (Dima) Dzhulgakov

Feb 16, 2025 · 18:55

Dmytro (Dima) Dzhulgakov, co-founder and CTO of Fireworks AI, argues that open source models are the future for production Gen AI applications, and Fireworks provides a platform to make them customized and production-ready. He explains that while proprietary models like GPT-4 are powerful, they are too large, expensive, and slow for many use cases, whereas open models like Llama or Gemma can be fine-tuned for specific domains to achieve better quality at up to 10x speed and lower cost. The main challenges of using open models—complex setup, performance tuning, and production scaling—are addressed by Fireworks' custom serving stack, which achieves the fastest long-prompt inference and serves SDXL fastest among providers, handling over 150 billion tokens per day. The platform supports fine-tuning and serving thousands of LoRA adapters on the same GPU with serverless pay-per-token pricing. Dzhulgakov also highlights the emerging architecture of compound AI systems, where function calling—exemplified by the open-source Fire Function model—connects LLMs to external tools and knowledge sources, enabling agentic applications like stock querying and chart generation. The episode…

The Adversarial Path to the Personal Assistant: Sumit Agarwal

Feb 15, 2025 · 18:46

Sumit Agarwal, founder of Ario AI, discusses building a personal AI assistant that uses adversarial ETL to extract users' data from services like Google, Amazon, and Doordash, enabling immediate personalization without manual input. He announces $16M in funding and demonstrates how the assistant generates data portraits, recommends travel and routines based on personal history, and manages schedules by detecting conflicts. Agarwal shares RAG insights: avoid LLMs for trivial math, use search and pre-processed profiles, and present the right data at the right time. Ario Boost, a browser extension, lets users download their data locally in developer mode without creating an account.

System Design for Next-Gen Frontier Models — Dylan Patel, SemiAnalysis

Feb 11, 2025 · 18:29

Dylan Patel of SemiAnalysis breaks down the inference challenges for next-generation frontier models like GPT-4 (1.8 trillion parameters) and upcoming models trained on 100,000+ GPU clusters. He emphasizes that prefill (prompt processing) is compute-intensive while decode (token generation) is memory bandwidth-intensive, creating a systems problem where serving 64 users at 30 tokens/second requires 60 terabytes/second of memory bandwidth. Patel details engineering strategies such as continuous batching to improve batch utilization by 10-100x, disaggregated prefill to isolate noisy neighbors and maintain time-to-first-token SLAs, and context caching (like Google's) to cache KV cache on CPU/storage instead of GPU memory, dramatically reducing prefill costs. He warns that open-source tools like LLaMA.cpp lack these optimizations, making high-performance serving of models like LLaMA 405b infeasible without libraries like vLLM or TensorRT-LLM. On scaling, Patel notes that 100,000 GPU clusters (e.g., Microsoft's Arizona data center consuming 150 MW) face reliability issues — optical transceivers fail every five minutes — and straggler chips (silicon lottery) can degrade training…

Unveiling the latest Gemma model advancements: Kathleen Kenealy

Feb 9, 2025 · 16:25

Kathleen Kenealy, technical lead of the Gemma team at Google DeepMind, unveils the latest advances in the Gemma model family, including the launch of Gemma 2 in 9B and 27B parameter sizes, which outperform models two to three times larger, such as LLaMA 3 70B. She also introduces PALI Gemma, a multimodal model combining Siglip Vision Encoder with Gemma 1.0 for image-text tasks. The episode highlights Gemma's responsible-by-design approach, broad framework support (TensorFlow, Jax, PyTorch, etc.), and the release of the Gemma cookbook with 20 recipes. Kenealy emphasizes that Gemma 2 is optimized for easy integration and fine-tuning, available on Google AI Studio, and invites the community to build and share their projects.

Making Open Models 10x faster and better for Modern Application Innovation: Dmytro (Dima) Dzhulgakov

Oct 9, 2024 · 18:55

Dmytro Dzhulgakov, CTO of Fireworks AI, argues that open-source models are the future for GenAI applications because they offer lower latency, lower cost, and domain adaptability compared to proprietary models. He explains that open models can be up to 10x faster for narrow domains and cut costs significantly, using examples like fine-tuned Llama 3 for function calling. Fireworks addresses the challenges of setup, optimization, and production readiness with a custom serving stack that delivers the fastest inference for long prompts and image generation (e.g., SDXL). He highlights FireFunction V2, an open-source model for function calling that combines chat and tool use, and notes that Fireworks serves over 150 billion tokens per day for companies like Quora and Cursor. The talk emphasizes that platforms like Fireworks enable developers to start with serverless inference, fine-tune models, and scale to enterprise-grade deployment with dedicated hardware.

Building and Scaling an AI Agent Swarm of low latency real time voice bots: Damien Murphy

Oct 8, 2024 · 1:07:23

Damien Murphy, Senior Applied Engineer at Deepgram, demonstrates building and scaling low-latency real-time voice bots using Deepgram's new voice agent API, which wraps speech-to-text, LLM, and text-to-speech into a single streaming endpoint. He shows a drive-thru ordering demo with function calling (add/remove items) using GPT-4o, achieving sub-second latency by co-locating components. Murphy explains scaling to millions of concurrent calls through regional Kubernetes clusters and multi-agent swarms (routing, booking, support agents) to reduce complexity and cost. He addresses endpointing challenges, VAD-based barge-in, and cost/quality trade-offs with hosted vs. self-hosted models, noting Deepgram offers 20x cheaper TTS than ElevenLabs and 50ms STT latency self-hosted. The talk emphasizes keeping agents simple, using smallest capable LLMs, and composability for reuse.

GraphRAG: The Marriage of Knowledge Graphs and RAG: Emil Eifrem

Aug 28, 2024 · 19:15

Emil Eifrem, Neo4j co-founder and CEO, argues that GraphRAG—combining knowledge graphs with vector search—significantly improves RAG application accuracy, ease of development, and explainability. Citing studies, he reports accuracy gains of 3x (Data.org), 75-77% (LinkedIn), and Microsoft's finding that GraphRAG enables answering entirely new question types. He demonstrates Neo4j's Knowledge Graph Builder, which auto-generates graphs from PDFs, Wikipedia, and YouTube, making graph creation accessible. Eifrem frames this as the next evolution in search after PageRank and Google's knowledge graph, urging developers to adopt GraphRAG for richer context and better AI outcomes.

[Workshop] AI Engineering 201: Inference

Nov 7, 2023 · 1:43:16

Charles Frye, instructor of the Full Stack LLM Bootcamp, leads a workshop on AI engineering inference, focusing on the build-versus-buy decision between proprietary and open models. He argues that proprietary models like OpenAI's GPT-4 and Anthropic's Claude are currently more capable but expensive, while open models like LLaMA 2 are less capable but offer hackability, though they may catch up if capabilities requirements saturate. Frye covers inference on end-user devices, noting that running models locally avoids network latency but faces tight memory and power constraints—e.g., a 7B parameter model requires 14 GB, too large for phone RAM. He explains inference-as-a-service (e.g., OpenAI, Replicate) versus self-serving on cloud GPUs or serverless platforms like Modal, highlighting that memory bandwidth is the bottleneck: GPUs have 1.5 TB/s memory bandwidth vs 312 TFLOPS compute, so batching is crucial for throughput. Frye also discusses inference arithmetic, custom silicon like TPUs which offer ~30% better efficiency but not drastic gains, and containerization challenges for GPU workloads.