Vercel

How to avoid disaster when vibe-coding a billing engine — Andrew Garvin, Stripe

Aug 28, 2026 · 17:49

Andrew Garvin, co-founder of Metronome, the usage-billing platform Stripe acquired this year in its largest deal ever, argues that billing engines carry deep business logic, so a coding agent should build a test environment while a human stays in the loop — demonstrated by a Stripe Projects workflow replicating Lovable's pricing model. He demonstrates a billing sandbox that provisions a customer, metered usage, scoped credit pools for builds, plan mode, cloud and gateway calls, and a draft invoice, using portable skills files and verbose errors so the agent can self-correct. He separates agent as product, buyer, and user, pointing to HubSpot's move from seats to credits as one agent does the work of many logins. Stripe's CLI use by coding agents has risen exponentially.

How We Got LLMs to Recommend Our Open Source Library — Christopher Burns, Inth

Aug 26, 2026 · 16:27

Christopher Burns, founder of Inth and creator of the open source consent banner library c15t, explains how he got LLMs to recommend his library: after April 13th, Claude, ChatGPT, Codex, and Gemini became its number one inbound source, with c15t at 3 million NPM downloads and 45% month-on-month growth. He argues no single fix works, so he built LeadType, a framework-neutral docs pipeline that generates agent-facing files from MDX. He recommends hand-writing llms.txt (40 good lines beat 1,000), shipping markdown via .md, content negotiation, or mode=agent, and bundling markdown docs plus AGENTS.md in the npm package since coding agents read node_modules, not websites, saving nearly 50% of tokens. He also covers Web MCP tools and Aura AI's agent-readiness score.

Designing Agents (The Floor Is the Frontier) — Ben Hylak, Raindrop

Aug 12, 2026 · 19:46

Ben Hylak, CTO and co-founder of Raindrop, argues that most eval advice is stuck in the chatbot era and that agents have effectively infinite issues, so the real question is which ones matter—measured by when an issue started and what share of users it hits. He reframes agent quality around raising the floor (the worst thing an agent can do, like recommending a competitor or deleting data) rather than the ceiling, and says evals belong in your repo as code tests, not prompt playgrounds, because the harness is the product. He offers three tactical lessons from Raindrop: clusters are not issues because boundaries drift and you don't control them; code mode scales to traces, letting you write classifiers and run them in a sandbox at production volume; and agents are poor at anomaly detection but good at investigating anomalies you surface deterministically, like keyword spikes. He also notes that continual learning is rare in the real world, and that your approach should depend on user…

Emulated: The Data for Fully Autonomous Software Engineers and Companies — Joseph Wang

Jul 31, 2026 · 16:33

Joseph Wang and his co-founder Sid from Emulated argue that AI agents struggle with infrastructure work because training data misses the messy reality of production, so Emulated simulates entire companies inside sandboxes. Tasks run 50 to 100 turns, with live traffic, failing nodes, data corruption, clock skew, deployments, and customer conversations — not clean code diffs. They argue single-node sandboxes break down when provisioning real resources like VPCs, subnets, and security groups, and meeting bars for throttling, auth, and authorization, plus managing costs and gradual rollouts. Their goal is making agents own entire companies by emulating the real world at full fidelity; they start with infra because domain expertise improves data quality and infra's problem statements are clear.

Full Workshop: Setting Yourself Up for Success —Jason Liu, OpenAI Codex

Jul 24, 2026 · 1:15:02

Jason Liu walks through how to set up OpenAI's Codex for maximum productivity, using voice dictation, app shots, computer use, and a personal memory vault to delegate and automate almost every task. He demonstrates creating skills and plugins from past work, using compaction to keep threads running for weeks with hundreds of sub-agents, and setting up automations with heartbeats and goals. Specific examples include auto-checking flights, editing iMovies via computer use, and having a chief-of-staff thread that monitors Slack and updates task lists. He also covers security permissions (auto-review vs full auto), tips for new users to start with low thinking mode to save tokens, and how to build skills that self-improve by editing their own files. The workshop emphasizes that threads can now communicate with each other, enabling manager-like orchestration and long-running work streams.

Notion's Token Town — Sarah Sachs, Notion

Jul 23, 2026 · 23:55

Sarah Sachs, Notion's AI engineering lead and contract negotiator, argues that AI companies must stop competing on token economics and instead build model-agnostic products that win on data flywheels, orchestration, and security. She advises treating every model supplier as a competitor, because frontier labs charge a markup on a markup for tokens they sell for first-party use. Notion's auto model routes 75% of traffic through a Switzerland-like system that swaps providers underneath, avoiding vendor lock-in. Sachs advocates routing by cost per capability per second, using open weight models for the moderate middle, and reaching for CPUs over GPUs (e.g., no LLM needed to turn a CSV into a PDF). She highlights the 'lethal trifecta' of private data, untrusted content, and external communication as the next security challenge, and demos Notion agents scoping a task, tagging teammates, and opening a PR. Her core message: optionality is leverage, and the product must transcend tokens.

"The biggest challenge in your stack? Evals, Evals, Evals" - 2026 State of AI Engineering results

Jul 21, 2026 · 19:47

In the 2026 State of AI Engineering survey presented by Amplify Partners' Barr Yaron, 1,048 AI engineers reveal that cost is now a first-class engineering constraint—40% say it regularly shapes how ambitiously they use AI. Agents have exploded: 95% of teams now use agents, and 89% of those agents have write access, tripling from last year. Image generation adoption doubled to 36%, while audio shows the strongest intent-to-adopt at 56%. Open-weight models augment rather than replace closed models—45% use open-weight, but over 90% of them also use closed models. Evals remain the top infrastructure challenge, and inference is the most bought layer, while prompt management (61% built in-house) stays close to product logic. Teams report 97% net positive impact, but 59% fear long-term liabilities from AI code, and over a third say non-developers now ship features.

Everything we knew about software has changed — Theo Browne, @t3dotgg ​

Jul 8, 2026 · 16:02

Theo Browne argues that AI model evolution—from Sonnet 3.5’s tool-calling to Opus 4.5's long-running tasks and Mythos's orchestration—requires engineers to think bigger and wider. He compares current developer habits to skeuomorphism in iOS 7, urging rejection of legacy constraints like Git's inability to commit environment files and terminal-centric workflows. Browne introduces a shifted tier system: what was a startup is now a side project; a Markdown file running on a cron job can replace a company's product. His own PR triage service became a Markdown file updated daily via cron. He advocates building breadth over depth—architecting products so users can extend features, enabling small teams to compete with AWS or Salesforce. 'If your idea doesn't feel stupid, it's because your idea is not big enough,' he concludes.

Why Eval++ Is the Next Great Compute Primitive — Sunil Pai & Matt Carey, Cloudflare

Jun 8, 2026 · 24:51

Matt Carey and Sunil Pai from Cloudflare's agents team argue that Durable Objects — stateful serverless with 15ms London latency — and Dynamic Workers — a safe, sandboxed eval for LLM-generated code — form the next compute primitive for AI agents. They explain how Durable Objects enable resumable streaming, multi-tab sync, and background scheduling out of the box without distributed systems engineering. Dynamic Workers allow running generated JavaScript strings in isolated sandboxes, reclaiming 30 years of avoided eval. The pair tease upcoming talks: one on collapsing Cloudflare's 2,600 API endpoints into a 1,000-token MCP tool, and another on a coding agent harness built entirely on Workers that they are already shipping.

Dark Factory: OpenClaw Ships Faster Than You Can Read the Diff — Vincent Koc, OpenClaw

Jun 5, 2026 · 16:44

Vincent Koc, a core maintainer at OpenClaw, explains how the open-source project ships code faster than humans can read diffs by running 60–70 autonomous coding agents across swim lanes. In a single night, he and Peter Steinberger executed 2,700 commits touching 82% of the core codebase, launching a plugin architecture and changing close to a million lines of code. Koc describes managing agents as managing people: the key skill is reading reasoning tokens to detect when an agent is bullshitting, gained through sheer volume of token maxing. He argues 2025 was about token maxing; 2026 is about not wasting them, with agent-in-the-loop processes and opinionated swim lanes replacing blind Ralph looping. The episode details his agent development environment using .skills files, Git work trees, and a semantic graph for triaging 60K PRs, emphasizing that the bottleneck shifts from engineering to taste and process.

Beyond Components: Designing Generative UI for MCP Apps — Ruben Casas, Postman

Jun 3, 2026 · 16:58

Ruben Casas from Postman argues that AI models can now write better frontend code than he can—his prompt to rewrite his blog produced a search box with blur animation and accessibility out of the box—yet most agent UIs still invoke static prebuilt components. He presents three levels of UI generation: static components (AG UI, Goose) passing props to predefined React elements; declarative UI where the model generates JSON or YAML for a rendering engine (e.g., Vercel's JSON Render), which he deems the current ideal balance; and fully generative UI where the model writes HTML, CSS, and JavaScript on demand, as in his weather agent that does so in one tool call. The key obstacle is trust, necessitating sandboxing, and MCP apps with their double iframe default are the best delivery mechanism. He likens today to early TV—radio shows with cameras—and predicts the future lies beyond components in collaborative human-agent interfaces on shared canvases, as seen with the Skeletro MCP app.

Introducing WebMCP: Agents in the Browser — RL Nabors

May 23, 2026 · 23:08

Rachel Nabors, former web standards contributor and now Principal Developer Experience Engineer at Arise, argues that chat-only agent interfaces (the 'starfish' design) are the CLI of the future and demonstrates how to replace them with rich interactive surfaces using MCP Apps and WebMCP. She explains STDIO vs HTTP transports, shows how MCP Apps bundle HTML, CSS, and JavaScript into single files rendered in agent iframes, and introduces WebMCP, which adds tool attributes to existing page elements so browser agents can call functions without screenshot parsing or DOM traversal. Using her own web comic site as a case study, she builds a comic reader with full panel navigation, transcript mode, and speech synthesis via the Web Speech API, and advocates for better client support of MCP resources to avoid inefficient tool calls for context loading.

Combine Skills and MCP to Close the Context Gap — Pedro Rodrigues, Supabase

May 15, 2026 · 18:27

Pedro Rodrigues from Supabase argues that combining agent skills with the Model Context Protocol (MCP) outperforms either alone, closing the context gap that makes agents unreliable on production systems. In a test with Claude Sonnet 4.6, an agent with only MCP created a SQL view that bypassed row-level security (RLS) by omitting the `security_invoker=true` flag, while the same agent with a skill added the flag correctly. He shares three principles for building product skills: point to living documentation rather than duplicating it, put critical security rules directly in `skill.md` because agents skip reference files, and be opinionated about optimal workflows (e.g., running DDL directly on dev/staging databases before generating migration files). Their evals across Claude Opus 4.6, Sonnet 4.6, GPT 5.4, and GPT 5.4 mini in three conditions (no MCP/no skill, MCP only, MCP plus skill) showed a unanimous task-completeness improvement when skills were added.

Give Your Agent a Computer — Nico Albanese, Vercel

May 12, 2026 · 1:08:53

Nico Albanese demonstrates building an agent with Vercel's AI SDK v6, centered on the insight that giving an agent a file system transforms its behavior—it follows through on long tasks, stays on track, and builds on prior work. He walks through creating a tool loop agent from scratch, adding provider-executed web search with typed UI components, and integrating Vercel's persistent named sandboxes that snapshot state after inactivity. The agent gains a bash tool for file system access, a memories.md file for persistent memory injected into instructions each turn, and instructions to generate Python scripts for repeatable tasks so it accumulates tools across sessions. Albanese explains how these patterns scale: an internal agent called D0 reduced customer support tickets by 90% with a 95% satisfaction rate, and his personal coding agent ran for 104 minutes in one turn, using 316 tool calls and only 32% of GPT-5.4's context window with zero compaction. The session concludes with a preview of a larger sub-agent system that uses durable workflow steps and background sub-agents to keep the main thread under 7,000 tokens.

Give Your Chat Agent a Voice — Luke Harries, Head of Growth, ElevenLabs

May 9, 2026 · 8:12

Luke Harries from ElevenLabs argues that the next upgrade for chat agents is a voice layer, not smarter prompts or RAG, and introduces the company's Voice Engine to wrap existing agents via a few lines of code. He demonstrates how text-in, text-out chat agents can be converted to voice agents using a single prompt, leveraging ElevenLabs' advanced turn-taking, emotion-aware interruption detection, and Scribe for speech-to-text. The Voice Engine provides server and client SDKs, plus Shadcn-based UI components, enabling omni-channel deployment like phone calls and Zoom. Harries also addresses tool calling, noting that the wrapper proxies calls to existing agent logic without rebuilding. He predicts that chat agents will either adopt voice or become obsolete.

Agentic Search for Context Engineering — Leonie Monigatti, Elastic

May 8, 2026 · 1:03:13

Leonie Monigatti from Elastic argues that context engineering is 80% agentic search—the search tool that decides what to pull from files, databases, memory, and the web. She identifies three failure modes: the agent not calling any tool, calling the wrong tool, or generating incorrect parameters, and shows how detailed tool descriptions and agent skills (progressive disclosure) reduce these. In demos, semantic search fails for keyword 'JEPA' due to embedding similarity; a general-purpose ESQL tool with an agent skill correctly queries the database. Shell/bash tool enables file system search but requires iterative grep, while the custom Gina Grap CLI provides semantic grep for fuzzy queries. Practical recommendations: start with general-purpose tools, log agent behavior, and add specialized tools for frequent queries to balance low floor (easy successes) and high ceiling (complex queries). Hybrid agents combining shell and database tools achieve higher accuracy by verifying results.

Full Walkthrough: Writing & Using Skills — Nick Nisi and Zack Proser

May 6, 2026 · 1:21:03

Zack Proser and Nick Nisi, Developer Experience Engineers at WorkOS, teach how to write portable, composable skills — single markdown files with optional scripts — that train AI tools like Claude, Codex, and Cursor to perform specific tasks consistently. They demonstrate building a repo roast skill that uses deterministic Git commands via script interpolation and progressive disclosure to load targeted references only when needed. The workshop covers structuring skills with front matter (name, description for LLM routing), adding confidence scoring and constraints to improve output, and sharing skills as .skill files or through marketplaces. Advanced topics include using skills in non-coding workflows (e.g., Slack–Linear automation) and composing skills in the WorkOS CLI, which uses the Claude Agent SDK to install auth. Attendees learn to iterate skills by reflecting on past conversations and leveraging Claude's built-in skill builder for evaluation.

Skill Issue: How We Used AI to Make Agents Actually Good at Supabase — Pedro Rodrigues, Supabase

May 4, 2026 · 1:18:41

Pedro Rodrigues, AI tooling engineer at Supabase, demonstrates how to write, test, and iterate on Agent Skills to make agents actually good at real systems like Supabase's performance-review app. He explains progressive disclosure—skills load only a short description first, letting the agent decide when to pull in full instructions—and shows that combining skills with MCP tools gives agents both context and actions. In a live demo, Rodrigues reveals a common failure: creating a database view without a security invoker flag bypasses row-level security, letting all users see everyone's salary data. Adding a skill with security checklists guides the agent to include the flag, fixing the bug. He then introduces an eval-driven development cycle using an eval.json file and an LLM-as-judge to automate testing across conditions with and without the skill, cautioning that writing accurate evals is tricky because non-deterministic outputs can mislead. The session offers a practical framework for validating skills, avoiding pitfalls, and measuring what actually improves agent behavior.

Agents for Everything Else — swyx

May 1, 2026 · 14:10

swyx, co-founder of AI Engineer Conferences, details how his team of nine full‑time people uses coding agents like Devin and Town to run the conference, moving from a non‑AI stack to agents that handle everything from Figma‑to‑website conversion and schedule management to buying a lobster for the event. He claims these agents save “yak shaving” and increase human productivity because employees enjoy faster feedback cycles, leading to more work like animations and polish. The team now manages a 1,000‑person conference with plans to handle 6,000 in San Francisco without growing headcount. Swyx also advocates for “agents for everything else,” predicting that the primary user of software will shift from humans to bots, making APIs and MCPs more important than dashboards. He urges attendees to adopt agents and “prescribe” them to skeptical coworkers.

The New Application Layer - Malte Ubl, CTO Vercel

Apr 20, 2026 · 18:52

Vercel CTO Malte Ubl argues that AI engineering is the legitimate successor to web development and that the real value lies in the application layer, not the model labs. He identifies four effective agent archetypes—24/7 support, compressed research, surfacing existing information, and eliminating boring work—and reveals that over 60% of vercel.com page views now come from AI agents. Ubl predicts model companies will commoditize, driving costs down and empowering engineers, while citing Europe's leadership in AI engineering through Vercel's AI SDK, Pi (a coding agent from Austria), and OpenClaw. He stresses the need for open-mindedness toward paradigm shifts and new infrastructure, such as sandboxed agent runtimes, and warns of impending security challenges.

Identity for AI Agents - Patrick Riley & Carlos Galan, Auth0

Jan 14, 2026 · 1:22:12

Patrick Riley and Carlos Galan from Auth0 (Okta) present new identity and access management features for AI agents, including Async Auth, Token Vault, and MCP integration. Token Vault securely stores and refreshes upstream resource tokens, enabling agents to access APIs on behalf of users. Async Auth allows agents to request user approval for risky operations, such as placing an order, via out-of-band notifications like push to the Guardian app. They demonstrate a Next.js chatbot that uses Token Vault to read a user's stock portfolio and Async Auth to require explicit consent before executing trades. The MCP server is secured with OAuth 2.0, supporting Dynamic Client Registration (DCR) and fine-grained scopes. The solution bridges user identity, agent identity, and upstream API authorization, ensuring agents can act autonomously but with user-defined boundaries.

Building durable Agents with Workflow DevKit & AI SDK - Peter Wielander, Vercel

Jan 6, 2026 · 1:09:49

Peter Wielander from Vercel introduces the Workflow DevKit, an open-source library that adds durability, observability, and resumability to AI agents by wrapping them in a workflow pattern. He demonstrates converting a coding agent into a workflow-supported agent, showing how steps like LLM calls and tool executions become isolated, retryable units. The toolkit enables long-running agents that can sleep for days, resume streams after disconnection, and integrate human-in-the-loop via webhooks. Deployable on Vercel or any cloud, it provides built-in observability through a local UI. The Workflow DevKit is currently in beta with general availability targeted for January.

No Vibes Allowed: Solving Hard Problems in Complex Codebases – Dex Horthy, HumanLayer

Dec 2, 2025 · 20:31

Dex Horthy of HumanLayer argues that with deliberate context engineering — specifically a technique he calls "frequent intentional compaction" — today's AI coding agents can handle large brownfield codebases, not just greenfield projects. He presents a three-phase workflow (research, plan, implement) that keeps agents in the "smart zone" of the context window, avoiding the diminishing returns that set in around the 40% mark. Horthy demonstrates this approach solving a real problem in a 300k-line Rust codebase, shipping a week's worth of work in seven hours with code that passed expert review. He cautions against outsourcing thinking to AI, emphasising that plans must include actual code snippets for reliable execution and team alignment. The talk also addresses the cultural rift where senior engineers clean up slop from juniors' AI tools, calling for top-down adoption and team-wide workflow adaptation.

Form factors for your new AI coworkers — Craig Wattrus, Flatfile

Aug 22, 2025 · 15:35

Craig Wattrus, AI design engineer at Flatfile, argues that designing AI interactions requires treating AI as a new coworker with distinct form factors — invisible, ambient, inline, and conversational — rather than traditional UI. He demonstrates how feeling the material by building tools like a 'chat tuner' to tune AI character shifts from helicopter-parenting to character coaching. Finding the grain involves designing tool UX where AI communicates visually, checks alignment, and hands back control. Courting emergence yields unexpected results: an agent combined two files without being asked and suggested the user contact HR for missing employee IDs. Looking forward, Wattrus explores auto-complete for data transformations, using LLMs to suggest fixes alongside human oversight.

Evals Are Not Unit Tests — Ido Pesok, Vercel v0

Aug 6, 2025 · 15:22

Ido Pesok, an engineer at Vercel working on v0, argues that evals—not unit tests—are the key to making LLM applications reliable, using the basketball court analogy of understanding your domain (the court), collecting user queries (data points), and scoring passes/fails. He illustrates with the Fruit Letter Counter app, which failed in production despite passing manual tests, because LLMs are inherently non-deterministic. Pesok advises collecting data from thumbs-up/down, logs, and forums, and structuring evals with constants in data and variables in task (e.g., system prompt). He emphasizes deterministic pass/fail scoring, adding extra prompt tags for easy parsing, and integrating evals into CI/CD to detect regressions when changing models or prompts. The episode stresses that improvement without measurement is limited, and evals enable systematic reliability gains.

Good design hasn’t changed with AI — John Pham, SF Compute

Jul 21, 2025 · 20:25

John Pham of SF Compute argues that good design principles—speed, trust, accessibility, and delight—remain unchanged even with AI. He demonstrates these through SF Compute's onboarding, which loads in under 300ms using server-side rendering and a 14KB fog image animated with stacked transparent layers and no third-party JavaScript, respecting reduced motion preferences. Trust is built by setting clear expectations (e.g., '3 steps, under a minute'), auto-filling forms via browser autocomplete, and avoiding layout shifts. Delight comes from color psychology to slow users, peak-end bias (ending with a beautiful San Francisco scene), and the GPU habitat: a live video feed of GPUs implemented as stacked divs and a looping video, with a prediction cone that smooths nested menu interactions. Accessibility includes semantic HTML, screen reader testing, and pausing animations for reduced motion. These techniques turn users into super fans while growing total addressable market.

Conquering Agent Chaos — Rick Blalock, Agentuity

Jul 1, 2025 · 14:40

Rick Blalock, founder of Agentuity, argues that deploying AI agents remains the number one headache for developers, citing common issues like serverless timeouts (agents running 15–30 minutes), statelessness, and networking complexities. He demonstrates Agentuity’s platform, which treats agents as first-class infrastructure citizens: users scaffold projects via a CLI (supporting Bun, Python/UV, Node.js), write a simple request handler, and deploy with automatic routing, tunneling, and a built-in AI gateway that tracks costs per run. The platform decouples inputs and outputs—agents can be triggered via email, SMS, webhooks, or cron—and provides human and agent-facing telemetry via OTel tracing. Blalock notes Agentuity already hosts 50–60 internal agents and is building infrastructure agents to replace tools like PagerDuty, with plans to add Slack/Discord integrations and service-level reasoning capabilities.

Vercel AI SDK Masterclass: From Fundamentals to Deep Research

Apr 20, 2025 · 59:52

Nico Albanese from Vercel delivers a workshop on building agents with the AI SDK, demonstrating how to use generateText, tools, and structured outputs to create a deep research clone in Node.js. He shows that switching between models like GPT-4o mini, Perplexity Sonar Pro, and Gemini Flash 1.5 requires only changing one line of code. Tools with maxSteps enable autonomous multi-step agents, as when the model fetches weather data for San Francisco and New York and sums the temperatures across three steps. The deep research workflow generates subqueries (e.g., 'requirements to become a D1 shotput athlete'), searches via Exa with live crawl, evaluates relevance with an evaluate tool that discards irrelevant results, and recursively deepens research using depth and breadth parameters. Finally, o3 mini synthesizes the accumulated research into a structured Markdown report, all in 218 lines of code.

Building an AI assistant that makes phone calls [Convex Workshop]

Feb 9, 2025 · 50:53

Tom Redmond, head of DX at Convex, builds Floyd, an AI phone-calling assistant using Convex, GPT-4, Google Cloud speech-to-text, and Twilio. Floyd lets users make voice requests (e.g., calling a school to report a sick child) via a web app; transcription, context retrieval, and real-time conversation with a human on the other end happen through a reactive database architecture. Redmond explains how Convex's WebSocket-based infrastructure solves serverless latency issues, enabling live transcript streaming and status updates without polling. He demonstrates two calls (school absence and flower order), noting that latency (2–5 seconds) comes from OpenAI text-to-speech and growing prompts, with plans to use OpenAI Threads and faster TTS services. The prototype explicitly discloses it's an AI to avoid misrepresentation, and Redmond envisions AI agents efficiently handling service bookings.

[Full Workshop] How to add secure code interpreting in your AI app: Vasek Mlejnsky

Feb 6, 2025 · 1:48:16

Vasek Mlejnsky, CEO of E2B, demonstrates how to add secure AI code execution to any app using E2B’s Code Interpreter SDK alongside Anthropic's Sonnet 3.5 and Vercel’s AI SDK. The workshop builds an open-source version of Claude’s artifacts UI, running Python code in isolated Firecracker VMs that start in ~900ms and support custom environments via Dockerfiles. Vasek explains how E2B’s Jupyter-server-based sandbox returns stdout, stderr, runtime errors, and rich outputs like PNG charts, which are streamed to the front end using Vercel’s stream data. He covers security architecture—full root access in a VM that self-destructs on escape—and plans for AWS/GCP self-hosting and snapshot-based agent branching. The episode also addresses customizing sandboxes with private packages, mounting cloud storage (S3, GCS), and interpreting table previews as HTML or images.

How Zapier Builds AI Products and Features with the Help of Braintrust: Ankur Goyal & Olmo Maldonado

Nov 7, 2024 · 14:59

In this talk, Olmo Maldonado (Senior AI Engineer at Zapier) and Ankur Goyal (CEO of Braintrust) explain how Zapier uses Braintrust's evaluation and observability platform to build and improve AI features like AI Zap Builder and Copilot. They moved from 7 manual unit tests to over 800 automated evals, improving accuracy by nearly 300%. Braintrust's tracing capabilities let them dissect Copilot's multi-tool agent performance, leading them to adopt GPT-4 Turbo for the message router despite latency trade-offs. When switching to GPT-4.0 caused a regression below 80% scores, they diagnosed the issue via evals—GPT-4.0 was ignoring system prompts—and fixed it by relaxing prompt engineering and adjusting tool choices, recovering accuracy and cutting response time from 14 to 3 seconds. The episode details how eval-driven development and observability enable rapid iteration and reliable AI products at scale.

The era of unbounded products: Designing for Multimodal IO: Ben Hylak

Sep 25, 2024 · 20:32

Ben Hylak, founder of Dawn and former Apple Vision Pro designer, argues that the key to building intuitive AI products in the era of unbounded interfaces is adding structure—highlighting what matters, establishing hierarchy, and leveraging familiarity—lessons from designing VisionOS. He shows how successful AI apps like Dot, Perplexity, and Claude use structure (e.g., Claude pulling code into artifacts) while the Vercel chatbot's inline dynamic UI is an anti-pattern because it disrupts conversation flow. For agents, spreadsheets (like Clay) make unfamiliar multi-step tasks familiar. Looking ahead, Hylak predicts less prompt engineering via sparse autoencoders for millions of ranked, personalized presets, shifting product evaluation from evals to user analytics as apps become increasingly personalized.

Hypermode Launch: Kevin Van Gundy

Aug 23, 2024 · 5:03

Kevin Van Gundy, founder of Hypermode, argues that iteration velocity is the compound interest of software and key to succeeding with AI, and presents Hypermode as a runtime that makes AI approachable by integrating models and data into AI functions with minimal friction. Drawing from his experience at Vercel, he explains how rapid iteration helped them win against competitors, and applies the same philosophy to AI development. Hypermode removes common pain points: it offers one-request RAG with in-memory embedding and search, model comparison and fine-tuning export, and strong defaults that work with existing stacks. The platform de-risks getting it wrong by making model switching and prompt changes frictionless. Kevin invites listeners to a workshop to build a demo and offers $1000 in Hypermode credits to get started.

The Weekend AI Engineer: Hassan El Mghari

Nov 22, 2023 · 21:49

Hassan El Mghari shares how he built viral AI apps like RoomGPT and AI Commit, arguing that simple off-the-shelf APIs and focused weekend sprints can attract millions. He describes building 11 side projects in a year, which grew from 20,000 visitors to over 8.5 million unique visitors and 2.8 million sign-ups. Key projects include RoomGPT, which used ControlNet for room redesign and reached 6 million visitors, and AI Commit, an open-source CLI tool adopted by 30,000 developers. He emphasizes using tools like the Vercel AI SDK and v0.dev to accelerate development, making apps free and open source to drive growth, spending 80% of time on UI, and launching quickly with minimal fine-tuning.

Supabase Vector: The Postgres Vector database: Paul Copplestone

Nov 3, 2023 · 16:05

Paul Copplestone, CEO of Supabase, makes the case for pgvector as an embedded vector database within Postgres, arguing it offers production-grade performance and unique advantages for AI applications. He recounts how pgvector was contributed by a single developer, Andrew Kane, and how Supabase integrated it, leading to 12,000 new databases launched weekly with 10-15% using pgvector. Addressing benchmarks that claimed pgvector was 20x slower, Copplestone shows that adding HNSW indexing brought accuracy to 0.99, matching specialized vector databases, and a cost comparison vs. Pinecone ($410 vs. $480 for comparable queries/sec) demonstrates Postgres competitiveness. He demonstrates Postgres's extensibility with a cat image filtering example using partitions and triggers to separate good cats (similarity >0.8) from bad, all in 14 lines of SQL. Finally, he outlines future work on sharding with Citus to handle billions of vectors, and invites design partners for enterprise use cases.