NVIDIA

Codex, Behind the Harness — Dominik Kundel, OpenAI

Aug 10, 2026 · 20:55

Dominik Kundel, an OpenAI engineer, explains the internals of the Codex agent harness, which is open source under Apache 2 and written in Rust. He details how context construction caps available skills at 2% of the context window and uses deferred tools with tool search to manage size and cost. For actions, Codex relies on an apply patch tool for file edits, a shell tool with ripgrep, and sandboxes: seatbelt on macOS, bubblewrap on Linux, and a custom open-source Windows sandbox. To reduce approval fatigue, an auto-review subagent with read-only permissions judges actions against user authorization and risk taxonomies. Speed improvements come from websocket mode in the responses API, which sends only changed items instead of full state, crucial when GPT 5.3 Codex Spark hit 1,000 tokens per second on Cerebras. Long-horizon goals work via a continuation prompt until the model calls an update goal tool, favoring concrete verifiable objectives, and auto compaction maintains performance…

The New Primitives: Building AI Native Software — Kwindla Kramer, Daily

Aug 7, 2026 · 21:14

Kwindla Hultman Kramer (Daily, Pipecat) argues that agents are the web pages of 1995—a primitive, not a destination—and the next target is AI-native software. He grounds this in 80 years of computing: Vannevar Bush's 1945 As We May Think predicted OCR, speech to text, and hypertext; the 1950s brought programming languages, the 1960s interactivity, the 1970s databases, then the personal computer. VisiCalc, he says, didn't eliminate accountants; it multiplied accounting work and created new roles, a counter to AI unemployment fears. He cites Apple's 1987 Knowledge Navigator and Tavas's real reimagining as prototypes. He closes with Gradient Bang, a multiplayer game he built with LLMs at the core of every interaction, showing asynchronous non-blocking context compression, long-running subagents, progressive skills loading, dynamic UI generation, and conversational voice.

Local Models: Trust, Control, Optimization — Carter Abdallah, NVIDIA

Aug 7, 2026 · 43:21

NVIDIA's Carter Abdallah, Prime Intellect's Vincent Weisser, Arcee's Lucas Atkins, and NVIDIA's Chris Alexiuk argue open-weight models are the trustworthy foundation for enterprise and local AI. Atkins separates trust from safety: when Anthropic pulled Fable, enterprises chose Chinese open models for guaranteed availability, and open models are inspectable unlike closed APIs. Arcee pretrained a 400B model in six months; Weisser cites a customer that specialized an open model for finance in a week or two, beating Opus at a fraction of Haiku's cost. Alexiuk calls open weights the fix for 'mismanaged genius' and expects capable local models on MacBooks within a year; the panel predicts Fable-level open models within a year and hopes local-model use rises from a rounding error to 10–15%.

The State of Model Routing — NVIDIA, Cognition, OpenRouter

Aug 6, 2026 · 48:17

Cognition's Walden Yan, OpenRouter's Alex Atallah, NVIDIA's Tanay Varshney and Carter Abdallah argue routing should orchestrate frontier and cheaper models, not per-task benchmark picks; Devin Fusion cuts Fable-level intelligence cost by 40%. Yan: task-type routing is fragile because a session shifts from codebase question to feature request to live debugging; Devin keeps a frontier model planning while a cheap sidekick executes. Atallah: OpenRouter's auto router sat unused for two years until OpenClaw heartbeats every ten minutes created an app with two intelligence needs; out-of-distribution, small models thrash: Opus scores three times better at a tenth of Haiku's cost on terminal bench. Varshney cites jagged capabilities for up to 10% higher accuracy; Abdallah adds local/cloud routing.

Building Turbopuffer: Gergely Orosz (@pragmaticengineer ) × Simon Eskildsen (CEO)

Aug 3, 2026 · 56:30

Gergely Orosz talks with Turbopuffer CEO Simon Eskildsen about how napkin-math engineering took him from high-school Shopify hire to building the S3-based vector database that cut Cursor's bill by 95%. Eskildsen recounts scaling Shopify's infra to 1M RPS, building failure-injection proxy ToxyProxy, and launching Turbopuffer at $1 per million vectors on S3 with an Nginx cache. Cursor became the first customer after he helped debug Postgres/autovacuum issues; he also tells how Jensen Huang ribbed him for choosing CPUs over GPUs. He argues RL workloads are gobbling CPU capacity, making CPU SKUs scarce, and lists six reasons to raise capital, noting his first raise funded R&D and the second let employees cash out. He closes on Turbopuffer's remote culture of campfires and turbo credits for business-class flights.

Learning on the Job: The Future of Post-Training — Raymond Feng, Applied Compute

Jul 31, 2026 · 18:20

Raymond Feng of Applied Compute argues post-training must move from controlled Q&A and synthetic environments into real enterprise harnesses, enabling models to learn on the job. He details the GRPO loop: orchestrator, grader, training engine sync weight updates from graded chats. Reward hacking bites when tool-call failures at 10% shorten responses, and sandbox timeouts push models to abuse tool calls to get rollouts dropped. Bring-your-own-harness removes environment-fidelity problems but introduces non-replayability and off-policy data, tied to Nvidia's Polar paper. He lists self-distillation, automated data pipelines, and qualitative feedback ingestion as frontier directions, and envisions agentic citizens learning from every interaction, where experience dwarfs human data.

First Steps Toward Automated AI Research — Richard Socher, CEO Recursive AI

Jul 30, 2026 · 20:24

Richard Socher, CEO of Recursive AI, presents his vision of a "Eureka machine" that automates scientific discovery through recursive self-improvement, arguing that automating research can compress centuries of progress into decades. He frames science as an evolutionary process driven by Popperian falsification, and proposes a four-pillar system covering existing knowledge, measurement, simulation, and physical experimentation. Socher shows early proof points from his lab: a NanoChat model improved from 0.93 to 0.91 bits per byte, a NanoGPT speedrun cut by over two seconds to 70 seconds, and CUDA kernels that beat NVIDIA's benchmark leaderboard across all categories. He emphasizes that while these are early wins, the direction points toward fully autonomous AI research that could ultimately tackle problems in medicine, economics, and astrophysics.

Why Large? Tiny LMs & Agents on Edge/Robotics — Cormac Brick, Google

Jul 25, 2026 · 21:45

Cormac Brick of Google AI Edge argues that the real constraint on edge AI is DRAM cost, not compute, making tiny models essential for widespread deployment. Small models (1–4 billion parameters) require 4–8 GB of RAM, but their quantized Gemma 2B model—at just 2.9 bits per weight—runs on a Raspberry Pi at 7.6 tokens per second and on a Qualcomm NPU at 31 tokens per second. For even lower-end devices, tiny models (50–500 million parameters) need under 2 GB of RAM and can be fine-tuned for specific tasks like voice-to-function calling, reaching over 86% reliability on ten actions. A shipped example is an offline voice dictation app that uses two fine-tuned sub-billion Gemma models to clean up ums and ahs without a subscription. The talk covers the Gemini-optimized toolchain (Lighter TLM, MediaPipe), synthetic data generation for fine-tuning, and the trade-off between zero-shot prompting in small models versus fine-tuning tiny models for reach and speed.

The Desktop Frontier — Ahmad Osman, Osmantic

Jul 21, 2026 · 18:02

Ahmad Osman, founder of Osmantic, argues that within roughly 18 months (by late 2027) a single RTX 5090 will run intelligence equivalent to GLM 5.2, driven by the Densing Law of increasing impact per parameter. He shows this trend through concrete examples: a 27B-parameter Qwen 3.5 now beats the 405B LLaMA 3, and the same eight RTX 3090s that once struggled with LLaMA 2 can now run 15 parallel Qwen 3.5 agents. Osman presents the Densing Law—every 3.5 months, 50% fewer parameters achieve the same capability—as a systematic pattern, not coincidence. He advocates for sovereign AI: owning your own hardware (like a DGX Station or RTX 5090) gives you control, avoids cloud limitations, and sees hardware appreciate in utility as models become more efficient. He asks why fund cloud data centers when local hardware can run frontier intelligence and grow more valuable over time.

Security Track Intro — Randall Degges, Snyk

Jul 20, 2026 · 4:16

Randall Degges of Snyk opens the World's Fair Security Track by identifying three barriers to building secure AI at scale: AI-generated code with security flaws, autonomous agents that can go off the rails in production, and geopolitical disruptions like access being pulled to frontier models such as Fable and GPT-5.6. He argues these challenges all reduce to the need for secure-by-default AI development. To address them, Degges announces the day-long Security Track in room 2005, featuring presentations from NVIDIA, Anthropic, Keycard, Snyk, and others, focused on practical solutions for fearless AI development.

Your LLM Stack Is a 2008 Database With Better Marketing — Lovina Dmello, NVIDIA

Jul 20, 2026 · 20:36

NVIDIA's Lovina Dmello argues that production ML security failures stem from infrastructure misconfigurations, not exotic AI attacks, citing 2023 research finding thousands of Ray clusters exposed because authentication was off by default. An audit of 50 systems showed 78% had critical mistakes, always the same three: overprivileged accounts, flat networks, and exposed secrets or model weights. She presents a maturity model tied to overhead budgets—basic controls under 8%, selective isolation 10–20%, real-time detection 15–30%—and insists the field needs deployable defenses, not new attack-defence pairs. Dmello concludes that LLM stacks should be secured like 2008 databases: lock down access, segment networks, protect data at rest.

State of the Union: Why Local, Why Now — NVIDIA, Osmantic, Roboflow, EXO Labs, @matthew_berman

Jul 11, 2026 · 44:29

Nader Khalil (NVIDIA), Joseph Nelson (Roboflow), Alex Cheema (Exo Labs), Matthew Berman, and Ahmad Osman (Osmantic, r/LocalLLaMA) argue that local AI is now useful, driven by stronger open models and better hardware. They cite inflection points like Llama 2, DeepSeek v3, and GLM 5.2, which closed the gap with frontier cloud models. Sovereignty and control are key: enterprises need to choose their own model versions and avoid lock-in. Specialized models, such as Roboflow's fine-tuned vision models for deep-sea fish discovery, outperform general ones for specific tasks. Optimization is critical: EXO Labs achieved 10x performance on the DGX Spark by tuning existing NVIDIA kernels. The panel emphasizes that simplicity remains a barrier—most users need point-and-click solutions—and advocates for open-source AI to ensure freedom and innovation.

The Prompt Is Still a Punch Card - Ted Johnson, JoinIn AI

Jul 2, 2026 · 20:13

Ted Johnson argues that AI interfaces still use the batch protocol of punch cards, forcing humans to adapt to machines rather than the reverse. He introduces three concepts—channel, expression, protocol—to show how expression exploded with LLMs but the protocol remained static. Examples include voice mode misinterpreting a side conversation, PersonaPlex's turn-taking, and a meeting AI that follows group dynamics without prompts. Johnson calls for designing interfaces where AI understands timing, ambiguity, and shared context, moving from prompting to genuine communication.

OpenClaw in Your Hand: Building a Physical AI Terminal - Lech Kalinowski, Callstack

Jun 28, 2026 · 24:36

Lech Kalinowski presents Vault, a dual-display handheld AI terminal built on an ESP32-S3 that runs a local LLM and OpenClaw agents without cloud reliance. The device pairs a fast OLED live surface with a bistable e-paper content display, offering four modes — shell, assist, control, and an LLM-native RPG — all powered by a single lithium-polymer cell. The backend serves GPT-oss 120b via NVIDIA TensorRT-LLM, keeping inference off the microcontroller. Kalinowski shares engineering war stories: blown OLEDs from unstable power supplies, software I2C issues, and noisy encoders. The RPG mode generates four worlds with NPCs and narrative state tracked by the LLM, not dice or HP. Built in three months with 130 commits, the terminal is intentionally distraction-free, targeting quiet spaces where users want text-first AI interaction.

Structuring the Unstructured - Cedric Clyburn, Red Hat

Jun 28, 2026 · 20:41

Cedric Clyburn (Red Hat) demonstrates how Docling, an open-source tool from the Linux Foundation, converts unstructured documents like PDFs, tables, and images into structured formats (Markdown/JSON) for AI workflows. He shows that naive PDF parsers lose table structure and image content, while proprietary VLMs are expensive and non-deterministic. Docling uses OCR and layout analysis to preserve context, achieving 50x cost savings compared to VLMs on CPU. The demo covers extracting tables and images from an 8-page PDF, using a local Granite vision model for image annotation, and implementing chunkless RAG where an LLM queries a Docling document outline directly. Scaling is addressed via Docling Serve (REST API) and Docling MCP server for agentic document processing with tools like Claude Code.

You Might Not Need 50 Diffusion Steps — Ziv Ilan, Nvidia

Jun 16, 2026 · 18:46

Nvidia's Ziv Ilan explains how combining quantization, caching, and step distillation enables near real-time video diffusion on a single Blackwell B200 GPU. Working with Black Forest Labs on Flux 2, dynamic quantization reduces memory and compute, caching skips redundant denoising steps, and distillation cuts steps from fifty to as few as one. The open-source FastGen repo packages these post-training and sharding techniques, achieving 10–200x speedups for real-time generation.

Dark Factory: OpenClaw Ships Faster Than You Can Read the Diff — Vincent Koc, OpenClaw

Jun 5, 2026 · 16:44

Vincent Koc, a core maintainer at OpenClaw, explains how the open-source project ships code faster than humans can read diffs by running 60–70 autonomous coding agents across swim lanes. In a single night, he and Peter Steinberger executed 2,700 commits touching 82% of the core codebase, launching a plugin architecture and changing close to a million lines of code. Koc describes managing agents as managing people: the key skill is reading reasoning tokens to detect when an agent is bullshitting, gained through sheer volume of token maxing. He argues 2025 was about token maxing; 2026 is about not wasting them, with agent-in-the-loop processes and opinionated swim lanes replacing blind Ralph looping. The episode details his agent development environment using .skills files, Git work trees, and a semantic graph for triaging 60K PRs, emphasizing that the bottleneck shifts from engineering to taste and process.

Beyond Transcription: Building Voice AI That Understands Conversations — Hervé Bredin, pyannoteAI

Jun 5, 2026 · 25:20

Hervé Bredin, chief science officer at pyannoteAI, argues that speaker diarization benchmarks are misleading because they use headset audio while users rely on table microphones—Nvidia Parakeet reports 11.4% word error rate on AMI headset data but gets 26% on the same dataset's table mic. The episode covers how speaker diarization (who speaks when) is harder than it looks, especially when combining with transcription: overlapping speech, timestamp disagreements, and words falling between speaker boundaries create errors. Bredin demonstrates pyannoteAI's Precision 2 model achieving 3% diarization error rate (DER) against a 5% baseline on a two-speaker phone call. State of the art today: 2% DER on clean telephone calls but 41% in a noisy restaurant, showing the problem is far from solved. The reconciliation between diarization and STT is handled by a proprietary orchestration that works with any STT model.

Engineering voice agents: Latency, quality, and scale — Rishabh Bhargava, Together AI

May 31, 2026 · 24:35

Rishabh Bhargava from Together AI outlines engineering voice agents, explaining that pipeline architecture with colocated models can achieve sub-500ms response times critical for user retention. Speech-to-text targets P90 under 100ms and 6% word error rate, while the LLM must stay within 200-300ms time-to-first-token using 8-30B parameter models—larger models blow the budget, smaller ones break tool calling. Network latency from distant data centers adds 75ms (30% overhead) versus 5ms when colocated in the same building. Pure speech-to-speech models are emerging but still struggle with instruction following and tool calling. The thinker-talker pattern uses a small LLM for fast conversational flow and issues a single tool call to a larger model for complex requests.

Run Frontier AI at Home — Alex Cheema, EXO Labs

May 26, 2026 · 1:45:02

Alex Cheema of EXO Labs argues that running frontier AI locally has 100x improvement potential in cost and performance, demonstrated with GLM 5.1—a trillion-parameter model—running across four Mac Studios at roughly 20 tokens per second for $40,000. He details kernel fusion that recovered 30% performance on Qwen 3.5 by eliminating unnecessary kernel launches, and RDMA integration that cut node-to-node latency from 300 microseconds to single digits, enabling tensor parallelism to actually scale. Cheema advocates splitting inference: prefill on compute-dense hardware (e.g., an RTX Spark) and decode on high-bandwidth hardware (e.g., Mac), cutting large-prompt inference roughly in half. He warns against misleading benchmarks like one-bit quantized models, and outlines how multi-agent setups, test-time scaling, and continual learning could further improve local inference efficiency. The talk includes a live demo of GLM 5.1 across four Mac Studios connected via Thunderbolt 5, and a preview of EXO's upcoming benchmarking site to track intelligence per Joule.

Fast Models Need Slow Developers — Sarah Chieng, Cerebras

May 22, 2026 · 18:02

Sarah Chieng from Cerebras argues that the 20x speed increase of models like Codex Spark (1,200 tokens/sec vs. 40-60 for Sonnet/Opus) forces developers to rethink workflows, or risk generating technical debt at unprecedented scale. She presents a practical playbook: validation and linting become free at every step, so must run continuously; developers can generate 75 component variations across five sub-agents and cherry-pick the best; and with context filling in 30 seconds instead of ten minutes, a four-file external memory system (agents, plan, progress, verify) maintains continuity between sessions. Chieng emphasizes real-time collaboration with the model rather than spawning agents and walking away, and advocates using a slower planner model with a fast executor to orchestrate agents effectively.

Lobster Trap: OpenClaw in Containers from Local to K8s and Back — Sally Ann O'Malley, Red Hat

May 22, 2026 · 21:56

Sally Ann O'Malley of Red Hat argues that running OpenClaw in containers with Podman and Kubernetes delivers secure, portable, and reproducible AI agent setups. She uses Podman secrets and OpenClaw's secret ref feature to manage API keys, ensuring secrets stay out of logs and configs. O'Malley demonstrates a local installer that spins up an OpenClaw container in two seconds and lifts the same workload to Kubernetes. She cites an Nvidia team of 10 engineers each running their own OpenClaw in Kubernetes for model evals, claiming it replaced the work of six people. Her vision is a team-standard containerized OpenClaw baseline with company-approved MCP servers and skills, enabling reproducible onboarding and personalization across an organization.

What Breaks When You Build AI Under Sovereignty Constraints - Bilge Yücel, deepset GmbH

May 19, 2026 · 19:09

Bilge Yücel, Senior Developer Relations Engineer at deepset, argues that sovereign AI requires explicit control over data flow, model choice, infrastructure, and operations, and retrofitting these pillars breaks existing systems in predictable ways. Replacing frontier APIs with self-hosted models forces re-evaluation from scratch, moving private data across jurisdictions creates multi-database search problems, replacing managed infra reveals vendor lock-in, and adding observability exposes black-box systems. She presents a sovereign architecture with guardrails, MCP tools, and Haystack's swappable components to mitigate these issues. The closing checklist asks whether you can swap models without changing application logic, have compliant run logs, and respond to incidents without calling a hyperscaler.

Voice AI: when is the "Her" moment? — Neil Zeghidour, CEO, Gradium AI

May 9, 2026 · 19:27

Neil Zeghidour, CEO of Gradium AI, argues that voice AI remains far from the 'Her' ideal because cascaded systems (speech-to-text, LLM, text-to-speech) suffer from high latency—tool calls alone add 500ms to 4 seconds—while human response time is ~200ms. Speech-to-speech models reduce latency but are half-duplex, meaning they cannot handle overlapping speech or backchanneling, unlike Moshi, Gradium's full-duplex model. However, Moshi lacked intelligence, tool calls, and paralinguistic understanding—the ability to infer tone, hesitation, or discomfort from voice, which is stripped away in text. Cost is another barrier: TTS bills burn through fundraising before user bases grow. Gradium's solution is Phonon, an on-device TTS model running on smartphone CPUs, offering privacy and eliminating API fees. The path forward requires combining full-duplex natural conversation with the reliability and smarts of cascaded systems.

State of the Claw — Peter Steinberger

Apr 17, 2026 · 44:12

Peter Steinberger, creator of OpenClaw, presents a five-month update on the world's fastest-growing open-source project. He details the project's staggering growth—30,000 commits, nearly 2,000 contributors—and the immense security burden: 1,142 advisories (16.6 per day) with 99 critical, often AI-generated slop that demands human vetting. Steinberger refutes media fearmongering, citing how researchers ignore security docs to fabricate scary scenarios. He clarifies OpenAI did not buy OpenClaw; he joined the company while establishing the OpenClaw Foundation to remain vendor-neutral. He emphasizes the importance of local models for data sovereignty and describes his coding workflow of running 5-6 agent sessions simultaneously, iterating on taste and personality. Future visions include ubiquitous agents, 'Dreaming' for memory reconciliation, and modular plugins. For engineers, he champions taste, system design, and learning to say no.

Running LLMs locally: Practical LLM Performance on DGX Spark — Mozhgan Kabiri chimeh, NVIDIA

Apr 10, 2026 · 10:16

NVIDIA’s Mozhgan Kabiri Chimeh demonstrates that running LLMs locally on the DGX Spark workstation, powered by the GB10 Grace Blackwell superchip and 128GB unified memory, achieves practical performance for models up to 14B parameters. Using a reproducible vLLM benchmarking methodology, she shows that the 14B NVFP4 quantized model delivers 20.19 tokens per second and a time-to-first-token 3.4× faster than the unoptimized 14B base model. The DGX Spark supports the same NVIDIA AI software stack as production environments, enabling local development, fine-tuning, and privacy-sensitive workloads before scaling to the cloud. NVFP4 quantization is highlighted as critical for balancing intelligence and throughput on single-system setups.

AI Kernel Generation: What's working, what's not, what's next – Natalie Serrino, Gimlet Labs

Dec 17, 2025 · 19:15

Natalie Serrino, cofounder of Gimlet Labs, presents how AI-generated kernels can automatically speed up custom PyTorch code by up to 24% on Apple M4 hardware using the Metal framework, with a 40% speedup from kernel fusion. The agentic system iterates through compilation, execution, correctness, and optimization, but faces challenges like validation of floating-point results and reliable benchmarking. Successes include rewriting average pool 1D as a convolution for 80% improvement, while failures occur on heavily optimized ops like matrix multiply. A real-world audio encoder model saw 70% faster inference on RTX 6000 Blackwell via six custom fused kernels. Serrino emphasizes that AI is best for rapidly searching optimizations and porting code to new hardware, not for surpassing human experts on novel algorithms.

Government Agents: AI Agents Meet Tough Regulations — Mark Myshatyn, Los Alamos National Lab

Dec 6, 2025 · 16:31

Mark Myshatyn, Enterprise AI Architect at Los Alamos National Laboratory, describes how the lab is building AI agents for scientific research under strict federal regulations. He showcases an agent that reads fusion capsule papers, designs a hypothesis, and runs simulations on high-performance computing assets, integrating 60+ years of physics models. He stresses the need for explainability, isolation, and governance in government AI tools, citing OMB memoranda M2521 and M2522 mandating faster AI adoption with real-world impact. The lab partners with frontier labs like OpenAI for chem-bio safety work, and with NVIDIA and HPE for the Venado supercomputer. He urges startups to build for DoD IL5 environments and continuous compliance to succeed in federal procurement, noting Los Alamos holds petabytes of never-internet data and specializes in sensors like the ChemCam on Mars.

Building Cursor Composer – Lee Robinson, Cursor

Dec 2, 2025 · 15:36

Lee Robinson explains how Cursor built Composer, its first agent model for real-world software engineering, by focusing on being both fast and smart — achieving 4x more efficient token generation than similarly intelligent models while matching open-source performance initially and approaching frontier models after reinforcement learning. The model's training posed infrastructure challenges: matching training and inference environments across thousands of GPUs, handling complex rollouts with hundreds of tool calls and up to millions of tokens, and ensuring consistency by using the same tool format and responses as production. Cursor solved these with custom kernels that sped up training by 3.5x on NVIDIA Blackwell chips for mixture-of-experts layers, load balancing across threads to avoid idle time, and co-designing RL infrastructure with its Cloud Agents product using virtual machines that mirror the production Cursor environment. This allowed the model to become a power user of tools like semantic search, which improved all models but especially Composer. RL also taught the model to parallelize tool calls (e.g., reading 10 files simultaneously) and to search more before editing,…

Full Workshop: Realtime Voice AI — Mark Backman, Daily

Aug 3, 2025 · 1:09:41

In this hands-on workshop, Mark Backman and Aleś from Daily demonstrate how to build real-time voice AI agents using Pipecat, their open-source Python framework, and Google's Gemini Multimodal Live API. They explain that Pipecat's modular pipeline orchestrates audio transport, speech-to-text, LLM, and text-to-speech services, allowing developers to plug and play vendors like Deepgram, OpenAI, or Cartesia. The session highlights how speech-to-speech models like Gemini Live simplify architecture by handling transcription, LLM logic, and voice generation in one step, reducing latency to under 800 milliseconds. Key topics include the critical role of voice activity detection (VAD) for natural turn-taking, strategies for managing context windows to maintain accuracy, and the trade-offs between speed and reliability when using tool calls in real-time. Despite conference Wi-Fi issues, the team live-codes a functional bot and demonstrates interruption handling and the Word Wrangler game, showcasing Pipecat's production readiness with hundreds of thousands of daily calls.

Hacking the Inference Pareto Frontier - Kyle Kranen, NVIDIA

Aug 1, 2025 · 20:25

Kyle Kranen, architect of NVIDIA Dynamo, explains how to break the inference Pareto frontier by manipulating trade-offs between quality, latency, and cost using a toolkit of techniques. Disaggregation separates prefill and decode phases, achieving up to 2× tokens per second per GPU at fixed latency for LLaMA 70B on 16 H100s. Smart routing maximizes KV cache hits, asymptotically reducing prefill work as deployments scale. Structure from agentic workloads, like inference-time scaling, enables smaller models (e.g., 8B queried 3–4 times) to match larger models' quality at lower cost. KV manipulation offloads caches during tool calls (e.g., 30-second delays) to avoid re-prefill. Dynamism adjusts worker specialization and autoscales prefill/decode ratios in real time to handle shifting user distributions, ensuring disaggregation reaches maximum potential.

What Is a Humanoid Foundation Model? An Introduction to GR00T N1 - Annika & Aastha

Jul 28, 2025 · 17:47

NVIDIA's Annika Brundyn and Aastha Jhunjhunwala introduce GR00T N1, an open-source Vision-Language-Action foundation model for humanoid robots. They argue that physical AI is key to addressing labor shortages in industries like healthcare and manufacturing, and that humanoid forms are needed because the world is built for humans. The model uses a data pyramid strategy combining limited real-world teleoperation data, synthetic simulation data, and internet video, with DreamGen for data multiplication. Its dual-system architecture, inspired by Kahneman's 'Thinking, Fast and Slow', has a System 2 planner for high-level reasoning and a System 1 for fast motor control at 120 Hz. The model is trained end-to-end via imitation learning and reinforcement learning, and features a generalist action decoder that enables cross-embodiment fine-tuning.

Continuous Profiling for GPUs — Matthias Loibl, Polar Signals

Jul 22, 2025 · 11:31

Matthias Loibl of Polar Signals explains how continuous profiling for GPUs maximizes GPU efficiency using low-overhead, always-on sampling via eBPF. He contrasts tracing (high cost) with sampled profiling (e.g., 100 Hz, <1% overhead) and details GPU metrics collected from NVIDIA NVMe, including utilization, memory, clock speed, power, temperature, and PCIe throughput. The platform correlates these with CPU stack traces to identify bottlenecks, such as Python and CUDA functions underutilizing the GPU. A new GPU time profiling feature records the duration of CUDA kernel executions, showing actual time spent by functions on the GPU. Deployment runs on Linux with a binary, Docker, or Kubernetes DaemonSet; early adopters like TurboPuffer use it to optimize their vector engine.

HybridRAG: A Fusion of Graph and Vector Retrieval - Mitesh Patel, NVIDIA

Jul 22, 2025 · 20:24

Mitesh Patel, Developer Advocate Manager at NVIDIA, presents HybridRAG, a fusion of knowledge graph-based GraphRAG and vector-based VectorRAG for improved question-answering from complex texts. He emphasizes that ontology engineering consumes 80% of development time and is critical for accurate triplet extraction, where fine-tuning LLaMA 3.1 with LoRA boosted triplet accuracy from 71% to 87% on 100 documents. Patel also highlights retrieval strategies like multi-hop graph traversal, which provides richer context but increases latency, and recommends QGraph acceleration via Networx to reduce latency. For evaluation, he suggests RAGAS for end-to-end pipeline metrics and the LLaMA-Nimotron reward model for response quality. Ultimately, he advises using GraphRAG when data has inherent structure or complex relationships, but notes it is compute-heavy, so the choice between GraphRAG, semantic RAG, or hybrid depends on the use case.

What every AI engineer needs to know about GPUs — Charles Frye, Modal

Jul 20, 2025 · 19:52

Charles Frye of Modal explains that AI engineers need to understand GPU hardware constraints to optimize inference, arguing GPUs embrace bandwidth over latency and that Tensor Cores for low-precision matrix-matrix multiplication are the key resource. He describes how GPUs achieve 16,000+ parallel threads per cycle on H100, and notes Patterson’s Law: bandwidth improves at the square of latency. The main insight: arithmetic intensity favors N² operations per N memory loads, so matrix-matrix operations are efficient while matrix-vector is wasteful. Frye demonstrates that running a small 8B model 1,000 times on the same prompt matches GPT-4 quality, and that multi-token prediction and multi-sample query become nearly free because Tensor Cores handle expanded batches as matrix-matrix multiplications. He recommends using smaller models that fit on a single GPU and scaling via multiple generations.

[Full Workshop] Reinforcement Learning, Kernels, Reasoning, Quantization & Agents — Daniel Han

Jul 19, 2025 · 2:42:28

Daniel Han of Unsloth presents a technical workshop covering reinforcement learning (RL), kernels, reasoning, quantization, and agents, arguing that RL with verifiable rewards (RLVR) is the key to unlocking LLM capabilities beyond supervised fine-tuning. He explains why open-source models plateaued after September 2024 until DeepSeek-R1 showed that RL can elicit reasoning, and breaks down PPO, GRPO, and the REINFORCE algorithm, emphasizing that GRPO removes the value model for efficiency. Han details how reward functions—not algorithms—are the hardest part, with examples like distance-based scoring for math. He demonstrates a free Colab notebook training a base model to reason, and shows that dynamic quantization can shrink models like DeepSeek-R1 from 730 GB to 140 GB with only ~1% accuracy loss, arguing that GPUs may stop getting faster after FP4 precision.

OpenThoughts: Data Recipes for Reasoning Models — Ryan Marten, Bespoke Labs

Jul 19, 2025 · 19:59

Ryan Marten, co-lead of the OpenThoughts collaboration and founding engineer at Bespoke Labs, reveals the missing data recipe for open-source reasoning models, presenting OpenThoughts 3, a state-of-the-art 7B reasoning dataset that outperforms DeepSeek R1 Qwen 7B and Nematron Nano on benchmarks like AIME, Live Code Bench, and GPQA Diamond. Through over 1,000 experiments and 5,000 datasets, key findings include that sampling multiple reasoning traces per question scales performance by 16x, Qwen 32B surpasses DeepSeek R1 as a teacher model, synthetic question generation is highly effective, and filtering by difficulty or response length works better than embeddings. Surprisingly, verification of answers in SFT distillation did not improve results, and focusing on fewer high-quality sources outperformed maximizing diversity. For domain-specific reasoning, Marten advises starting with the OpenThoughts recipe, using synthetic data generation (via the open-source Curator library), and rigorous evaluation (via EvalComet). A legal reasoning example shows that distillation can surpass the teacher model. All resources are open-source.

Dream Machine: Scaling to 1m users in 4 days — Keegan McCallum, Luma AI

Jul 19, 2025 · 19:03

Keegan McCallum, Head of ML Infrastructure at Luma AI, details how the company's Dream Machine model scaled from 500 to 9,000 H100 GPUs within hours to handle 1 million users in four days, outpacing ChatGPT's initial growth. He explains that their initial Triton inference server setup was brittle and ill-suited for multi-GPU, multi-node video models, prompting a re-architecture to a custom serving stack on vanilla PyTorch. To solve work starvation across user tiers, they implemented an SLO-based aging system that ranks jobs by the percentage of their worst-case wait time elapsed. For managing dozens of model versions, they store immutable full Python environments and checkpoints in object storage, with a YAML file controlling active deployments and enabling zero-downtime rollouts across thousands of GPUs. McCallum also discusses partnerships with Nvidia, AMD, and Grok, and how Luma's broader mission is to build general multimodal intelligence that generates, understands, and operates in the physical world.

ComfyUI Full Workshop — first workshop from ComfyAnonymous himself!

Jul 19, 2025 · 51:25

ComfyAnonymous and Yedrick Kosinski present ComfyUI, an open-source node-based canvas for generative AI that has become the top 150 most popular GitHub repos with 78,000 stars and 3 to 4 million active users. Started as a personal project in January 2023, it now supports image, video, audio, 3D, and text models, with 22,000 custom nodes from 3,000 developers. A key feature is embedding full workflows into generated files for easy sharing. The team discusses its story, including ComfyAnonymous's time at Stability AI and founding ComfyOrg, and addresses questions on CFG, VAEs, LoRAs, control nets, API nodes for remote generation, and future plans for cloud inference and improved onboarding.

2025 in LLMs so far, illustrated by Pelicans on Bicycles — Simon Willison

Jul 9, 2025 · 18:30

Simon Willison reviews the past six months of LLM releases — including AWS Nova, Llama 3.3 70B, DeepSeek R1, Mistral Small 3, Claude 3.7 Sonnet, GPT 4.5, Gemini 2.5 Pro, GPT-4o, Llama 4, GPT 4.1, O3/O4 Mini, and Claude 4 — using his 'pelican on bicycle' SVG benchmark to argue that local models have become good enough to run GPT-4 class models on a laptop and that combining tools with reasoning is the most powerful technique in AI engineering, while noting risks like prompt injection and the 'lethal trifecta'. He tracks 30 significant model releases, highlighting that Mistral Small 3 (24B) matches Llama 3 70B's performance, which itself matched the 405B model, enabling local inference. DeepSeek's R1 caused a $500B+ Nvidia stock drop on January 27. GPT 4.1 Nano is the cheapest model yet at a fraction of a cent per pelican. He also examines bugs: ChatGPT's sycophantic 'shit-on-a-stick' incident and Claude 4's tendency to snitch to authorities when given ethical instructions and email tools. Willison concludes that while the pace is accelerating, control over context and security remain critical.

Trends Across the AI Frontier — George Cameron, ArtificialAnalysis.ai

Jul 8, 2025 · 17:52

George Cameron of Artificial Analysis presents multiple frontiers—reasoning, open-weights, cost, speed—across the AI stack, arguing that trade-offs between intelligence, latency, and expense are critical for building applications. Reasoning models like O4 mini high use an order of magnitude more output tokens (72M vs GPT-4.1's 7M) and take over 40 seconds per response versus 4.7 seconds, impacting agentic workflows where 30 sequential calls multiply latency. The open-weights gap has nearly closed, with China-based labs like DeepSeek R1 and Alibaba's Qwen 3 leading. O3 cost roughly $2,000 to run the intelligence index, while GPT-4.1 nano is over 500 times cheaper. Output speeds have jumped from GPT-4's 40 tokens/s in 2023 to over 1,000 on a B200 accelerator. Despite efficiency gains, demand for compute will keep rising due to larger models, reasoning’s extra tokens, and multi-step agents.

From Mixture of Experts to Mixture of Agents with Super Fast Inference - Daniel Kim & Daria Soboleva

Jun 27, 2025 · 53:15

Daria Soboleva and Daniel Kim of Cerebras explain how Mixture of Experts (MoE) architectures enable scaling large language models efficiently by replacing monolithic feedforward networks with specialized experts, a technique used by GPT-4 and Claude. They then introduce Mixture of Agents (MoA), which combines multiple LLMs with custom prompts to outperform frontier models like GPT-4o on complex tasks, reducing a 293-second reasoning problem to 7.4 seconds using Cerebras' ultra-fast inference. The workshop guides participants to build their own MoA system, configure agents for bug fixing and performance optimization on a Python function, and achieve scores up to 120/120. Daniel details Cerebras' wafer-scale chip with 900,000 cores and distributed memory that eliminates memory bandwidth bottlenecks, enabling linear scaling and 15.5x faster inference on Llama 3.3-70B versus GPUs. Daria discusses ongoing research in diffusion models and sparsity, while Daniel notes plans for multimodal APIs and LoRA fine-tuning support.

Forget RAG Pipelines—Build Production Ready Agents in 15 Mins: Nina Lopatina, Rajiv Shah, Contextual

Jun 27, 2025 · 1:15:43

Rajiv Shah, Nina Lopatina, and Matthew from Contextual AI demonstrate how developers can build production-ready RAG agents in minutes using Contextual AI's managed RAG platform, emphasizing that RAG should be treated as a managed service to avoid reinventing infrastructure. They walk through ingesting documents like NVIDIA financials and spurious correlation reports, then querying the agent with questions requiring quantitative reasoning across tables. The platform handles extraction, layout analysis, image captioning, hybrid retrieval, and a state-of-the-art reranker, capped by a grounded language model that avoids hallucinations and provides attribution. Evaluation is done via LM Unit, a model-as-judge that scores responses on criteria like accuracy and causation. The episode also shows integrating the agent with Claude Desktop via MCP and answers audience questions on pricing (consumption-based with a $25 credit), scalability, entitlements, HIPAA, and domain-specific language.

Foundry Local: Cutting-Edge AI experiences on device with ONNX Runtime/Olive — Emma Ning, Microsoft

Jun 27, 2025 · 22:52

Emma Ning, Principal PM at Microsoft, presents Foundry Local, a platform for building cross-platform on-device AI applications using ONNX Runtime and Olive. She argues local AI is essential for low-bandwidth, privacy-sensitive, cost-efficient, and real-time latency scenarios, noting that modern hardware and optimized models like Phi-4-mini and DeepSeek make it viable. Ning demonstrates Foundry Local's CLI, model benchmarking (Qwen 1.5B at 90 tok/s vs. Phi-4-mini with richer output), and a cross-platform document summarization app built with JavaScript SDK, running identically on Windows and macOS. She also previews a local agent with MCP servers (file system and OCR) that extracts receipt totals using Phi-4-mini. Customer testimonial from Sava (CEO of Pieces) highlights improved memory management and tokens-per-second, while another partner underscores ease of install and hybrid cloud-local solutions.

The Geopolitics of AI Infrastructure - Dylan Patel, SemiAnalysis

Jun 19, 2025 · 18:29

Dylan Patel of SemiAnalysis argues that despite US sanctions, Huawei has engineered a 384-chip cluster (Cloud Matrix 384) that Nvidia failed to deploy, while accessing TSMC via Softgo and HBM from Samsung via shell companies — all legally. China's SMIC will soon produce 7nm AI chips in high volumes, debunking the notion that China lacks compute. Meanwhile, Middle East players like G42 (UAE) and Datavolt (Saudi Arabia) are building multi-gigawatt data centers, with G42's deal letting it keep 20% of 500,000 GPUs yearly for itself while 80% goes to US companies like OpenAI. Patel highlights the US's 63-gigawatt power shortfall vs. 100 GW of planned data centers, explaining why US companies rely on Middle East capacity and why China's superior power buildout gives it a geopolitical edge.

The End of Awkward AI Transcriptions - Travis Bartley and Myungjong Kim

Jun 3, 2025 · 16:24

NVIDIA speech AI researchers Travis Bartley, Myungjong Kim, and Jae-hun detail how their Fast Conformer backbone powers both streaming (Parakeet) and multitask (Canary) models, achieving top-5 Hugging Face ASR leaderboard rankings by prioritizing customization over one-size-fits-all. They explain CTC/TDT for low-latency streaming and attention encoder-decoder for high-accuracy multitask models, with Subformer enabling unified speaker diarization and target-speaker ASR. Training uses open-source and proprietary data with pseudo-labeling via the Nemo toolkit. Deployment on NVIDIA Riva NIM with TensorRT optimization supports low-latency streaming and offline processing. Customization includes fine-tuning acoustic models, language models, punctuation, and inverse text normalization for domain-specific terms like medical and food ordering.

The RAG Stack We Landed On After 37 Fails - Jonathan Fernandes

Jun 3, 2025 · 18:52

Jonathan Fernandes, independent AI engineer, details the RAG stack his team settled on after 37 failed attempts, covering orchestration (LlamaIndex), embeddings (BAAI BGE small), vector database (Qdrant), LLMs (GPT-4, Qwen, Llama), reranking (Cohere), monitoring (Arize Phoenix), and evaluation (RAGAS). He demonstrates a live prototype in Google Colab using a London railway knowledge base, showing how a naive RAG returns irrelevant results (e.g., suggesting black cabs for "where can I get help at the station"). By swapping components—replacing in-memory storage with Qdrant, using an open-source embedding model, upgrading to GPT-4, and adding Cohere reranking—the answer improves to "go to booth number five next to the Eurostar ticket gates." For production, he deploys via Docker Compose with NVIDIA embedding/reranking models and Ollama for serving. The episode also stresses the importance of tracing latency per component and using RAGAS for systematic evaluation across many queries.

Effective AI Agents Need Data Flywheels, Not The Next Biggest LLM – Sylendran Arunagiri, NVIDIA

Jun 3, 2025 · 16:41

Sylendran Arunagiri of NVIDIA argues that effective AI agents rely on data flywheels, not the largest LLMs, enabling smaller models to achieve top accuracy at a fraction of cost. He details how NVIDIA's NeMo microservices power a continuous cycle of data curation, fine-tuning, evaluation, and guardrailing. Using an internal employee support agent (NVINFO), they achieved 96% accuracy with a 70B model but found that fine-tuning a smaller 8B model matched that accuracy, while a 1B model reached 94% with 98% lower inference cost and 70x latency reduction. The process involved curating 685 ground truth data points from user feedback and error analysis. He provides a framework: monitor user feedback, attribute errors, plan model experimentation, and execute regular retraining cycles.

How to Build Your Own AI Data Center in 2025 — Paul Gilbert, Arista Networks

Apr 27, 2025 · 23:00

Paul Gilbert, tech lead at Arista Networks, explains the key considerations for building AI data center networks in 2025, emphasizing the stark differences from traditional enterprise networks. He details the backend GPU network (eight 400G ports per H100 server, no over-subscription), the front-end storage network (calmer, 100-200G), and the need for lossless Ethernet with ECN and PFC flow control to prevent packet drops during synchronized GPU bursts. Gilbert highlights power challenges (10.2kW per GPU server, requiring 100-200kW water-cooled racks) and the importance of telemetry like RDMA error monitoring and an AI agent that correlates GPU and network issues. He also covers advanced load balancing (cluster-aware, up to 93% utilization) and smart system upgrades without downtime, while noting the upcoming Ultra Ethernet Consortium (v1.0 in 2025) that shifts congestion control to NICs.

Keynote: The AI developer experience doesn't have to suck – why and how we built Modal

Feb 22, 2025 · 21:38

Eric Bernhardson, CEO of Modal, explains why and how his company replaced Kubernetes and Docker with a custom container system to deliver sub-second cold starts for AI developers. Modal turns any Python function into a serverless function with a decorator, runs on thousands of H100s, and fans out to 10,000 parallel calls. To achieve fast startup, Modal built content-addressable storage for deduplication, lazy file loading with prefetching, and uses gVisor for CPU memory snapshotting, cutting Stable Diffusion startup to seconds. The company built its own scheduler and file system, and uses mixed integer programming to manage a global GPU pool across cloud vendors. Customers like Suno use Modal for AI-generated music inference. Modal offers $30/month free credits.

Keynote: Why people think "agent" is a buzzword but it isn't

Feb 22, 2025 · 28:07

Chip Huyen argues that agents are not a buzzword but a practical yet hard technology, facing three core challenges: the curve of complexity, tool use translation, and context management. Even simple queries require multiple steps, and models' success rates drop rapidly past 5 steps—newer reasoning models like DeepSeek R1 are pushing the boundary, but most still fail after 10. Tool use requires translating ambiguous natural language to precise API calls, worsened by poor documentation; she advises narrow functions and asking for clarifications. Context is another bottleneck: agents must juggle instructions, tool docs, and outputs, often exceeding a model's efficient context (many hallucinate beyond 30K tokens), necessitating external memory like RAG. Her benchmark shows planning-specialized models struggle with long context and vice versa, and she recommends breaking tasks into subtasks and using test-time compute scaling.

Training Albatross An Expert Finance LLM: Leo Pekelis

Feb 13, 2025 · 16:20

Leo Pekelis, chief scientist at Gradient, explains how they transformed an open-source model into Albatross, a finance LLM that tops leaderboards on both general and domain-specific tasks. The key was an automated data pipeline using membership inference to curate finance data from a massive corpus, followed by continual pre-training and alignment via supervised fine-tuning and preference optimization. He also details a one-million-token context extension on a Llama 3-based model that achieves 100% needle-in-the-haystack scores, enabling in-context learning with thousands of examples to reduce hallucinations. The models, v-alpha-tross and the extended-context Llama 3, are open-sourced on Hugging Face.

Accelerating Mixture of Experts Training With Rail Optimized InfiniBand Networking in Crusoe Cloud

Feb 12, 2025 · 17:45

Ievgen Bakulenko, product manager at Crusoe Cloud, explains how their rail-optimized InfiniBand networking accelerates training for sparse mixture of experts models. By leveraging NVIDIA's PXN feature, which allows GPUs to communicate across different rails using the internal NVSwitch in a single hop, Crusoe achieves a 50% improvement in synthetic benchmark latency and bandwidth for both small and large messages. In a real-world test fine-tuning the Mixtral model (8 feed-forward blocks, 7 billion parameters) on 240 H100 GPUs, this topology reduced training time by 14%, directly lowering cost and time-to-train. Bakulenko also outlines Crusoe's AI cloud platform, its climate-aligned mission using stranded energy, and its focus on easy-to-use infrastructure for AI engineers.

System Design for Next-Gen Frontier Models — Dylan Patel, SemiAnalysis

Feb 11, 2025 · 18:29

Dylan Patel of SemiAnalysis breaks down the inference challenges for next-generation frontier models like GPT-4 (1.8 trillion parameters) and upcoming models trained on 100,000+ GPU clusters. He emphasizes that prefill (prompt processing) is compute-intensive while decode (token generation) is memory bandwidth-intensive, creating a systems problem where serving 64 users at 30 tokens/second requires 60 terabytes/second of memory bandwidth. Patel details engineering strategies such as continuous batching to improve batch utilization by 10-100x, disaggregated prefill to isolate noisy neighbors and maintain time-to-first-token SLAs, and context caching (like Google's) to cache KV cache on CPU/storage instead of GPU memory, dramatically reducing prefill costs. He warns that open-source tools like LLaMA.cpp lack these optimizations, making high-performance serving of models like LLaMA 405b infeasible without libraries like vLLM or TensorRT-LLM. On scaling, Patel notes that 100,000 GPU clusters (e.g., Microsoft's Arizona data center consuming 150 MW) face reliability issues — optical transceivers fail every five minutes — and straggler chips (silicon lottery) can degrade training…

LLM Quality Optimization Bootcamp: Thierry Moreau and Pedro Torruella

Feb 8, 2025 · 53:05

Thierry Moreau of OctoAI demonstrates how to fine-tune Llama 3 8B on a PII redaction task using OpenPipe and OctoAI, achieving 47% better accuracy and a 200x cost reduction (from $30 to $0.15 per million tokens) compared to GPT-4 Turbo. He explains that fine-tuning should follow prompt engineering and RAG, and works best for specialized tasks like function calling. The talk walks through building a fine-tuning dataset from the PI Masking 200k dataset, using OpenPipe to train a LoRA for $40, deploying it on OctoAI, and evaluating it to show the fine-tuned model scores 0.97 accuracy versus GPT-4’s 0.68. Moreau emphasizes that this continuous deployment cycle requires monitoring data drift and retraining, but tools like OpenPipe and OctoAI make it accessible even for teams without deep ML expertise.

Insights from Snorkel AI running Azure AI Infrastructure: Humza Iqbal and Lachlan Ainley

Feb 8, 2025 · 20:46

Humza Iqbal of Snorkel AI explains how the company uses Azure AI infrastructure powered by NVIDIA GPUs to fine-tune foundation models for enterprise customers, achieving better performance per dollar by switching from A100s to H100s. He details their distributed training stack (PyTorch, Horovod, NFS) and lessons learned such as balancing node count for batch size and monitoring GPU utilization to avoid networking or data-loading bottlenecks. A cost comparison found two H100s outperformed four A100s on both training and inference, enabling faster iteration through more synthetic data. Azure's dedicated VMs, reliable NFS throughput, and flexible capacity allowed Snorkel to scale experiments from single-node to dozens of GPUs. Future work includes programmatic preference signals and multimodal retrieval algorithms, all planned on Azure.

Mastering LLM Inference Optimization From Theory to Cost Effective Deployment: Mark Moyou

Jan 1, 2025 · 33:39

NVIDIA solutions architect Mark Moyou explains that LLM inference differs fundamentally from standard deep learning deployment, requiring careful management of KV Cache, attention mechanisms, and GPU memory to control cost. He details how tokens are processed: prefill computes attention across the entire prompt, then generation produces one token at a time, with KV Cache storing key-value pairs to avoid recomputation. Llama's 32 attention heads and FP8 quantization (halving memory with near-identical accuracy) are cited as key optimizations. Moyou emphasizes measuring time to first token, inter-token latency, and input/output sequence length distributions to size inference engines. He presents NVIDIA's TRT-LLM (model compilation for LLMs) and Triton inference server as tools to maximize throughput, and discusses how query patterns like long-input-short-output or short-input-long-output impact GPU utilization and deployment cost.

From model weights to API endpoint with TensorRT LLM: Philip Kiely and Pankaj Gupta

Sep 13, 2024 · 1:40:01

Philip Kiely and Pankaj Gupta of Baseten lead a workshop on TensorRT-LLM, NVIDIA's high-performance inference framework for LLMs, arguing its use delivers best-in-class throughput and latency on NVIDIA GPUs. They explain that TensorRT-LLM optimizes computational graphs via plugin kernels and in-flight batching, achieving 216 tokens per second and 180ms time to first token on Mistral 7B. The workshop demonstrates building an engine for TinyLlama 1.1B, including FP8 quantization that reduced engine size from 2 GB to 1.2 GB with minimal quality loss. They benchmark a deployed model, showing 7,000 total tokens per second at batch size 64 on an A10G. The presenters compare TensorRT-LLM favorably to VLLM for high-throughput production use and introduce Truss, Baseten's open-source packaging tool, alongside their managed platform for automatic scaling and fast cold starts.

Low Level Technicals of LLMs: Daniel Han

Jul 31, 2024 · 2:52:26

Daniel Han of Unsloth explains how to find and fix bugs in open-source LLMs like Gemma, Phi-3, and Llama, covering tokenizer issues, architecture pitfalls, and finetuning optimizations. He details the eight Gemma bugs Unsloth fixed, including a critical RoPE downcasting error that broke positional encoding, and a 2048 sliding window bug in Phi-3. Han walks through transformer internals: attention masking, layer norms, RoPE embeddings, and SwiGLU activation, showing how to derive gradients for custom kernels. He demonstrates Unsloth's 2x faster finetuning with 70% less memory via Triton kernels, and introduces new features: automatic Ollama model file creation, CSV fine-tuning with merged columns, and chunked cross-entropy for large vocabularies. The session includes live Q&A on learning rate schedules, precision trade-offs, and mechanistic interpretability.

Unlocking Developer Productivity across CPU and GPU with MAX: Chris Lattner

Jul 25, 2024 · 18:33

Chris Lattner, CEO of Modular, presents MAX, a unified AI framework that accelerates Gen AI inference by combining CPU and GPU programming into a single Pythonic model, and Mojo, a new programming language that extends Python to systems programming with 100–1000x speedups. Lattner argues that current fragmentation across PyTorch, ONNX, TensorRT, and hardware-specific libraries slows innovation, and MAX replaces the entire stack—including cuDNN and Intel MKL—with a consistent, compiler-driven approach. He demonstrates that MAX's Int4/Int6 quantization achieves 5x faster performance than llama.cpp on cloud CPUs, and that its GPU matrix multiplication beats NVIDIA's cuBLAS by up to 30%. Mojo enables developers to write for loops and tokenizers (e.g., for LLaMA 3) in Python-like syntax without dropping to C++ or Rust. MAX is free and available now for CPU inference; GPU support launches in September with early access via Discord.

Open Challenges for AI Engineering: Simon Willison

Jul 17, 2024 · 18:49

Simon Willison argues the GPT-4 barrier has been broken as GPT-4o, Claude 3.5 Sonnet, Gemini 1.5 Pro, and open models like LLaMA 3 70B now compete, making GPT-4-class models a commodity. He highlights the AI trust crisis with examples of Dropbox and Slack being falsely accused of training on user data, and notes Anthropic trained Claude 3.5 Sonnet without customer data. Willison warns about prompt injection vulnerabilities, citing the Markdown image exfiltration bug affecting six major chatbots, and defines slop as unreviewed AI-generated content, calling for accountability and responsible use patterns.

Llamafile: bringing AI to the masses with fast CPU inference: Stephen Hood and Justine Tunney

Jul 16, 2024 · 17:25

Stephen Hood and Justine Tunney present Mozilla's Llamafile project, which turns AI model weights into single-file executables that run on any OS and CPU without installation, democratizing access to AI. They claim CPU inference can match GPU performance through techniques like outer-loop unrolling in matrix multiplication and using a GPU-like programming model with sync threads, achieving 30-500% speed increases. Justine demonstrates a summarization task where the optimized version completes in seconds versus the old version's many seconds. Hood announces the Mozilla Builders accelerator offering $100,000 in non-dilutive funding for open-source local AI projects, emphasizing that individuals and small groups can still make impactful contributions in AI.

Retrieval Augmented Generation in the Wild: Anton Troynikov

Nov 15, 2023 · 12:20

Anton Troynikov, co-founder of Chroma, explains that retrieval augmented generation (RAG) requires more than simple vector search—it needs human feedback, self-updating memory, and agent interaction to handle dynamic data. He covers challenges like choosing the right embedding model, chunking strategies (including using language model perplexity), and determining result relevance without distractors. Chroma is building a horizontally scalable cluster, a cloud technical preview by December, and support for multimodal data. The episode argues that a capable memory system is key to making AI agents truly functional, citing the Voyager paper where Chroma stored learned skills for Minecraft agents.

Domain adaptation and fine-tuning for domain-specific LLMs: Abi Aryan

Nov 14, 2023 · 25:09

Abi Aryan's talk covers domain adaptation and fine-tuning for large language models, contrasting prompting, RAGs, and three fine-tuning methods: adaptive, behavioral, and parameter-efficient. Adaptive fine-tuning adds small adapter modules (0.15% of parameters) for new domains like biochemical engineering; behavioral fine-tuning optimizes label space for a single task; and parameter-efficient methods like LoRA and QLoRA reduce model size via low-rank adaptation and four-bit precision, ideal for low-resource devices. Aryan emphasizes data quality—deduplication reduces memorization—and practical tips: batch size of 32 or 64, starting with 100 epochs, using Adam optimizer, gradient checkpointing for memory savings, and in-context learning with dynamic examples to handle drift. Evaluation should combine metric-based (Bleu, Rouge), tool-based (Weights & Biases), model-based, and human-in-the-loop approaches, though full pipeline considerations (data collection, base model choice, storage) are critical for robust applications.

[Workshop] AI Engineering 201: Inference

Nov 7, 2023 · 1:43:16

Charles Frye, instructor of the Full Stack LLM Bootcamp, leads a workshop on AI engineering inference, focusing on the build-versus-buy decision between proprietary and open models. He argues that proprietary models like OpenAI's GPT-4 and Anthropic's Claude are currently more capable but expensive, while open models like LLaMA 2 are less capable but offer hackability, though they may catch up if capabilities requirements saturate. Frye covers inference on end-user devices, noting that running models locally avoids network latency but faces tight memory and power constraints—e.g., a 7B parameter model requires 14 GB, too large for phone RAM. He explains inference-as-a-service (e.g., OpenAI, Replicate) versus self-serving on cloud GPUs or serverless platforms like Modal, highlighting that memory bandwidth is the bottleneck: GPUs have 1.5 TB/s memory bandwidth vs 312 TFLOPS compute, so batching is crucial for throughput. Frye also discusses inference arithmetic, custom silicon like TPUs which offer ~30% better efficiency but not drastic gains, and containerization challenges for GPU workloads.

Building AI For All: Amjad Masad & Michele Catasta

Oct 23, 2023 · 25:13

Amjad Masad and Michele Catasta announce Replit's AI for all, giving free AI-powered coding to millions of users. They detail the training of Replit Code v1.5, a 3.3B parameter model trained on 1 trillion tokens of code, which outperforms StarCoder 3B and approaches Code LLaMA 7B in Human Eval despite being half the size. The model is optimized for low-latency inference, generating over 200 tokens per second per GPU. Catasta explains the data pipeline using permissively licensed code from The Stack and Replit public repos, with five epochs of repeated high-quality data. The model is released open-source with a commercially permissive license. Partnerships with Glaive AI for instruction fine-tuning, Morph Labs for a novel fill-in-syntax-tree format, and Perplexity for fast model serving are also unveiled.