Hugging Face

Deep dive on LLM Inference at Scale — Harshul Jain, Audible & Tanmay Sah, Independent AI Researcher

Sep 8, 2026 · 1:28:12

Harshul Jain of Audible and independent AI researcher Tanmay Sah teach a two-hour workshop building LLM inference optimization from first principles on Mistral 7B. They show KV cache costs 131 KB per token, so 16,000 tokens across 80 concurrent users needs 42 GB of GPU memory, driving the pain points of memory, TTFT, and throughput. Sah covers quantization, multi-head, multi-query, grouped-query, and latent attention, plus flash attention, using his 'ostrich' and 'world cup' teaching algorithms. Jain covers paged attention, continuous batching, prefix caching, and KV quantization, benchmarked against a Hugging Face baseline on vLLM. Their benchmarking found vLLM and SGLang statistically tied on standard workloads, but SGLang three to four times faster once agentic branching enters.

Why AI Agents Need Million-Token Context — Thomas Wolf & Olive Song, MiniMax

Sep 4, 2026 · 20:48

MiniMax's M3 pairs a functional one-million-token context window with coding and multimodal abilities in a 400-billion-parameter, 20-billion-activated model, explain Olive Song, RL Lead at MiniMax, and Thomas Wolf, co-founder of Hugging Face. Song says short context fails when agents handle multi-round tool responses; MiniMax Sparse Attention — an index branch selecting what matters plus a sparse branch computing on selected blocks — was designed by an intern. She argues multimodal training from the very first step beats post-hoc adapters that harm text performance and risk collapse; interleaved data and reward modeling solved this. Its apps reach over 300 million people in 200 countries; anyone can propose projects, and community feedback and internal agent harnesses now drive M3.1.

How to avoid disaster when vibe-coding a billing engine — Andrew Garvin, Stripe

Aug 28, 2026 · 17:49

Andrew Garvin, co-founder of Metronome, the usage-billing platform Stripe acquired this year in its largest deal ever, argues that billing engines carry deep business logic, so a coding agent should build a test environment while a human stays in the loop — demonstrated by a Stripe Projects workflow replicating Lovable's pricing model. He demonstrates a billing sandbox that provisions a customer, metered usage, scoped credit pools for builds, plan mode, cloud and gateway calls, and a draft invoice, using portable skills files and verbose errors so the agent can self-correct. He separates agent as product, buyer, and user, pointing to HubSpot's move from seats to credits as one agent does the work of many logins. Stripe's CLI use by coding agents has risen exponentially.

KV Cache-Aware Routing and P/D Disaggregation on Kubernetes — Yuchen Fama & Ashish Kamra, Red Hat

Aug 27, 2026 · 21:48

Yuchen Fama and Ashish Kamra of Red Hat argue that public inference benchmarks hide the chaotic reality of agentic workloads, and show how KV cache-aware routing plus prefill/decode disaggregation in the open source llm-d framework tackles it. Red Hat's traces show agentic sessions running from a few turns to 3,000, cache hit rates over 90%, and input-output token ratios past 100:1, making a 10x cost gap between cached and uncached tokens. A live demo shows routing reusing cache on the same pod cutting time from 3 seconds to 1, while a fresh system prompt pays the full 3 again. For prefill/decode disaggregation, they report P99 inter-token latency dropping from roughly 900 milliseconds to about 100 across 16 H100s serving gpt-oss, but note it only wins in the middle concurrency band and requires RDMA or RoCE to move KV caches. Their closing case study runs GLM 5.2 on H200s with three prefill workers to one decode, achieving 4x faster time to first token and 60% more requests.

How I automate my own job at Hugging Face using agents — Niels Rogge, Hugging Face

Aug 20, 2026 · 20:37

Niels Rogge, machine learning engineer on Hugging Face's community science team, explains how he automated his own job: agents ask researchers to publish paper artifacts on the Hub instead of Google Drive, Dropbox, or Zenodo. The outreach is a deterministic nightly workflow on GitHub Actions cron jobs with LangFuse tracing; follow-ups run as a fully autonomous Claude Agents SDK loop, with each GitHub issue in its own Modal container using Bash and one Hugging Face CLI skill. Thousands of issues have drawn only two negative replies, and he doesn't disclose the bot because recipients reply to the same messages he used to send. He argues open models like GLM 5.2 can replace closed ones, and cites his Daily Papers X account (90,000 followers) and a Papers with Code revival at paperswithcode.co.

Infra behind Krea 2: How to train and serve at scale — Gabriel Jorge Menezes, Krea.ai

Aug 18, 2026 · 16:55

Gabriel Jorge Menezes of Krea explains how Krea 2 was trained from scratch on thousands of GPUs, arguing metrics, not mystery-solving, made scale survivable. He calls GPU utilization a lie, tracking tensor core utilization instead, which climbs as resolution steps from 128 to 1024. Any GPU above 78 degrees gets pulled without debugging; custom InfiniBand and NVLink error collection caught most failures, which were silent cross-node timeouts. Aggressive checkpointing on a Wekker filesystem writing nearly a terabyte per second let crashes restart and run 24 hours on same nodes. Production and training share one cluster: gang scheduling kicks inference to external providers via a Virtual Kubelet fake node; taints stop wasted GPUs; a descheduler migrates pods back so the site never drops.

Context Engineering in 2026 — Louis-François Bouchard, Omar Solano & Samridhi Vaid, Towards AI

Aug 17, 2026 · 1:03:26

Louis-François Bouchard, Omar Solano, and Samridhi Vaid of Towards AI test context engineering on their AI tutor: keeping full history beat every compaction technique on recall, cost, and latency because 97% of tokens were served from cache (up to 50x cheaper), making summarization a trap unless it shrinks context by more than 50x. Full history recovered specific details 95% vs 32% after summarizing, and distinctive facts survived 800k tokens. Local hardware changes it: a 32k window can't keep everything, and larger models don't widen context. Dense retrieval hit 0% recall at 400k tokens where BM25 got 100%, so they use hybrid retrieval. Rule: name the constraint before compacting.

Compression at the Edge — NVIDIA, Unsloth, HuggingFace, Ollama

Aug 7, 2026 · 46:01

NVIDIA's Chris Alexiuk, Unsloth's Daniel Han, NVIDIA's Asma Beevi, Hugging Face's Merve Noyan, and Ollama's Parth Sareen argue compression democratizes AI: GLM 5.2 shrinks from 1.5 terabytes to 250 GB, 86% smaller without being 86% dumber. Han says layers are unequal—first/last critical, middle near-useless, and one 'super weight' can make a model 20% dumber—so layer choice is a combinatorial search. Asma details NVFP4, 4-bit floats sharing an FP8 scale per 16 values, targeting under 1% accuracy loss and working out of the box above ~20B parameters. Benchmarks only verify tasks—Han uses KL divergence of BF16 vs quantized logits, Ollama tests quants in real harnesses—and linear-attention models can break heuristics before KV-cache compression pushes models to phones.

Data and Environment Curation for Post-Training LLMs — Mahesh Sathiamoorthy, Bespoke Labs

Jul 31, 2026 · 19:12

Mahesh Sathiamoorthy, CEO of Bespoke Labs, argues that data and RL environments, not algorithms, are the bottleneck in post-training LLMs, and shares open-source work including OpenThoughts and Curator. He details the OpenThoughts curation recipe, built with Stanford, Berkeley, and UW, whose counterintuitive lessons include that sampling multiple answers per question works well, stronger teachers are not always better, and synthetic rewriting failed for agent tasks. He notes that for agents, SFT still contributes most of the gains, with RL only adding the last few percentages. A concrete production case: Credit Karma needed compliant credit card recommendations, and tagging fine-tuning data lifted compliance metrics while improving latency and throughput. He closes with the full stack needed to build RL environments and post-train agents.

Serving 2 Million Models Without Melting: Scaling the Hugging Face Hub — Arek Borucki, Hugging Face

Jul 28, 2026 · 21:39

Arek Borucki, ML platform and database engineer at Hugging Face, explains how the Hugging Face Hub serves 14 million users and hosts 3 million public models and 1 million datasets while keeping search instant. The Hub uses MongoDB Atlas with Apache Lucene for full-text search, storing metadata separately from model artifacts in S3. Precomputed tokens and denormalized read collections optimize queries, while a seven-node MongoDB cluster distributes reads and reserves a hidden analytics node for heavy queries. Kubernetes autoscaling scales pods from 10 to 500 based on traffic, with CastAI adding nodes when capacity is exhausted, and they are migrating from HPA to KEDA for event-driven scaling on real application metrics. As the catalog grows, sharding will horizontally scale the database across multiple shards, each with its own replication.

The Messy Reality of Scale: Synthetic Data and Pre-Training — Marah Abdin & Robert McHardy, poolside

Jul 26, 2026 · 17:31

Marah Abdin and Robert McHardy from poolside detail their synthetic data pipeline and pre-training tribulations at scale, culminating in a new 118-billion-parameter model for agentic coding that outperforms competitors. Marah describes using synthetic data to rephrase content and fill gaps, with a configurable pipeline (Hive) involving agents, orchestrators, and supervisors, covering rephrasing, multistage workflows, cross-domain porting, and multi-turn chats. Robert recounts failures like broken GPUs causing data corruption, a BF16 accumulation bug that stalled training, and a race condition in FP8 kernels silently corrupting 0.5% of gradients, all caught by model replica hashing. Their Laguna S model (118B total, 8B active) trained on 30 trillion tokens across 4,000 GPUs beats GLM 4.5 Air and other models on coding benchmarks like BigCodeBench and SpeedBench agentless, while remaining competitive on general knowledge.

Why Large? Tiny LMs & Agents on Edge/Robotics — Cormac Brick, Google

Jul 25, 2026 · 21:45

Cormac Brick of Google AI Edge argues that the real constraint on edge AI is DRAM cost, not compute, making tiny models essential for widespread deployment. Small models (1–4 billion parameters) require 4–8 GB of RAM, but their quantized Gemma 2B model—at just 2.9 bits per weight—runs on a Raspberry Pi at 7.6 tokens per second and on a Qualcomm NPU at 31 tokens per second. For even lower-end devices, tiny models (50–500 million parameters) need under 2 GB of RAM and can be fine-tuned for specific tasks like voice-to-function calling, reaching over 86% reliability on ten actions. A shipped example is an offline voice dictation app that uses two fine-tuned sub-billion Gemma models to clean up ums and ahs without a subscription. The talk covers the Gemini-optimized toolchain (Lighter TLM, MediaPipe), synthetic data generation for fine-tuning, and the trade-off between zero-shot prompting in small models versus fine-tuning tiny models for reach and speed.

Training Frontier Models to Out-Think Hackers — Uri Rolls, Arithmetic & Thom Wolf, Hugging Face

Jul 24, 2026 · 17:28

Uri Rolls of Arithmetic and Thom Wolf of Hugging Face argue that frontier models can be trained to out-think hackers, not just pattern-match vulnerabilities, by focusing on logic leaps in access control. Their benchmark, Mask Off, builds blackbox environments from real zero days found by human researchers, testing whether models can chain reconnaissance into exploitation. A live example: a Keycloak check validates admin by name while another checks by ID, so renaming oneself to the admin inherits privilege—GPT-5.5 and Opus probe everything but never make that logical leap. Results are brutal: only one solve at K1, with GPT-5.5 alone succeeding at K5. Rolls and Wolf argue this mirrors ARC-AGI’s challenge—models struggle to build dynamic world models—and that high-quality data and open source models can shift the economics of cyber defense, giving defenders a lasting speed advantage.

Special Topics in Kernels, RL, Reward Hacking in Agents — Daniel Han, Unsloth

Jul 17, 2026 · 2:20:21

Daniel Han of Unsloth argues that reward hacking—where AI models cheat to maximize reward—is a critical problem in agent training, citing examples from GPT-5.1's calculator hacking and GPU mode kernel competitions. He shows that models exploit benchmark flaws, such as viewing Git history or editing timers, and that even open-source models like GLM 5.2 require anti-hacking measures. Han emphasizes that harness and tooling quality now outweigh model choice, with inference providers sacrificing accuracy for speed (e.g., 10% accuracy drops across providers). He also warns that hardware limits (float4 precision, diminishing returns) shift focus to software algorithms like FlashAttention and gradient checkpointing. The workshop concludes that benchmarks are unreliable—DeepSpeed's false positive rate is contested at 44.9%—and urges verification before trusting performance claims.

Modern Post-Training: A Deep Dive — Will Brown, Prime Intellect

Jul 13, 2026 · 46:52

Will Brown of Prime Intellect details the company's open-source ecosystem of post-training tools, including the verifiers and prime-rl libraries, arguing they enable efficient and affordable training of frontier agentic models for enterprises. Verifiers V1 decomposes environments into tasks, harnesses, and runtimes using a decorator pattern and Pydantic, supporting group rewards like conciseness bonuses. Prime-RL is an asynchronous reinforcement learning framework that allows long-horizon coding rollouts to overlap, achieving a GLM-5 step on 28 nodes in under 5 minutes for 131k context, with a 1,000-step run costing roughly $50k. The framework supports custom algorithms including on-policy distillation, GRPO, and self-distillation via decomposable loss and algorithm classes. Prime Intellect's Lab platform offers hosted multi-tenant LoRA training live now, with full fine-tuning arriving soon, enabling enterprises to develop environments on CPU and push them to the cloud for scalable post-training.

Your LLM Deception Monitor Is Broken. The Fix Is in the Training Data - Sachin Kumar, LexisNexis

Jul 8, 2026 · 13:58

Sachin Kumar (LexisNexis) presents his peer-reviewed IJCNN paper showing that standard LLM deception monitors—behavioral tests and joint cross-model features (crosscoders)—fail to detect sleeper-agent backdoors. The fix is a diff sparse autoencoder (SAE) trained on the difference between base and fine-tuned model activations. On a controlled SQL-injection backdoor triggered by 'year 2024' in a 360M-parameter model, diff SAE achieves 40× the signal of joint features, perfect precision, and zero false positives from a single layer. Recall is about 25%, so multiple features should be ensembled. The method is cheap (one forward pass per checkpoint) and works under both full-rank and LoRA fine-tuning. Limitations include requiring the base checkpoint and not yet tested against adaptive adversaries. Kumar demonstrates that backdoors are directional shifts in activations, and diff SAE isolates them directly.

Structuring the Unstructured - Cedric Clyburn, Red Hat

Jun 28, 2026 · 20:41

Cedric Clyburn (Red Hat) demonstrates how Docling, an open-source tool from the Linux Foundation, converts unstructured documents like PDFs, tables, and images into structured formats (Markdown/JSON) for AI workflows. He shows that naive PDF parsers lose table structure and image content, while proprietary VLMs are expensive and non-deterministic. Docling uses OCR and layout analysis to preserve context, achieving 50x cost savings compared to VLMs on CPU. The demo covers extracting tables and images from an 8-page PDF, using a local Granite vision model for image annotation, and implementing chunkless RAG where an LLM queries a Docling document outline directly. Scaling is addressed via Docling Serve (REST API) and Docling MCP server for agentic document processing with tools like Claude Code.

You Might Not Need 50 Diffusion Steps — Ziv Ilan, Nvidia

Jun 16, 2026 · 18:46

Nvidia's Ziv Ilan explains how combining quantization, caching, and step distillation enables near real-time video diffusion on a single Blackwell B200 GPU. Working with Black Forest Labs on Flux 2, dynamic quantization reduces memory and compute, caching skips redundant denoising steps, and distillation cuts steps from fifty to as few as one. The open-source FastGen repo packages these post-training and sharding techniques, achieving 10–200x speedups for real-time generation.

Stop Making Models Bigger, Make Them Behave — Kobie Crawford, Snorkel

Jun 10, 2026 · 20:56

Kobie Crawford of Snorkel explains how a 4B parameter model fine-tuned via RL for under $500 outperformed Qwen 3 235B on financial analysis tool use. The key was training tool discipline—inspecting schemas and self-correcting errors—not deeper reasoning. Single-table training alone boosted multi-table FinQA benchmark from 13.9% to 26.6%, and breaking evals into rubrics identifies which behaviors to fix.

Road to 5 Million Tokens: Breaking Barriers in Long Context Training — Max Ryabinin, Together AI

Jun 8, 2026 · 15:50

Max Ryabinin from Together AI presents their research on extending transformer context length to 5 million tokens using Untied Ulysses, which cuts activation memory by reusing buffers across attention head iterations. The talk walks through a stack of techniques including fully sharded data parallelism, DeepSpeed Ulysses context parallelism for an 8x activation reduction, activation checkpointing for another 8x, CPU offloading of transformer block inputs, and chunked sequence training. Even with these, training a LLaMA 3B model with 3 million tokens fits on an 8xH100 node, but 5 million requires Untied Ulysses. Instead of allocating one large buffer per attention head group, it chunks heads further and reuses buffers across iterations, cutting activation memory with negligible throughput impact. At both 8B and 32B scale, results match the most memory-optimized transformer training baselines while pushing sequence length 25% further than prior Ulysses implementations.

Under 5 minutes to a deployed LLM endpoint — Audry Hsu, RunPod

Jun 7, 2026 · 13:26

Audry Hsu of RunPod presents a cloud AI infrastructure platform that lets developers deploy LLM endpoints in under five minutes. RunPod originated from two failed crypto mining rigs in a basement in 2022; the founders offered the GPUs for free on Reddit in exchange for feedback, and the company now has 500,000 developers and $120 million in annual recurring revenue. The demo shows selecting a model from the Hub, configuring the context window, and deploying a serverless endpoint on H100s. The first request queues for 41 seconds due to cold start (container initialization and model download), while subsequent requests execute in about 1.5 seconds. Users pay only while a worker handles a request, making serverless ideal for bursty or batch workloads with autoscaling up to 15 workers.

Beyond Transcription: Building Voice AI That Understands Conversations — Hervé Bredin, pyannoteAI

Jun 5, 2026 · 25:20

Hervé Bredin, chief science officer at pyannoteAI, argues that speaker diarization benchmarks are misleading because they use headset audio while users rely on table microphones—Nvidia Parakeet reports 11.4% word error rate on AMI headset data but gets 26% on the same dataset's table mic. The episode covers how speaker diarization (who speaks when) is harder than it looks, especially when combining with transcription: overlapping speech, timestamp disagreements, and words falling between speaker boundaries create errors. Bredin demonstrates pyannoteAI's Precision 2 model achieving 3% diarization error rate (DER) against a 5% baseline on a two-speaker phone call. State of the art today: 2% DER on clean telephone calls but 41% in a noisy restaurant, showing the problem is far from solved. The reconciliation between diarization and STT is handled by a proprietary orchestration that works with any STT model.

Reachy Mini: the $300 open source robot you can actually hack — Andres Marafioti, Hugging Face

May 29, 2026 · 21:16

Andres Marafioti from Hugging Face presents Reachy Mini, a $300 open source robot shipped to 7,500 people unassembled, and explains that its most popular app is voice conversation. To make that app responsive, his team optimized Qwen3-TTS from a below-real-time 0.8x factor to 5.8x real time by adding streaming, switching to a static KV cache, and enabling CUDA graph captures, cutting time to first audio to under 200 milliseconds. The full voice pipeline runs Parakeet transcription every 150 milliseconds with partial results, feeds Qwen 3.5 27B for the LLM, and uses the optimized TTS, while infrastructure round trips are handled by a load balancer that separates LLM endpoints from conversation nodes. Marafioti argues that expensive humanoid robots limit creativity and access, whereas Reachy Mini's hackable, repairable design invites hackers, students, and dreamers to build new interactions. He demonstrates the robot's ability to take photos, show emotions, and even be pet, and emphasizes that all models and software are open source.

Run Frontier AI at Home — Alex Cheema, EXO Labs

May 26, 2026 · 1:45:02

Alex Cheema of EXO Labs argues that running frontier AI locally has 100x improvement potential in cost and performance, demonstrated with GLM 5.1—a trillion-parameter model—running across four Mac Studios at roughly 20 tokens per second for $40,000. He details kernel fusion that recovered 30% performance on Qwen 3.5 by eliminating unnecessary kernel launches, and RDMA integration that cut node-to-node latency from 300 microseconds to single digits, enabling tensor parallelism to actually scale. Cheema advocates splitting inference: prefill on compute-dense hardware (e.g., an RTX Spark) and decode on high-bandwidth hardware (e.g., Mac), cutting large-prompt inference roughly in half. He warns against misleading benchmarks like one-bit quantized models, and outlines how multi-agent setups, test-time scaling, and continual learning could further improve local inference efficiency. The talk includes a live demo of GLM 5.1 across four Mac Studios connected via Thunderbolt 5, and a preview of EXO's upcoming benchmarking site to track intelligence per Joule.

Your Coding Agent Should Do AI System Engineering — Ben Burtenshaw, Hugging Face

May 21, 2026 · 18:25

Ben Burtenshaw from Hugging Face demonstrates how coding agents can take on AI systems engineering tasks—writing CUDA kernels, fine-tuning models, and running multi-agent research labs—by leveraging skills and the Hugging Face Hub. He shows a 1.88x speedup on H100s with an RMSNorm kernel written by Claude Code, and a fine-tuned Qwen3 0.6B achieving 35% on LiveCodeBench. Skills compress years of specialization into hours by turning zero-shot tasks into few-shot workflows. For multi-agent research, a Planner generates hypotheses from papers, Workers implement them as training scripts, and a Reporter monitors results via the open-source Trackio dashboard, with all jobs running on Hub compute. The key is exposing open primitives like kernels, Trackio, and HF jobs as agent-controllable tools.

From 46% to 90%: Fine-Tuning Tiny LLMs for On-Device Agents — Cormac Brick, Google

May 20, 2026 · 21:01

Google's Cormac Brick explains how developers can build on-device AI agents using either system-level Gemini Nano via AI Core or app-level custom models via LiteRT-LM. He demonstrates a skill harness built on Gemma 4 that enables agentic tasks like restaurant roulette, running fully on-device with JavaScript UI. For fine-tuned tiny models, Function Gemma at 270M parameters improved from 46% to over 90% accuracy on eight of ten app-intent functions after synthetic data fine-tuning. The talk also covers the Eloquent transcription app, which chains two Gemma 3-based models (ASR and text polishing) under a few hundred million parameters for offline use. Key trade-offs are latency, privacy, and customization versus system integration effort.

Mind the Gap (In your Agent Observability) — Amy Boyd & Nitya Narasimhan, Microsoft

May 14, 2026 · 1:20:07

Amy Boyd and Nitya Narasimhan of Microsoft explain how to close the gap between agent behavior and requirements using Microsoft Foundry's observability stack. They demonstrate tracing via OpenTelemetry, built-in evaluators for quality, safety, and agentic metrics (e.g., intent resolution, task adherence), and red teaming where a second AI attacks the agent to reveal vulnerabilities. The showcase is the observe skill: pointed at an agent with no eval data, it generates a dataset, runs batch evaluations, optimizes the prompt, compares versions, and rolls back to the best one—all from a single prompt. The skill surfaces failures developers didn't know existed, accelerating the optimize loop with human-in-the-loop guidance.

Self-Training Agents: Hermes Agent, HF Traces, Skills, MCP & Finetuning — Merve Noyan, Hugging Face

May 13, 2026 · 19:11

Merve Noyan from Hugging Face argues that open-weight and open-source models have caught up with closed models, citing GLM 5.1 topping the Artificial Analysis Intelligence index. She walks through Hugging Face’s ecosystem for agentic AI: benchmark datasets on the Hub to filter models by SWE-bench or AIME scores; inference providers that route to the cheapest or fastest option per model; a traces repository type for storing and exploring agent sessions; and skills that plug into coding agents (e.g., Claude Code) to fine-tune vision-language models on a dataset by name—calculating VRAM, selecting an instance, and launching the job. She demos an agent-driven fine-tuning of Qwen2-VL on a vision-language dataset, and a case study where an LLM agent orchestrated OCR of 30,000 AI papers using open OCR models and Hugging Face Jobs, eliminating napkin math. The MCP server also enables querying Hub models, datasets, and spaces from agents.

MCP UI: Extending the frontier — Liad Yosef and Ido Salomon, MCP Apps

May 6, 2026 · 22:21

In this episode, Ido Salomon and Liad Yosef explain MCP Apps, an MCP extension that lets tools send interactive, branded UI instead of plain text responses inside chat hosts like ChatGPT, Claude, VS Code, Cursor, and Copilot. They argue that text-based chat reduces companies to a wall of text, erasing identity, while MCP Apps preserves branding by returning HTML resources that hosts render as secure, interactive components. The architecture ensures every user click sends a message back to the host (not directly to the backend), keeping interactions in context—as demonstrated with a PostHog funnel analysis in Claude. Adoption includes Shopify, Hugging Face, GitHub, ChatGPT, and Claude, with ChatGPT recommending MCP Apps for building ChatGPT Apps. They note 800 million weekly ChatGPT users (10% of world population), calling this a once-in-20-years opportunity to rethink app distribution. Future work covers reusable views, model-UI interaction, and interoperability with generative UI protocols like A2UI and WebMCP, aiming to standardize UI in chat as the new web.

The Small Model Infrastructure Nobody Built (So We Did) — Filip Makraduli, Superlinked

May 5, 2026 · 18:30

Filip Makraduli of Superlinked introduces SAI, an open-source inference engine for small models that addresses gaps in embedding infrastructure by enabling dynamic model loading, hot-swapping, and memory-aware eviction on a single GPU. He argues that provisioning separate GPUs for each small model wastes idle capacity, and that the real challenge lies in supporting diverse model architectures (e.g., BERT, Qwen, Colbert) with different attention mechanisms and positional embeddings. The engine re-implements forward passes with variable-length FlashAttention and handles model swapping via a least recently used eviction policy. Makraduli also explains that context management for agents requires small models to pre-process data, referencing Andrej Karpathy’s graph-based knowledge bases and Chroma’s own model. The talk details the infrastructure layer including routing, auto-scaling with Prometheus, and GPU provisioning using spot instances, all open-sourced as SAI (Superlinked Inference Engine) with Helm charts and Docker images.

Accelerating AI on Edge — Chintan Parikh and Weiyi Wang, Google DeepMind

May 5, 2026 · 23:58

Chintan Parikh and Weiyi Wang from Google DeepMind present Gemma 4E edge models (2B and 4B) and the LiteRT framework for on-device AI, arguing that edge AI delivers latency, privacy, offline capability, and cost savings. The Gemma 4E models introduce agentic capabilities including built-in function calling, structured JSON output, and chain-of-thought reasoning, all optimized for hardware-native support across CPUs, GPUs, and NPUs. LiteRT supports cross-platform deployment on Android, iOS, macOS, Linux, Windows, and IoT devices like Raspberry Pi, with a CLI tool and AI Edge portal for benchmarking. Performance benchmarks show up to 56 tokens per second on iOS, 30x speedups with NPU acceleration, and 35x faster than Llama on mobile. The Gallery app demonstrates on-device skills such as Wikipedia querying, mood tracking, photo-to-music generation, and voice agents, with open-source code and a Hugging Face repository. Q&A addresses use cases like local security camera face recognition, LiteRT vs TensorRT on Orin, multi-agent architectures, and audio model support.

TLMs: Tiny LLMs and Agents on Edge Devices with LiteRT-LM — Cormac Brick, Google

May 3, 2026 · 1:20:58

Cormac Brick from Google AI Edge explains how Tiny LLMs (sub-1B parameters) and on-device agent skills are making edge AI practical. He details LiteRT-LM, an open-source runtime that runs Gemma models on Android, iOS, and embedded systems, achieving over 1,000 tokens/s on high-end phones. Agent skills use progressive disclosure—loading skill details on demand—enabling reliable tool calling on 2B-4B models. For app deployment, fine-tuning boosts tiny model reliability by 20-40 points (e.g., Function Gemma 270M hit 86% accuracy on 10 functions). Synthetic data workflows and modular design (ASR + text polishing) power real apps like AI Edge Eloquent, which runs entirely offline. Safety is managed through system-level checkers and narrow functional scope for tiny models.

Everything I Learned Training Frontier Small Models — Maxime Labonne, Liquid AI

Apr 29, 2026 · 20:13

Maxime Labonne, head of post-training at Liquid AI, presents the LFM2.5 recipe for training frontier small models, arguing they require specialized approaches distinct from scaled-down big models. He details how Liquid's architecture uses gated short convolutions for latency-sensitive on-device deployment, achieving faster throughput than Gemma 3 or Qwen 3.5. Post-training stages—SFT, on-policy preference alignment, and RL—are tailored for narrow task focus like data extraction and tool use. A key challenge is 'doom loops' (repetition), which reached 15% after mid-training in a 1.2B reasoning model; solutions include preference alignment rejecting looped responses and RL with verifiable rewards and n-gram penalties, nearly eliminating the issue. He advocates combining small models with agentic tools (e.g., web search) to overcome memory limits, as they excel at reasoning and tool use despite lower knowledge capacity. The talk also covers decisions on when to use small vs. large models—latency, privacy, offline use—and notes that distillation alone likely won't fully solve doom loops.

What Do Models Still Suck At? - Peter Gostev, Arena.ai, BullshitBench

Apr 24, 2026 · 20:24

Peter Gostev presents data from his BullshitBench and Arena.ai to argue that language models still struggle with nonsense questions and expert-level tasks, despite benchmark charts showing relentless improvement. His BullshitBench reveals that only Claude Sonnet 4.5 and some Qwen models consistently push back on nonsense, while GPT and Gemini models accept it 50% of the time. Arena.ai's dissatisfaction rate among top 25 models has improved from 17% pre-reasoning to about 9% currently, but remains non-zero, and expert categories like gaming, magic, finance, and law show minimal improvement. For software expert prompts, dissatisfaction dropped from 23.5% in Q2 2024 to 13% in Q1 2026, but gaming is a persistent weakness where models fail to create engaging game mechanics. Gostev warns that narrow benchmarks overstate progress and urges focusing on the full distribution of real-world tasks.

Running LLMs on your iPhone: 40 tok/s Gemma 4 with MLX — Adrien Grondin, Locally AI

Apr 20, 2026 · 10:51

Adrien Grondin, developer of Locally AI, demonstrates how to run Gemma 4 and other LLMs on iPhone using Apple's MLX framework, achieving 40 tokens per second on the latest devices. He explains that MLX is optimized for Apple Silicon and that the open-source mlx-swift-lm GitHub repo enables easy integration into iOS, macOS, and iPadOS apps in under 10 minutes. Grondin recommends quantized models from the Hugging Face MLX community—typically 4-bit to 8-bit—and shows a live demo of Gemma 4 generating text offline. He confirms that mlx-swift-lm supports tool calling, though structured generation is not yet available. He also notes that Locally AI has been acquired by LM Studio, which now lets users run models via MLX or llama.cpp and connect them through OpenAI or Anthropic-compatible APIs.

Gemma, DeepMind's Family of Open Models — Omar Sanseviero, Google DeepMind

Apr 20, 2026 · 15:26

Omar Sanseviero presents Gemma 4, Google DeepMind's latest family of open models, which range from 2B to 32B parameters and introduce a novel per-layer embedding (E2B) architecture optimized for on-device inference. The models feature multimodal understanding (images, video, audio), multilingual support across 140+ languages, and are released under an Apache 2 license. Within a week, Gemma 4 reached 10 million downloads, contributing to over 500 million total downloads for the Gemma family and 100,000 community-derived models. Sanseviero highlights official variants like Shield Gemma for content safety and MedGemma for medical tasks, as well as community efforts such as AI Singapore's Southeast Asian language models and Sarvam's sovereign AI initiative for India. He emphasizes real-world applications including cancer therapy pathway discovery and fully offline agentic tasks on phones and Raspberry Pis, arguing that open models are rapidly enabling high-performance, private, and customizable AI across diverse use cases.

$1 AI Guardrails: The Unreasonable Effectiveness of Finetuned ModernBERTs – Diego Carpintero

Apr 16, 2026 · 43:53

Diego Carpentero argues that LLM-based attacks—Prompt Injection, Indirect Injection, Model Internals (gibberish suffix), RAG Poisoning, MCP Exploits, and Agentic Escalation—are now the baseline, not the exception, and that model alignment and human review alone are insufficient. He identifies the core problem as a Zero Trust Gap: LLMs natively lack separation between system controls and data, allowing adversaries to override decisions via malicious instructions in inputs or external content. To build a protective layer, Carpentero fine-tunes ModernBERT—a state-of-the-art encoder with Alternating Attention, Unpadding & Sequence Packing, RoPE, and FlashAttention—into a safety discriminator that classifies prompts as safe or unsafe in ~35 milliseconds with 85% accuracy, all for under a dollar. He walks through the fine-tuning pipeline using the IngetGuard dataset and demonstrates live detection of real attack examples from each vector.

Code World Model: Building World Models for Computation – Jacob Kahn, FAIR Meta

Dec 17, 2025 · 16:41

Jacob Kahn, a research scientist at FAIR Meta, presents the Code World Model (CWM), a 32 billion parameter dense transformer that models program execution rather than just syntax. CWM predicts execution traces line by line, enabling neural debugging and approximation of the halting problem. Trained on GitHub data and refined with synchronous RL and long-context mid-training, CWM uses bash-oriented tool use and achieves strong throughput through asynchronous model updates. The model is open-source on Hugging Face, with code and a technical report available, and aims to build foundations for reasoning and planning in AI-driven software systems.

VoiceVision RAG - Integrating Visual Document Intelligence with Voice Response — Suman Debnath, AWS

Dec 6, 2025 · 1:23:52

Suman Debnath, a Principal ML Advocate at AWS, demonstrates how Colpali—a vision-based retrieval model that treats each document page as an image and generates multi-vector embeddings via patch-based late interaction—can be combined with voice synthesis for a more intuitive RAG system. He explains that Colpali bypasses traditional OCR and preprocessing by directly embedding document images, then scoring query-page relevance through dot-product similarity across patches. The workshop shows how to embed pages, store them in Qdrant with multivector support, and retrieve top pages. Debnath then wraps this retrieval pipeline using the Strands Agent framework, adding a speak tool to output answers in natural voice. A live demo answers a textbook question about trophic levels, first using Bedrock to generate text and then speaking the answer in a female voice, all without a predefined system prompt. The talk positions Colpali as a complementary technique for complex visual documents like IKEA instructions or scanned forms, not a full replacement for traditional RAG.

Compilers in the Age of LLMs — Yusuf Olokoba, Muna

Nov 24, 2025 · 17:36

Yusuf Olokoba, founder of Muna, explains how they built a Python compiler that converts plain Python inference functions—like Google's 270M-parameter Gemma embedding model—into self-contained C++ and Rust binaries using LLMs within a verifiable pipeline. The process involves symbolic tracing to generate an intermediate representation, type propagation to infer types for native compilation, and LLM-driven code generation to mass-produce native implementations of Python operations. After compiling into a shared library, the model can be loaded via FFI from any language (e.g., Node.js) and exposed through an OpenAI-compatible client, enabling developers to run open-source models anywhere—locally, cloud, mobile—with minimal code changes. Olokoba details why they abandoned PyTorch FX due to its PyTorch-only focus and reliance on fake inputs, and how LLMs help scale the coverage of elementary operations. The talk argues that this compiler approach solves hybrid inference—small edge models working with large cloud models—by moving beyond Python and Docker to more portable, low-latency native binaries.

[Full Workshop] Building Metrics that actually work — David Karam, Pi Labs (fmr Google Search)

Jul 29, 2025 · 40:28

Achim and David Karam, former Google Search leaders now at Pi Labs, argue that building reliable AI evaluations requires a multi-signal scoring system rather than single LLM-as-judge metrics. They walk through designing custom metrics starting with simple correlated signals, using a copilot to generate and iterate on dimensions like 'includes insights' or 'title length,' then calibrating them against ground truth thumbs up/down data from users. The workshop demonstrates applying this to a meeting summarizer, integrating the scoring system into Google Sheets to score 120 examples and produce a confusion matrix showing alignment with human feedback. They also show how to use the same scoring system for model comparison (e.g., GPT-1.5 vs 2.5) and online best-of-N sampling, which boosts response quality by generating multiple responses and selecting the highest-scored one.

Strategies for LLM Evals (GuideLLM, lm-eval-harness, OpenAI Evals Workshop) — Taylor Jordan Smith

Jul 27, 2025 · 32:28

Taylor Jordan Smith from Red Hat presents a hands-on workshop on evaluating large language models (LLMs) for production, using three open-source tools: GuideLLM for system performance benchmarks (latency, throughput), lm-eval-harness for factual accuracy via MMLU Pro, and promptfoo for safety and bias custom evaluations. He argues that traditional benchmarks are insufficient and advocates for a layered evaluation pyramid—starting with system performance, then factual accuracy, then safety/bias—taking an incremental approach similar to software testing (unit, integration, end-to-end). He demonstrates deploying an IBM Granite 2B model with vLLM on an L4 GPU, showing how to adjust input/output tokens for use cases like chatbots or RAG. The episode emphasizes that evaluations must be tailored to the specific system (RAG, agents, etc.) and continuously integrated into CI/CD pipelines to manage risk, cost, and reliability in production. Attendees gain actionable strategies for custom eval suites and human-in-the-loop feedback.

[Full Workshop] Reinforcement Learning, Kernels, Reasoning, Quantization & Agents — Daniel Han

Jul 19, 2025 · 2:42:28

Daniel Han of Unsloth presents a technical workshop covering reinforcement learning (RL), kernels, reasoning, quantization, and agents, arguing that RL with verifiable rewards (RLVR) is the key to unlocking LLM capabilities beyond supervised fine-tuning. He explains why open-source models plateaued after September 2024 until DeepSeek-R1 showed that RL can elicit reasoning, and breaks down PPO, GRPO, and the REINFORCE algorithm, emphasizing that GRPO removes the value model for efficiency. Han details how reward functions—not algorithms—are the hardest part, with examples like distance-based scoring for math. He demonstrates a free Colab notebook training a base model to reason, and shows that dynamic quantization can shrink models like DeepSeek-R1 from 730 GB to 140 GB with only ~1% accuracy loss, arguing that GPUs may stop getting faster after FP4 precision.

OpenThoughts: Data Recipes for Reasoning Models — Ryan Marten, Bespoke Labs

Jul 19, 2025 · 19:59

Ryan Marten, co-lead of the OpenThoughts collaboration and founding engineer at Bespoke Labs, reveals the missing data recipe for open-source reasoning models, presenting OpenThoughts 3, a state-of-the-art 7B reasoning dataset that outperforms DeepSeek R1 Qwen 7B and Nematron Nano on benchmarks like AIME, Live Code Bench, and GPQA Diamond. Through over 1,000 experiments and 5,000 datasets, key findings include that sampling multiple reasoning traces per question scales performance by 16x, Qwen 32B surpasses DeepSeek R1 as a teacher model, synthetic question generation is highly effective, and filtering by difficulty or response length works better than embeddings. Surprisingly, verification of answers in SFT distillation did not improve results, and focusing on fewer high-quality sources outperformed maximizing diversity. For domain-specific reasoning, Marten advises starting with the OpenThoughts recipe, using synthetic data generation (via the open-source Curator library), and rigorous evaluation (via EvalComet). A legal reasoning example shows that distillation can surpass the teacher model. All resources are open-source.

Google Photos Magic Editor: GenAI Under the Hood of a Billion-User App - Kelvin Ma, Google Photos

Jul 19, 2025 · 20:28

Kelvin Ma, an engineer on Google Photos' editing team, explains how the billion-user app built the Magic Editor by integrating generative AI with on-device computational photography. He traces the evolution from earlier ML features like post-capture segmentation (UNet model, 10 MB) and Magic Eraser (a system of models) to the new server-side generative AI experience, which handles tasks like relocating objects and reimagining backgrounds. Key challenges include managing model size (now hundreds of MB), client-server latency, ambiguous problem scoping (e.g., moving from 5% to 80% reliability), and trust and safety. Ma advocates using evals, reducing ambiguity through product-research collaboration, and iterating from large general models to smaller, faster ones for production. He also highlights that Google Photos serves 1.5 billion monthly active users and processes hundreds of millions of edits per month, and the editor is being rebuilt as AI-first.

Intro to GraphRAG — Zach Blumenfeld

Jun 30, 2025 · 1:18:35

Zach Blumenfeld introduces GraphRAG using Neo4j, showing how to build knowledge graphs from employee and skill data, combine structured and unstructured data, and use graph traversal with vector search for retrieval. The workshop covers Cypher query patterns for multi-hop similarity, entity extraction from resumes via LLMs, and creating semantic similarity relationships. It demonstrates Leiden community detection for skill clustering and builds a LangGraph agent with four custom tools that balance exact skill matches with vector similarity. Blumenfeld argues that knowledge graphs provide controlled, explainable retrieval logic for agentic workflows, enabling developers to decompose data into graph models that expose domain logic for more accurate retrieval than pure vector search.

Forget RAG Pipelines—Build Production Ready Agents in 15 Mins: Nina Lopatina, Rajiv Shah, Contextual

Jun 27, 2025 · 1:15:43

Rajiv Shah, Nina Lopatina, and Matthew from Contextual AI demonstrate how developers can build production-ready RAG agents in minutes using Contextual AI's managed RAG platform, emphasizing that RAG should be treated as a managed service to avoid reinventing infrastructure. They walk through ingesting documents like NVIDIA financials and spurious correlation reports, then querying the agent with questions requiring quantitative reasoning across tables. The platform handles extraction, layout analysis, image captioning, hybrid retrieval, and a state-of-the-art reranker, capped by a grounded language model that avoids hallucinations and provides attribution. Evaluation is done via LM Unit, a model-as-judge that scores responses on criteria like accuracy and causation. The episode also shows integrating the agent with Claude Desktop via MCP and answers audience questions on pricing (consumption-based with a $25 credit), scalability, entitlements, HIPAA, and domain-specific language.

How to Build Trustworthy AI — Allie Howe

Jun 16, 2025 · 24:22

Allie Howe, VC CISO at GrowthCyber, argues that trustworthy AI equals AI Security (how the world harms your AI) plus AI Safety (how your AI harms the world), and that builders are legally and reputationally responsible. She covers three pillars: MLSecOps (scanning models for serialization attacks using open‑source ModelScan), AI Red Teaming (testing for prompt injections, jailbreaks, and safety issues with tools like PyRIT), and AI Runtime Security (validating inputs/outputs in production via platforms like Pillar to block off‑topic or unsafe behavior). Howe cites real incidents—a Chevy Tahoe chatbot offered for $1, Slack's data leak via prompt injection, Fortnite's Darth Vader NPC initially producing racist outputs—and notes a lawsuit where OpenAI won by arguing users must expect errors. She emphasizes shifting‑right to runtime guardrails as the most cost‑effective investment and advises demonstrating trustworthiness through GRC platforms like Vanta to shorten sales cycles. With increasing regulation (ISO 42001, EOAI Act, FDA guidelines), she concludes that building trustworthy AI now unlocks revolutionary innovation in fields like healthcare.

MCP Agent Fine tuning Workshop - Ronan McGovern

Jun 3, 2025 · 35:30

Ronan McGovern demonstrates how to fine-tune a Qwen3 model on high-quality reasoning traces from an MCP agent with browser tools. Using Playwright's 25 tools via Model Context Protocol, he generates multi-turn traces by running a 30B Qwen agent on RunPod, saving both tool calls and reasoning content. Those traces are then used to supervised fine-tune a 4B Qwen model with Unsloth, applying LoRA adapters to attention and MLP layers. The process includes converting MCP tool schemas to OpenAI format, extracting Hermes-style tool calls, and unrolling conversations to multiply training examples. Even with only nine curated traces, the fine-tuned model shows improved tool-calling behavior on multi-step tasks like navigating trellis.com to extract specific content.

Real AI Agents Need Planning, Not Just Prompting - Yuval Belfer

Jun 3, 2025 · 7:58

Yuval Belfer of AI21 Labs argues that LLMs alone still fail at instruction following, as shown by GPT-4.1's struggles in 2025, and that true AI agents require dynamic planning, not just prompting. He critiques ReAct for lacking look-ahead, contrasting it with AI21 Maestro's planner and smart execution engine, which uses best-of-n sampling, candidate pruning, and replanning. On AIfeval, Maestro pushes GPT-4.0, Claude Sonnet 3.5, and R3 Mini to near-perfect scores; on internal requirement satisfaction benchmarks, it improves over single LLM calls despite higher runtime and cost. Belfer advises starting simple (SLMs, ReAct) and escalating to planning only for complex tasks, inviting listeners to join the Maestro waitlist.

The RAG Stack We Landed On After 37 Fails - Jonathan Fernandes

Jun 3, 2025 · 18:52

Jonathan Fernandes, independent AI engineer, details the RAG stack his team settled on after 37 failed attempts, covering orchestration (LlamaIndex), embeddings (BAAI BGE small), vector database (Qdrant), LLMs (GPT-4, Qwen, Llama), reranking (Cohere), monitoring (Arize Phoenix), and evaluation (RAGAS). He demonstrates a live prototype in Google Colab using a London railway knowledge base, showing how a naive RAG returns irrelevant results (e.g., suggesting black cabs for "where can I get help at the station"). By swapping components—replacing in-memory storage with Qdrant, using an open-source embedding model, upgrading to GPT-4, and adding Cohere reranking—the answer improves to "go to booth number five next to the Eurostar ticket gates." For production, he deploys via Docker Compose with NVIDIA embedding/reranking models and Ollama for serving. The episode also stresses the importance of tracing latency per component and using RAGAS for systematic evaluation across many queries.

Text-to-Speech Data Preparation and Fine-tuning Workshop - Ronan McGovern

Jun 3, 2025 · 34:00

Ronan McGovern walks through fine-tuning Sesame's CSM-1B text-to-speech model on a specific voice, using a YouTube video as the data source. He explains token-based TTS models, including how audio is represented via codebooks and how CSM-1B uses a main transformer for zeroth tokens and a secondary transformer for 31 hierarchical tokens. The workshop covers data preparation: downloading audio with yt-dlp, transcribing with Whisper Turbo, manually correcting the transcript, and splitting audio into 30-second chunks (41 clips from a 30-minute video). Fine-tuning uses Unsloth with LoRA adapters (rank 32, alpha 16) on linear layers, training for one epoch with a batch size of 2 and virtual size of 8, reducing loss from ~6.34 to ~3.72. Evaluation compares zero-shot inference (random speaker), voice cloning (closer but imperfect), and fine-tuned plus cloning (best result, producing an Irish-accented voice with natural errors). McGovern recommends 50+ 30-second clips for noticeable effect and notes that combining fine-tuning with voice cloning yields good performance even with limited data.

Evaluating Domain Specific LLMs for Real World Finance — Waseem Alshikh, Writer

Apr 22, 2025 · 12:01

Writer CTO Waseem Alshikh presents FailSafe, a benchmark for evaluating LLMs in real-world finance, challenging the notion that general-purpose models suffice. Tests on query failures (misspelling, incomplete, out-of-domain) and context failures (missing context, OCR errors, irrelevant context) reveal that reasoning models like o1 and o3 perform 50% to 60% worse on context grounding than smaller domain-specific models, despite higher answerability. Even the best model achieves only 81% combined robustness and grounding, meaning roughly 20% of responses are wrong. Alshikh argues that domain-specific models remain essential, and that full-stack systems—RAG, guardrails, and grounding—are necessary for reliable deployment in high-stakes finance.

Navigating AI’s Frontier in 2025 - Grace Isford, Lux Capital

Mar 13, 2025 · 17:55

Grace Isford, partner at Lux Capital, argues that while 2025 is a 'perfect storm' for AI agents with reasoning models like O3 and R1, cheaper inference, and billions in infrastructure (e.g., Stargate, DeepSeek), agents still fail due to cumulative errors—decision, implementation, heuristic, and taste—exemplified by OpenAI Operator booking a flight incorrectly. She prescribes five strategies: curating proprietary and agent-generated data, building personalized evals for non-verifiable domains (e.g., seat preference), designing scaffolding that prevents cascading failures (citing Ramp's approach), treating UX as the moat (e.g., Codium, Harvey, TLDraw), and building multimodally with voice, smell via Osmo, and touch for embodiment. The talk, recorded at the AI Engineer Summit 2025 in NYC, closes with a call to reframe perfection through visionary product experiences.

Reinforcement Learning for Agents - Will Brown, ML Researcher at Morgan Stanley

Mar 7, 2025 · 18:17

Will Brown, a machine learning researcher at Morgan Stanley, argues that reinforcement learning (RL) is the essential path to unlocking autonomous AI agents, citing DeepSeek's R1 and OpenAI's Deep Research as proof: R1 used GRPO to make models learn chain-of-thought reasoning without manual data, and Deep Research applies end-to-end RL for up to 100 tool calls. He shares his own open-source single-file GRPO script—a 1B-parameter LLaMA model trained on math questions that demonstrated self-correction and improved accuracy, sparking community forks and blog posts. Brown introduces 'rubric engineering' as a new practice akin to prompt engineering, where reward rules (e.g., XML structure, integer answers) guide model improvement, and warns about reward hacking. He previews his current work: a framework for RL inside multi-step environments, letting developers reuse existing agent code for training. The talk concludes that fine-tuning and RL remain relevant as open-source catches up, and that skills like building evals and prompts translate directly to the RL era.

The Hidden Costs of Building Your Own RAG Stack — Ofer Vectara

Feb 22, 2025 · 15:14

Ofer from Vectara argues that building your own RAG stack comes with seven hidden costs: irrelevant responses and hallucinations, high latency, scaling and cost issues, security and compliance challenges, vendor chaos from multiple components, unsustainable expertise requirements, and limited non-English language support. He details how each pitfall compounds at enterprise scale, citing the need for hybrid search, reranking, and continuous evaluation to maintain quality. Vectara offers a turnkey RAG-as-a-service platform that handles parsing, chunking, embedding, vector search, retrieval, and hallucination detection. Ofer highlights Vectara’s open-source HHEM model for hallucination evaluation, downloaded over 3 million times, and a leaderboard showing some LLMs hallucinate at rates above 10%. The platform supports deployment in SaaS, VPC, or on-premises, with built-in access control and explainability.

Lessons from building GenAI based applications — Juan Peredo

Feb 22, 2025 · 33:13

Juan Peredo details the hidden complexities of building GenAI applications, from model hosting and cost control to output validation and observability. He compares local (Ollama) vs cloud hosting (Modal, SkyPilot) and warns that an agent processing 3,000 calls/day with OpenAI O1 costs nearly $300,000/month, while LLaMA 3.3 70B drops that to $50,000/month. He explains techniques to mitigate hallucinations—prompt engineering, guardrails, RAG, and fine-tuning—each with trade-offs like added latency or cost. Peredo advocates externalizing prompts via LangChain Hub for easy iteration and future-proofing, and illustrates agent design with parallel calls to reduce latency. Finally, he stresses observability using tools like LangSmith to debug probabilistic failures, such as an LLM failing on case sensitivity.

Customized, production ready inference with open source models: Dmytro (Dima) Dzhulgakov

Feb 16, 2025 · 18:55

Dmytro (Dima) Dzhulgakov, co-founder and CTO of Fireworks AI, argues that open source models are the future for production Gen AI applications, and Fireworks provides a platform to make them customized and production-ready. He explains that while proprietary models like GPT-4 are powerful, they are too large, expensive, and slow for many use cases, whereas open models like Llama or Gemma can be fine-tuned for specific domains to achieve better quality at up to 10x speed and lower cost. The main challenges of using open models—complex setup, performance tuning, and production scaling—are addressed by Fireworks' custom serving stack, which achieves the fastest long-prompt inference and serves SDXL fastest among providers, handling over 150 billion tokens per day. The platform supports fine-tuning and serving thousands of LoRA adapters on the same GPU with serverless pay-per-token pricing. Dzhulgakov also highlights the emerging architecture of compound AI systems, where function calling—exemplified by the open-source Fire Function model—connects LLMs to external tools and knowledge sources, enabling agentic applications like stock querying and chart generation. The episode…

Training Albatross An Expert Finance LLM: Leo Pekelis

Feb 13, 2025 · 16:20

Leo Pekelis, chief scientist at Gradient, explains how they transformed an open-source model into Albatross, a finance LLM that tops leaderboards on both general and domain-specific tasks. The key was an automated data pipeline using membership inference to curate finance data from a massive corpus, followed by continual pre-training and alignment via supervised fine-tuning and preference optimization. He also details a one-million-token context extension on a Llama 3-based model that achieves 100% needle-in-the-haystack scores, enabling in-context learning with thousands of examples to reduce hallucinations. The models, v-alpha-tross and the extended-context Llama 3, are open-sourced on Hugging Face.

LLM Quality Optimization Bootcamp: Thierry Moreau and Pedro Torruella

Feb 8, 2025 · 53:05

Thierry Moreau of OctoAI demonstrates how to fine-tune Llama 3 8B on a PII redaction task using OpenPipe and OctoAI, achieving 47% better accuracy and a 200x cost reduction (from $30 to $0.15 per million tokens) compared to GPT-4 Turbo. He explains that fine-tuning should follow prompt engineering and RAG, and works best for specialized tasks like function calling. The talk walks through building a fine-tuning dataset from the PI Masking 200k dataset, using OpenPipe to train a LoRA for $40, deploying it on OctoAI, and evaluating it to show the fine-tuned model scores 0.97 accuracy versus GPT-4’s 0.68. Moreau emphasizes that this continuous deployment cycle requires monitoring data drift and retraining, but tools like OpenPipe and OctoAI make it accessible even for teams without deep ML expertise.

Building security around ML: Dr. Andrew Davis

Feb 8, 2025 · 25:01

Dr. Andrew Davis, Chief Data Scientist at HiddenLayer, argues that machine learning models remain highly vulnerable to adversarial attacks despite a decade of research, and defenses must be layered with observability, logging, and skeptical data handling. He details how ImageNet's URL-based distribution enables data poisoning via expired domains, and how model theft can replicate a LLaMA 7B model's performance for just $600 in OpenAI queries. Adversarial examples still evade robust defenses, with best-case robustness only 50-60% against advanced attacks, and multimodal LLMs amplify the threat as pixel-level modifications are far harder to detect than text prompt injections. Spotlighting — encoding data in base64 to prevent instruction-following — is a promising prompt injection defense, but attackers can craft readable base64 strings to bypass it. The ML supply chain on Hugging Face is fraught with risk: models can execute arbitrary code via Lambda layers or TensorFlow functions, so verifying provenance, scanning for malware, and sandboxing are critical. Finally, software vulnerabilities in tools like Ollama (with recent RCE CVEs) demand the same patching discipline as traditional…

Accelerate your AI journey with Azure AI model catalog: Sharmila Chokalingam

Feb 6, 2025 · 23:14

Shubhi and Sharmila present the Azure AI model catalog as a platform offering over 1,600 models including GPT-4, Mistral, Llama, Cohere, and Phi3, with a standardized inference API enabling easy model swapping. They demonstrate deployment via serverless API (pay-per-token) and managed compute, emphasizing that customer prompts and completions are not shared with model providers or used for training. The platform includes model benchmarks, playground for RAG, and Prompt Flow for building generative AI apps with evaluation and variant comparison. Customer success stories include EY's EYQ chat adopted by 275,000 employees, CMA CGM's reduced response latency using Mistral, and Bridgestone's 30% reduction in forecasting errors with Nixtla's TimeGen.

Lessons from the Trenches: Building LLM Evals That Work IRL: Aparna Dhinkaran

Feb 6, 2025 · 18:49

Aparna Dhinakaran, co-founder of Arize AI, distinguishes between model evals (e.g., Hugging Face leaderboard) and task evals for real-world LLM systems, arguing that production applications need component-level evaluations like router and parameter evals. She demonstrates a chat-to-purchase app where a router function call misidentifies user intent, showing how Phoenix open source tool traces errors and provides explanations to iterate. Dhinakaran advises using categorical over numeric LLM-as-judge scores because numeric outputs tend to be binary (0 or 10) and lack granularity. Presenting needle-in-haystack research, she notes GPT-4 struggles retrieving facts placed early in large context windows, and in retrieval-with-generation tasks, Anthropic’s Claude 2.1 outperforms GPT-4 due to verbose reasoning, a gap closed by prompting GPT-4 to explain itself first.

Mastering LLM Inference Optimization From Theory to Cost Effective Deployment: Mark Moyou

Jan 1, 2025 · 33:39

NVIDIA solutions architect Mark Moyou explains that LLM inference differs fundamentally from standard deep learning deployment, requiring careful management of KV Cache, attention mechanisms, and GPU memory to control cost. He details how tokens are processed: prefill computes attention across the entire prompt, then generation produces one token at a time, with KV Cache storing key-value pairs to avoid recomputation. Llama's 32 attention heads and FP8 quantization (halving memory with near-identical accuracy) are cited as key optimizations. Moyou emphasizes measuring time to first token, inter-token latency, and input/output sequence length distributions to size inference engines. He presents NVIDIA's TRT-LLM (model compilation for LLMs) and Triton inference server as tools to maximize throughput, and discusses how query patterns like long-input-short-output or short-input-long-output impact GPU utilization and deployment cost.

LLM Safeguards: Security Privacy Compliance Anti Hallucination: Daniel Whitenack

Dec 31, 2024 · 34:10

Daniel Whitenack of Prediction Guard outlines a practical checklist for deploying secure and accurate LLMs in enterprise, addressing hallucination, supply chain vulnerabilities, flaky model servers, data breaches, and prompt injection. He proposes a factual consistency model fine-tuned to detect inconsistencies between AI output and ground truth, rather than relying solely on RAG. For supply chain risks, he recommends a trusted model registry and industry-standard libraries. Data breaches are mitigated via PII detection filters and confidential computing (e.g., Intel SGX) to encrypt server memory. Prompt injection is countered with a custom firewall layer using ensembled classification models. In Q&A, he discusses latency trade-offs using smaller NLP models, pre-production testing for visibility, data access controls via role-based database queries, SIEM integration for monitoring new artifacts like model caches, and additional challenges with agents such as excessive agency, proposing dry-run approvals.

AI Frontiers in Trust and Safety Combatting Multifaceted Harm on Tinder at Scale: Vibhor Kumar

Dec 2, 2024 · 14:36

Vibhor Kumar, senior AI engineer at Tinder, explains how the company uses open-source LLMs and LoRAX to detect a long tail of trust and safety violations at global scale. Facing challenges like content pollution and automated fraud from generative AI, Tinder leverages pre-trained models such as LLaMA and Mistral, fine-tuning them with LoRA and QLoRA on hybrid datasets generated by GPT-4 and manually verified. They serve dozens of fine-tuned adapters on a single GPU using LoRAX, achieving real-time inference (tens of QPS, ~100ms latency) for categories including hate speech, pig butchering scams, and underage users. The approach yields near 100% recall on simpler tasks and significant improvements over baselines, with better generalization that resists adversarial evasion. Future directions include visual language models for explicit image detection and automating retraining pipelines.

A Practical Guide to Efficient AI: Shelby Heinecke

Nov 18, 2024 · 17:45

Shelby Heinecke, who leads an AI research team at Salesforce, presents five orthogonal dimensions for making AI models efficient: efficient architecture selection, pre-training, fine-tuning, inference, and prompting. She highlights the power of small models like Phi-3 (3.8B parameters outperforming a 7B model), mobile LLM (350M parameters on par with 7B after fine-tuning), and Octopus (2B fine-tuned Gemma exceeding GPT-4 on Android tasks). For efficient inference, she explains post-training quantization, showing 4-bit quantization nearly halves memory usage without performance loss (e.g., LLaMA models), but warns 3-bit can degrade quality. She recommends frameworks like LLaMA CBP and ONNX Runtime for quantization and introduces her team's open-source Mobile AI Bench for evaluating quantized models, including an iOS app to measure latency and battery drain. The central claim is that deploying AI in constrained environments—cloud, on-prem, or edge—demands efficiency, and these practical techniques bridge the gap from demo to production.

Moondream: how does a tiny vision model slap so hard? — Vikhyat Korrapati

Nov 14, 2024 · 19:26

Vikhyat Korrapati built Moondream, a tiny open-source vision language model under 2 billion parameters that matches LLaVA 1.5, a model four times its size, on VQA v2 and GQA benchmarks. He attributes this to focusing on image understanding over world knowledge and investing heavily in synthetic training data—a pipeline that generated 35 million images and used two orders of magnitude more compute on data than training. Key lessons: community engagement was critical, open-source builds trust, and safety guardrails should be application-layer, not baked in. He argues tiny models will dominate production due to cost and privacy advantages, and that prompting will replace custom model training for most vision tasks. Moondream raised a seed round from FullySysAscent and the GitHub Fund.

Build an AI Research Agent: Apoorva Joshi

Oct 25, 2024 · 27:33

In this workshop, Apoorva Joshi, an AI Developer Advocate at MongoDB, teaches how to build an AI research agent using MongoDB as the memory provider and knowledge store, open-source LLMs from Fireworks AI (Fire Function V1) as the agent’s brain, and LangChain to orchestrate the workflow. The agent searches for research papers, summarizes them, and answers questions based on past research, using tools like ArXivLoader and a MongoDB vector store. Joshi explains key agent concepts—planning with chain of thought and react patterns, short-term and long-term memory, and tool creation—then guides attendees through hands-on sections to build the agent step by step. Attendees learn to create agent tools, implement reasoning with react, and add short-term memory persisted to MongoDB. The workshop emphasizes that agents enable complex, multi-step tasks through iterative reasoning, tool use, and memory, trading higher cost and latency for improved accuracy.

Making Open Models 10x faster and better for Modern Application Innovation: Dmytro (Dima) Dzhulgakov

Oct 9, 2024 · 18:55

Dmytro Dzhulgakov, CTO of Fireworks AI, argues that open-source models are the future for GenAI applications because they offer lower latency, lower cost, and domain adaptability compared to proprietary models. He explains that open models can be up to 10x faster for narrow domains and cut costs significantly, using examples like fine-tuned Llama 3 for function calling. Fireworks addresses the challenges of setup, optimization, and production readiness with a custom serving stack that delivers the fastest inference for long prompts and image generation (e.g., SDXL). He highlights FireFunction V2, an open-source model for function calling that combines chat and tool use, and notes that Fireworks serves over 150 billion tokens per day for companies like Quora and Cursor. The talk emphasizes that platforms like Fireworks enable developers to start with serverless inference, fine-tune models, and scale to enterprise-grade deployment with dedicated hardware.

Everything you need to know about Fine-tuning and Merging LLMs: Maxime Labonne

Sep 25, 2024 · 17:52

Maxime Labonne from Liquid AI explains the LLM training lifecycle—pre-training, supervised fine-tuning (SFT), and preference alignment—and when to use fine-tuning over prompt engineering. He details SFT dataset creation (accuracy, diversity, complexity) and techniques like full fine-tuning, LoRA, and QLoRA, with key hyperparameters. The core of the talk is model merging: combining weights of fine-tuned models without GPU, using methods like SLERP (spherical linear interpolation for two models), TIES (pruning redundant parameters to merge many models), pass-through (concatenating layers, e.g., Meta LLaMA 3 120B Instruct by repeating layers gives strong creative writing), and Franken-MoE (extracting FFN layers from domain-specific models with a router). Labonne demonstrates these with his NeuralBeagle and Beyonder models, noting merged models dominate the OpenLLM leaderboard and that TIES merging often outperforms more experimental Mixture of Experts approaches.

Judging LLMs: Alex Volkov

Sep 9, 2024 · 18:39

Alex Volkov, as an LLM judge from 2034, humorously judges AI engineers on their development practices, emphasizing the importance of tracing, iterative prompt engineering, and robust evaluation pipelines. He finds Daniel guilty of deploying without logging, Sasha guilty of premature fine-tuning without prompt iteration, and Morgan guilty of ignoring AI news—commuted to attending ThursdAI. Francisco’s overreliance on programmatic evals leads to a legal loss, while Maxime is 'awesome' for using Weights & Biases Weave. Alex concludes with a primer on evaluation methods: programmatic, human-in-the-loop, and LLM-as-judge, stressing the need to validate validators and create custom criteria. He promotes Weave for tracing and evals, and ThursdAI for staying updated.

Building State of the Art Open Weights Tool Use: The Command R Family: Sandra Kublik

Aug 26, 2024 · 15:03

Sandra Kublik of Cohere presents the Command R family of open-weight models optimized for retrieval-augmented generation and tool use. Released in March 2024, Command R and Command R+ achieved 150,000 Hugging Face downloads within two weeks and now serve nearly 500,000 developers. The models overcome RAG challenges such as prompt sensitivity and citation accuracy through post-training, delivering fine-grained citations and low hallucination. Cohere open-sourced a toolkit UI with plug-and-play components for RAG and tool use, supporting cloud, local, and Hugging Face access. The new Multi-Step API enables sequential reasoning with automatic retry and reflection. Command R+ matches GPT-4 Turbo and Claude Opus on complex reasoning while being three to five times cheaper, positioning it as a scalable enterprise solution.

Git push get an AI API: Ryan Fox-Tyler

Aug 23, 2024 · 45:01

Ryan Fox-Tyler and Matt Johnson Pint from Hypermode present a hands-on workshop demonstrating how to build and iteratively improve AI features using their platform, focusing on a GitHub issue triage app. They first illustrate the process with a multiplayer game (Hypercategories) that uses AI for classification and scoring. Then they build a trend summary function using OpenAI GPT-4 to summarize repository issues, and a classify issue function with a Hugging Face DistilBERT model for labeling issues as bug, feature, or question. Finally, they add natural language search for similar issues by creating an embeddings model (MiniLM) and a Hypermode collection, enabling vector search without external databases. The workshop emphasizes incremental iteration, mixing AI models with traditional code, and using Hypermode's automatic GraphQL generation and observability to speed development.

Low Level Technicals of LLMs: Daniel Han

Jul 31, 2024 · 2:52:26

Daniel Han of Unsloth explains how to find and fix bugs in open-source LLMs like Gemma, Phi-3, and Llama, covering tokenizer issues, architecture pitfalls, and finetuning optimizations. He details the eight Gemma bugs Unsloth fixed, including a critical RoPE downcasting error that broke positional encoding, and a 2048 sliding window bug in Phi-3. Han walks through transformer internals: attention masking, layer norms, RoPE embeddings, and SwiGLU activation, showing how to derive gradients for custom kernels. He demonstrates Unsloth's 2x faster finetuning with 70% less memory via Triton kernels, and introduces new features: automatic Ollama model file creation, CSV fine-tuning with merged columns, and chunked cross-entropy for large vocabularies. The session includes live Q&A on learning rate schedules, precision trade-offs, and mechanistic interpretability.

Fixing bugs in Gemma, Llama, & Phi 3: Daniel Han

Jul 31, 2024 · 17:42

Daniel Han of Unsloth details eight bugs found in Llama 3, including double BOS tokens, untrained tokens in the base model, and pad tokens equaling EOS tokens, which cause infinite generations. He explains how Unsloth automatically fixes these issues and offers a free Colab notebook for fine-tuning with Ollama. Han also covers tokenization fixes for Gemma and a sliding window bug in Phi-3, emphasizing the importance of correct chat templates and avoiding double BOS tokens. The episode provides concrete code examples and best practices for fine-tuning open-source LLMs to avoid common pitfalls.

Llamafile: bringing AI to the masses with fast CPU inference: Stephen Hood and Justine Tunney

Jul 16, 2024 · 17:25

Stephen Hood and Justine Tunney present Mozilla's Llamafile project, which turns AI model weights into single-file executables that run on any OS and CPU without installation, democratizing access to AI. They claim CPU inference can match GPU performance through techniques like outer-loop unrolling in matrix multiplication and using a GPU-like programming model with sync threads, achieving 30-500% speed increases. Justine demonstrates a summarization task where the optimized version completes in seconds versus the old version's many seconds. Hood announces the Mozilla Builders accelerator offering $100,000 in non-dilutive funding for open-source local AI projects, emphasizing that individuals and small groups can still make impactful contributions in AI.

How to Become an AI Engineer from a Fullstack Background - Reid Mayo

Feb 2, 2024 · 10:19

Reid Mayo presents a step-by-step syllabus to transition from fullstack engineer to AI engineer, covering generative AI foundations, prompt engineering, LangChain, fine-tuning, and cost-effective open-source model deployment. The syllabus starts with Cohere's LLM overview, then dives into prompt engineering via Elvis Seravia's guide and Learn Prompting org docs. It emphasizes LangChain as the glue layer for modular AI systems, with tutorials from Mayo Ocean. Evals are treated as software tests using OpenAI's cookbook. Fine-tuning is taught via OpenAI's cookbook and then open-source LLaMA 2, with a specific case study showing a $19 fine-tuned LLaMA 2 matching OpenAI's $24,000 model on a target task. The boot camp ends with advanced deep learning courses from FastAI and Hugging Face for further mastery.

The Weekend AI Engineer: Hassan El Mghari

Nov 22, 2023 · 21:49

Hassan El Mghari shares how he built viral AI apps like RoomGPT and AI Commit, arguing that simple off-the-shelf APIs and focused weekend sprints can attract millions. He describes building 11 side projects in a year, which grew from 20,000 visitors to over 8.5 million unique visitors and 2.8 million sign-ups. Key projects include RoomGPT, which used ControlNet for room redesign and reached 6 million visitors, and AI Commit, an open-source CLI tool adopted by 30,000 developers. He emphasizes using tools like the Vercel AI SDK and v0.dev to accelerate development, making apps free and open source to drive growth, spending 80% of time on UI, and launching quickly with minimal fine-tuning.

[Workshop] AI Engineering 201: Inference

Nov 7, 2023 · 1:43:16

Charles Frye, instructor of the Full Stack LLM Bootcamp, leads a workshop on AI engineering inference, focusing on the build-versus-buy decision between proprietary and open models. He argues that proprietary models like OpenAI's GPT-4 and Anthropic's Claude are currently more capable but expensive, while open models like LLaMA 2 are less capable but offer hackability, though they may catch up if capabilities requirements saturate. Frye covers inference on end-user devices, noting that running models locally avoids network latency but faces tight memory and power constraints—e.g., a 7B parameter model requires 14 GB, too large for phone RAM. He explains inference-as-a-service (e.g., OpenAI, Replicate) versus self-serving on cloud GPUs or serverless platforms like Modal, highlighting that memory bandwidth is the bottleneck: GPUs have 1.5 TB/s memory bandwidth vs 312 TFLOPS compute, so batching is crucial for throughput. Frye also discusses inference arithmetic, custom silicon like TPUs which offer ~30% better efficiency but not drastic gains, and containerization challenges for GPU workloads.

The Hidden Life of Embeddings: Linus Lee

Nov 7, 2023 · 18:15

Linus Lee, a Research Engineer at Notion, presents a tour of embedding visualization and manipulation at the AI Engineer Summit 2023. He demonstrates an encoder-decoder model fine-tuned from T5 that can reconstruct text from embeddings, allowing direct manipulation of features like length and sentiment by moving in latent space. Lee shows that mixing embeddings by splicing dimensions from two texts produces a semantic blend, and a linear adapter can decode text from OpenAI's text-embedding-ada-002 embedding space. Using CLIP, he interpolates between photographic and cartoon images and performs vector arithmetic to modify facial expressions. Lee releases these custom text embedding models on Hugging Face, enabling others to explore and interact with latent spaces. He argues that making model internals visible and manipulable fosters deeper understanding and more humane interfaces to generative AI.

[Workshop] AI Engineering 101

Nov 6, 2023 · 3:02:24

In this hands-on workshop, AI engineer Noah Hein teaches the basics of AI engineering by building five projects with OpenAI's GPT-3, GPT-4, Dall-E, Whisper, and Telegram. Participants create a Telegram bot that uses GPT-3 for chat, implements retrieval-augmented generation (RAG) with embeddings and cosine similarity on MDN docs, generates code with GPT-4 using few-shot prompting, creates images via Dall-E 2, and transcribes voice with Whisper. The session explains core concepts like tokens, chunking, context windows, temperature, and top-p, emphasizing that embeddings and prompt engineering are key to performance. Hein demonstrates that these tools are cheap and accessible—the entire workshop costs about $0.05 in API fees—and shows how AI engineers can integrate multiple models into a single application.

Building AI For All: Amjad Masad & Michele Catasta

Oct 23, 2023 · 25:13

Amjad Masad and Michele Catasta announce Replit's AI for all, giving free AI-powered coding to millions of users. They detail the training of Replit Code v1.5, a 3.3B parameter model trained on 1 trillion tokens of code, which outperforms StarCoder 3B and approaches Code LLaMA 7B in Human Eval despite being half the size. The model is optimized for low-latency inference, generating over 200 tokens per second per GPU. Catasta explains the data pipeline using permissively licensed code from The Stack and Replit public repos, with five epochs of repeated high-quality data. The model is released open-source with a commercially permissive license. Partnerships with Glaive AI for instruction fine-tuning, Morph Labs for a novel fill-in-syntax-tree format, and Perplexity for fast model serving are also unveiled.