DeepSeek

Deep dive on LLM Inference at Scale — Harshul Jain, Audible & Tanmay Sah, Independent AI Researcher

Sep 8, 2026 · 1:28:12

Harshul Jain of Audible and independent AI researcher Tanmay Sah teach a two-hour workshop building LLM inference optimization from first principles on Mistral 7B. They show KV cache costs 131 KB per token, so 16,000 tokens across 80 concurrent users needs 42 GB of GPU memory, driving the pain points of memory, TTFT, and throughput. Sah covers quantization, multi-head, multi-query, grouped-query, and latent attention, plus flash attention, using his 'ostrich' and 'world cup' teaching algorithms. Jain covers paged attention, continuous batching, prefix caching, and KV quantization, benchmarked against a Hugging Face baseline on vLLM. Their benchmarking found vLLM and SGLang statistically tied on standard workloads, but SGLang three to four times faster once agentic branching enters.

Why AI Agents Need Million-Token Context — Thomas Wolf & Olive Song, MiniMax

Sep 4, 2026 · 20:48

MiniMax's M3 pairs a functional one-million-token context window with coding and multimodal abilities in a 400-billion-parameter, 20-billion-activated model, explain Olive Song, RL Lead at MiniMax, and Thomas Wolf, co-founder of Hugging Face. Song says short context fails when agents handle multi-round tool responses; MiniMax Sparse Attention — an index branch selecting what matters plus a sparse branch computing on selected blocks — was designed by an intern. She argues multimodal training from the very first step beats post-hoc adapters that harm text performance and risk collapse; interleaved data and reward modeling solved this. Its apps reach over 300 million people in 200 countries; anyone can propose projects, and community feedback and internal agent harnesses now drive M3.1.

Can LLMs Write Fast Multi-GPU Kernels? — Simran Arora, Together AI

Aug 27, 2026 · 30:00

Simran Arora, principal scientist at Together AI, argues multi-GPU communication, not single-GPU compute, is AI's bottleneck, and introduces ParallelKittens and ParallelKernelBench to test whether LLMs can write fast multi-GPU kernels. From A100 to B200, BF16 tensor core throughput improved 7.2x vs 3x intra-node communication, leaving PyTorch+NCCL baselines below 50% of communication-aware roofline on most problems. ParallelKittens adds a dozen lines to a single-GPU kernel and runs in production at Together and Cursor; in ParallelKernelBench's 87 problems, the best model solves 28 zero-shot (22 faster), and more samples lift correctness to 36 but fast-and-correct stalls near 31%. Models compile after retries but stall on ordering and transfer choices; a bash-based agent hits 35 correct.

Trading Desks to Clinical Trials: Parallels in Applied Vertical AI — Ayush Bhardwaj, Allos AI

Aug 19, 2026 · 20:02

Ayush Bhardwaj, who applied AI at a hedge fund before pharma tech startup Allos AI, argues vertical AI projects die from not being able to judge whether they work. His seven-step recipe: narrow tasks, proprietary data, expert-modeled prompts, observability, then hiring the user—a domain expert—because using LLM-as-judge is a "stupid mistake" that jargons its way out. Verifiable rewards fail in finance and pharma: no answer keys, and data is withheld—30% of firms never disclose clinical trials, and the FDA publicly reminded over 2,000 sponsors in 2026. He says 89% of enterprise AI agents never reach production; the fix is endless expert-in-the-loop learning, starting with error analysis as highest ROI. Moat is never model or infra; only curated domain expertise and proprietary data matter.

Scaling to Long Horizons — Ross Taylor & Chengxi Taylor, General Reasoning

Jul 31, 2026 · 18:07

Ross Taylor and Chengxi Taylor, co-founders of London-based RL company General Reasoning, argue that scaling to long horizons demands a mindset shift and better simulation, not just bigger context windows. Ross recounts how Galactica's 2022 thinking tokens were prescient, but its base-model demo backfired, while RLHF made LLMs products; his Meta team's PPO with verifiable rewards worked, but DeepSeek-R1 later showed better base models were key. Chengxi details long-horizon obstacles: sparse rewards, credit assignment, and 1M token limits, solved by value models that reduce variance and enable bootstrapping. Kelly Bench gave frontier models $100K to trade Premier League matches; all lost money, exposing how little current environments simulate real competition. Pipeline RL trades off off-policy staleness (8 steps okay) against GPU utilization, and they point to openreward.ai, with 350+ environments, for long-horizon RL.

The Base Model Is Dead — Varun Singh, Arcee AI

Jul 31, 2026 · 17:45

Varun Singh, pre-training lead at Arcee AI, argues the base model is dead: it no longer just mirrors web text but must carry the prior that reinforcement learning builds on. He traces how web text fell from 85% of GPT-3's mix to 15% in MAI Thinking 1, with code and STEM dominating, and how Nemotron 3 Ultra pulls SFT-style Q&A data back into pre-training. Synthetic rephrasing, as used in Arcee's Trinity Large and Kimi K2, upsamples information to teach task shapes early. He warns that without post-training-flavored data early, MoE load balancing can break when SFT distributions differ, citing MAI's cranking of the balancing coefficient. He frames training as supervised learning vs RL, noting RL compute now rivals or exceeds pre-training, as with Compose 2.5, and argues the base model's job is to provide atomic skills for RL to compose.

The Desktop Frontier — Ahmad Osman, Osmantic

Jul 21, 2026 · 18:02

Ahmad Osman, founder of Osmantic, argues that within roughly 18 months (by late 2027) a single RTX 5090 will run intelligence equivalent to GLM 5.2, driven by the Densing Law of increasing impact per parameter. He shows this trend through concrete examples: a 27B-parameter Qwen 3.5 now beats the 405B LLaMA 3, and the same eight RTX 3090s that once struggled with LLaMA 2 can now run 15 parallel Qwen 3.5 agents. Osman presents the Densing Law—every 3.5 months, 50% fewer parameters achieve the same capability—as a systematic pattern, not coincidence. He advocates for sovereign AI: owning your own hardware (like a DGX Station or RTX 5090) gives you control, avoids cloud limitations, and sees hardware appreciate in utility as models become more efficient. He asks why fund cloud data centers when local hardware can run frontier intelligence and grow more valuable over time.

You Might Not Need 50 Diffusion Steps — Ziv Ilan, Nvidia

Jun 16, 2026 · 18:46

Nvidia's Ziv Ilan explains how combining quantization, caching, and step distillation enables near real-time video diffusion on a single Blackwell B200 GPU. Working with Black Forest Labs on Flux 2, dynamic quantization reduces memory and compute, caching skips redundant denoising steps, and distillation cuts steps from fifty to as few as one. The open-source FastGen repo packages these post-training and sharding techniques, achieving 10–200x speedups for real-time generation.

Mind the Gap (In your Agent Observability) — Amy Boyd & Nitya Narasimhan, Microsoft

May 14, 2026 · 1:20:07

Amy Boyd and Nitya Narasimhan of Microsoft explain how to close the gap between agent behavior and requirements using Microsoft Foundry's observability stack. They demonstrate tracing via OpenTelemetry, built-in evaluators for quality, safety, and agentic metrics (e.g., intent resolution, task adherence), and red teaming where a second AI attacks the agent to reveal vulnerabilities. The showcase is the observe skill: pointed at an agent with no eval data, it generates a dataset, runs batch evaluations, optimizes the prompt, compares versions, and rolls back to the best one—all from a single prompt. The skill surfaces failures developers didn't know existed, accelerating the optimize loop with human-in-the-loop guidance.

Let LLMs Wander: Engineering RL Environments — Stefano Fiorucci

Apr 8, 2026 · 40:35

Stefano Fiorucci demonstrates how to build Reinforcement Learning environments for language models using the open-source Verifiers library, arguing that training small models with verifiable rewards can surpass large closed models on specific tasks. He maps classic RL concepts to LLMs, introduces Verifiers components for single-turn, multi-turn, and tool environments, and then walks through an experiment where he takes LiquidAI's LLM 2 — a small open model — and transforms it into a tic-tac-toe master via supervised fine-tuning and GRPO-based reinforcement learning. After training, the model dominates random opponents and draws 85% of games against optimal ones, eventually outperforming GPT-5 Mini against identical optimal opponents. Fiorucci shares practical lessons: large batch sizes (≥256) ensure stable training, hidden biases in opponent algorithms can skew results, and starting from a base model (not a reasoning model) avoids truncated thinking traces. He concludes that if you can define a clear reward signal, you can build an environment and train a small specialized model to beat a large closed model at a fraction of the cost.

Fuzzing in the GenAI Era — Leonard Tang, Haize Labs

Aug 22, 2025 · 19:12

Leonard Tang of Haize Labs argues that standard evaluation methods fail for GenAI systems due to brittleness (Lipschitz discontinuity), and proposes 'Haizing'—fuzz testing that simulates diverse inputs to uncover corner cases. He details two core problems: scoring outputs via 'judges', where Haize's Verdict library stacks GPT-4 Mini in a self-verified debate ensemble to beat O1 at a third the cost, and RL-tuned judges like a 1.7B parameter model achieving 80.7% on RewardBench. For stimuli generation, he frames it as discrete optimization over natural language, using gradient-based methods and tree search. Case studies include Haizing the largest Hungarian bank's loan calculator, discovering prompt injections in minutes, and boosting a voice agent's ground-truth human agreement by 38% using rubric fanout.

State of Startups and AI 2025 - Sarah Guo, Conviction

Aug 2, 2025 · 23:52

Sarah Guo of Conviction argues that AI's value creation is massive and early, with companies like Cursor reaching $100M ARR in 12 months and Harvey exceeding $70M ARR. She predicts that by end of 2026, AI agents will ship code directly to production, voice AI will replace text for most business communication, and inference costs will drop below a cent per million tokens. Reasoning is a new scaling vector unlocking higher-stakes use cases, and agent startups have increased 50% in the last year. Multimodal models from HeyGen and Eleven are already rocketing past $50M ARR. The model market is more competitive than ever, with GPT-4 costs falling from $30 to $2 per million tokens in 18 months and open-source like DeepSeek competing. Guo advises builders to focus on thick wrappers around LLMs, leveraging domain and workflow knowledge, and warns against generic text boxes: 'The prompt is a bug, not a feature.' Execution, not first-mover advantage, is the moat.

Introduction to LLM serving with SGLang - Philip Kiely and Yineng Zhang, Baseten

Jul 26, 2025 · 43:42

Philip Kiely and Yineng Zhang introduce SGLang, an open-source fast serving framework for LLMs and VLMs, arguing it offers production-ready performance with day-zero support for new models like DeepSeek and Qwen, and a customizable codebase for contributions. Yineng, a core maintainer, traces SGLang's rapid growth from a December 2023 paper to nearly 15,000 GitHub stars and adoption by xAI, AMD, and Meituan. The workshop demonstrates deploying a first model via Baseten's trunk packaging, then tuning the CUDA Graph max batch size flag on an L4 GPU to maintain CUDA Graph-enabled decoding during higher concurrency, boosting generation throughput. They also cover Eagle 3 speculative decoding, where a draft model derived from the target model speculates tokens; users can benchmark different step and top-k configurations on representative prompts to find optimal settings for production. Finally, they invite contributions via GitHub's 'good first issue' tags and highlight Baseten's job openings.

Latent Space Paper Club: AIEWF Special Edition (Test of Time, DeepSeek R1/V3) — VIbhu Sapra

Jul 25, 2025 · 53:54

VIbhu Sapra presents DeepSeek's latest models and announces a new curriculum-based Test of Time Paper Club. DeepSeek's May 28th R1 update doubles reasoning tokens (12k→25k) and matches O3/Gemini 2.5 Pro, improving AIME 2024 from 70% to 87.5%. A new distillation of R1 traces into Qwen 3 8b achieves performance on par with Qwen's 235b thinking model, proving reasoning models distill efficiently. The Test of Time Paper Club will cover 50–100 foundational papers over six months in San Francisco and remote, targeting core AI engineering topics from attention and scaling laws to inference techniques. Sapra emphasizes that pure RL (GRPO) on verifiable math and code unlocks emergent reflection and aha moments, shifting scaling from pre-training compute to inference-time reasoning.

Building Agents (the hard parts!) - Rita Kozlov, Cloudflare

Jul 23, 2025 · 21:12

Rita Kozlov, VP of Product for Cloudflare's developer platform, presents the building blocks of AI agents—client, AI reasoning, workflows, and tools—arguing that effective agents require all four components. She highlights the Model Context Protocol (MCP) as a standard for exposing APIs to LLMs, and demonstrates Cloudflare's Agents SDK, which simplifies hosting remote MCP servers with built-in OAuth, state management via durable objects, and real-time WebSocket communication. Kozlov cites real-world impact: companies using agents for sales automation see 20% revenue increases, 90% faster support response times, and 50–75% time savings. She walks through a human-in-the-loop credit card approval workflow built with Nok, showing how durable objects maintain long-running state, prevent duplicate actions, and route approvals across Slack, email, or in-app notifications. The talk emphasizes that once an MCP server is deployed, it can be used directly from Cursor, Claude, ChatGPT, or a custom client, including voice interfaces via WebRTC-to-WebSocket translation.

What every AI engineer needs to know about GPUs — Charles Frye, Modal

Jul 20, 2025 · 19:52

Charles Frye of Modal explains that AI engineers need to understand GPU hardware constraints to optimize inference, arguing GPUs embrace bandwidth over latency and that Tensor Cores for low-precision matrix-matrix multiplication are the key resource. He describes how GPUs achieve 16,000+ parallel threads per cycle on H100, and notes Patterson’s Law: bandwidth improves at the square of latency. The main insight: arithmetic intensity favors N² operations per N memory loads, so matrix-matrix operations are efficient while matrix-vector is wasteful. Frye demonstrates that running a small 8B model 1,000 times on the same prompt matches GPT-4 quality, and that multi-token prediction and multi-sample query become nearly free because Tensor Cores handle expanded batches as matrix-matrix multiplications. He recommends using smaller models that fit on a single GPU and scaling via multiple generations.

A Taxonomy for Next-gen Reasoning — Nathan Lambert, Allen Institute (AI2) & Interconnects.ai

Jul 19, 2025 · 19:21

Nathan Lambert, senior research scientist at AI2 and founder of Interconnects.ai, argues that next-generation reasoning models require a taxonomy of four traits: skills, calibration, strategy, and abstraction. While current models excel at math and code (skills), they overthink easy problems, wasting tokens and latency—calibration is needed to match output length to task difficulty. The real frontier is planning: models must learn strategy (choosing the right direction) and abstraction (breaking problems into tractable sub-tasks) to enable long-horizon agents like Deep Research and Claude Code. Lambert traces how OpenAI's Q*→Strawberry→o1 took 12–18 months of human data to teach backtracking and verification; planning should be easier because humans can write 5-step plans. He predicts post-training compute could reach parity with pre-training, citing DeepSeek's shift from 0.18% post-training compute in V3 to an estimated 10–20% for R1. The path forward: collect diverse verifiable questions, filter by difficulty, run stable RL—then scale.

2025 in LLMs so far, illustrated by Pelicans on Bicycles — Simon Willison

Jul 9, 2025 · 18:30

Simon Willison reviews the past six months of LLM releases — including AWS Nova, Llama 3.3 70B, DeepSeek R1, Mistral Small 3, Claude 3.7 Sonnet, GPT 4.5, Gemini 2.5 Pro, GPT-4o, Llama 4, GPT 4.1, O3/O4 Mini, and Claude 4 — using his 'pelican on bicycle' SVG benchmark to argue that local models have become good enough to run GPT-4 class models on a laptop and that combining tools with reasoning is the most powerful technique in AI engineering, while noting risks like prompt injection and the 'lethal trifecta'. He tracks 30 significant model releases, highlighting that Mistral Small 3 (24B) matches Llama 3 70B's performance, which itself matched the 405B model, enabling local inference. DeepSeek's R1 caused a $500B+ Nvidia stock drop on January 27. GPT 4.1 Nano is the cheapest model yet at a fraction of a cent per pelican. He also examines bugs: ChatGPT's sycophantic 'shit-on-a-stick' incident and Claude 4's tendency to snitch to authorities when given ethical instructions and email tools. Willison concludes that while the pace is accelerating, control over context and security remain critical.

Training Agentic Reasoners — Will Brown, Prime Intellect

Jul 7, 2025 · 19:17

Will Brown of Prime Intellect argues that reasoning and agents are fundamentally the same, and reinforcement learning (RL) is the key to advancing both. He explains that RL now works at scale, as shown by DeepSeek's GRPO and OpenAI's o3, and that agentic tasks like tool calling are natural RL environments. Brown warns against reward hacking and emphasizes designing evals that are harder to game than the task itself. He introduces his open-source toolkit 'verifiers' (now on pip) which lets users build trainable agent loops with a simple API, and demonstrates training a 7B Wordle agent in a few turns on just a couple GPUs.

The Geopolitics of AI Infrastructure - Dylan Patel, SemiAnalysis

Jun 19, 2025 · 18:29

Dylan Patel of SemiAnalysis argues that despite US sanctions, Huawei has engineered a 384-chip cluster (Cloud Matrix 384) that Nvidia failed to deploy, while accessing TSMC via Softgo and HBM from Samsung via shell companies — all legally. China's SMIC will soon produce 7nm AI chips in high volumes, debunking the notion that China lacks compute. Meanwhile, Middle East players like G42 (UAE) and Datavolt (Saudi Arabia) are building multi-gigawatt data centers, with G42's deal letting it keep 20% of 500,000 GPUs yearly for itself while 80% goes to US companies like OpenAI. Patel highlights the US's 63-gigawatt power shortfall vs. 100 GW of planned data centers, explaining why US companies rely on Middle East capacity and why China's superior power buildout gives it a geopolitical edge.

Frontier Feud: Anthropic, Google DeepMind, Meta FAIR, Thinking Machines — Barr Yaron, Amplify

Apr 19, 2025 · 22:26

Teams from Anthropic, Google DeepMind, Meta FAIR, and Thinking Machines compete in a Frontier Feud game hosted by Barr Yaron at the AI Engineer Summit 2025. Surveyed AI engineers name Ilya Sutskever as the most influential AI researcher, with Andrej Karpathy, Jeff Hinton, and Yann LeCun also on the board. Cost is the top consideration when choosing a model, followed by latency, eval scores, and open vs. closed source. The buzzword everyone is tired of hearing is AGI, with RAG and prompt engineering trailing. In the fast money round, Cursor tops favorite AI tools, 'Attention Is All You Need' wins most influential paper, and hardware failure is the biggest 2 a.m. nightmare. The winning team, Rocco's Basilisk, takes home a massive llama and other prizes.

Navigating AI’s Frontier in 2025 - Grace Isford, Lux Capital

Mar 13, 2025 · 17:55

Grace Isford, partner at Lux Capital, argues that while 2025 is a 'perfect storm' for AI agents with reasoning models like O3 and R1, cheaper inference, and billions in infrastructure (e.g., Stargate, DeepSeek), agents still fail due to cumulative errors—decision, implementation, heuristic, and taste—exemplified by OpenAI Operator booking a flight incorrectly. She prescribes five strategies: curating proprietary and agent-generated data, building personalized evals for non-verifiable domains (e.g., seat preference), designing scaffolding that prevents cascading failures (citing Ramp's approach), treating UX as the moat (e.g., Codium, Harvey, TLDraw), and building multimodally with voice, smell via Osmo, and touch for embodiment. The talk, recorded at the AI Engineer Summit 2025 in NYC, closes with a call to reframe perfection through visionary product experiences.

Reinforcement Learning for Agents - Will Brown, ML Researcher at Morgan Stanley

Mar 7, 2025 · 18:17

Will Brown, a machine learning researcher at Morgan Stanley, argues that reinforcement learning (RL) is the essential path to unlocking autonomous AI agents, citing DeepSeek's R1 and OpenAI's Deep Research as proof: R1 used GRPO to make models learn chain-of-thought reasoning without manual data, and Deep Research applies end-to-end RL for up to 100 tool calls. He shares his own open-source single-file GRPO script—a 1B-parameter LLaMA model trained on math questions that demonstrated self-correction and improved accuracy, sparking community forks and blog posts. Brown introduces 'rubric engineering' as a new practice akin to prompt engineering, where reward rules (e.g., XML structure, integer answers) guide model improvement, and warns about reward hacking. He previews his current work: a framework for RL inside multi-step environments, letting developers reuse existing agent code for training. The talk concludes that fine-tuning and RL remain relevant as open-source catches up, and that skills like building evals and prompts translate directly to the RL era.

The Model Isn’t Wrong—You’re Just Bad at Prompting

Feb 22, 2025 · 8:54

Dan from PromptHub argues that prompt engineering remains critical for improving LLM outputs, covering Chain of Thought, few-shot, and meta prompting techniques. Chain of Thought breaks problems into sub-problems and is built into reasoning models; few-shot prompting works best with just one or two diverse examples, but can degrade performance on reasoning models like O1 and R1. Meta prompting uses LLMs to write or refine prompts, with PromptHub offering model-specific enhancers. For reasoning models, Dan advises minimal prompting, encouraging more reasoning instead of few-shot, and avoiding instructing the model on how to reason. Free resources include PromptHub's templates, the AutoReason prompt, and the Prompt Engineering Substack.

How Coding Agents change Software Development Forever - Hailong Zhang

Feb 22, 2025 · 8:50

Hailong Zhang presents how coding agents, particularly the unit test agent Gru, transform software development by automating routine tasks while humans focus on creative work. In the future workflow, synchronous agents like GitHub Copilot and Cursor assist in real-time, while asynchronous agents like Gru autonomously handle tasks such as writing tests, fixing bugs, and submitting pull requests. Gru, built to boost unit tests, detects code changes from pull requests, generates and runs tests, and submits a new PR with a summary and coverage improvement. In production, over 50% of Gru's PRs are accepted by humans, and it handles 80% of unit tests in its own repo, making it the top contributor. To build such agents, Zhang emphasizes defining a clear problem (e.g., unit tests, not generic software engineering), creating datasets and evaluation harnesses, selecting and fine-tuning LLMs per stage, building task-specific context from environment data, and developing an agent operating system to reuse common infrastructure across different tasks.

Keynote: Why people think "agent" is a buzzword but it isn't

Feb 22, 2025 · 28:07

Chip Huyen argues that agents are not a buzzword but a practical yet hard technology, facing three core challenges: the curve of complexity, tool use translation, and context management. Even simple queries require multiple steps, and models' success rates drop rapidly past 5 steps—newer reasoning models like DeepSeek R1 are pushing the boundary, but most still fail after 10. Tool use requires translating ambiguous natural language to precise API calls, worsened by poor documentation; she advises narrow functions and asking for clarifications. Context is another bottleneck: agents must juggle instructions, tool docs, and outputs, often exceeding a model's efficient context (many hallucinate beyond 30K tokens), necessitating external memory like RAG. Her benchmark shows planning-specialized models struggle with long context and vice versa, and she recommends breaking tasks into subtasks and using test-time compute scaling.