SGLang

Deep dive on LLM Inference at Scale — Harshul Jain, Audible & Tanmay Sah, Independent AI Researcher

Sep 8, 2026 · 1:28:12

Harshul Jain of Audible and independent AI researcher Tanmay Sah teach a two-hour workshop building LLM inference optimization from first principles on Mistral 7B. They show KV cache costs 131 KB per token, so 16,000 tokens across 80 concurrent users needs 42 GB of GPU memory, driving the pain points of memory, TTFT, and throughput. Sah covers quantization, multi-head, multi-query, grouped-query, and latent attention, plus flash attention, using his 'ostrich' and 'world cup' teaching algorithms. Jain covers paged attention, continuous batching, prefix caching, and KV quantization, benchmarked against a Hugging Face baseline on vLLM. Their benchmarking found vLLM and SGLang statistically tied on standard workloads, but SGLang three to four times faster once agentic branching enters.

Local Models: Trust, Control, Optimization — Carter Abdallah, NVIDIA

Aug 7, 2026 · 43:21

NVIDIA's Carter Abdallah, Prime Intellect's Vincent Weisser, Arcee's Lucas Atkins, and NVIDIA's Chris Alexiuk argue open-weight models are the trustworthy foundation for enterprise and local AI. Atkins separates trust from safety: when Anthropic pulled Fable, enterprises chose Chinese open models for guaranteed availability, and open models are inspectable unlike closed APIs. Arcee pretrained a 400B model in six months; Weisser cites a customer that specialized an open model for finance in a week or two, beating Opus at a fraction of Haiku's cost. Alexiuk calls open weights the fix for 'mismanaged genius' and expects capable local models on MacBooks within a year; the panel predicts Fable-level open models within a year and hopes local-model use rises from a rounding error to 10–15%.

What Lies Beneath the API — Benjamin Cowen, Modal

Jun 2, 2026 · 12:40

In this episode, Ben Cowen from Modal argues that as AI products mature, fine-tuning becomes essential, citing cases like Intercom beating their frontier API at 1/10th the cost. He identifies three signals it's time to fine-tune: paying more for the API than customers pay you, evaluative plateaus, and latency requirements that shared endpoints can't meet. Cowen explains that supervised fine-tuning now fits in 300 lines of Python and that reinforcement learning rollouts can scale to 50,000 sandboxes using serverless platforms like Modal. He contends that frontier labs aim to win at everything, while businesses need to win at their specific logic, making fine-tuning a natural destination. The episode provides practical guidance on when and how to make the leap, emphasizing that building an agent harness and collecting eval data already sets the stage for training.

Gemma, DeepMind's Family of Open Models — Omar Sanseviero, Google DeepMind

Apr 20, 2026 · 15:26

Omar Sanseviero presents Gemma 4, Google DeepMind's latest family of open models, which range from 2B to 32B parameters and introduce a novel per-layer embedding (E2B) architecture optimized for on-device inference. The models feature multimodal understanding (images, video, audio), multilingual support across 140+ languages, and are released under an Apache 2 license. Within a week, Gemma 4 reached 10 million downloads, contributing to over 500 million total downloads for the Gemma family and 100,000 community-derived models. Sanseviero highlights official variants like Shield Gemma for content safety and MedGemma for medical tasks, as well as community efforts such as AI Singapore's Southeast Asian language models and Sarvam's sovereign AI initiative for India. He emphasizes real-world applications including cancer therapy pathway discovery and fully offline agentic tasks on phones and Raspberry Pis, arguing that open models are rapidly enabling high-performance, private, and customizable AI across diverse use cases.

Z.ai GLM 4.6: What We Learned From 100 Million Open Source Downloads — Yuxuan Zhang, Z.ai

Nov 22, 2025 · 19:39

Yuxuan Zhang from zAI details the technical roadmap behind the GLM 4.6 open-source model series, which has surpassed 100 million downloads and tied for number one on the LMSYS Chatbot Arena alongside GPT-4o and Claude 3.5 Sonnet. The training pipeline uses 15 trillion tokens of pre-training data, followed by 7 trillion tokens of code and reasoning data, repo-level contexts at 32,000 tokens, and 100 billion tokens of long-context agent trajectories up to 128,000 tokens. Zhang introduces SLIME, a hybrid synchronous/asynchronous RL framework that decouples agent-environment interaction from GPU training to avoid bottlenecks. He explains why single-stage RL at 64,000 tokens outperforms multi-stage approaches for preserving long-context abilities, and shows that token-weighted loss converges faster than sequence-average loss for code RL. The multimodal GLM 4.5V model handles native resolution images and video with temporal index tokens, enabling GUI agent capabilities. Deployment is supported via vLLM and SGLang, with an API at z.ai and an open-source coding assistant.

Introduction to LLM serving with SGLang - Philip Kiely and Yineng Zhang, Baseten

Jul 26, 2025 · 43:42

Philip Kiely and Yineng Zhang introduce SGLang, an open-source fast serving framework for LLMs and VLMs, arguing it offers production-ready performance with day-zero support for new models like DeepSeek and Qwen, and a customizable codebase for contributions. Yineng, a core maintainer, traces SGLang's rapid growth from a December 2023 paper to nearly 15,000 GitHub stars and adoption by xAI, AMD, and Meituan. The workshop demonstrates deploying a first model via Baseten's trunk packaging, then tuning the CUDA Graph max batch size flag on an L4 GPU to maintain CUDA Graph-enabled decoding during higher concurrency, boosting generation throughput. They also cover Eagle 3 speculative decoding, where a draft model derived from the target model speculates tokens; users can benchmark different step and top-k configurations on representative prompts to find optimal settings for production. Finally, they invite contributions via GitHub's 'good first issue' tags and highlight Baseten's job openings.

The Rise of Open Models in the Enterprise — Amir Haghighat, Baseten

Jul 24, 2025 · 16:50

Amir Haghighat, CTO of Baseten, argues that enterprises are increasingly moving from closed frontier models like OpenAI and Anthropic toward open source models, driven by four specific cracks in the assumption that closed models will work indefinitely: quality for specialized tasks (e.g., medical document extraction), latency requirements (especially for voice), unit economics ballooning from agentic use cases where a single user action triggers 50 inference calls, and the desire for competitive differentiation. Drawing on conversations with over 100 enterprises, he explains that while most started with dedicated deployments on Azure/AWS for toying around in 2023, by 2024 about 40-50 had production use cases, and in 2025 the shift accelerated. However, adopting open models forces enterprises to build inference infrastructure, facing challenges like speculative decoding, prefix caching, guaranteeing four-nines reliability with hardware failures and VLM crashes, and scaling replicas—with one Fortune 50 soft drink company reporting an eight-minute spin-up time. Haghighat concludes by contrasting the simple API-call world with the complexities of mission-critical inference, where…

What every AI engineer needs to know about GPUs — Charles Frye, Modal

Jul 20, 2025 · 19:52

Charles Frye of Modal explains that AI engineers need to understand GPU hardware constraints to optimize inference, arguing GPUs embrace bandwidth over latency and that Tensor Cores for low-precision matrix-matrix multiplication are the key resource. He describes how GPUs achieve 16,000+ parallel threads per cycle on H100, and notes Patterson’s Law: bandwidth improves at the square of latency. The main insight: arithmetic intensity favors N² operations per N memory loads, so matrix-matrix operations are efficient while matrix-vector is wasteful. Frye demonstrates that running a small 8B model 1,000 times on the same prompt matches GPT-4 quality, and that multi-token prediction and multi-sample query become nearly free because Tensor Cores handle expanded batches as matrix-matrix multiplications. He recommends using smaller models that fit on a single GPU and scaling via multiple generations.

360Brew: LLM-based Personalized Ranking and Recommendation - Hamed and Maziar, LinkedIn AI

Jul 16, 2025 · 22:00

LinkedIn AI's Hamed Firooz and Maziar Sanjabi present BrewXL, a 150-billion-parameter foundation model for ranking and recommendation that personalizes the platform's feeds, jobs, and search. They show that training a large model then distilling to a 3B model outperforms training a small model from scratch. Scaling data, model size (from 7B to 8x22B), and context length all improve performance, though context beyond the trained length degrades accuracy. The model achieves zero-shot generalization on out-of-domain tasks, matching or beating task-specific production models, and significantly reduces the cold-start gap for users with fewer than five interactions. For serving, they combine gradual pruning, mixed-precision quantization (FP8 for most layers, FP32 for the LM head), and 4D attention masks to score up to 500 items without cross-attention, yielding a 7X latency reduction and 30X throughput increase per GPU.

How fast are LLM inference engines anyway? — Charles Frye, Modal

Jun 27, 2025 · 16:07

Charles Frye presents benchmarks from hundreds of runs on Modal comparing open-source inference engines VLM, SGLang, and TensorRTLM across models like Qwen 3 and Gemma 27B, arguing open weights models have caught up to proprietary ones, making self-hosting viable. He shows, for example, that Qwen 3 (MoE) on VLM achieves ~1 request/sec with 128 input tokens and 1024 output tokens, while switching to a RAG-like workload (1024 in, 128 out) yields a 4x throughput improvement. Frye warns that optimizing for context over reasoning can improve latency without sacrificing quality, and notes that the engines' out-of-the-box performance varies by model—e.g., SGLang underperforms VLM on Gemma due to less optimization. He also highlights the gap between prefill (parallel) and decode (autoregressive) speeds, which a rationalist would expect from transformer architecture. The benchmarks, available at modal.com/llmalmanac, aim to help engineers choose hardware and engines, with contributions welcome for optimized configs like TensorRTLM's knobs.