VLM

Taking Reinforcement Learning Cross Datacenter — Nan Jiang, Modal

Aug 10, 2026 · 19:50

Nan Jiang from Modal explains how reinforcement learning post-training can run across datacenters by shipping sparse weight deltas instead of full checkpoints. He argues that less than 1% of rollout-visible weights change between versions because Adam steps are tiny relative to BF16 rounding boundaries, a mechanism he calls Adam absorption. Modal's implementation, Stitch, lets rollout engines sync via patches (e.g., 500 MB instead of 500 GB) and operate as an elastic fleet across regions and providers. He cites internal runs showing 0.15% weight changes initially, settling near 0.05% for GLM 4.7 Air in FP8, and notes gradients are dense but updates small. He also explores whether sparsity holds for Muon and async RL scalability.

How fast are LLM inference engines anyway? — Charles Frye, Modal

Jun 27, 2025 · 16:07

Charles Frye presents benchmarks from hundreds of runs on Modal comparing open-source inference engines VLM, SGLang, and TensorRTLM across models like Qwen 3 and Gemma 27B, arguing open weights models have caught up to proprietary ones, making self-hosting viable. He shows, for example, that Qwen 3 (MoE) on VLM achieves ~1 request/sec with 128 input tokens and 1024 output tokens, while switching to a RAG-like workload (1024 in, 128 out) yields a 4x throughput improvement. Frye warns that optimizing for context over reasoning can improve latency without sacrificing quality, and notes that the engines' out-of-the-box performance varies by model—e.g., SGLang underperforms VLM on Gemma due to less optimization. He also highlights the gap between prefill (parallel) and decode (autoregressive) speeds, which a rationalist would expect from transformer architecture. The benchmarks, available at modal.com/llmalmanac, aim to help engineers choose hardware and engines, with contributions welcome for optimized configs like TensorRTLM's knobs.

The GenAI Maturity Curve or You Probably Don't Need Fine Tuning: Kyle Corbitt

Feb 9, 2025 · 18:03

Kyle Corbitt, CEO of OpenPipe, argues that most teams don't yet need fine-tuning and should start with prompted models like GPT-4. He presents a GenAI maturity curve where the trigger to fine-tune is when you hit constraints on cost, latency, or quality consistency—for example, if GPT-4 is 80-90% correct but inconsistent on the last 10-20%. Fine-tuning shifts the paradigm frontier outward, enabling models like fine-tuned LLaMA 38B to outperform GPT-4 at 1/25th the cost. The process has four steps: capture production logs to know your input distribution, prepare high-quality data (using GPT-4 outputs or iterative labeling), train with one-click tools, and evaluate with inner-loop (LLM-as-judge) and outer-loop (business metrics) evals. OpenPipe and other providers make deployment trivial via OpenAI-compatible APIs. The talk delivers a concrete decision framework and a walkthrough so any engineer can fine-tune in under an hour.