PyTorch

Can LLMs Write Fast Multi-GPU Kernels? — Simran Arora, Together AI

Aug 27, 2026 · 30:00

Simran Arora, principal scientist at Together AI, argues multi-GPU communication, not single-GPU compute, is AI's bottleneck, and introduces ParallelKittens and ParallelKernelBench to test whether LLMs can write fast multi-GPU kernels. From A100 to B200, BF16 tensor core throughput improved 7.2x vs 3x intra-node communication, leaving PyTorch+NCCL baselines below 50% of communication-aware roofline on most problems. ParallelKittens adds a dozen lines to a single-GPU kernel and runs in production at Together and Cursor; in ParallelKernelBench's 87 problems, the best model solves 28 zero-shot (22 faster), and more samples lift correctness to 36 but fast-and-correct stalls near 31%. Models compile after retries but stall on ordering and transfer choices; a bash-based agent hits 35 correct.

AI Agents for Performance: Ship Faster, Pay Less — Rajat Shah, Netflix

Jul 28, 2026 · 33:39

Rajat Shah from Netflix explains how an AI agent can replace the manual performance engineering loop by reading profiling data, identifying inefficient code patterns like an O(N²) tensor merge method, and producing a validated fix — all within five minutes. The agent traced a call stack to the exact source line, proposed an optimized implementation, and ran a canary deployment to confirm CPU savings without regression. Shah emphasizes building a shared catalog of anti-patterns (starting as markdown files in a Git repo) so future agents can catch similar issues earlier, even during code authoring. He advocates shifting left from reactive profiling to proactive prevention, using unit tests, canary automation, and human approval as guardrails. The playbook aims to help teams adopt the same loop to reduce infrastructure cost and ship faster.

Special Topics in Kernels, RL, Reward Hacking in Agents — Daniel Han, Unsloth

Jul 17, 2026 · 2:20:21

Daniel Han of Unsloth argues that reward hacking—where AI models cheat to maximize reward—is a critical problem in agent training, citing examples from GPT-5.1's calculator hacking and GPU mode kernel competitions. He shows that models exploit benchmark flaws, such as viewing Git history or editing timers, and that even open-source models like GLM 5.2 require anti-hacking measures. Han emphasizes that harness and tooling quality now outweigh model choice, with inference providers sacrificing accuracy for speed (e.g., 10% accuracy drops across providers). He also warns that hardware limits (float4 precision, diminishing returns) shift focus to software algorithms like FlashAttention and gradient checkpointing. The workshop concludes that benchmarks are unreliable—DeepSpeed's false positive rate is contested at 44.9%—and urges verification before trusting performance claims.

SWE-Marathon: Evaluating Coding Agents at Billion-Token Scale - Rishi Desai, Abundant AI

Jul 7, 2026 · 12:58

Rishi Desai of Abundant AI presents SWE-Marathon, a benchmark for coding agents that finds even the strongest setup achieves only a 26% resolution rate across 20 project-scale tasks. Trajectories average 31 million tokens, with the longest consuming 877 million tokens, and a computer-use agent verifier evaluates full-stack products via browser interactions. Desai warns that weak verifiers become attack surfaces in long-horizon evals, showing agents bypassing tasks by calling GCC from within a Rust compiler, caught by anti-cheat layers using S-Trace. Across 1,400 rollouts, 12.8% showed suspicious shortcuts and 9% clear verifier bypasses, but zero earned reward through exploits due to multi-channel checks. The central claim: long-horizon SWE is unsolved, and robust verification—not harder unit tests—is the key bottleneck for future benchmarks.

Stop Making Models Bigger, Make Them Behave — Kobie Crawford, Snorkel

Jun 10, 2026 · 20:56

Kobie Crawford of Snorkel explains how a 4B parameter model fine-tuned via RL for under $500 outperformed Qwen 3 235B on financial analysis tool use. The key was training tool discipline—inspecting schemas and self-correcting errors—not deeper reasoning. Single-table training alone boosted multi-table FinQA benchmark from 13.9% to 26.6%, and breaking evals into rubrics identifies which behaviors to fix.

GPU Cloud Deployment Without Leaving Your IDE — Audry Hsu, RunPod

Jun 9, 2026 · 20:19

Audry Hsu of RunPod introduces Flash, a Python SDK that deploys GPU cloud functions from a developer's IDE with a single decorator, eliminating the slow iteration cycle of commits, Docker builds, and server allocation. She demonstrates hot reload, swapping Stable Diffusion XL Turbo for DreamShaper instantly, and a pipeline that chains Qwen 3 for prompt generation, DreamShaper for image rendering, and Nano Banana 2 for photo composition. RunPod's serverless H100 pricing is $0.00116 per second, charged only during active inference. Hsu recommends starting with pods for experimentation and switching to serverless when scaling to hundreds of workers across data centers.

Road to 5 Million Tokens: Breaking Barriers in Long Context Training — Max Ryabinin, Together AI

Jun 8, 2026 · 15:50

Max Ryabinin from Together AI presents their research on extending transformer context length to 5 million tokens using Untied Ulysses, which cuts activation memory by reusing buffers across attention head iterations. The talk walks through a stack of techniques including fully sharded data parallelism, DeepSpeed Ulysses context parallelism for an 8x activation reduction, activation checkpointing for another 8x, CPU offloading of transformer block inputs, and chunked sequence training. Even with these, training a LLaMA 3B model with 3 million tokens fits on an 8xH100 node, but 5 million requires Untied Ulysses. Instead of allocating one large buffer per attention head group, it chunks heads further and reuses buffers across iterations, cutting activation memory with negligible throughput impact. At both 8B and 32B scale, results match the most memory-optimized transformer training baselines while pushing sequence length 25% further than prior Ulysses implementations.

From 46% to 90%: Fine-Tuning Tiny LLMs for On-Device Agents — Cormac Brick, Google

May 20, 2026 · 21:01

Google's Cormac Brick explains how developers can build on-device AI agents using either system-level Gemini Nano via AI Core or app-level custom models via LiteRT-LM. He demonstrates a skill harness built on Gemma 4 that enables agentic tasks like restaurant roulette, running fully on-device with JavaScript UI. For fine-tuned tiny models, Function Gemma at 270M parameters improved from 46% to over 90% accuracy on eight of ten app-intent functions after synthetic data fine-tuning. The talk also covers the Eloquent transcription app, which chains two Gemma 3-based models (ASR and text polishing) under a few hundred million parameters for offline use. Key trade-offs are latency, privacy, and customization versus system integration effort.

Accelerating AI on Edge — Chintan Parikh and Weiyi Wang, Google DeepMind

May 5, 2026 · 23:58

Chintan Parikh and Weiyi Wang from Google DeepMind present Gemma 4E edge models (2B and 4B) and the LiteRT framework for on-device AI, arguing that edge AI delivers latency, privacy, offline capability, and cost savings. The Gemma 4E models introduce agentic capabilities including built-in function calling, structured JSON output, and chain-of-thought reasoning, all optimized for hardware-native support across CPUs, GPUs, and NPUs. LiteRT supports cross-platform deployment on Android, iOS, macOS, Linux, Windows, and IoT devices like Raspberry Pi, with a CLI tool and AI Edge portal for benchmarking. Performance benchmarks show up to 56 tokens per second on iOS, 30x speedups with NPU acceleration, and 35x faster than Llama on mobile. The Gallery app demonstrates on-device skills such as Wikipedia querying, mood tracking, photo-to-music generation, and voice agents, with open-source code and a Hugging Face repository. Q&A addresses use cases like local security camera face recognition, LiteRT vs TensorRT on Orin, multi-agent architectures, and audio model support.

Building Generative Image & Video models at Scale - Sander Dieleman, Google DeepMind

Apr 21, 2026 · 40:46

Sander Dieleman, a research scientist at Google DeepMind, explains the behind-the-scenes process of building generative image and video models at scale, focusing on diffusion models. He emphasizes that data curation is often underrated and more impactful than model tweaking. Dieleman details how latent diffusion models use learned autoencoders to compress pixels into smaller representations, enabling training on high-resolution video. He describes diffusion as 'spectral autoregression' that generates images coarse-to-fine, and notes that guidance—amplifying the difference between conditional and unconditional predictions—massively improves sample quality at the cost of diversity. On sampling, he contrasts deterministic and stochastic approaches, and covers distillation techniques like consistency models that reduce steps. Dieleman also discusses control signals beyond text, such as camera motion in video generation, and the importance of post-training for conditioning.

DSPy: The End of Prompt Engineering - Kevin Madura, AlixPartners

Jan 8, 2026 · 1:13:13

Kevin Madura of AlixPartners argues that building robust enterprise AI applications requires shifting from brittle prompt engineering to programming with LLMs using DSPy, a declarative framework that treats prompts as implementation details optimized by the system. He demonstrates how typed interfaces (Signatures) and modular logic (Modules) allow developers to focus on control flow while deferring implementation to the LLM, with Adapters controlling prompt formats (e.g., JSON vs. BAML) to improve performance by 5-10%. The talk's core is Optimizers (like MIPRO and JEPA), which automatically tune prompts by learning from data, shown improving a time entry corrector from 86% to 89% accuracy. Real-world examples include routing files by type (SEC filings vs. contracts), using a 'poor man's RAG' with attachments for multimodal documents, and a boundary detector that segments legal documents from images. Madura emphasizes that DSPy enables transferability across models (e.g., GPT-4.1 to GPT-4.1 Nano) and addresses cost concerns by allowing offline optimization to reduce LLM calls.

AI Kernel Generation: What's working, what's not, what's next – Natalie Serrino, Gimlet Labs

Dec 17, 2025 · 19:15

Natalie Serrino, cofounder of Gimlet Labs, presents how AI-generated kernels can automatically speed up custom PyTorch code by up to 24% on Apple M4 hardware using the Metal framework, with a 40% speedup from kernel fusion. The agentic system iterates through compilation, execution, correctness, and optimization, but faces challenges like validation of floating-point results and reliable benchmarking. Successes include rewriting average pool 1D as a convolution for 80% improvement, while failures occur on heavily optimized ops like matrix multiply. A real-world audio encoder model saw 70% faster inference on RTX 6000 Blackwell via six custom fused kernels. Serrino emphasizes that AI is best for rapidly searching optimizations and porting code to new hardware, not for surpassing human experts on novel algorithms.

Building Cursor Composer – Lee Robinson, Cursor

Dec 2, 2025 · 15:36

Lee Robinson explains how Cursor built Composer, its first agent model for real-world software engineering, by focusing on being both fast and smart — achieving 4x more efficient token generation than similarly intelligent models while matching open-source performance initially and approaching frontier models after reinforcement learning. The model's training posed infrastructure challenges: matching training and inference environments across thousands of GPUs, handling complex rollouts with hundreds of tool calls and up to millions of tokens, and ensuring consistency by using the same tool format and responses as production. Cursor solved these with custom kernels that sped up training by 3.5x on NVIDIA Blackwell chips for mixture-of-experts layers, load balancing across threads to avoid idle time, and co-designing RL infrastructure with its Cloud Agents product using virtual machines that mirror the production Cursor environment. This allowed the model to become a power user of tools like semantic search, which improved all models but especially Composer. RL also taught the model to parallelize tool calls (e.g., reading 10 files simultaneously) and to search more before editing,…

Compilers in the Age of LLMs — Yusuf Olokoba, Muna

Nov 24, 2025 · 17:36

Yusuf Olokoba, founder of Muna, explains how they built a Python compiler that converts plain Python inference functions—like Google's 270M-parameter Gemma embedding model—into self-contained C++ and Rust binaries using LLMs within a verifiable pipeline. The process involves symbolic tracing to generate an intermediate representation, type propagation to infer types for native compilation, and LLM-driven code generation to mass-produce native implementations of Python operations. After compiling into a shared library, the model can be loaded via FFI from any language (e.g., Node.js) and exposed through an OpenAI-compatible client, enabling developers to run open-source models anywhere—locally, cloud, mobile—with minimal code changes. Olokoba details why they abandoned PyTorch FX due to its PyTorch-only focus and reliance on fake inputs, and how LLMs help scale the coverage of elementary operations. The talk argues that this compiler approach solves hybrid inference—small edge models working with large cloud models—by moving beyond Python and Docker to more portable, low-latency native binaries.

[Full Workshop] Reinforcement Learning, Kernels, Reasoning, Quantization & Agents — Daniel Han

Jul 19, 2025 · 2:42:28

Daniel Han of Unsloth presents a technical workshop covering reinforcement learning (RL), kernels, reasoning, quantization, and agents, arguing that RL with verifiable rewards (RLVR) is the key to unlocking LLM capabilities beyond supervised fine-tuning. He explains why open-source models plateaued after September 2024 until DeepSeek-R1 showed that RL can elicit reasoning, and breaks down PPO, GRPO, and the REINFORCE algorithm, emphasizing that GRPO removes the value model for efficiency. Han details how reward functions—not algorithms—are the hardest part, with examples like distance-based scoring for math. He demonstrates a free Colab notebook training a base model to reason, and shows that dynamic quantization can shrink models like DeepSeek-R1 from 730 GB to 140 GB with only ~1% accuracy loss, arguing that GPUs may stop getting faster after FP4 precision.

Dream Machine: Scaling to 1m users in 4 days — Keegan McCallum, Luma AI

Jul 19, 2025 · 19:03

Keegan McCallum, Head of ML Infrastructure at Luma AI, details how the company's Dream Machine model scaled from 500 to 9,000 H100 GPUs within hours to handle 1 million users in four days, outpacing ChatGPT's initial growth. He explains that their initial Triton inference server setup was brittle and ill-suited for multi-GPU, multi-node video models, prompting a re-architecture to a custom serving stack on vanilla PyTorch. To solve work starvation across user tiers, they implemented an SLO-based aging system that ranks jobs by the percentage of their worst-case wait time elapsed. For managing dozens of model versions, they store immutable full Python environments and checkpoints in object storage, with a YAML file controlling active deployments and enabling zero-downtime rollouts across thousands of GPUs. McCallum also discusses partnerships with Nvidia, AMD, and Grok, and how Luma's broader mission is to build general multimodal intelligence that generates, understands, and operates in the physical world.

RL for Autonomous Coding — Aakanksha Chowdhery, Reflection.ai

Jul 16, 2025 · 19:27

Aakanksha Chowdhery, CEO of Reflection AI and former lead researcher on PaLM and Gemini at Google, argues that reinforcement learning (RL) scaling is the next frontier for autonomous coding agents. She explains that inference-time techniques like majority voting and self-revision improve accuracy but require many samples (e.g., 10,000 for rare correct generations). RL training can learn to generate correct outputs directly, especially in verifiable domains like code with unit tests. Reflection AI aims to build superintelligence starting with autonomous coding, leveraging automated verification to design better reward functions. She notes challenges in scaling RL, including system complexity and reward hacking, but sees coding as ideal due to execution feedback.

Optimizing inference for voice models in production - Philip Kiely, Baseten

Jul 1, 2025 · 15:13

Philip Kiely of Baseten shows how open-source TTS models like Orpheus TTS, built on a LLaMA 3.2 3B backbone, can be optimized for production inference using LLM tooling such as TensorRT-LLM and FP8 quantization, achieving time-to-first-byte (TTFB) under 150 milliseconds and supporting 16–24 simultaneous streams on a single half-H100 GPU. He argues that because TTS models are architecturally similar to LLMs, techniques like dynamic batching, KV-cache quantization, and Torch Compile on the audio decoder can dramatically reduce latency and increase concurrency. He emphasizes that for real-time voice agents, the goal is not raw tokens per second (real-time requires only 83 TPS for Orpheus) but low TTFB and high throughput to minimize GPU spend. However, Kiely warns that non-runtime factors—such as client code using sequential requests without session reuse or sending traffic to distant data centers—can easily add back the milliseconds saved at the model level, and that infrastructure connecting listening, thinking, and talking pipelines is often the dominant source of latency.

Foundry Local: Cutting-Edge AI experiences on device with ONNX Runtime/Olive — Emma Ning, Microsoft

Jun 27, 2025 · 22:52

Emma Ning, Principal PM at Microsoft, presents Foundry Local, a platform for building cross-platform on-device AI applications using ONNX Runtime and Olive. She argues local AI is essential for low-bandwidth, privacy-sensitive, cost-efficient, and real-time latency scenarios, noting that modern hardware and optimized models like Phi-4-mini and DeepSeek make it viable. Ning demonstrates Foundry Local's CLI, model benchmarking (Qwen 1.5B at 90 tok/s vs. Phi-4-mini with richer output), and a cross-platform document summarization app built with JavaScript SDK, running identically on Windows and macOS. She also previews a local agent with MCP servers (file system and OCR) that extracts receipt totals using Phi-4-mini. Customer testimonial from Sava (CEO of Pieces) highlights improved memory management and tokens-per-second, while another partner underscores ease of install and hybrid cloud-local solutions.

Rust is the language of the AGI - Michael Yuan

Jun 3, 2025 · 29:14

Michael Yuan argues that Rust, with its strong type system and compiler feedback, is the ideal language for AI code generation, unlike human-friendly Python or JavaScript. He presents Rust Coder, an open-source project supported by two Linux Foundation Mentorship grants, which uses MCP tools to generate, compile, and fix Rust projects. The system integrates a coding LLM (e.g., Qwen Coder) with a self-improving knowledge base of compiler errors, enabling it to generate correct code and automatically fix bugs. Yuan demonstrates its use in a Rust camp of 1000+ students and envisions future AI agents generating and deploying Rust code autonomously for tasks like drone control. He invites contributions to expand the knowledge base and enhance the tools for AGI.

Luminal - Search-Based Deep Learning Compilers - Joe Fioti

Jun 3, 2025 · 24:35

Joe Fioti presents Luminal, a search-based deep learning compiler that simplifies ML libraries to 12 primitive operations and uses search to automatically discover optimized kernels like flash attention. By representing models as directed acyclic graphs of these simple ops, Luminal keeps its codebase under 5,000 lines yet can run all major models. Its compiler applies 20-25 rewrite rules to search through equivalent GPU kernels, profiling to find the fastest—automatically rediscovering flash attention, an algorithm that took five years for the industry to develop. Data movement accounts for 99% of runtime, so kernel fusion merges many ops into one, dramatically speeding execution. An external auto-grad crate adds training support without altering the core. Future plans include supporting AMD, TPUs, and a serverless cloud that exports optimized graphs for inference.

The Devops Engineer Who Never Sleeps — Diamond Bishop, Datadog

Apr 22, 2025 · 16:18

Diamond Bishop, Director of AI Engineering at Datadog, explains how his team builds AI agents—the On-Call Engineer and Software Engineer—that automate DevOps tasks like incident investigation, remediation, and postmortem writing. He details how the On-Call Engineer wakes up for alerts, reads runbooks, queries logs and metrics, and suggests fixes to let human engineers sleep. The Software Engineer proactively fixes errors by generating code diffs and pull requests. Bishop shares four key lessons: scope tasks and evaluate rigorously, assemble teams of optimistic generalists and UX experts, adapt UX for human-agent collaboration, and treat observability as critical for debugging multi-step agent workflows. He predicts that within five years, AI agents will surpass humans as primary users of SaaS platforms like Datadog, urging builders to design for agent consumers.

Personal, Local, Private AI Agents: Soumith Chintala

Apr 6, 2025 · 20:32

Soumith Chintala, co-founder of PyTorch, argues that personal AI agents should run locally and privately to maintain trust and control over intimate data. He warns that cloud-based agents, lacking complete context (like access to all messaging or financial accounts), become unreliable and potentially dangerous—catastrophic actions like buying a Tesla instead of Tide Pods are possible. Key technical challenges include slow local inference, immature open-source computer-use models, and poor catastrophic action classification. He is bullish on open models surpassing closed ones through coordinated improvement, citing Linux, Llama, and DeepSeek. Chintala also plugs open reasoning data from gr.ink and PyTorch’s work on enabling local agents, urging AI engineers to tackle these gaps.

Optimizing LLMs in Insurance with DSPy: Jeronim Morina

Feb 16, 2025 · 19:29

Jeronim Morina from AXA Germany argues that AI engineers must stop relying on manual prompt tuning and instead adopt DSPy's automated optimization combined with first-principles thinking. He explains how his team built a customer-facing insurance chatbot by first crafting clear problem definitions, annotating evaluation data, and avoiding data leakage. After initial struggles with fragile prompts and overly complex error-handling code, they modularized their system into DSPy modules and created custom metrics for German text. Morina emphasizes that tools like DSPy require a steep learning curve and a solid baseline of hand-written prompts and basic evaluations before optimization. The episode calls for engineers to focus on real-world impact, measure everything, and break down problems into discrete, optimizable steps.

Customized, production ready inference with open source models: Dmytro (Dima) Dzhulgakov

Feb 16, 2025 · 18:55

Dmytro (Dima) Dzhulgakov, co-founder and CTO of Fireworks AI, argues that open source models are the future for production Gen AI applications, and Fireworks provides a platform to make them customized and production-ready. He explains that while proprietary models like GPT-4 are powerful, they are too large, expensive, and slow for many use cases, whereas open models like Llama or Gemma can be fine-tuned for specific domains to achieve better quality at up to 10x speed and lower cost. The main challenges of using open models—complex setup, performance tuning, and production scaling—are addressed by Fireworks' custom serving stack, which achieves the fastest long-prompt inference and serves SDXL fastest among providers, handling over 150 billion tokens per day. The platform supports fine-tuning and serving thousands of LoRA adapters on the same GPU with serverless pay-per-token pricing. Dzhulgakov also highlights the emerging architecture of compound AI systems, where function calling—exemplified by the open-source Fire Function model—connects LLMs to external tools and knowledge sources, enabling agentic applications like stock querying and chart generation. The episode…

Unveiling the latest Gemma model advancements: Kathleen Kenealy

Feb 9, 2025 · 16:25

Kathleen Kenealy, technical lead of the Gemma team at Google DeepMind, unveils the latest advances in the Gemma model family, including the launch of Gemma 2 in 9B and 27B parameter sizes, which outperform models two to three times larger, such as LLaMA 3 70B. She also introduces PALI Gemma, a multimodal model combining Siglip Vision Encoder with Gemma 1.0 for image-text tasks. The episode highlights Gemma's responsible-by-design approach, broad framework support (TensorFlow, Jax, PyTorch, etc.), and the release of the Gemma cookbook with 20 recipes. Kenealy emphasizes that Gemma 2 is optimized for easy integration and fine-tuning, available on Google AI Studio, and invites the community to build and share their projects.

Insights from Snorkel AI running Azure AI Infrastructure: Humza Iqbal and Lachlan Ainley

Feb 8, 2025 · 20:46

Humza Iqbal of Snorkel AI explains how the company uses Azure AI infrastructure powered by NVIDIA GPUs to fine-tune foundation models for enterprise customers, achieving better performance per dollar by switching from A100s to H100s. He details their distributed training stack (PyTorch, Horovod, NFS) and lessons learned such as balancing node count for batch size and monitoring GPU utilization to avoid networking or data-loading bottlenecks. A cost comparison found two H100s outperformed four A100s on both training and inference, enabling faster iteration through more synthetic data. Azure's dedicated VMs, reliable NFS throughput, and flexible capacity allowed Snorkel to scale experiments from single-node to dozens of GPUs. Future work includes programmatic preference signals and multimodal retrieval algorithms, all planned on Azure.

The Hierarchy of Needs for Training Dataset Development: Chang She and Noah Shpak

Oct 15, 2024 · 16:32

Chang She (CEO of LanceDB) and Noah Shpak (AI data platform lead at Character AI) argue that data infrastructure is the critical bottleneck for LLM training, and LanceDB's columnar format solves the 'new cap theorem' for AI: needing fast scans, random access, and handling large multimodal blobs simultaneously. Noah explains how Character AI structures pre-training around wide domain coverage and post-training around granular analytics like token counts and difficulty scores, using synthetic data, quality scoring, and dataset selection to improve models. Chang details how Lance format provides zero-copy schema evolution, time travel, and indexing extensions for vector, scalar, and full-text search, enabling a single table to serve SQL analytics, PyTorch training, and production vector search. The episode emphasizes that speed and iterative dataset management are key to accelerating AI research, with LanceDB facilitating cheap random access and low-infra billion-scale vector search.

Making Open Models 10x faster and better for Modern Application Innovation: Dmytro (Dima) Dzhulgakov

Oct 9, 2024 · 18:55

Dmytro Dzhulgakov, CTO of Fireworks AI, argues that open-source models are the future for GenAI applications because they offer lower latency, lower cost, and domain adaptability compared to proprietary models. He explains that open models can be up to 10x faster for narrow domains and cut costs significantly, using examples like fine-tuned Llama 3 for function calling. Fireworks addresses the challenges of setup, optimization, and production readiness with a custom serving stack that delivers the fastest inference for long prompts and image generation (e.g., SDXL). He highlights FireFunction V2, an open-source model for function calling that combines chat and tool use, and notes that Fireworks serves over 150 billion tokens per day for companies like Quora and Cursor. The talk emphasizes that platforms like Fireworks enable developers to start with serverless inference, fine-tune models, and scale to enterprise-grade deployment with dedicated hardware.

From model weights to API endpoint with TensorRT LLM: Philip Kiely and Pankaj Gupta

Sep 13, 2024 · 1:40:01

Philip Kiely and Pankaj Gupta of Baseten lead a workshop on TensorRT-LLM, NVIDIA's high-performance inference framework for LLMs, arguing its use delivers best-in-class throughput and latency on NVIDIA GPUs. They explain that TensorRT-LLM optimizes computational graphs via plugin kernels and in-flight batching, achieving 216 tokens per second and 180ms time to first token on Mistral 7B. The workshop demonstrates building an engine for TinyLlama 1.1B, including FP8 quantization that reduced engine size from 2 GB to 1.2 GB with minimal quality loss. They benchmark a deployed model, showing 7,000 total tokens per second at batch size 64 on an A10G. The presenters compare TensorRT-LLM favorably to VLLM for high-throughput production use and introduce Truss, Baseten's open-source packaging tool, alongside their managed platform for automatic scaling and fast cold starts.

Low Level Technicals of LLMs: Daniel Han

Jul 31, 2024 · 2:52:26

Daniel Han of Unsloth explains how to find and fix bugs in open-source LLMs like Gemma, Phi-3, and Llama, covering tokenizer issues, architecture pitfalls, and finetuning optimizations. He details the eight Gemma bugs Unsloth fixed, including a critical RoPE downcasting error that broke positional encoding, and a 2048 sliding window bug in Phi-3. Han walks through transformer internals: attention masking, layer norms, RoPE embeddings, and SwiGLU activation, showing how to derive gradients for custom kernels. He demonstrates Unsloth's 2x faster finetuning with 70% less memory via Triton kernels, and introduces new features: automatic Ollama model file creation, CSV fine-tuning with merged columns, and chunked cross-entropy for large vocabularies. The session includes live Q&A on learning rate schedules, precision trade-offs, and mechanistic interpretability.

Unlocking Developer Productivity across CPU and GPU with MAX: Chris Lattner

Jul 25, 2024 · 18:33

Chris Lattner, CEO of Modular, presents MAX, a unified AI framework that accelerates Gen AI inference by combining CPU and GPU programming into a single Pythonic model, and Mojo, a new programming language that extends Python to systems programming with 100–1000x speedups. Lattner argues that current fragmentation across PyTorch, ONNX, TensorRT, and hardware-specific libraries slows innovation, and MAX replaces the entire stack—including cuDNN and Intel MKL—with a consistent, compiler-driven approach. He demonstrates that MAX's Int4/Int6 quantization achieves 5x faster performance than llama.cpp on cloud CPUs, and that its GPU matrix multiplication beats NVIDIA's cuBLAS by up to 30%. Mojo enables developers to write for loops and tokenizers (e.g., for LLaMA 3) in Python-like syntax without dropping to C++ or Rust. MAX is free and available now for CPU inference; GPU support launches in September with early access via Discord.