Multimodal Models

Why AI Agents Need Million-Token Context — Thomas Wolf & Olive Song, MiniMax

Sep 4, 2026 · 20:48

MiniMax's M3 pairs a functional one-million-token context window with coding and multimodal abilities in a 400-billion-parameter, 20-billion-activated model, explain Olive Song, RL Lead at MiniMax, and Thomas Wolf, co-founder of Hugging Face. Song says short context fails when agents handle multi-round tool responses; MiniMax Sparse Attention — an index branch selecting what matters plus a sparse branch computing on selected blocks — was designed by an intern. She argues multimodal training from the very first step beats post-hoc adapters that harm text performance and risk collapse; interleaved data and reward modeling solved this. Its apps reach over 300 million people in 200 countries; anyone can propose projects, and community feedback and internal agent harnesses now drive M3.1.

SOTA Generative Media Panel — Dumitru Erhan, Shane Gu & Nicole Brichtova, Google DeepMind

Aug 30, 2026 · 56:59

Google DeepMind's Dumitru Erhan, Shane Gu and Nicole Brichtova tell swyx that the new Nano Banana 2 Lite and Gemini Omni Flash APIs are steps toward world models, not just prettier videos. They see video models as zero-shot learners that should mature like language models, with understanding and generation unified once cost allows. Language is a lossy intermediary for audio, taste, smell and skin tone; a wine taster Shane consulted borrowed dating vocabulary to describe flavor. Dumitru says people preferred AI versions of real videos because they are sharper and more saturated, an 'Instagram filter', and warns of reward hacking like models adding wedding rings. Evaluation stays manual: Nicole describes ten-person side-by-side video comparisons and asks for real task data and FDE feedback.

Video Has No Memory. Here's How We Built One. — James Le, TwelveLabs

Jul 23, 2026 · 20:27

James Le of TwelveLabs argues that video AI systems lack memory because they treat video as a bag of frames rather than a spatial temporal volume, losing continuity and context. He presents three core problems—wrong context, wrong memory, and weak reasoning—and five properties that make video memory hard: temporality, multimodality, density, ambiguity, and expense. The TwelveLabs stack solves this with Marengo (multimodal embedding encoder), a spatial temporal context store, and Pegasus (video language model), exposed as an API. Le introduces a context graph that connects time-bounded moments, entities, appearances, relationships, and corpus-level themes, enabling traversable video memory. He demonstrates five design principles: ingest once and reason many times, store primitives not answers, ground every claim to a timestamp, let intent shape memory, and keep the layer composable. In demos, he shows a video agent (Joki) that tracks Lionel Messi across 67 World Cup videos, identifies near misses and dramatic goals, classifies vehicles in traffic footage, and suggests ad placement moments in an Adidas clip, all grounded to specific timestamps.

FLUX, Open Research, and the Future of Visual AI — Stephen Batifol, Black Forest Labs

May 8, 2026 · 22:32

Black Forest Labs (BFL), the team behind Stable Diffusion and Latent Diffusion, has released a series of open FLUX image models pushing toward visual intelligence. After FLUX.1 and the first open-source editing model FLUX Kontext (7–8 second edits), FLUX.2 achieved state-of-the-art text-to-image and multi-reference editing, and FLUX.2 Klein generates and edits in 300–500 milliseconds for near real-time use. BFL also published Self-Flow, a scalable self-supervised approach that trains multimodal models across images, video, audio, and actions without external encoders, outperforming baselines in all modalities. The episode explains how Self-Flow reduces artifacts (e.g., corrects text rendering and anatomy) and converges 70× faster with representation alignment. BFL’s roadmap includes world models that simulate geometry and interaction, aiming to train agents for robotics and automation.

Real-time Experiments with an AI Co-Scientist - Stefania Druga, fmr. Google Deepmind

Jul 28, 2025 · 18:42

Stefania Druga, formerly at Google DeepMind, demonstrates a real-time AI co-scientist system built with under $300 in open-source hardware and two weeks of effort, capable of live scientific experimentation. Using a micro:bit with Jack DAC temperature sensor, ReCamera for object tracking, and a microscope, the system streams sensor data, images, and voice to Gemini via a React app that dynamically assembles multimodal context. Druga presents two home experiments: crystal growth, where she discovered that formation occurs in bursts once critical saturation is reached, and fermentation, monitoring CO2 and temperature to optimize dough rise. She references DeepMind's AI co-scientist paper, which replicated a 12-year gene transfer discovery in two days and identified novel drug targets for liver fibrosis that proved effective in wet labs. Her vision moves from async data analysis to real-time empirical hypothesis generation, and she highlights open-source platforms like Jubilee and open bioreactors for automating lab equipment, with future plans to use real-time data to inform realistic simulations.

Multi model multimodal and multi agent innovations in Azure AI: Cedric Vidal

Feb 6, 2025 · 28:56

Cedric Vidal, Principal AI Advocate at Microsoft, demonstrates Azure AI's multi-model, multimodal, and multi-agent capabilities in a session packed with live demos. He shows how GPT-4 Omni mixes text and vision to read handwritten French menus and translate them, and diagnoses infrastructure damage from photos for energy and insurance industries. A new video translation service lets him speak German, Spanish, Italian, and Japanese in his own voice while preserving tone, such as whispering or yelling. The Azure AI model catalog now offers 1,600 models, including serverless deployment options, and Phi-3 Vision, a small 3.8B parameter model running locally in a browser via WebGPU. He also demonstrates code interpreter analyzing a GPX file from a kitesurfing session, plotting a map with turn markers, and GitHub Workspaces preview generating a Java GUI from Python code.

State Space Models for Realtime Multimodal Intelligence: Karan Goel

Oct 29, 2024 · 14:26

Karan Goel, founder of Cartesia, argues that state space models (SSMs) are key to real-time multimodal intelligence, offering cheaper, faster, and higher-quality alternatives to transformers for streaming applications like conversational voice and on-device assistants. He contrasts batch intelligence (cloud APIs) with streaming intelligence needed for low-latency tasks, emphasizing that SSMs compress information linearly rather than storing all tokens, enabling efficient long-context processing. Cartesia’s voice generation model achieves instant latency in the data center and is being optimized for on-device Mac and desktop deployment. Goel asserts that compression helps long-context tasks (e.g., 24-hour security footage analysis) more than retrieval, and that SSMs now match transformer quality while scaling better for multimodal data.