Generative Media & Multimodal

Why AI Agents Need Million-Token Context — Thomas Wolf & Olive Song, MiniMax

Sep 4, 2026 · 20:48

MiniMax's M3 pairs a functional one-million-token context window with coding and multimodal abilities in a 400-billion-parameter, 20-billion-activated model, explain Olive Song, RL Lead at MiniMax, and Thomas Wolf, co-founder of Hugging Face. Song says short context fails when agents handle multi-round tool responses; MiniMax Sparse Attention — an index branch selecting what matters plus a sparse branch computing on selected blocks — was designed by an intern. She argues multimodal training from the very first step beats post-hoc adapters that harm text performance and risk collapse; interleaved data and reward modeling solved this. Its apps reach over 300 million people in 200 countries; anyone can propose projects, and community feedback and internal agent harnesses now drive M3.1.

SOTA Generative Media Panel — Dumitru Erhan, Shane Gu & Nicole Brichtova, Google DeepMind

Aug 30, 2026 · 56:59

Google DeepMind's Dumitru Erhan, Shane Gu and Nicole Brichtova tell swyx that the new Nano Banana 2 Lite and Gemini Omni Flash APIs are steps toward world models, not just prettier videos. They see video models as zero-shot learners that should mature like language models, with understanding and generation unified once cost allows. Language is a lossy intermediary for audio, taste, smell and skin tone; a wine taster Shane consulted borrowed dating vocabulary to describe flavor. Dumitru says people preferred AI versions of real videos because they are sharper and more saturated, an 'Instagram filter', and warns of reward hacking like models adding wedding rings. Evaluation stays manual: Nicole describes ten-person side-by-side video comparisons and asks for real task data and FDE feedback.

The Next Medium: Why Real-Time Interactive Video Changes Everything — Ahmed Ahres, Reactor

Aug 18, 2026 · 17:30

Ahmed Ahres, head of go-to-market at Reactor, argues that world models are real-time interactive video, a change of medium rather than a speedup, because video becomes programmable like software. GPS enabling Uber and viewfinders enabling Instagram and TikTok show how real time unlocks new applications. Reactor's platform serves infinite interactive video (Helios from ByteDance), controllable worlds (Lingbot from Alibaba, LongLive 2 from Nvidia), and live avatars he admits are still uncracked. Users build interactive live streams, medical and cooking simulations, and video-to-video editing. Real-time infrastructure means streaming pixels, live sessions with memory, and sub-100-millisecond latency; he offers promo code AIE2026 for $75 credits.

Generative Video at the Speed of Light — Keegan McCallum, uRun

Aug 18, 2026 · 8:43

Keegan McCallum, founder of inference provider uRun, argues that generative video's interesting axis is no longer quality but efficiency and long-horizon generation. He shows Helios, a distillation of Wan 2.1 14b, generating clips in real time for ~one-hundredth the cost of a slower frontier-quality clip, and notes at least 40 real-time/long-horizon models released this year. Ten dollars buys three hours of continuous generative video, and fifty buys fifteen; this unlocks magic-mirror webcam transformations, visual mediums for people who don't think in text, and content creation where you steer a generation in under a second. The hard part is serving: global GPUs, WebRTC with ICE/TURN, and synchronized streaming pipelines; uRun is building a React component, Python runtime, and MCP/CLI.

Voice agents with Realtime Video — Sidney Primas, LemonSlice

Aug 18, 2026 · 26:36

LemonSlice CTO Sidney Primas explains how his startup builds real-time video avatars by pointing world models at humans. A Microsoft partnership put a Teddy Roosevelt avatar in a replica Oval Office, generating continuously for eight hours with no reset. He argues audio embeddings drive emotion, and because avatars only look backward, errors compound, so LemonSlice trains with an attention mask and collapses roughly 30 denoising steps to one. He says serving video costs about the same as a voice model, and that the model harness, which orchestrates GPU/CPU threads and queues to avoid stutters, holds much durable value. He predicts an emotion engine for better reactions and an end-to-end EQ layer within two or three years, taking user video and audio and outputting avatar video and audio.

The Next Game Engine Won't Have a Manual — Arturo Nunez, Nereu

Aug 18, 2026 · 19:33

Arturo Nunez, creator of the AI-assisted game engine Nereu, argues the next game engine won't have a manual: instead of coding, users describe intent. Drawing on a decade at Unity, he built Nereu around an asset tag system (ATS) inspired by Entity Component System, where everything is an asset tagged with intent (e.g. character, animated, double jump) and systems query tags—so a building can be tagged drivable and dropped into a Mario Kart race. The assistant, Bibi, helps users get unstuck rather than one-shot finished games, and context is assembled using level of detail: assets near the edited object get full tag values, distant ones collapse to position and type, and hundreds of grass pieces are omitted. Nereu also uses vision models to tag 6,000–7,000 assets, runs in the browser with JavaScript, and is in closed alpha. Nunez contrasts this with world models, which he says remain far from real-time 4K 60 fps games.

Building an Agentic Video Editor for Mass Consumer — Ekaterina Deyneka, Reelful

Aug 18, 2026 · 12:45

Ekaterina Deyneka, founder and CEO of Reelful, presents her agentic video editor as structurally identical to an agentic app builder, with editing real footage harder than generation. At Reelful, users drop in media plus a prompt, and the agent understands the media, proposes a creative plan for approval, then spins up a remote sandbox where agent skills encode cut rules, font pairings, and b-roll generation. Video is composed as React code via Remotion, a verification layer catches composition errors and sends the agent back to reiterate, and the result renders to a polished clip. To hide this complexity from consumers, Reelful is mobile-first with directional templates and a building editor for tweaks. Deyneka demonstrated clips made purely by the agent and announced Reelful's funding from a16z Speedrun.

Training Krea 2: What matters in generative model training — Sangwu Lee, Krea.ai

Aug 18, 2026 · 21:46

Sangwu Lee of Krea.ai explains what went into training Krea 2, its open-sourced image foundation model, arguing that once architecture is locked, data is everything. To keep stylistic diversity over the 'most boring average person' consistency of production models, Krea filters billions of images: no AI images, OCR and vision-language captions, hash and embedding dedup, distilled VLM classifiers, sparse autoencoders as unsupervised taggers for watermarks and borders. World knowledge is checked against Wikipedia concepts by PageRank. Training goes from 256 to 1K resolution through pre-training, mid-training, SFT, preference optimization, and GRPO-style RL, plus a prompt expander. Lee wants a single clean transformer without VAEs/text encoders and VLM-generated bounding boxes or scene graphs.

Agents at Scale: Inside MiniMax's Model and the Infrastructure Behind It — Dan Fu and Olive Song

Jul 31, 2026 · 20:14

Olive Song, RL lead at MiniMax, and Dan, Together AI's VP of Kernels, argue that open-weight MiniMax M3 is closing the gap to frontier labs via community feedback and inference optimization. Dan details the day-zero stack: GPU kernels for M3's sparse attention, KV cache at million-token contexts, and adapting to agentic workloads that upload codebases. Song says M3 trains multimodal from scratch to prevent text-vision collapse, and uses RL with environment design and rewards for long-horizon tasks like replicating a 12-hour iClear paper, plus validation/test splits to avoid hacking. Together's Parallel Kernel Bench holds unsolved problems, so overfitting it yields kernels the company deploys. Both see self-evolution and better GPU utilization as why open models keep accelerating.

Video Has No Memory. Here's How We Built One. — James Le, TwelveLabs

Jul 23, 2026 · 20:27

James Le of TwelveLabs argues that video AI systems lack memory because they treat video as a bag of frames rather than a spatial temporal volume, losing continuity and context. He presents three core problems—wrong context, wrong memory, and weak reasoning—and five properties that make video memory hard: temporality, multimodality, density, ambiguity, and expense. The TwelveLabs stack solves this with Marengo (multimodal embedding encoder), a spatial temporal context store, and Pegasus (video language model), exposed as an API. Le introduces a context graph that connects time-bounded moments, entities, appearances, relationships, and corpus-level themes, enabling traversable video memory. He demonstrates five design principles: ingest once and reason many times, store primitives not answers, ground every claim to a timestamp, let intent shape memory, and keep the layer composable. In demos, he shows a video agent (Joki) that tracks Lionel Messi across 67 World Cup videos, identifies near misses and dramatic goals, classifies vehicles in traffic footage, and suggests ad placement moments in an Adidas clip, all grounded to specific timestamps.

HTML Is All Agents Need — James Russo, HeyGen

Jul 21, 2026 · 15:13

James Russo, software engineer at HeyGen, argues that HTML, CSS, and JavaScript are the native languages of LLMs and all agents need to create great videos—leading to HyperFrames, an open-source framework that turns agent-authored HTML into deterministic MP4s. By letting the small Gemini 3 Flash model author code first, they ensured the thinnest wrapper won, with only a few data attributes for timing. Rendering works by freezing the browser clock and seeking frame by frame, ensuring everything loads before capturing each frame—enabling deterministic video from any web technology (Three.js, WebGL, Lottie). HyperFrames skills focus on taste and video craft rather than teaching frameworks, raising the floor for single-shot output. The framework has already rendered over 1.3 million videos in 90 days from 267,000 creators, with 15,000 daily renders and 32,000 GitHub stars—proving the approach at scale.

"The biggest challenge in your stack? Evals, Evals, Evals" - 2026 State of AI Engineering results

Jul 21, 2026 · 19:47

In the 2026 State of AI Engineering survey presented by Amplify Partners' Barr Yaron, 1,048 AI engineers reveal that cost is now a first-class engineering constraint—40% say it regularly shapes how ambitiously they use AI. Agents have exploded: 95% of teams now use agents, and 89% of those agents have write access, tripling from last year. Image generation adoption doubled to 36%, while audio shows the strongest intent-to-adopt at 56%. Open-weight models augment rather than replace closed models—45% use open-weight, but over 90% of them also use closed models. Evals remain the top infrastructure challenge, and inference is the most bought layer, while prompt management (61% built in-house) stays close to product logic. Teams report 97% net positive impact, but 59% fear long-term liabilities from AI code, and over a third say non-developers now ship features.

Running a Chess YouTube Channel entirely by AI — Stephan Steinfurt, TNG

Jul 8, 2026 · 16:31

Stephan Steinfurt, from TNG, presents their AI system that automatically creates daily chess puzzle explanation videos for YouTube. The agent combines a chess engine (for analysis) with Google's Gemini 3.1 Pro (for natural language) to generate detailed move-by-move commentary, including arrows and highlights. It downloads games from LeChess nightly, runs checks/captures/threats analysis, and uses 11Labs for text-to-speech with emotional tags. The YouTube channel has 500k views and 4,000 subscribers, with each video costing 20-30 cents (not yet monetized). Steinfurt notes the error rate is low (1 in 20 videos has a mistake) and discusses balancing explanations for different skill levels using the Maya engine to suggest human-like moves.

HTML is All You Need (for Agents to Make Graphics) - Amol Kapoor, Nori

Jun 28, 2026 · 7:00

Amol Kapoor, CEO of Nori Agentic, argues that coding agents can create high-quality visual artifacts like slides, docs, and videos by using HTML instead of traditional design tools. He explains that agents think in language and structure, not pixels, so tools like PowerPoint or Figma force them into a human-centric approach that fails. By giving agents HTML—a language they understand intuitively—they can produce properly aligned, themed graphics without manually placing coordinates. Kapoor demonstrates this with the 'pelican riding a bicycle' test, where models fail at raw SVG but succeed with HTML. He applies this method at Nori to build board decks, sales decks, and even animated videos purely with HTML and CSS. The episode advocates for 'thinking like the model' and concludes that for graphics, HTML is all you need.

From Transcription to Live Music: Gemini's Audio Stack — Thor Schaeff, Google DeepMind

Jun 9, 2026 · 19:34

Thor Schaeff from Google DeepMind presents the Gemini audio stack—Gemini 3 Flash Preview for deep audio understanding, Gemini 3.1 Flash Live for real-time sound-to-sound multimodal interaction, and Lyria 3 for music generation. He shows how a single API call extracts speaker labels, timestamps, emotions, language detection, and translation, and how speech generation uses a 'director's note' to modify a base voice's accent and tone. The talk culminates in a live demo where the Gemini Live model uses Lyria via tool calls to generate a German techno schlager about the UK startup scene.

Prompt to Pipeline: Building with Google's Gen Media Stack — Paige & Guillaume, Google DeepMind

May 23, 2026 · 1:54:35

Paige Bailey, Guillaume Vernade, and Ian Valentine from Google DeepMind demonstrate the company's full generative media stack—from Gemini 3.1 Flash Light's video analysis at $0.25 per million tokens to Genie 3's playable world models and Gemma 4's on-device agentic coding—showing developers how to build multimodal apps without cloud APIs. Paige shows AI Studio's Build feature creating a bookshelf scanning app with Firestore and OAuth. Guillaume walks through a workshop using Nano Banana 2 for character portraits, VO 3.1 Lite for video generation at $0.05 per image, LIA 3 for chapter scores, and text-to-speech with distinct voices. Ian runs Gemma 4's 26B mixture-of-experts model on a MacBook to generate 10 SVGs in parallel and build a game from a spec, all locally. The episode argues that Google's models absorb common agent patterns, making custom fine-tunes less necessary.

Any-to-Any: Building Native Multimodal Agents - Patrick Löber, Google DeepMind

May 20, 2026 · 16:21

Patrick Löber, a member of the technical staff at Google DeepMind, explains how to build native multimodal agents using the Gemini API ecosystem, covering multimodal understanding, native image and speech generation, and real-time interaction via the Live API. He demonstrates constructing a NotebookLM clone as an agentic system where a reasoning Gemini model decides whether to generate an infographic or podcast-style audio using function calls to specialized models like Nano Banana for images and a text-to-speech model for speech. The episode details practical implementation: uploading PDFs, video, and audio files; using context caching to reduce costs by 90%; and generating infographics or multi-speaker audio directly from prompts. Löber highlights that native generation models understand world context—like drawing arrows on a map to produce the Golden Gate Bridge—and that the Live API enables audio-to-audio interactions with a single architecture, supporting multiple languages and accents.

Let's go Bananas with GenMedia — Guillaume Vernade, Google DeepMind

May 18, 2026 · 1:17:14

Guillaume Vernade from Google DeepMind demonstrates the full GenMedia stack—Nano Banana for images, Veo for video, Lyria for music, and TTS—by illustrating an open-source book live. The core insight is that Gemini acts as the prompt engineer for every other model, and this works partly because the gen media models were trained on prompts written by Gemini. He shows how to generate character portraits, chapter scenes, animated video clips using images as first frames, and distinct music per chapter, with Lyria Realtime allowing continuous music that responds to new prompts like a DJ. A new interactions API caches context server-side to make chained multi-turn calls cheaper. For TTS, he uses a trick: assigning different speaking styles (e.g., long poetic pauses vs. breathless stutter) to the same two voices to create four distinct-sounding characters.

FLUX, Open Research, and the Future of Visual AI — Stephen Batifol, Black Forest Labs

May 8, 2026 · 22:32

Black Forest Labs (BFL), the team behind Stable Diffusion and Latent Diffusion, has released a series of open FLUX image models pushing toward visual intelligence. After FLUX.1 and the first open-source editing model FLUX Kontext (7–8 second edits), FLUX.2 achieved state-of-the-art text-to-image and multi-reference editing, and FLUX.2 Klein generates and edits in 300–500 milliseconds for near real-time use. BFL also published Self-Flow, a scalable self-supervised approach that trains multimodal models across images, video, audio, and actions without external encoders, outperforming baselines in all modalities. The episode explains how Self-Flow reduces artifacts (e.g., corrects text rendering and anatomy) and converges 70× faster with representation alignment. BFL’s roadmap includes world models that simulate geometry and interaction, aiming to train agents for robotics and automation.

Agents on the Canvas in tldraw — Steve Ruiz, tldraw

May 1, 2026 · 19:54

Steve Ruiz of tldraw details the evolution of AI on the infinite canvas, from early one-shot demos like MakeReal (draw to functional prototype) to multi-agent 'fairies' that collaborate, delegate, and even rewrite the canvas in real time. He explains how the tldraw SDK enables agents to act as virtual collaborators, using structured outputs and tool use to draw, animate, and modify designs. Ruiz demonstrates fairy agents that can work independently or in leader-follower mode, coordinating on tasks like creating wireframes or filling out forms. He also unveils a desktop prototype that lets Claude directly inject JavaScript into the canvas, enabling agents to edit code, modify UI, and even hack other apps like Spotify. The talk emphasizes the shift from sidebar agents to canvas-native collaborators, the challenges of vision model training for 2D spaces, and the safety trade-offs of giving LLMs runtime access in local-first apps.

Build & deploy AI-powered apps — Paige Bailey, Google DeepMind

Apr 29, 2026 · 1:03:20

Paige Bailey, engineering lead for developer relations at Google DeepMind, demonstrates how the company's latest multimodal models and AI Studio platform enable rapid prototyping of AI-powered apps at minimal cost. She showcases Gemini 3.1 Flash Lite analyzing YouTube dinosaur videos for under a penny, the new Build feature constructing a fully functional bookshelf cataloger with Firebase database and Google login, and Gemini Live providing real-time multilingual screen interaction. Other demos include Genie 3 generating playable world models from text prompts, NanoBanana 2 performing cost-effective image editing, Veo 3.1 Lite producing stock footage (e.g., a "vegan basketball food truck" featuring Chef Curry), Lyria 3 creating custom music tracks, and Gemma 4 offering open-source multimodal models that run on mobile devices. Bailey emphasizes that the entire stack—grounded by Google Search and code execution—allows developers to ship ideas that would have been startups years ago, with all code exportable for production use.

Gemma 4 Deep Dive — Cassidy Hardin, Researcher, Google DeepMind

Apr 27, 2026 · 19:03

Cassidy Hardin, a researcher at Google DeepMind, details the Gemma 4 family of open-source models, claiming they set a new precedent for small-scale performance. The family includes two on-device effective models (E2B and E4B) with Per Layer Embeddings (PLE) stored in flash memory, and two larger models: a 26B mixture-of-experts (MoE) with 128 total experts (8 active) and a 31B dense model that ranked #3 on the LM arena leaderboard, outperforming models 20x its size. Architectural innovations include interleaved local/global attention (5:1 ratio, sliding windows of 512/1024 tokens), grouped query attention (8 queries per key-value in global layers), and native multimodal support with variable aspect ratios and resolutions for vision encoders (550M params for large, 150M for small) plus a 305M parameter conformer for audio. All models are released under Apache 2.0 license, available for self-hosting on Hugging Face and Ollama, or cloud deployment on Vertex AI.

Building Generative Image & Video models at Scale - Sander Dieleman, Google DeepMind

Apr 21, 2026 · 40:46

Sander Dieleman, a research scientist at Google DeepMind, explains the behind-the-scenes process of building generative image and video models at scale, focusing on diffusion models. He emphasizes that data curation is often underrated and more impactful than model tweaking. Dieleman details how latent diffusion models use learned autoencoders to compress pixels into smaller representations, enabling training on high-resolution video. He describes diffusion as 'spectral autoregression' that generates images coarse-to-fine, and notes that guidance—amplifying the difference between conditional and unconditional predictions—massively improves sample quality at the cost of diversity. On sampling, he contrasts deterministic and stochastic approaches, and covers distillation techniques like consistency models that reduce steps. Dieleman also discusses control signals beyond text, such as camera motion in video generation, and the importance of post-training for conditioning.

Gemma, DeepMind's Family of Open Models — Omar Sanseviero, Google DeepMind

Apr 20, 2026 · 15:26

Omar Sanseviero presents Gemma 4, Google DeepMind's latest family of open models, which range from 2B to 32B parameters and introduce a novel per-layer embedding (E2B) architecture optimized for on-device inference. The models feature multimodal understanding (images, video, audio), multilingual support across 140+ languages, and are released under an Apache 2 license. Within a week, Gemma 4 reached 10 million downloads, contributing to over 500 million total downloads for the Gemma family and 100,000 community-derived models. Sanseviero highlights official variants like Shield Gemma for content safety and MedGemma for medical tasks, as well as community efforts such as AI Singapore's Southeast Asian language models and Sarvam's sovereign AI initiative for India. He emphasizes real-world applications including cancer therapy pathway discovery and fully offline agentic tasks on phones and Raspberry Pis, arguing that open models are rapidly enabling high-performance, private, and customizable AI across diverse use cases.

How Google DeepMind is researching the next Frontier of AI for Gemini — Raia Hadsell, VP of Research

Apr 18, 2026 · 20:37

Google DeepMind VP of Research Raia Hadsell presents three non-language-model frontiers of AI: Gemini Embeddings 2, an omnimodal embedding model that unifies text, video, audio, and PDFs into a single semantic vector for fast retrieval; weather forecasting models including GraphCast (spherical GNN for 15-day forecasts), GenCast (probabilistic model 97% more accurate than benchmarks), and FGN (directly predicts cyclones, used by the US National Hurricane Center); and Genie world models that create interactive, memory-rich 3D environments from prompts, enabling real-time dynamic changes for gaming and education.

Building in the Gemini Era – Kat Kampf & Ammaar Reshi, Google DeepMind

Dec 15, 2025 · 17:57

Kat Kampf and Ammaar Reshi from Google DeepMind present Gemini 3 and Nano Banana Pro, arguing that these models enable anyone to build complex, aesthetic applications through natural language alone. Gemini 3 achieves state-of-the-art results in one-shot UI design and agentic tool calling, with SweBench outperformance. Nano Banana Pro integrates Google Search for world knowledge and renders text accurately, handling up to 14 consistent people per image. They demonstrate vibe coding in AI Studio, building a personalized comic book with precise text, laptop stickers grounded in search, and a 3D racing game that scaled to 23 players live. Upcoming full-stack runtime adds backend support and automatic database integration, further democratizing software creation.

VoiceVision RAG - Integrating Visual Document Intelligence with Voice Response — Suman Debnath, AWS

Dec 6, 2025 · 1:23:52

Suman Debnath, a Principal ML Advocate at AWS, demonstrates how Colpali—a vision-based retrieval model that treats each document page as an image and generates multi-vector embeddings via patch-based late interaction—can be combined with voice synthesis for a more intuitive RAG system. He explains that Colpali bypasses traditional OCR and preprocessing by directly embedding document images, then scoring query-page relevance through dot-product similarity across patches. The workshop shows how to embed pages, store them in Qdrant with multivector support, and retrieve top pages. Debnath then wraps this retrieval pipeline using the Strands Agent framework, adding a speak tool to output answers in natural voice. A live demo answers a textbook question about trophic levels, first using Bedrock to generate text and then speaking the answer in a female voice, all without a predefined system prompt. The talk positions Colpali as a complementary technique for complex visual documents like IKEA instructions or scanned forms, not a full replacement for traditional RAG.

Perceptual Evaluations: Evals for Aesthetics — Diego Rodriguez, Krea.ai

Aug 23, 2025 · 16:28

Diego Rodriguez, cofounder of Krea.ai, argues that current AI evaluations fail to capture human perception and aesthetics, leading to metrics that misjudge generative media. He traces how compression standards like JPEG and MP3 exploit human sensory limits yet their artifacts poison AI training data, making models blind to what humans instantly see wrong. Standard metrics like FID score penalize perceptually identical images, while models cannot evaluate subjective qualities like artistic meaning. Rodriguez calls for new "perceptually aware" metrics trained on human opinions, echoing a friend's insight that predicting cars was easy but traffic was hard — the real challenge is evaluating AI's impact on creative expression. Krea.ai, an 8-person startup, invites researchers to join work on aesthetics research and hyper-personalization for generative multimedia.

The Next Unicorns: 7 Top AI startups from the HF0 Residency

Aug 21, 2025 · 22:16

Diego Rodriguez (Krea) presents an AI creative suite that generated 1M images/day for a Fox ad, while OpenHome debuts the first AI smart speaker with 10K developers and 500 free dev kits. Josh’s Coframe made $20M for a travel firm by making websites adaptive, and Eugene’s Featherless AI built QWERTY 72B without transformer attention, claiming scale is dead in favor of reliability. Jonas Bauer’s Upside uses LLMs to structure enterprise data, Lengyue’s OpenAudio introduces S1, the first instructable voice model beating ElevenLabs, and Alex Atallah’s OpenRouter provides a single API for all LLMs, growing 10–100% monthly.

Vision AI in 2025 — Peter Robicheaux, Roboflow

Aug 3, 2025 · 17:24

Peter Robicheaux, ML lead at Roboflow, argues vision models are not smart because they fail at basic visual tasks like telling time on a watch or identifying a school bus direction. He attributes this to saturated evals like ImageNet and COCO, and to vision models not leveraging large pre-training like language models. Roboflow introduces RFDetter, a real-time detector using a Dynav2 backbone, achieving second SOTA on COCO and large gains on their new RF100VL dataset of 100 diverse domains. The benchmark shows specialist models fine-tuned on 10-shot examples outperform large VLMs like Qwen2.5 VL 72B, indicating VLMs are strong linguistically but weak visually. Roboflow's platform is free for researchers contributing data back; dataset at rf100vl.org.

What Is a Humanoid Foundation Model? An Introduction to GR00T N1 - Annika & Aastha

Jul 28, 2025 · 17:47

NVIDIA's Annika Brundyn and Aastha Jhunjhunwala introduce GR00T N1, an open-source Vision-Language-Action foundation model for humanoid robots. They argue that physical AI is key to addressing labor shortages in industries like healthcare and manufacturing, and that humanoid forms are needed because the world is built for humans. The model uses a data pyramid strategy combining limited real-world teleoperation data, synthetic simulation data, and internet video, with DreamGen for data multiplication. Its dual-system architecture, inspired by Kahneman's 'Thinking, Fast and Slow', has a System 2 planner for high-level reasoning and a System 1 for fast motor control at 120 Hz. The model is trained end-to-end via imitation learning and reinforcement learning, and features a generalist action decoder that enables cross-embodiment fine-tuning.

Real-time Experiments with an AI Co-Scientist - Stefania Druga, fmr. Google Deepmind

Jul 28, 2025 · 18:42

Stefania Druga, formerly at Google DeepMind, demonstrates a real-time AI co-scientist system built with under $300 in open-source hardware and two weeks of effort, capable of live scientific experimentation. Using a micro:bit with Jack DAC temperature sensor, ReCamera for object tracking, and a microscope, the system streams sensor data, images, and voice to Gemini via a React app that dynamically assembles multimodal context. Druga presents two home experiments: crystal growth, where she discovered that formation occurs in bursts once critical saturation is reached, and fermentation, monitoring CO2 and temperature to optimize dough rise. She references DeepMind's AI co-scientist paper, which replicated a 12-year gene transfer discovery in two days and identified novel drug targets for liver fibrosis that proved effective in wet labs. Her vision moves from async data analysis to real-time empirical hypothesis generation, and she highlights open-source platforms like Jubilee and open bioreactors for automating lab equipment, with future plans to use real-time data to inform realistic simulations.

Robotics: why now? - Quan Vuong and Jost Tobias Springberg, Physical Intelligence

Jul 26, 2025 · 18:07

Physical Intelligence's Quan Vuong and Jost Tobias Springberg describe their mission to build a model that can control any robot to do any task, arguing that software intelligence is the main bottleneck in robotics. They explain Vision Language Action models (VLAs) as adaptations of vision language models that output robot actions instead of text. To train these models, they built a data engine from scratch, collecting 10,000 hours of successful episodes via teleoperation in six months. Their latest model, PAIO-5, achieves open-world generalization by training on data from multiple homes, matching or surpassing performance on held-out scenes. They demonstrate this with a policy that performs long-horizon tasks like cleaning an unseen bedroom for up to 10 minutes autonomously. They also highlight a remote coffee-making demonstration on a robot they never touched, showing model portability across hardware.

Google Photos Magic Editor: GenAI Under the Hood of a Billion-User App - Kelvin Ma, Google Photos

Jul 19, 2025 · 20:28

Kelvin Ma, an engineer on Google Photos' editing team, explains how the billion-user app built the Magic Editor by integrating generative AI with on-device computational photography. He traces the evolution from earlier ML features like post-capture segmentation (UNet model, 10 MB) and Magic Eraser (a system of models) to the new server-side generative AI experience, which handles tasks like relocating objects and reimagining backgrounds. Key challenges include managing model size (now hundreds of MB), client-server latency, ambiguous problem scoping (e.g., moving from 5% to 80% reliability), and trust and safety. Ma advocates using evals, reducing ambiguity through product-research collaboration, and iterating from large general models to smaller, faster ones for production. He also highlights that Google Photos serves 1.5 billion monthly active users and processes hundreds of millions of edits per month, and the editor is being rebuilt as AI-first.

ComfyUI Full Workshop — first workshop from ComfyAnonymous himself!

Jul 19, 2025 · 51:25

ComfyAnonymous and Yedrick Kosinski present ComfyUI, an open-source node-based canvas for generative AI that has become the top 150 most popular GitHub repos with 78,000 stars and 3 to 4 million active users. Started as a personal project in January 2023, it now supports image, video, audio, 3D, and text models, with 22,000 custom nodes from 3,000 developers. A key feature is embedding full workflows into generated files for easy sharing. The team discusses its story, including ComfyAnonymous's time at Stability AI and founding ComfyOrg, and addresses questions on CFG, VAEs, LoRAs, control nets, API nodes for remote generation, and future plans for cloud inference and improved onboarding.

The State of Generative Media - Gorkem Yurtseven, FAL

Jul 16, 2025 · 17:14

Gorkem Yurtseven, co-founder and CTO of fal.ai, argues generative media lowers creation's marginal cost to near zero, transforming advertising and e-commerce. He traces evolution from DALL·E 2 to open-source models like Stable Diffusion and FLUX, with the playing field evening quickly. Video models (Sora, Veo 3) are accelerating: fal's platform saw video usage jump from near zero to 30% of revenue, and he predicts the generative video market 100x-250x bigger than image. Advertising leads with hyper-personalized, interactive ads, like fal's A24 campaign turning selfies into toy soldiers. In e-commerce, virtual try-on is a proven product-market fit. Future real-time video generation will blur games and movies.

Building Multimodal AI Agents From Scratch — Apoorva Joshi, MongoDB

Jun 27, 2025 · 36:58

Apoorva Joshi, an AI developer advocate at MongoDB, leads a hands-on workshop on building multimodal AI agents from scratch. She defines agents as systems using LLMs to reason, plan, and execute tasks with tools, contrasting them with simple prompting and RAG. The workshop focuses on processing mixed-media documents (text and images) by converting each page to a screenshot, embedding it with Voyage AI's multimodal (VLM-based) model to avoid the modality gap of CLIP, and storing embeddings in MongoDB for vector search. The agent uses Google's Gemini 2.0 Flash as the multimodal LLM, implements REACT (reasoning and acting) for planning with feedback, and manages short-term memory via session IDs. Participants build a tool-calling agent that retrieves relevant page screenshots, passes them with conversation history to Gemini, and generates answers for queries like analyzing charts or extracting insights from documents.

Veo 3 for Developers — Paige Bailey, Google DeepMind

Jun 21, 2025 · 20:37

Paige Bailey, engineering lead for Google DeepMind's DevRel team, presents Veo 3, the latest video generation model that natively produces synchronized audio—dialogue, sound effects, and music—from text and image prompts, marking a leap in coherence, physics simulation, and cinematic control. She demonstrates Veo 2's creative controls like reference-powered videos, outpainting, and character consistency, and shows how video generation has evolved from 2023's choppy clips to Veo 3's polished raccoon dancing in front of pyramids. A key demo replicates a Chick-fil-A commercial: Veo 2 required stitching multiple tools, while Veo 3 produced the full ad from a single prompt. The models are accessible via Vertex AI private preview and Gemini Ultra plans.

AI Music Generation, From Prompt to Production: Phlo Young

Feb 11, 2025 · 54:34

Phlo Young, an AI engineer and musician, demystifies AI music generation by distinguishing text-to-music, audio-to-music, and style transfer, and demonstrates hands-on use of Suno and Udio to turn prompts into songs. He showcases examples like a Kanye West voice conversion, a Randy Travis AI comeback, and a BBL Drizzy meme song generated entirely from text. Phlo explains prompt engineering techniques, provides live access to paid accounts for attendees to generate unlimited music, and covers tools like Wave Tool for extracting stems. He also touches on ethical considerations and the RIAA lawsuits against Suno and Udio. The workshop equips beginners with practical skills to create and refine AI-generated music.

Unveiling the latest Gemma model advancements: Kathleen Kenealy

Feb 9, 2025 · 16:25

Kathleen Kenealy, technical lead of the Gemma team at Google DeepMind, unveils the latest advances in the Gemma model family, including the launch of Gemma 2 in 9B and 27B parameter sizes, which outperform models two to three times larger, such as LLaMA 3 70B. She also introduces PALI Gemma, a multimodal model combining Siglip Vision Encoder with Gemma 1.0 for image-text tasks. The episode highlights Gemma's responsible-by-design approach, broad framework support (TensorFlow, Jax, PyTorch, etc.), and the release of the Gemma cookbook with 20 recipes. Kenealy emphasizes that Gemma 2 is optimized for easy integration and fine-tuning, available on Google AI Studio, and invites the community to build and share their projects.

Multi model multimodal and multi agent innovations in Azure AI: Cedric Vidal

Feb 6, 2025 · 28:56

Cedric Vidal, Principal AI Advocate at Microsoft, demonstrates Azure AI's multi-model, multimodal, and multi-agent capabilities in a session packed with live demos. He shows how GPT-4 Omni mixes text and vision to read handwritten French menus and translate them, and diagnoses infrastructure damage from photos for energy and insurance industries. A new video translation service lets him speak German, Spanish, Italian, and Japanese in his own voice while preserving tone, such as whispering or yelling. The Azure AI model catalog now offers 1,600 models, including serverless deployment options, and Phi-3 Vision, a small 3.8B parameter model running locally in a browser via WebGPU. He also demonstrates code interpreter analyzing a GPX file from a kitesurfing session, plotting a map with turn markers, and GitHub Workspaces preview generating a Java GUI from Python code.

EyeLevel Launch: Your RAG is Tripping, Here's the Real Reason Why

Feb 6, 2025 · 6:13

Benjamin Over, co-founder of EyeLevel.ai, argues that 95% of RAG hallucinations stem from content ingestion problems, not LLMs or prompts, and introduces his platform's approach to solve this. EyeLevel avoids vector databases entirely, instead creating semantic objects with auto-generated metadata and rewriting text for search and completion. Its pipeline uses a fine-tuned vision model to extract tables, figures, and text, then applies nine fine-tuned models for ingestion and search, including a re-ranking LLM. Air France uses EyeLevel to build a call-center co-pilot, achieving over 95% accuracy on hundreds of thousands of complex documents. In a study, EyeLevel reached 98% accuracy on real-world documents, outperforming popular solutions by up to 120%.

Moondream: how does a tiny vision model slap so hard? — Vikhyat Korrapati

Nov 14, 2024 · 19:26

Vikhyat Korrapati built Moondream, a tiny open-source vision language model under 2 billion parameters that matches LLaVA 1.5, a model four times its size, on VQA v2 and GQA benchmarks. He attributes this to focusing on image understanding over world knowledge and investing heavily in synthetic training data—a pipeline that generated 35 million images and used two orders of magnitude more compute on data than training. Key lessons: community engagement was critical, open-source builds trust, and safety guardrails should be application-layer, not baked in. He argues tiny models will dominate production due to cost and privacy advantages, and that prompting will replace custom model training for most vision tasks. Moondream raised a seed round from FullySysAscent and the GitHub Fund.

State Space Models for Realtime Multimodal Intelligence: Karan Goel

Oct 29, 2024 · 14:26

Karan Goel, founder of Cartesia, argues that state space models (SSMs) are key to real-time multimodal intelligence, offering cheaper, faster, and higher-quality alternatives to transformers for streaming applications like conversational voice and on-device assistants. He contrasts batch intelligence (cloud APIs) with streaming intelligence needed for low-latency tasks, emphasizing that SSMs compress information linearly rather than storing all tokens, enabling efficient long-context processing. Cartesia’s voice generation model achieves instant latency in the data center and is being optimized for on-device Mac and desktop deployment. Goel asserts that compression helps long-context tasks (e.g., 24-hour security footage analysis) more than retrieval, and that SSMs now match transformer quality while scaling better for multimodal data.

The Multimodal Future of Education: Stefania Druga

Oct 24, 2024 · 20:05

Stefania Druga, a research scientist on Google's Gemini team, argues that multimodal AI can transform education by making learning more interactive and critical. She presents Cognimates, an open-source platform she built that expands Scratch to let kids train custom AI models and program smart devices, showing a longitudinal study where kids became more skeptical of AI's intelligence after tinkering. Druga also introduces a benchmark for math misconceptions and demonstrates live demos using the Gemini API—a science tutor that responds to drawings, a math coach that avoids giving direct answers, and a curiosity-driven object identifier—all in under 100 lines of code. She emphasizes the need to cultivate AI literacy early, as 70% of generative AI users are Gen Z, and invites engineers to build tinkering tools that balance agency between learners and AI.

From Text to Vision to Voice Exploring Multimodality with Open AI: Romain Huet

Jul 10, 2024 · 23:39

OpenAI's Romain Huet demonstrates GPT-4o's real-time audio, vision, and screen-sharing capabilities at the AI Engineer World's Fair, showing how the omni-model achieves human-like conversational latency, understands emotion, and can be interrupted naturally. He walks through live demos of ChatGPT Desktop recognizing drawings, reading book pages, and pair-coding a responsive travel app using Tailwind CSS. Huet also previews Sora's text-to-video generation and OpenAI's unreleased Voice Engine, which can clone a voice from seconds of audio and translate it into multiple languages while preserving the original speaker's tone. He outlines OpenAI's four focus areas: advancing textual intelligence, faster/cheaper models (GPT-4o is twice as fast as GPT-4 Turbo at half the price), model customization via fine-tuning, and enabling multimodal agents. The talk emphasizes that today's models are the dumbest they'll ever be and urges developers to build for a future where AI works across text, voice, and video.

Using AI to Build an Infinite Game: Jeff Schomay

Feb 1, 2024 · 11:08

Jeff Schomay built a game with 100% AI-generated content, using OpenAI's fine-tuning to generate JSON scene definitions and Leonardo to create stylistically consistent images. He fine-tuned a model on 50 examples for about $1–$2, enabling shorter prompts and reliable JSON output. To maintain visual consistency across scenes with varied elements (people, animals, buildings), he trained multiple image models. He implemented a caching pipeline that prefetches scenes to compensate for 10–30 second generation times, ensuring a smooth player experience. The final game procedurally generates every scene and image, so each playthrough is unique.

120k players in a week: Lessons from the first viral CLIP app: Joseph Nelson

Nov 22, 2023 · 15:59

Joseph Nelson (Roboflow CEO) details building paint.wtf, a viral AI Pictionary game using OpenAI CLIP that attracted 120,000 players in its first week, peaking at 7 submissions per second. The architecture uses GPT-3 to generate prompts, a browser canvas for drawing, and CLIP to judge similarity between text and image embeddings. Key lessons include CLIP's ability to read text, requiring a moderation hack that penalizes images more similar to handwriting than the prompt; CLIP's conservative similarity scores (ranging 8% to 48%); and the necessity of using CLIP itself to block NSFW content. Nelson also live-codes a minimal MVP in under 50 lines of Python using Roboflow's open-source inference server, demonstrating CLIP embedding and cosine similarity. The talk underscores foundation models' new paradigm of open-set understanding.

See, Hear, Speak, Draw: Logan Kilpatrick & Simón Fishman

Oct 24, 2023 · 18:43

OpenAI's Simón Fishman and Logan Kilpatrick argue that 2024 will be the year of multimodal models, showcasing how text can currently bridge vision, audio, and image generation. They demo a DALL-E vision loop where GPT-4 with vision describes an image, DALL-E 3 generates a synthetic version, then the models compare and iterate to improve accuracy—all in roughly 50 lines of code. Another demo extracts frames from a video, uses GPT-4 with vision to describe them, combines with Whisper's transcript, and generates a blog post with matching DALL-E images, making video content more accessible. They emphasize that while today's multimodal capabilities are separate, future unified models will directly reason across modalities, eliminating the need for manual integration.