Video Has No Memory. Here's How We Built One. — James Le, TwelveLabs
Jul 23, 2026 · 20:27
James Le of TwelveLabs argues that video AI systems lack memory because they treat video as a bag of frames rather than a spatial temporal volume, losing continuity and context. He presents three core problems—wrong context, wrong memory, and weak reasoning—and five properties that make video memory hard: temporality, multimodality, density, ambiguity, and expense. The TwelveLabs stack solves this with Marengo (multimodal embedding encoder), a spatial temporal context store, and Pegasus (video language model), exposed as an API. Le introduces a context graph that connects time-bounded moments, entities, appearances, relationships, and corpus-level themes, enabling traversable video memory. He demonstrates five design principles: ingest once and reason many times, store primitives not answers, ground every claim to a timestamp, let intent shape memory, and keep the layer composable. In demos, he shows a video agent (Joki) that tracks Lionel Messi across 67 World Cup videos, identifies near misses and dramatic goals, classifies vehicles in traffic footage, and suggests ad placement moments in an Adidas clip, all grounded to specific timestamps.
