# Video Has No Memory. Here's How We Built One. — James Le, TwelveLabs AI Engineer · 2026-07-23 James Le of TwelveLabs argues that video AI systems lack memory because they treat video as a bag of frames rather than a spatial temporal volume, losing continuity and context. He presents three core problems—wrong context, wrong memory, and weak reasoning—and five properties that make video memory hard: temporality, multimodality, density, ambiguity, and expense. The TwelveLabs stack solves this with Marengo (multimodal embedding encoder), a spatial temporal context store, and Pegasus (video language model), exposed as an API. Le introduces a context graph that connects time-bounded moments, entities, appearances, relationships, and corpus-level themes, enabling traversable video memory. He demonstrates five design principles: ingest once and reason many times, store primitives not answers, ground every claim to a timestamp, let intent shape memory, and keep the layer composable. In demos, he shows a video agent (Joki) that tracks Lionel Messi across 67 World Cup videos, identifies near misses and dramatic goals, classifies vehicles in traffic footage, and suggests ad placement moments in an Adidas clip, all grounded to specific timestamps. ## Questions this episode answers ### What are James Le's five design principles for building a memory layer for video intelligence? James Le outlines five principles: first, ingest once and reason many times, paying upfront interpretation cost like a database to avoid repeated processing. Second, store primitives (moments, entities, appearances) not just answers to enable downstream workflows. Third, ground every claim to a specific timestamp for evidence. Fourth, let intent shape memory—configure the layer for different workflows like sports or compliance. Fifth, keep the layer composable and API-first to serve structured metadata to applications. [9:04](https://aiengineer.podhood.com/0e86acd2-9625-4f32-bd49-a3f3f7bd8e1f?t=544000) ### How does TwelveLabs represent video collections as a context graph, and what are its layers? James Le describes a context graph as a durable, queryable representation connecting video moments, entities, appearances, relationships, and corpus-level context. At the bottom are time-bound moments (scenes, shots). Above are appearances (where and when each entity appears), then entities (people, brands, places). Next, relationships (co-occurrences, sequences, causality) and timeline. At the top, corpus-level context covers themes and patterns. Different questions traverse different parts: a simple search hits moments, entity workflows expand from appearances, and narrative storytelling follows relationships across time. [7:44](https://aiengineer.podhood.com/0e86acd2-9625-4f32-bd49-a3f3f7bd8e1f?t=464000) ### What is the difference between video search and video memory, according to James Le? James Le explains that search recovers relevant moments from video libraries, giving candidates without continuity. Memory, in contrast, enables answering questions requiring preserved entities, timelines, and evidence across an entire corpus. While search returns a time-bound clip, memory returns structured knowledge, timelines, explanations, and composable output. This shifts the unit from clip retrieval to corpus memory, supporting multi-hop timelines, episodic recall, and follow-up questions without reprocessing the whole archive. [5:56](https://aiengineer.podhood.com/0e86acd2-9625-4f32-bd49-a3f3f7bd8e1f?t=356000) ### How did James Le's video memory system track Lionel Messi across World Cup footage? James Le ingested 67 videos from the 2022 World Cup and asked the system to track Lionel Messi across the entire corpus, including shots where he is a marginal figure, describing camera framing. It highlighted specific moments: Messi dribbling past a sliding defender, scoring against Australia in the round of 16, and scoring the third goal in the final. This demonstrates how the memory layer can preserve entities and retrieve timeline-based evidence across multiple videos. [14:49](https://aiengineer.podhood.com/0e86acd2-9625-4f32-bd49-a3f3f7bd8e1f?t=889000) ## Key moments - **[0:00] Memory Gap** - [0:24] "Video has no memory," James Le argues, noting most video AI systems answer each query from scratch. - **[0:52] Spatial-Temporal Volume** - [0:52] "Video is not a bag of frames," James Le insists; treating it as such throws away the continuity that makes video unique. - **[2:07] Wrong Stack** - [2:25] Forcing video into text tokens causes wrong context, wrong memory, and wrong reasoning, says James Le, losing spatial-temporal relations and continuity. - **[3:34] Five Challenges** - [3:48] Five properties make video memory hard: temporal dependence, multimodality, density, ambiguity, and expense of grounding claims to source moments. - [4:53] TwelveLabs' stack uses Marengo multimodal encoder, a spatial-temporal context store, and Pegasus VLM for video reasoning, exposed as an API. - **[4:57] Architecture** - **[5:41] Search vs Memory** - [5:56] Search returns time-bound moments, but memory yields structured knowledge, timelines, and composable output from the entire video corpus. - **[7:44] Context Graph** - [7:44] A context graph is a durable, queryable representation that connects video moments, entities, appearances, relationships, timestamps, and corpus-level themes. - **[8:56] Design Principles** - [9:04] Five design principles for video memory: ingest once, store primitives, ground claims, let intent shape memory, and keep the layer composable. - **[10:38] Video Worker** - [10:45] A video worker is a deterministic system that plans tasks, retrieves evidence, synthesizes outputs, and is evaluable, unlike stateless models producing single answers. - [11:36] Hardness engineering for video workers requires memory, task planning, retrieval, expert tools, operating envelope, output contract, and evaluation. - **[12:56] Demos** - [13:09] Ingesting 67 World Cup 2022 videos, the system returns near misses—shots that almost scored—each with a reason, like goalkeeper saves and offsides. - [14:49] Tracking Messi across World Cup videos, the system finds scenes of him dribbling past a sliding defender and scoring, describing camera framing. - [15:43] Analyzing traffic footage, the system classifies vehicles and pedestrians, counts them, and detects safety events like a red SUV almost struck. - [17:04] Analyzing an Adidas commercial, the system classifies ad-insertion points: reveals, hard cuts, high-impact actions, and logo appearances for brand safety. - **[18:16] Use Cases** - [18:16] With a video memory layer, developers can build discovery, reasoning, organization, and action workflow applications across media, security, and advertising. - **[19:13] Product & Beta** - [19:13] TwelveLabs' Joki product features knowledge store as the video memory layer, configurable ingestion, corpus digest, and resolution engine with search/response APIs. ## Speakers - **James Le** (guest) ## Topics Multimodal Models ## Mentioned TwelveLabs (company), Joki (product), Marengo (product), Pegasus (product) ## Transcript ### Memory Gap **James Le** [0:13] Thanks so much for having me and inviting me to be a speaker at the World Fair. I attended last year and was so impressed by the quality it presented, so glad to have a chance to be here and present. So the title of my talk is, you know, "Video Has No Memory,"right? And, you know, this might sound strange because video is already like a preservation of the past,right? If you think about, like, you have footage you preserve: recording, training data, incident, creative work, history, etc. But actually, most of the video AI systems these days do not have memory in the system sense. So actually, for this talk, I will try to answer the question: what could it take to build a memory layer for video intelligence? To start, I want to be clear about what makes video different from other data types,right? So this is the first mental model that I want to highlight, which is that video is not a bag of frames. So in many of my conversations with, like, developers, you know, who are using our product, a lot of them still treat video as like a stack of images, maybe a transcript being attached, or, you know, like, you know, but essentially like a frame level,right? ### Spatial-Temporal Volume **James Le** [1:17] And that is useful approximation for some tasks, but it throws away the thing that makes video very unique, which is continuity,right? So meaning in video derives from space, time, modalities, and sequence. So a better mental model for video is a spatial temporal volume. So what I mean is that inside that volume, you have visual information, speech, sound, motion, OCR, camera changes, scene transition, metadata, and time,right? So the hard part here is really about how can you preserve relationships across this volume so that later an application can traverse it. And then, you know, especially at the enterprise scale, you know, across, like, industry, like entertainment, sport, you know, short-form content, then you're sitting on petabytes of footage,right? So finding moment is already hard. So how can you preserve meaning across millions of moments in the deeper platform? ### Wrong Stack **James Le** [2:07] So I work at TwelveLabs, which is a seriously startup. We build foundation models that understand, you know, video the way that humans do. And the way we talk about our positioning is, like, the existing stack of dealing with video is not equivalent to that,right? Obviously, language models are very powerful. They are good reasoning interfaces. They are increasingly multimodal as well. But the supporting stack,right, around that is very, I would say, limited. And that creates three problems. Number one is wrong context,right? So video is not naturally a sequence of text tokens. If we force it into that sequence by sampling frames, by extracting a transcript, by dumping everything into a prompt, you lose the spatial temporal relationships,right, that actually define the event. Second, wrong memory. So if you think about text system, memory here is often mean reachable augmented generation, vector search, or probably, like, larger context window. Those are very useful, but video memory has a different requirement. It needs to link to the scene for something that happened in another file, another episode, another camera angle, another season, another year. So it actually needs durable continuity,right? And the last part here is wrong reasoning. Like I said, you know, text for system cannot reason over, you know, natively over motion, causality, all of that. So, you know, they do not automatically build, like, a persistent structure on, you know, who appeared, what happened, what changes, etc. ### Five Challenges **James Le** [3:34] And so my argument is that video intelligence needs a memory layer that decides what to traverse, how to connect it, and how to reach it later. So I want to kind of ground it into the properties of video,right, to make it even clearer. There are five challenges dealing with video. Number one is temporal,right? So meaning depends on before and after. So a frame by itself can be misleading,right? The same expression, product shot, physical action can mean different things depending on the sequence around it,right? Second is that video is obviously multimodal. I explained already, you know, a transcript alone may miss, you know, the logo. A frame alone may miss the spoken claim. Video is also very dense,right? So a few minutes can contain dozens of shots, people, objects, action, location, claims. The useful signal is uneven across the distribution on the frame. Some seconds are decisive, others are noisy. Fourth is that video is also ambiguous,right? People reappear under different lighting and angles. Brands are partially visible. Locations are implied. Concepts emerge over time rather than being named in a single moment. And lastly, it is expensive because in a lot of big enterprises and complex workflows, you need to, you know, point back to the source moment, like where it comes from,right? So these are the five properties explaining why video memory is very complex. You need to preserve temporal span, multimodal evidence, continuity, all that. This is a very simple stack of how we build things at TwelveLabs. At the bottom, we have these semantic chunks that capture, you know, meaningful temporal units. Above that is our multimodal embedding encoder called Marengo, which essentially turns those spans into spatial temporal relations. ### Architecture **James Le** [5:20] Basically, vector embeddings that represent video content. And then we have a spatial temporal context store, which is where it preserves reusable structures like moment, entities, metadata, all of that. We also build our own VLM, Video Context Aware Language Model, called Pegasus, that essentially serves as the reasoning layer,right, that can preserve prepare over video content. ### Search vs Memory **James Le** [5:41] So think about summaries, like metadata, synthesis comparison. And we expose our models as API because, you know, we want to get developers to use them as infrastructure. Now, moving beyond, like, kind of the stackright here, I want to talk about the difference between search and memory,right? Very quickly speaking, search is obviously super important. It's how you recover relevant moments from large video libraries. But then it gives you a candidate. It actually does not give you, like, any continuity. So memory, on the other hand, is all the, you know, the things that enable the system to answer a different class of questions, as you see here on theright side of my screen. So these are not the single reachable call,right? They require the system to preserve entities, timeline, evidence across an entire corpus. And so, like, you can actually build products moving beyond from, like, show me something like this to, you know, tell me what this collection knows,right? And so that might sound, you know, simple and subtle, but the output is completely different. Like, with search, you get, like, an output, like a time-bound moment. But with memory, you actually return, like, structured knowledge, timeline, explanation, composable output. And that, like, you know, is very important because we can now move the unit output from clip retrieval to corpus memory,right? There are two scaling dimensions shown here in the slide. The first is time scaling. So a real video system should be able to reason over years of footage without reprocessing the whole archive every time,right? That means memory-first retrieval, be reusable representation once, and then support multi-op timeline, episodic recall, follow-up questions at lower latency and cost. And then the second dimension is in space,right? So many real workflows actually involve multiple perspectives, like different camera angles, you know, live stream, creator, broadcasting content, body cam, stock cameras, event feed,right? So how can you build a system that can fuse evidence across all the sources and then maintain coherent understanding,right? And so that is the challenges here. How can you build a representation that lets applications traverse video across time and across sources? See, this is, you know, a track on graph,right? So the best mental model that I can come up with is to represent, you know, video collection as a context graph. So a context graph is a durable, queryable representation that connects video moment, entities, appearances, relationship, timestamp, metadata, and corpus-level context,right? ### Context Graph **James Le** [8:02] So if you take a look here on the screen, all the way in the bottom, you've got time-bound moment. These are, like, the scene, the shot,right? These are evidence units. One level up are the appearances, where and when each entity shows up. And then you've got the actual entity itself. So think about the people on the video, the brand, the places, the concept. Next, you have relationships, co-occurrences to the same brand, sequences between different places, causality, and timeline. And finally, at the top, you have corpus-level context. What are the main themes, the patterns, the gap, the coverage that this video collection covers,right? This matters because different questions traverse different parts of the graph. If you ask a simple search question, then that might go directly into the moment. But, like, an entity workflow might start with a person and then expand into appearances,right? And if you ask a question like a storyline, like narrative storytelling of a certain, you know, person, then it may follow relationships across time,right? ### Design Principles **James Le** [8:56] So the key idea here is that memory, in the context of video understanding, is a navigable structure over the entire video volume. From that concept, I come up with these five principles for building a memory layer for video intelligence. Number one is to ingest once and reason many times. So you don't want to, like, do every single query from scratch. Like, you want to pay the cost upfront, do one interpretation from the video content, upfront pay the cost, and then you move expensive understanding into ingestion. So this is the same mental model of database,right? You do not repeatedly have to parse your entire source of data for, you know, every application request. Second principle is to store primitives, not just answers. So, you know, moments, entities, appearances, I already talked about that. Those are the primitives,right? That allows you to do downstream workflows, like search, editing, you know, analytics, all of that. Third is to ground every claim. Like, basically, if you ask a question, you need to decide back into where that scene is happening in the video. So evidence, like, you know, should be grounded to a specific timestamp within the video,right? Fourth is to let intent shape memory. This is important because the same footage means different things in different workflows. We work across sports, applications, brand safety, compliance review, creator analytics. All of them require different primitives from the same video. So the memory layer should be configurable,right? Developers should be able to tell the system what matters. And lastly, keep the layer composable. So basically, being API-first, you know, it should provide the layers that allow those applications, on top of that, to serve it structured, grounded metadata that can be plugged into any sort of application. So moving beyond these five principles, I want to talk about, like, kind of the hardness around building a memory layer,right? There's a lot of talk these days about, you know, building theright hardness for the context of language models. ### Video Worker **James Le** [10:50] So what does it look like for video and a static model,right? A model code produces a single answer. It is stateless. It starts fresh each time. It doesn't have any constraint. So the output is largely based on what the model decides to produce. A video worker, on the other hand, operates inside a very deterministic system, understands what is available. It can plan the task, retrieve evidence, inspect the relevant moments, synthesize, validate, return an output, and then the entire workflow can be evaluated,right? So for video understanding, this is very important because the worker needs to know what memory is available, what evidence matters, and also, like, how deep to inspect because that will determine how much cost to spend, what output contract to satisfy,right? Talking about hardness engineering for video understanding, I come up with these, like, different capabilities for, like, a video worker,right? Number one is memory. I talked about that already. Number two is task planning. So given a query from an end user, you have to decide, like, what task to execute. Is it, like, search, or is it, like, summarization, or, like, you know, multi-step reasoning? Third is retrieval. Like, every single system should be able to, like, select theright evidence from your video corpus to read for a specific task. Expert tools,right? So we work with customers where they require, like, you know, zoom in, zoom out, comparing different, you know, frames, enriching content with, like, additional metadata. So building expert tools inside, like, a video worker is very important. Operating envelope. So these are, like, explicit limits on time, cost, depth, scope, autonomy. An output contract. So sometimes natural language is enough. Sometimes the application needs structured data with references and timestamps. And of course, finally, you have evaluation,right? Like, you know, did the retrieval find theright evidence? Did the synthesis preserve it,right? Did the worker stay within the budget? Allright. So that's a lot of, like, you know, slide and talk. I want to quickly jump into some demos that I actually built using TwelveLabs, you know, video agent product. ### Demos **James Le** [12:56] So there will be three demos. The video agent product that we've been building is called Joki. So this first example here is for sport understanding. You know, obviously, everyone is super excited about the World Cup that's happeningright now. So what I did is I ingest 67 videos from the 2022 World Cup in Qatar. And you can see here, I asked it to find the near misses, the shots that almost became goals but did not, for each, explain why it was not a goal, but do not include the actual goals,right? So these are the top output that it returned. So that's hitting the good work. This is a save from the goalkeeper. I don't know if the cell is up, but, like, I'm playing the video, by the way. Right, this is another save from the goalkeeper. It even catched, like, you know, the offside from one of the goals. And then I asked the question, okay, when is the goal? Find the most dramatic actual goals. Show the view of the play and the finish. For each goal, describe the sequence,right? So if you know this one, this is the first goal of the World Cup final, like, four years ago. And it actually, like, returned, like, you know, it understands who are the passers, like, it was Alvarez passing to Mac Allister and passing to Di Maria to score the goal. Take a look at this one from Richarlison. This is called a tournament from Brazil against South Korea, I believe,right? And it returned, like, you know, an average skill in a build-up,right? Name the player who did the return pass. You can even do player tracking. So I asked it to track Lionel Messi across this entire corpus, including the shot where he's one of the managers' figures on the screen. Describe the camera framing,right? So this highlights all the important moments in the game. And this is the scene where Messi dribbles past a sliding defender. You can see here. It picked up the scene where he scored the first goal against Australia in the round of 16, I believe. Right. This is another scene where he scored the third goal in the final. Yeah, so that's one example on sport understanding. But then you can obviously build more interesting and more, like, real practical applications, of which security is one that we encounter a lot. So on this example, I ingest it, you know, publicly available camera footage. For context, these are the clips. You have traffic jams, suburban, you know, urban area. And given this footage,right, I asked Joki to carve and classify every vehicle in the intersection, break it down by type plus pedestrian, and it returned the number of vehicles and the peak foot traffic as well. You can detect safety events,right? So you see there, a red SUV turned and almost got struck. Another scene here. Turned left into an upcoming car. Yeah, so that's a clearest red line entry. It works well in, you know, different scenarios. This scene is a very crowded area in Bangkok. It asked it. I also asked it to work on the, you know, the rain,right? So this is another scene where it understands the rainy condition. It identified the busiest intersection vehicle window. So yeah, those are some examples for camera security surveillance footage. Finally, advertising. So you've probably seen this Adidas clip in all the commercials leading up to the World Cup recently. It's a five-minute Adidas footage. And I asked it to classify all the points where you can put an add-on. So it finds the reveal, the hard cut, the impact, energy pick,right? It finds the scene where a certain player appears on the screen. It identified, like, you know, high-impact action like this. Condition, the hard cut to knife football under lights. And of course, it points to the logo of Adidas. So, you know, from the perspective of an advertiser, these are very important moments because they can, you know, find the scene with slow-motion hero. Hard cut on a beat. Or pick action. In which they can advertise their brand content against this footage,right? ### Use Cases **James Le** [18:16] Yeah, so those are three sample demo applications that I want to highlight using TwelveLabs. And again, now, what can you build with this sort of video memory layer? Based on the example, these are the categories of applications that I believe developers can build. You can discover things. You can build reasoning experience. You can organize your content across different video libraries. And it can be action workflow. Assemble a different scene together, do compliance review, data operation, etc. The same framework applies for different vehicles, in media entertainment and sport, segmentation, highlight generation, in commercial security, evidence review, contextual analysis, in advertising, brand safety, creative intelligence,right? And yeah, so this is our product that is coming up. One quick highlight is that we try to code as a video cognition infrastructure. So we have a knowledge store that basically becomes the video memory layer. We have configurable ingestion that lets viewers shape what the system can extract. Corpus digest so that you can understand what is in the library and the resolution engine search responses API. ### Product & Beta **James Le** [19:27] So the thing I want to highlight here is it's not an application layer. It's not an editing platform, not a compliance product. It's the cognition infrastructure with the layer and the harnesses that enable, like, those products to become available. And if you found the content of this talk interesting, I definitely recommend you to scan this QR code. The product is currently in private betaright now. If you're building any sort of workflow that touches video content, especially around content assembly, content organization, you know, think about media archive, content creator, YouTube, TikTok, sport analysis, media workflow, definitely either scan this QR code and register for the interest or come talk to me after the talk. So that should be my time. Thanks a lot. --- This library is powered by PodHood (https://podhood.com), the podcast website platform.