# Memory Harnesses for Long-Running Research Agents — Stefania Druga, Sakana.ai AI Engineer · 2026-08-12 Stefania Druga, a research scientist at Sakana AI in Tokyo, presents experiments on memory harnesses for long-running research agents running on local models like Qwen 27B and DeepSeek V4 Flash on an M3 Ultra. She frames memory as a write-manage-read control loop, not a database, and tests a recall ladder—no recall, vector RAG, a ranked decisions ledger, and an oracle—across 68 xbench questions. The ranked ledger performed best, beating even gating on whether memory is needed, while the oracle didn't hit max because giving the right memory doesn't force its use. When tasks fit in context, memory only added cost with no accuracy gain, but for long-horizon tasks where answers sit far outside the window, good recall policy became essential and cheaper. She urges treating recall policy as a first-class metric and highlights the broader memory technique landscape, including over 30 runnable cookbooks from Diamond, while noting local models run serially, which is why her Tokyo machine… ## Questions this episode answers ### What is a memory harness for long-running research agents and how does it work? Stefania Druga from Sakana AI describes memory as a write-manage-read control loop around the model, not just a database. Her harness includes a core of traces always shown to the agent, a recall block with different modes, and an archival block for cross-session information. She tested recall policies: no recall, vector RAG, a decisions ledger, and an oracle with ground-truth memory. [3:45](https://aiengineer.podhood.com/a5695ac2-9183-494a-a6af-44f8e7728194?t=225000) ### Why did adding a memory harness not improve performance on a literature review task? In a literature review task where all papers fit in the context window, adding a memory harness changed nothing—same accuracy, higher cost. Stefania Druga found that when the task fits in context, the harness doesn't add capability. The negative result marks the boundary: memory only pays off when the relevant context exceeds the window, as in long-horizon tasks. [5:43](https://aiengineer.podhood.com/a5695ac2-9183-494a-a6af-44f8e7728194?t=343000) ### Which memory recall policy performed best on the xbench long-horizon tasks? On 68 xbench questions, the ranked decisions ledger performed best, beating even gating the harness on whether memory seemed necessary. The oracle, which provides correct memory but doesn't force use, didn't hit max performance because the model can ignore or misuse it. Ranked recall also cost less, showing bad memory is expensive in tokens and direction. [7:29](https://aiengineer.podhood.com/a5695ac2-9183-494a-a6af-44f8e7728194?t=449000) ### Why did the oracle memory policy not achieve the highest performance in Stefania Druga's experiments? The oracle provides the right memory to the model but doesn't force it to use it. Stefania Druga explains that the model can retrieve the correct memory yet still choose wrong information, ignore it, or get confused. This is why the oracle didn't hit max performance on xbench tasks, while the ranked recall policy performed best. [8:39](https://aiengineer.podhood.com/a5695ac2-9183-494a-a6af-44f8e7728194?t=519000) ## Key moments - **[0:00] Context rot** - [0:35] Long-horizon agents hit context rot: models contradict themselves, redo work, and drift from the original questions, says Stefania Druga. - **[1:04] Longer tasks** - **[1:56] Local models** - **[2:50] Tokyo rig** - [2:51] Stefania Druga runs nonstop evals on an M3 Ultra in Tokyo, controlling it from her phone with fans stacked around it. - **[3:44] Memory loop** - [4:35] Stefania Druga's recall ladder holds the model fixed and varies memory modes: none, vector RAG, decisions ledger, and oracle. - **[4:36] Recall ladder** - **[5:27] Literature test** - **[6:19] Fits in context** - [6:27] When all papers fit the context window, memory harness added no capability and only cost, says Stefania Druga. - **[6:47] Long-horizon** - [7:49] Stefania Druga's ranked decisions-ledger recall beat no-memory and oracle conditions across 68 xbench long-horizon questions. - **[8:00] Ranked recall** - **[9:33] Cheaper memory** - [9:33] "Bad memory is expensive because it spends more token and it can send the agent the wrong way," said Stefania Druga. - **[11:38] Sovereign AI** - [11:38] Stefania Druga's local model sovereignty has a cost: DeepSeek V4 Flash lacks batch querying, so evals run serially in Tokyo. ## Speakers - **Stefania Druga** (guest) ## Topics Agent Memory, Autonomous Agents, RAG Evaluation ## Mentioned Sakana AI (company), DeepSeek V4 Flash (product), Diamond (product), M3 Ultra (product), Qwen 27B (product), SPIDER (product), xbench (product) ## Transcript ### Context rot **Stefania Druga** [0:13] Hello, welcome. Uh, this is a big room, so if you're in the back, don't hesitate to come closer. My name is Stefania Druga, I'm a research scientist at Sakana AI in Tokyo. I used to be based here, and AI engineering is a home community for me before being the hyper loop, so it's very good to be back. And today I'm going to talk to you about memory harnesses for long-running research agents on-device. So if you work with long-horizon tasks, you probably run into this issue of context blow,right? Like when the model starts contradicting itself, or it has to redo the work because it forgot it did that task in the first place, or it starts to drift from your questions because it forgot them. And this, this matters now more than ever because from this recent projections from Meter, we see that the trend is to solve longer and longer horizon tasks, and also that we're getting fewer and fewer model releases. So at some point later this year, we're going to have this convergence,right, where we'll get many more long-term horizon tasks and fewer model releases. ### Longer tasks **Stefania Druga** [1:29] So that makes this issue of dealing with context wrought a priority. And why did I wanted to, to tackle this problem on local models and with a local harness? Uh, maybe some of you have seen this tweet, it's only 2 days old. The CEO of Coinbase actually shared how their company managed to reduce their AI spend while actually increasing the AI usage. And the way they did that was by transitioning to use many more local models, but also having better practices, like using better routing, better caching, keeping the context clean, and then having better visibility for what people are using it for what, uh, what kind of task. ### Local models **Stefania Druga** [2:17] So we are seeing the local models like crossing the line,right? Like GLM is on everyone's minds, like especially with Fable going away. Uh, DeepSeek V4 Flash can now be run on M3 Ultra, and there's still a bottleneck for RAM, it's tricky. But these local models are starting to be useful for agentic tasks and for tool use. So I wanted to show you what has been my setup for the experiments I'm going to share with you today. Uh, this, this is my Mac. ### Tokyo rig **Stefania Druga** [2:51] It's still running evaluationsright now, uh, back in my desk in Tokyo, and I'm controlling it from my phone. Um, and after running evals nonstop for a couple of days, it started to get hot. So I had my husband put fans around it. Um, we're running out of fans, but the, the machine is still running and the evals are still giving results. Um, on this M3 Ultra with 96 gigabytes and 28 core CPUs, I'm using two models. I'm using the Qwen 27B Quantize at 4-bit and the DeepSeek V4 Flash. And before I show you how I built the memory harness on this machine, I wanted to tell you what this, what is this an example of,right? Like memory, when we design a harness for memory, this is the mental model I want you to have in mind. ### Memory loop **Stefania Druga** [3:45] Um, you can think of memory as a write-manage-read loop. So it's not just the database store, it's actually this control loop around the model. More concretely, how did I take that loop and customize it? So this is my harness design. Like I started with research agents that are the small agents because they have zero durable memory, and I wanted all the memory to come from the harness. And then, um, in the middle I have a core, which is always shown to, to the agent, um, of traces. And then I have a recall block where I'm testing different modes, and an archival block where I'm kee-keeping track of information across different, um, sessions. And in that recall block, I'm actually going through a ladder of modes that I'm testing. The baseline is like not to use memory at all, no recall at all, so I'm, I'm testing for that. Uh, next is to use RAG, vector, vector RAG, um, just to see whatever, like the harness would pull in terms of similarity. ### Recall ladder **Stefania Druga** [4:51] Then is to use a decisions, uh, ledger where I actually keep track of what decisions are being made for every turn, and then I can prioritize them. And last but not least, and this piece is very important, I have what I call an oracle, but basically this is the ground truth. So this is like telling the harness for every loop what the correct memory that needs to be retrieved is. And the model is fixed across all the different tasks. So the only things that I'm changing is like these different variables in the recall block. ### Literature test **Stefania Druga** [5:28] And I wanted to, to give you an example of a first task that I tested. So I wanted to see if I give the agent a task of doing literature review, and I'm including a lot of papers in the corpus where there was a big scientific claim. Like this is actually a nature paper where they said they discovered 742,000 promising materials. Like it was a very big claim, which got retracted later. But the retraction, it's a, it's a much smaller like haystack needle in that corpus than the headlines and the citations. So I wanted to see if, if the system can retrieve theright answer for these type of questions. And what I found was because like for these tasks, all the papers and all the information fit into the context, the memory actually didn't add more capability. ### Fits in context **Stefania Druga** [6:27] It was the same performance with memory and without memory, and it only added more cost. So when your task fits in context, the harness doesn't add much. However, if I start to run tasks that are longer term horizon and the entire task and the relevant context doesn't, uh, fit, then having a good memory harness really starts to pay off. ### Long-horizon **Stefania Druga** [6:55] So this is another example of a task that I ran. This is actually from an established benchmark for long-horizon, uh, tasks memory. It's called xbench. And this is an example of a question,right? So I'm asking a question and then like theright answer is in a like step 124, but the moment when I ask the question, I'm asking it like at step 500. So it's completely outside of the context window, and the model needs to use the memory harness to retrieve the specific answer from theright step. So I'm testing this by, uh, changing the different policy ladder that I explained before with memory off, uh, by deploying recall, different types of recall, and by using the oracle as a reference. And what I found was that with the ranked recall, the model gets theright answer, um, more frequently than without. And here's a breakdown of the decomposition of performance on this xbench tasks. So I ran over, uh, 68 questions, and for each of these questions, there were like multiple, um, cells and lots of different seeds. ### Ranked recall **Stefania Druga** [8:17] And what I found was that the rank only ledger performed the best. And it performed better than like just gating the harness by saying, do you need to use memory or do you not need to use memory. And you're probably going to ask like why is the oracle not hitting like the max? And I'm going to explain that too. So the oracle, what it does, it provides theright information, theright memory to the model, but it doesn't force it to use it. So the model can get theright memory but still retrieve the wrong information or choose to ignore it or be confused. So that's why the oracle in this case doesn't hit the max performance. And I've done lots of ablations on these tasks to see like what happens if I give arbitrary, um, examples. What happens if I give it the wrong step? What happens if I give it the most recent step? And I still found that the best performing condition was the one with the ranked policy for recall. And this actually works on several models, not only on the Qwen 27B, but also on the DS4 Flash, and it also works across different benchmarks. ### Cheaper memory **Stefania Druga** [9:33] I also tried it on the SPIDER V2 benchmark. And it's not just that it gives you better recall, it actually costs less. So maybe a good heuristic to have here is that bad memory is expensive because it spends more token and it can send the agent the wrong way. But having like a good structural policy for recall can save you a lot of tokens and, uh, budget. So one thing that I want to encourage you from this experiment is to consider the recall policy as a first class metric and to start to think about how you might use it in your systems. Like what are the type of memories that you want to store? What, how do you rank them? Like how do you design your recall function? And then, um, what are the type, what survives when you run this over and over and over and, um, multiple sessions, multiple runs? And this is just a simple first kind of experiment, um, but the memory technique landscape is very rich. Um, so there's over 30 runnable cookbooks that are shared in this open source repository from, um, Diamond. And memory is complex. We have short-term, long-term, different cognitive techniques. Uh, we can use, start to use evaluation results as well. Um, andright now there's actually a, a pretty broad landscape of solutions,right? So going from simple file system retrieval to training memory models, um, there's, there's a wide spectrum of solutions from less structural to completely structured. Um, so I think there's a lot of research we're going to see in this space. Uh, it's important. Uh, it becomes more and more relevant. And for me, it's been super fun to, to test this on local models, um, because I got to control everything. ### Sovereign AI **Stefania Druga** [11:38] I got to control the data I was using, the entire traces of compute and evaluations. And, um, yeah, I, I see that as an example of sovereignty. And it comes at a cost. Uh, I didn't tell you that these local models, I can only what, uh, run them in serial. Like they don't support batch querying for the DeepSeek V4 Flash. So that's why I am still running evaluations back on my computer in Tokyo, or I wa I was doing it on the flight on my way here because it takes a long time. Um, but I still think it's very powerful and it's a very good test for what memory can do when you can control every single step of the pipeline. And this sovereign capability is part of a bigger ecosystem that is very important for us at Sakana AI in Japan. Um, we believe in the importance of sovereign AI today more than ever, and we are also hiring. So if you're interested and want to hear more about this, and if you want to come join us in Japan, come talk to me. Uh, thank you very much. --- This library is powered by PodHood (https://podhood.com), the podcast website platform.