Claude for Long-Horizon Tasks — Lance Martin, Anthropic

Good to go? Allright, well, take a quick sip and then let's start. It is great to be here. This is, like, my third year coming to this conference, and I always really enjoy it. And thank you for coming to this workshop; I know there's many interesting talks.

Let me talk a little bit about our view of asynchronous agents at Anthropic, and some things we've been up to lately. So this is kind of a way I think about models and product. So you can think about Claude as a light source, and you can think about products as windows that allow the light to pass through.

And what's kind of interesting is, over time, the window that you need to actually kind of see the light of the model kind of shifts. And we've seen this over the past few years. So I'm plotting here different Claude models and their task horizon.

So how much autonomous work can they do over time? And you might recall back in, like, the Opus 3 days, this was kind of like 2024, models could only do, you know, maybe 10 to 20 minutes of autonomous work.

This is measured by meter. And in that regime, only certain product surfaces made sense. Things like autocomplete, things like chat, where your human is very in the loop. Because the model's really only doing a very short amount of work before you're steering it.

Now, over the past year, we saw the rise of synchronous coding agents like Claude Code, and this is, you know, kind of a shift because then models could do maybe an hour of work. So it made sense to have them run, but typically locally, where you could still steer them easily.

And it's kind of interesting because during this regime, I remember efforts, and I was involved in some efforts to build kind of asynchronous agents. But when models can only do, like, an hour of work, async as an experience is kind of bad.

The model goes off and it, like, hits an error and it comes back to you over a short period of time. In order to really unlock async, we needed longer task horizons. And so we're starting to see that now.

And kind of with this shift in capability and time horizon came a shift in the API surfaces. So if you look at the lower left, message API came out, like, two years ago. It's basically prompt response. It's great for building harnesses, but it's a very simple API.

Again, you're just passing in set of messages, you get a response out. There's no sense of deployment with that, so you basically take messages API and you can roll your own harness, you can deploy that harness and you have an agent.

Now, over the past year, we saw the rise of, you know, coding agents in particular. So we released Asian SDK. So that's basically a way to programmatically call Claude Code. And that's, like, basically us giving you a harness.

But over the past few months, since April, as we've seen longer and longer task horizons, we released a new API called Managed Agents, which basically packages both the harness as well as all the managed deployment infrastructure for you.

And I want to talk about some of the themes that underpin this new surface, Claude Managed Agents, and some of the themes that kind of extend beyond just Managed Agents broadly to think about this kind of new type of asynchronous agents, which can apply, of course, to Claude and other types of kind of longer running, long horizon agents.

So theme one is decoupling the brain from the hands.

So when we first set out to build Managed Agents, we started with a container. We put the harness in the sandbox in the same container. Now, the problem here is, what happens if the harness dies or the container dies?

What we saw is we actually lose the session. So basically, this architecture is kind of tricky for long horizon agents because what can happen is your agent's running, and if that container dies, you lose everything with it. Also, as models get more capable, putting the credentials in the same container with the agent itself can be problematic.

So, for example, giving Claude access to a bunch of your secrets and letting it run for 10 hours and you're not watching it can be a little bit spooky and has some security concerns, especially as models get extremely capable.

So for this reason, we kind of decouple, we call the brain, that's the harness, from the hands, the execution environments, and Managed Agents is set up like this. So the story here is that the harness becomes a stateless process that talks to a session.

The session is an append-only event log, and that can reach out to hands, which are just containers. So those are sandboxes, work is done. And one thing that's interesting is Claude is increasingly capable of managing many hands. So that is, you can give one harness access to many different containers to perform tool execution, and Claude can manage this very easily and effectively.

If the session, sorry, if the harness dies or sandbox dies, it's completely fine because the session is always backed up in this append-only log, and credentials are never actually added to the sandbox. They're stored in a separate vault.

So this decoupling actually makes it quite reliable and safe, particularly for long horizon tasks. And this is kind of one of the core ideas that underpins Managed Agents architecturally. And I think an interesting thing that falls out of this is related to, you guys may have kind of seen or come across the recursive language models work, the session becomes an external context object that the model interrogates.

And this has all sorts of benefits for context management. So you think about it, when you're doing something like compaction, you're choosing some logic to retain some amount of context, and naively, in a typical, you know, kind of a typical step, you're discarding all the context that you didn't compact.

In this architecture, and also more broadly with recursive language models, the idea is that the context object is persistent and is unadulterated. So it's append-only, and the model can always go back and fetch old context. So it basically creates a very, very nice architecture for context engineering because the core context object is immutable in the sense that it's non-destructive, and you can only, and you only append to it over time.

So we've seen this be to be quite nice in terms of long horizon context engineering as well.

So the second theme is use verifiers. And one of the problems that we've seen with Claude and other models in general is that when you ask them to do a bunch of work and then say, "Okay, grade your work," if that same context is being used to both do the work and grade, you can get lots of odd artifacts and confabulation and basically odd behavior.

For example, this is just an image showing, you can think about that context window is filled with lots of different information, and the model is grading itself. Often, it's not properly tuned to do kind of critical verification. And so what we found is it's quite effective to separate verification into a separate context window.

This is a very general trend. We talk about it in a number of different engineering blogs. And the reason is the verifier context can be tuned very specifically for the critique or verification task. And so the way this works in practice is, when you build loops, you can have a loop of a build context and a verify context.

And this can be a build agent, verifier agent. And what happens is the verifier has some goal or rubric, and it's verifying the result or work of the build agent. And this continues in a loop until verification is complete.

And this is really the big idea behind this whole loops trend that you might have heard about. And we found it to be a kind of a very powerful paradigm, especially for working with some of the higher capacity models.

And here's some of the primitives. So in Claude Code, you have goal. In Managed Agents, you have outcomes. And the principles are really the same. You're setting up a measurable end state in both cases. You're using an independent context model.

You're using independent context to grade over the course of this loop. The loop can run, and you only exit the loop until this independent verifier has verified that it has the outcomes or outputs that you want. That's the key idea.

Now, let me tell you a story about how I've used this. So this is a kind of a fun and interesting challenge called Parameter Golf. It's a benchmark that's set up, that was put up by OpenAI. And it tests models' ability to effectively do kind of ML research.

So it asks the model to basically take a small kind of model and train it with eight H100 GPUs in less than 10 minutes. And what you see on the Y is basically you can think about it as loss.

So lower is better, okay? And what I did was I set up a kind of a verifier loop using Managed Agents and outcomes to test the ability for Opus 4.7 and one of our frontier models, like Mythos-class models, on this task.

And what you see is, basically, I allow the model to continue to iterate until the outcome that I specify is satisfied, which is it finished exactly 20 iterations and kind of met, it kind of met all the experimental criteria as defined by the benchmark.

What you see is the frontier capability models are extremely good with this pattern of kind of loops and kind of verification. Because what happens is, instead of encoding steering into, like, me as the human, you're encoding the signal into the environment so the model can self-correct when it receives feedback from, for example, the verifier.

And using this kind of paradigm with very high capacity models, you can get very strong results. So the main point I'm trying to make here is that this paradigm of loops, which a lot of people have been talking about today, paired with very high capacity models, is a very good general primitive for long running asynchronous work.

That's really the key point here.

Now, let me talk another, about another theme of self-learning. So the human brain has two kind of interesting systems for memory. So one is, as you go about your day, the hippocampus is kind of writing traces of kind of short-term, very fast kind of experiential memory.

Like, you might remember what you had for lunch today. You had lunch an hour ago, you kind of remember. That's kind of written to short-term memory. When you go to bed at night, though, an offline process or out-of-band process dreams.

And dreaming stores certain important details to long-term memory in the cortex. So, for example, tomorrow, you might not remember what you ate for lunch today. That's kind of a local trace. But, like, if you had a very important experience today, maybe this talk, maybe not this talk.

But if you had an interesting experience today, that might be written to long-term memory. That's kind of the point. So these two subsystems in human work like this. And we actually found memory systems with Claude actually can employ these same two principles.

So this is showing Claude's capacity as an in-band memory writer. So you basically give Claude memory tools. And when I say memory tools, I mean is basically the ability to write to a file system that is basically a memory directory.

That's really it. Now, this is showing some work on Claude-based Pokémon with Claude Sonnet 3.5. And here's the key point. When Sonnet 3.5 is given this access to a memory directory and it can write memory, quote-unquote, "in-band" as it progresses through this game, it's not very good.

So the memories it writes are pretty crappy. It's kind of, it's kind of tactical notes. It's not very strategic. And the game progress is quite limited. But with more recent models, like this is looking at 4.6, the notes are much more strategic, and game progress is much further.

So the key point I'm making here is that Claude has gotten much better at this in-band memory writing across model generations.

And this is another way to show that same result. So this is a benchmark that I ran called Continual Learning Bench. It's an open-source benchmark. I took one of the tasks. This is a task that basically asks the model to perform a sequential question answering with a SQL database, and it can write memory in between each step.

And what you see is, basically, the performance improves across models.

And so what this is kind of showing is that models get natively better at this in-band memory writing with respect to model capability. And some of the most interesting things I found from this are that the main differentiation between, there we go, the main differentiation between, like, a lower capacity model and a high capacity model is kind of this distillation step.

And so basically, higher capacity models have a better sense of, like, what abstraction to save to memory that'll be useful later. Like, they're not just writing a specific fact. They're writing, how do I, how does this generalize to future sessions?

That's kind of the key difference that I found that higher capacity models kind of have when they're writing memory. So this is a very important thing to keep in mind, that models are getting better and better at this kind of in-band memory writing across model generations.

Now, there's a little trick here, which is very important. So we talked about kind of in-band memory, and we talked about dreaming. So at night, I dream and I write things to, like, long-term memory. Dreaming is very important because when I'm writing memory in-band over the course of a day, over the course of a session, sometimes you can write incorrect memories.

And, or you're writing things that are locally optimal, but not globally optimal. So you're writing over the course of a task to, like, kind of help you solve that task, but not necessarily looking forward to future tasks. This is a very important nuance.

And this process of dreaming is kind of an offline or out-of-band process that we've used to consolidate and improve memory. And I want to show you a fun example that I've used dreaming for. So this is, again, Pokémon.

And this is, I played a lot of games of Pokémon with Claude to find this. So this was a very hard one lesson, so I hope you appreciate it. Okay, here's the point. So basically what happened is, Claude wrote an incorrect memory, okay?

And what happened is, this incorrect memory was related to the location of, the details don't necessarily matter. The point is that this incorrect memory causes Claude to mislocalize itself, or the Pokémon to mislocalize itself, and it falls through this trap door, okay?

That's the key point. So it writes this incorrect memory. This incorrect memory causes it to mislocalize in the game, and it falls down this trap. And this is very consistent. So I saw this in five replicates, five out of five replicates with raw memory store fell down this trap.

With dreaming, this error is corrected, and it's able to properly localize itself and not fall down this trap. And I'll show you kind of a fun visualization of this. This is looking at kind of memory traces, or basically traces of game progress.

So going upwards on the Y-axis is improvement. That's like moving to the next level. Going down is backtracking. So what's interesting here, the no memory baseline, which is that gray bar, kind of doesn't make much progress at all.

It just kind of like is stuck. It's like a particularly hard level, okay? The memory, which is the orange, actually keeps falling down this trap door and falls back. So it backtracks. The dreaming traces, though, consistently kind of fix this error in its memory and proceed to the next level.

This is a very practical example of how dreaming can kind of work out-of-band on your memory store to fix corrections. Because what it does is, it looks at your memory store, and it looks at all your prior traces or sessions and kind of can find incorrect errors.

That's the key point. And that's why this dreaming process can be very helpful, because in-band, while Claude is writing to memory, it can make mistakes. And those mistakes get stuck in memory unless you have an offline process to kind of correct them.

That's the key intuition.

And theme four, and I'll open up for questions after this, is what I think is kind of this trend that we're going to see moving towards or global harnesses with async agents. And so we released Claude Tag, and a lot of the reaction was like, "Ah, Slackbot."

And like, look, I have actually created a lot of Slackbots myself. I understand. Not every Slackbot is particularly interesting or great. In fact, I've created many Slackbots that are quite bad. But what's interesting about Claude Tag is not the fact that it's accessible through Slack.

What's interesting about it is the fact that it has a very, very rich kind of system underneath it, which I want to just touch on briefly. And in particular, what's interesting about it is it represents what I would consider an org-level harness.

So agents historically have been kind of single player. So you have an agent like Claude Code on your machine with your local context that you've tuned and configured for yourself. What's interesting about Claude Tag is it is a harness that everyone in the organization has access to and can use.

So it is a multiplayer harness. And what's nice about that is it has its own identity. Its identity and credentials are not tied to a given user. And it has access to organization-level context, not just my local context.

This has many interesting and useful implications, including the ability to, like, check others' work before you do an experiment, the ability to kind of deduplicate findings, the ability to, like, do internal research, the ability to give everyone access to kind of a very well-developed harness on day one.

Whereas when you have your own personal harness, often new employees take them weeks or maybe even months to kind of ramp up fully to configure all theright connectors and so forth. So org-level harnesses are a real leveler of the playing field.

And I think it was, people kind of saw the Slackbot piece, but they didn't really appreciate the depth of kind of, the depth of benefit you get from building out org-level harnesses. So I do think that was kind of an important thing to note.

And I think we're going to see the rise of kind of harnesses that operate across orgs, across many different users that can operate increasingly on longer async

agents that can operate on longer timeframes. That's kind of one clear kind of follow-up that I think we're going to see from this. And another thing I think we're going to see is that asynchronous agents are going to be increasingly proactive.

So typically with, for example, like, locally scoped agents, they tend to be reactive. They're responsive to how you steer it. Versus async agents increasingly have the ability to steer proactivity. And that's one very nice thing about Claude Tag, where basically you can configure it to tell you things when, like, looking at this org-level context, alert me with things I might need to know about.

And this is a very important kind of new kind of UX that I think is going to be more and more common with async agents that kind of have access to organizational context. And of course, multiplayer. So the ability for a single harness to be steered by many, many different people kind of concurrently is an important shift in agent UX that I think will be quite interesting going forward.

So

yeah, let me just open up for questions. And thank you for listening.

Sure.

One thing that we see sort of incorrectly and also on the benchmarks is that the front-tier models perform better on these long-horizon tasks and are somewhat better at abstracting what they need to do, but not that source is not.

What is your view on, like, the Sonnet cycle lightweight model?

Yeah.

Like the open-source models so I can see how it's going to be on those tasks. What's your view on why that is, and then how long the front-tier will maintain that gap?

I see. So the question was kind of on the gap between the frontier models kind of on, for example, a benchmark like METER on, like, long-horizon tasks.

Yeah.

Okay.

So in the latest results that I saw from, like, for example, CODEX, like, 5.6, I think it also kind of is in that 12-plus-hour regime on METER. So I think you'reright that, like, the frontier models, like the, you know, mythos-class models, strong models from OpenAI, kind of are in this, like, 12-plus-hour regime.

Why is it that non-front-tier models are not kind of in that regime? I am actually not necessarily sure. I do think, I do think in order to build agents that can effectively operate in this regime, it's important to know that it's not just the model capability.

Like, for example, with the Claude Tag product, it's actually a combination of improvement in memory, because memory is very important. If you have agents working for, for example, 12 hours, you want to make sure that your proactivity preferences are well encoded in memory so it knows when to reach out to you if it gets stuck, for example.

So memory is very important. Security is very important. So resistance to prompt injection. Also, like, model architecture, like, kind of their agent architecture is very important. That decoupling of brain and hand so it's secure and safe and, like, resistant to failure.

So actually, I think to build real agents that can operate in these long time horizons, a bunch of things need to come together in terms of, like, architecture, infrastructure, security, memory. That might be why, but frontier labs invested in all these areas.

So that might be why you see kind of a gap in terms of, like, the agent products that we've released. And so we spent a lot of time, for example, building managed agents to kind of have these kind of considerations baked in.

Yeah, sure.

I'm Saiya. I have a follow-up. I have a set of questions. So I guess for long-running agents, I have a DPIA that is memory is playing a role in what should it look like. Is there any particular reason, you know, generally the design focus has been towards building memory systems rather than building memory databases?

Yeah, okay. This is interesting. The question was about kind of, like, the best memory substrate. So, like, why file systems versus, for example, databases?

This is kind of a subtle point that actually I want to think about carefully. So I don't necessarily think that it has to be the case that you use a file system for memory. I think what's quite important that we've seen is that it's, you want something that is highly programmable with simple primitives that the model can manipulate to, like, write and manage its own memory.

So, for example, a database could work fine relative to the file system. But what I've seen doesn't work is when you specify the structure of memory for the model very explicitly, whether that's in a file system or database or whatever, like a memory schema.

I kind of pre-populate, here's the types of memories you need to save. Because that ends up being not very bitter lesson built in the sense that models can learn to manage their own memory much better than you can intuit these memory types for the model ahead of time.

So I think what we've seen is that very general substrates for memory, be it just your database or file system, are good because the model can manage them freely versus a very, very kind of, like, prescriptive memory schema that you're trying to pigeonhole the model into.

That's when you see performance drop. That's the key differentiation.

So you're seeing there to be let the model have a medium scope of memory.

Right.

And decide how it wants to store it.

That's the key point. Let the model structure and maintain its own memory. Don't give it a prescribed memory schema. And that's, like, a common failure because models are getting good enough that they can manage their own memory much more effectively.

You can reason about types of, this is, like, very classically bitter lesson built, but, like, you can, models can reason about their own memory and context structure much better than you can prescribe for them a way to structure their own memories.

That's the key observation. Yep.

I think what you are saying here, any medium as a tool to.

Yes.

This is where you store memory to how you want.

That'sright. Exactly. General substrates for memory management. Yep.

Yeah. So I'm just curious. So you mentioned, like, reading.

Yes.

So actually describe what.

Okay. So that's a good point. Basically, the question is, so you do this dreaming thing. You look at the sessions, you look at the memory store, you update the memory store. How do you know those are correct? So evaluations, obviously, are one way to do it.

This is kind of a fun anecdotal example from Pokémon showing that, like, you can perform corrections via dreaming. The key point is that we've actually run a lot of different evals showing that dreaming can indeed improve performance for very intuitive reasons, as you see here.

But of course, evals are important in, like, your own context to confirm it's actually worth the offline compute. Yep. I guess we're done. Thank you all.