Beyond Static Intelligence: Evaluating Continual Learning — Parth Asawa, UC Berkeley

Hi everyone. Uh, my name is Parth Asawa, I'm a PhD student at UC Berkeley. We're all at these AI and machine learning conferences, but everyone is just talking about how smart these agents are. We're not actually talking about learning ability or how much these agents learn, and so today I want to talk about that.

You've seen the way we evaluate language models today. Every time there's a new model release, we see charts and graphs that look something like this,right? The way we evaluate these language models is we ask them to do one task, and then completely independently we ask them to do another task, and then another.

And then we repeat this across a variety of different benchmarks,right? And so you've probably seen a variety of all of these. In the end, those benchmarks in aggregate, when we take all of those independent evaluation scores, give us leaderboards that look like this.

Now, the challenge here is that if you look at the way we actually come to these single independent scores, what we've done is we've kind of told the models, imagine that every time you do something, you completely forget your memory, okay?

Like, imagine if your life was like that, you're restarting from scratch every single time you do anything. That's the premise under which we're evaluating language models today. I'm going to argue that it's not necessarily what it should look like.

If you were to look at learning ability, you might see charts where across different instances performance doesn't look like a scattered line, but it looks like as a function of prior experience, models are actually able to improve their performance in contrast to what they've done in the past.

Let's take a step back. What is continual learning? Continual learning, to me, is sample-efficient online learning that is stable over long horizons. It's a challenging problem because you have to deal with both your ability to retain prior information without forgetting, while simultaneously being able to update from new information and adjust your priors to new in new data you're receiving over time.

In language models specifically, you know, the way we train language models is we take a bunch of data we want to train them on and we kind of put them in a box in some offline training process. And then maybe for a few weeks or something, we train the model and then we extract a frozen checkpoint,right?

And this frozen checkpoint is what's deployed to the world. The weights don't really change after that in today's paradigm. And continual learning is all about trying to change that, trying to enable mechanisms for models to learn over time.

So in language models, people have looked to approaches like in-context learning. Put more information in the model's context and the model can simulate some amount of learning in context. People have looked towards external memory stores,right? You could externalize some of your memory to a notepad that you then get to refer to every single time, or a key-value store, or your choice of external memory.

Or people look towards the parametric approaches. What would it actually look like if we had ways to update the model's weights online such that it could improve and learn from the new information? Now, today I'm not going to make a case for which one of these can solve con continual learning, though I do have an opinion and I'll share it at the end.

But what I am going to make a case for is that we're actually not evaluating it at all today. And the question that we need to ask ourselves is that if continual learning doesn't look like point capabilities, we need to be measuring it theright way to optimize for theright objective as a field.

So let's get into it,right? What does continual learning kind of look like today when you read the literature? To be clear, continual learning by itself is not a new idea. Continual learning for neural networks has been around for decades,right?

There have been famous problems in catastrophic forgetting that people have spent years trying to solve. But with language and, you know, an example of that is some of these sorts of papers where you'll see graphs like this where you train the models on a particular task distribution, task A, then you train it on the task B distribution, and then you train it on task C.

And the objective you're looking for is does the does the model's performance not degrade on prior tasks while it's still able to learn the new tasks? And people have taken this similar methodology and applied it to language models too,right?

In some of the more recent papers we've seen, you're you're starting to see similar graphs of you train on one task and you train on the next. There's also a different class of evaluations for language models that look more like a factual recall sort of test where over these extremely long horizons can models remember particular parts of the conversation and improve from that.

The case I'm going to make, though, is that these sorts of evaluations aren't actually sufficient for continual learning, okay? They don't consider sample efficiency a fr a first-order requirement. They don't necessarily always measure learning. And perhaps most importantly, they won't work for frontier language models.

And the reason being that frontier language models are pre-trained on vast distributions of the entire internet or economically valuable tasks we care about. And so if you're expecting to see improvement on new tasks, it's it's actually really hard to come up with what those sorts of tasks should look like.

So given all of that, what should a continual learning benchmark look like? The biggest question that I get when I when I talk about this with people is why can't I just chain existing benchmarks together? Like, we've we've seen all of these AIME datasets.

Why can't I just take my AIME problems and solve them in a sequence? And the fundamental problem there is that benchmark instances are in traditional language model evaluation are designed to be independent. That means they don't have shared structure across tasks.

And as a result, you can't meaningfully expect them to improve from earlier experience in future instances. And this is the fundamental problem that exists with any approach that tries to chain benchmarks together.

Different.

In contrast, when we're designing continual learning benchmarks, we looked at three main design criteria as things that are kind of a fundamental requirement for evaluating continual learning. The first one here is headroom. And this kind of goes to my last point.

Frontier language models are trained on almost everything in the internet. If you want to evaluate continual learning, you need some sort of task that actually requires online adaptation or learning from the language models. And that doesn't exist for a wide variety of the benchmarks where the models are just trained offline on that data.

If the model can improve on your tasks by just training offline and not actually require any online learning, then it's not a good task for measuring continual learning. The second point that I'm going to talk about is shared structure.

What this means is when going back to my point of pre prior language modeling benchmarks all being independent instances. If you want there to be an expectation of improvement in how you're evaluating continual learning, then there needs to exist some shared latent structure between the tasks.

You can kind of think of this as some sort of shared latent in the environment that your models or agents are learning over time, and they're seeking to exploit these these latent structures that exist in the environments to improve their performance on future tasks as a result of their prior information.

And the third criteria that I'll talk about is a learning mechanism. And what this means is just there has to be some realistic expectation that the models are able to learn as a result of their prior experience. This could look like scalar reward.

It could look like error messages. It could look like textual feedback. The point being, there needs to exist something in the environment that's giving agents signal to learn and improve on future tasks. Otherwise, it's not really a fair measure of cont continual learning.

Let's talk a little bit more about the metrics before we get into examples. Obviously, there's a simple metric for a lot of tasks that we've seen recently, which is reward. And in our benchmark, it's no different. Every task defines a per-instance reward metric where reward serves as a proxy for learning.

Higher reward is better. You could think of this as efficiency. You could think of this as profit. You could think of this as how close you were to the actual predictions. There's some single unit of reward that operates on a per-instance basis.

And you might ask, like, why can't I just evaluate continual learning by looking at cumulative reward across the sequences of tasks we're constructing? And the answer is that total reward alone might confound continual learning ability with base model strength.

Let's look at a diagram to kind of explain why this might work. Here we have two different continual learning systems, okay? The the red one and the black one. The one's connected by the dotted lines. The gray system at the bottom is the black system, except it wasn't allowed to maintain state across time, okay?

So that's what the black system looks like if its memory was wiped every single time. But if its memory wasn't wiped, if it had some mechanism of learning, then you can kind of see that it tends to improve over time.

The challenge here is that if we looked at the cumulative reward metric alone, the red system would look like it was the best the best performing system on this task,right? It let's assume for a second that there is per room to do better and it hasn't saturated it.

It has a higher cumulative reward, but you can kind of tell that in comparison to, uh, to the black system, it doesn't necessarily improve over the stateless baseline. It's just a better system to begin with. And this is one of the confounds we run into when we're trying to measure cumulative reward.

To deal with this, we look at the gain metric in addition to reward. Gain refers to the difference between stateful reward and stateless reward. What that means practically in our benchmark is that for any system, we run it through the benchmark twice.

Once in the normal way where it's allowed to maintain state across all of the instances in the benchmark. That might mean it's learning and updating its policy. It might mean that it's updating its notes. It might mean that it's just growing its context length, but it's allowed to maintain state.

The second thing we run is a stateless baseline. What that means is we reset the system between every instance of a task so that the model isn't actually allowed to continually learn in any meaningful way. The difference between these two numbers then for any instance intuitively is kind of a question.

It answers the question of how much did if we're looking at gain on task five, how much did my prior experience on the first four tasks actually lead to an improvement in my performance on the fifth task? It isolates out what your benefit of from actually learning was versus your base model's initial capability.

To to be clear, I think reward, gain, and cost are all measured on Pareto frontiers. There isn't one single metric that I think defines continual learning because we still care about the base model strength. We still care about the ability to learn, and we still care about the cost we're expending for these systems.

And so we try to measure everything on Pareto frontiers in this benchmark. Let's jump into an example of a concrete task to ground some of what we've been talking about. Here I'm going to walk through the database exploration task, which is hopefully a little approachable for, uh, which is hopefully approachable to understand.

If you guys have ever worked in data engineering or know data engineers, you know that one of the things they do is operate over these vast databases with large set many different sets of tables, schemas they're unfamiliar with, and idiosyncrasies in the data.

And if you're a human data engineer, you learn those schemas over time. You learn how tables link together. You learn the idiosyncrasies in the data. But if I was an agent where my memory was being reset every single time, I wouldn't learn any of that,right?

And so here what we can use as a measure for our ability to continually learn is the efficiency in the the task here is you're given a natural language question over the data, and you have to answer it by performing some set of SQL queries,right?

That gives you the answer from the from the databases. But if you don't know anything about the databases, it's going to take you a lot of SQL queries at the start. And at the first questions, you're going to be doing a lot of different exploration into figuring out what do the schemas look like, what is the data format, how do these link together?

But as you progress through the benchmark, if you're a continually learning system, maybe let's say by the 10th instance of a task, it'll look a little different,right? You'll you'll be able to know a lot more about the schemas, the idiosyncrasies of the data.

You'll know how things link together, and it'll take you much less to actually answer the question. But there's a bit of a there's a bit of a conundrum here. We're going to add something a little more challenging to this task too.

In the real world, there's often concept drift in the things you're doing. In the database example, let's say there's a migration of your database,right? Columns get dropped. There's new columns with different names added. The data format changes. As a human, you reason through all this uncertainty and you're able to update your priors.

You know that there's still some information from the past that may be relevant, but from your exploration, you learn what to forget and you learn what's actually relevant to maintain over time. You kind of have an innate ability to maintain the stability and plasticity trade-off in your mind.

But this isn't native to a lot of language models. And so if we give it a task after the database migration, it might struggle to require it might struggle in its ability to detect and discard stale experience and simultaneously update from new experience.

And so this nate this notion of concept drifts is something we try to add a lot to our tasks to further test the limits of what memory and continual learning might look like in language model systems.

In Continual Learning Bench 1.0, which is the benchmark I'm talking about today, we have tasks from six different domains of tasks, okay? We have blind spectrum monitoring, which is a signal processing task, codebase adaptation, which is a software engineering efficiency, cohort studies and epidemiology task, exploitable poker, a strategic game playing task and because my friends like to play poker, and a database exploration task, the one I just showed you, and sales prediction, which is more of a data sciencey prediction error style task.

Across each of these task domains, we construct sequences of individual task instances with defined reward metrics that agents or models are tasked with solving gradually over time. And all of these task instances across domains, we validate with domain experts to see is this learnable?

Is this realistic? Are these the sorts of drifts you would expect and things you would expect to remember in an environment? Let's jump quickly into the results. In the first initial release of the Continual Learning Bench, we evaluated a lot of these context management systems to begin with.

And there's some parametric systems down the line too. If we look at just aggregate reward for a second and ignore gain and cost, you'll kind of see that the in-context learning systems, this is vanilla in-context learning where you just put the experience in the context and you don't do any of the fancy context management that some of these other systems do.

It tops the leaderboard. And and it's not just on reward. It's actually also when we look at the Pareto frontiers, this kind of holds across reward, uh, versus cost and gain versus cost. I don't necessarily think this is what the end state of continual learning might look like.

You could argue that these were medium horizon tasks and they didn't push the frontiers of the in-context learning systems enough. And I I would say that's fair. And that's one of the things on our roadmap to push those even further.

But it was still surprising that these more expensive context management systems perform a lot poorly compared to just vanilla in-context learning on these sets of tasks where you have to do real learning. These are real tasks that you might expect a normal person to do.

I think it's also interesting to chat briefly about what some of the failure modes look like when we observed in this initial version of the benchmark. I'm going to argue that most failure modes in continual learning fall on one side of the stability plasticity trade-off, okay?

And stability is your ability to retain new information in a stable way and use that for future tasks, while plasticity is your ability to actually learn from new information. We see usually that with most continual learning methods, any sort of failure mode comes from the inability to do one of these things.

And I'll dive into examples from both. You don't have to read too much of the text here, but this is our sales prediction task where the model's task making some predictions or forecasting for sales over the course of, I think, five years.

The model starts by getting feedback that it overpredicted for the last, uh, the last submission it set. And so what it does is that it then leads to it revises its prediction downwards to a lower prediction for the set of five years.

And then it gets feedback that actually its underprediction was too much of an underprediction. And so as a human might naturally do, you would go for the middle. But that's not what the model does. The model kind of forgot that it had the overprediction to begin with the start, and it just re jumps backright to the overprediction.

This is kind of example of what stability failures look like where it's not retaining past information in improving its future experience. Plasticity is a bit of a different story. This is from our epidemiology task. And the only words I really want you guys to read, this is a this is a system with with a notepad, okay?

It's a context management system that's given a notepad to externalize its memory to. The highlighted text is is the important thing to read here. Quote, "These seem to be cohort definitions from a different study schema that doesn't apply here."

The study schema did in fact apply here. The the model was unable to update its priors and data from new information and meaningfully improve over time. It just didn't even recognize that this is something that's relevant to the task and use that to update its priors.

Before I finish, I want to offer a little bit of my broader thoughts on continual learning. I alluded to this before that I have some opinions on what methods for continual learning might actually look like. I'm personally quite excited about the parametric methods for continual learning that look at alternative architectures, data, and algorithms jointly to optimize what continual learning should look like.

If you look at language models today, the training stack used to look something simple like this. Or maybe it never did, but I'm simplifying for for the sake of this. Pretraining, then we did supervised fine-tuning, and then we do RLHF.

But the stack kind of adapted over the course of the last few years. Now it might look something more like this where you have mid-training, you have RL for different teacher models, and you finish everything off with multi-teacher on-policy distillation.

A lot of the work in continual learning today is how do we take these models that are already trained and then kind of that frozen checkpoint, how do we figure out methods for continual learning that work after the fact?

But these models were never designed to be continual learners to begin with. And one of my hypotheses here at least is that we're operating in a bit of a sunk cost fallacy that because we've trained models the way we are today, we need continual learning methods that work on top of that.

If you were to think of designing what continual learning looks like as a first-order principle and a first-order requirement, it might look nothing like this. The in the purest sense, continual learning might just be one set, one phase of training for continual learning, and everything after that is deployment.

You're just the model is interacting in the environment. It's updating its weights. There's only one phase of learning. And we don't necessarily have to assume that everything is built on top of the way we do language modeling today just because we've spent so much doing that.

There there are there are different ways to explore where if you think about continual learning as a first-order requirement for your systems. More broadly here for a second, I think that continual learning is just one example of my group's broader effort to challenge the way we do things in training today.

I think that there are larger questions going on though and an opportunity to challenge how we do AI research as a whole. There's a lot of questions around open science, consolidation of power, safety, what the future of things like academia or these third-party institutions might look like.

And I think that there's a huge opportunity for us to rethink a lot of that. And I've spent some time doing some more writing about that that I'd encourage people to check out. But the crux being there there are opportunities to reimagine what third-party institutions for AI research look like and what the future of open science should look like that I think we all need to spend more time getting involved in.

To wrap up, uh, I want to thank a lot of my contributors on Continual Learning Bench. This is a collaboration across Berkeley, Snorkel, UW Madison, uh, thanking the Snorkel AI for their open their grant through the Open Benchmarks Grant Program and the Laude Institute via their Laude Slingshots program.

Uh, in the roadmap, we have more domain-specific and longer horizon tasks on the benchmark. We want to add more of the OSS models and the parametric approaches. We want to improve what simulation for task environments outside of easily verifiable deterministic domains look like.

What would continual learning where you're personalizing to a user model do? I don't think we have models that are capable enough of simulating thatright now to figure it out, but I think we should. If there's one thing to take away, continual learning doesn't look like point capabilities.

We need to measure it theright way to optimize for theright objective as a field because that's a history of how machine learning has progressed. Thank you guys very much for listening. Please feel free to email me or follow for updates.

I would love to stick around, but I'm going to catch a flight, but I'm happy to take anything afterwards via email. Thank you guys very much.