Reinforcement Learning without Verifiable Rewards — Will Brown, Prime Intellect

Uh, thanks all for coming to AI Engineer and checking out the post-stream session. Um, hopefully lots of fun stuff today and throughout the conference. I'm Will Brown, I lead applied research at Prime Intellect, and today I want to talk about reinforcement learning without verifiable rewards.

And so, many people may have been learning about RLVR over the past, uh, year or so, year and a half, as this stuff has really taken off and become the main way that we think about scaling reinforcement learning.

Um, but often we don't actually have verifiable rewards. And so, messy real-world tasks often we're kind of figuring it out as we go, we're having our agents run around, and we kind of, in hindsight, maybe can like look at what they did and say like, okay, this was good, this was bad.

Um, but sometimes sitting down and just like specifying, okay, this is the rule, this is the goal, is not always so straightforward. And so, this is going to be sym- uh, synthesizing a lot of work we've been doing as well as from the broader, uh, research literature and some of the things we're building to, uh, to kind of support extending RL into more messy real-world tasks.

And so, recap quickly of how reinforcement learning works. I would imagine if you're in the post-stream session here, you've probably heard a little bit about RL, but for those of you at home and, uh, for those, uh, who are kind of still just kind of looking for the crash course, generally we have an agent, which we're going to call a mo- model plus a harness, which we place into an environment.

And so, an environment we're going to call a task plus a world, a world you could think of as maybe it's a Docker image, maybe it's a code base, maybe it is a collection of, uh, task-specific tools, maybe it's some skills, maybe it is a bunch of applications or browser tabs or things like this, as well as a scoring rule, uh, verifiers or rewards, whatever you want to call them.

Uh, and the agent and the environment are going to interact in a loop, and at the end we'll have some reward of how well the agent did, uh, in the environment for this task. And then reinforcement learning is all about, uh, creating an advantage, and the advantage is really about taking the reward, minusing some baseline, maybe doing some scaling, and then now you have a set of, uh, rollouts from the agent in the environment that you can use to then update the policy.

The policy here is just the model weights, uh, themselves. Uh, and the goal here is to take a gradient which nudges the model towards getting higher reward. And so, all the RL stuff people talk about, whether it's GRPO or reinforce or CISPO or any of the other new algorithms people come up with, they're all kind of in this policy gradient framework, which is just about saying, okay, how do I make the model do things that have higher reward.

And so, at Prime Intellect, we build a lot of tooling to power all of this, uh, at every layer. We kind of go both from the we start at the compute layer and do lots of large-scale GPU orchestration.

Uh, we build the Prime RL training framework, which powers all of our large-scale reinforcement learning and other algorithms running. Uh, we build environments, uh, we have task sets and harnesses and verifiers, uh, as tools that you can com uh, mix and match to assemble, uh, complex worlds for agents to, to learn from real-world feedback.

Uh, we have a training platform called LAB, which is anchored around environments where we do both hosted training and evaluations as well as inference. Uh, and this is to allow people to monitor their experiments and manage their training runs and, uh, iterate on their evals, uh, and deploy these models.

And ultimately the models are starting generally from some open-source base model, uh, and you're optimizing it for your task. And the goal that we're really trying to enable is for more people to be able to, uh, become their own research lab, become, uh, take ownership over the, uh, intelligence of their own, uh, model weights and, uh, the optimize for the tasks that they care about with themselves as the experts steering the model, which means we need to make it way easier for people to do this.

Currently, for a lot of people, it's still really hard. I think, uh, you can go like we're all here learning more about how it works because it's hard. We don't know how it all works, and we're figuring it out as we go in many cases, but we've spent a lot of effort and a lot of time building stuff that hopefully makes this a bit easier for people.

Um, and so, what's an environment? An environment is tasks, har- harness, and rewards, uh, but it's not just for RL. So, I think a lot of people think RL when they think environment, but environments and evals are really the same thing.

You can use these same objects for generating synthetic data, which then you could use for SFT. You can do RL or you could do algorithms like on-policy distillation. You could do prompt optimization like JEPA. You can use it as a kind of scientific test bed to iterate on your agents and your harnesses.

Um, and verifiable rewards are the easy case where we just kind of can check exactly was something done correctly or not. And so, for math, often if you have a numerical answer, you can just parse this out of like a box in the answer from the model and check.

For code, maybe you want to use test cases or a linter or something like this. For tool use, often you have some database state, which you kind of know what you sh what you're expecting at the end, and you can just kind of like check this deterministically.

And so, these are kind of the easy cases where the, the, uh, reward design problem is not so difficult. Um, but most real-world tasks are not this verifiable. Uh, for a lot of real agent wor tasks, what we're having agents do things like write reports that maybe are analyzing a bunch of documents or research.

Um, we're maybe asking it to do things like book flights or buy things, but there isn't always like a clean best answer here. Uh, and there's also notions that are fuzzier of like interacting with users, like handling a refund.

Like, what does it mean to handle this well? Um, and so here the signal is less clear, and there's a lot of different tricks we might want to explore and techniques we want to develop to ensure that this can be done reliably and scalably.

Uh, and making evals is hard because oftentimes the benchmarks out there that we might like look at in the kind of, uh, new model releases, it's like a set of a few hundred tasks that a bunch of researchers spent months kind of handcrafting and talking to experts.

Maybe they worked with data vendors and kind of like spent lots and lots of money kind of getting these, uh, to be very like precisely refined. And, and this isn't very scalable, uh, out of the box, um, especially for things that are more open-ended where there's no kind of clean check for what's good or not.

Um, often the, the, the real-world situations can be unbounded. You don't always know what things are going to be, uh, in the distribution. Like a lot of these cases, we are figuring out the distribution as we go, and classical machine learning will tell you you can train for the distribution, but generalizing outside of the distribution is kind of an undefined problem.

And then especially with RL, we have to be very careful about reward hacking. And so, reward hacking is when you have a kind of loose proxy for your objective that is, uh, undefined at the boundaries. And then models, if you train with RL, they can learn to exploit this and, uh, find, uh, weaknesses where there's some path towards, uh, climbing the, the reward that doesn't actually give you what you want.

Uh, and so, really the goal of what we would hope all of this builds into is continual learning, which is a big buzzword that I think a lot of people like talking about in many different ways, but I'm going to use it to mean a very particular thing, which is that, uh, we want models to be deployed in relatively realistic, complex, messy settings and to be able to learn as they go where they are doing things, they are making mistakes, they are then able to observe and catch these mistakes after they happen and use this to not do the same thing again.

In some cases, people want to try to do this at the harness layer or the prompt layer, um, but ultimately you want a system that can evolve autonomously, uh, to be able to get better over time with humans in the loop at thatright level of abstraction.

And I think currently the level of abstraction for doing this is far too low for it to be practical for most people. Uh, and so, this means we need new methods to be able to, uh, automatize as much of the difficult processes as possible.

And many of these actually are automatizable. They just, uh, are kind of difficult problems to solve. And so, there's a few techniques you can, you can use to kind of, uh, start making progress here. But one of the goals here is to do online reinforcement learning so that you can kind of iterate on this process as you go, as well as beyond just RL.

There are other things where you might want to incorporate world knowledge into the model itself, not just in terms of skill refinement. RL is great for refining skills, uh, but less so for incorporating like dense new knowledge. And so, blending these two together is also an important goal.

Uh, and ultimately the what we want from this is to be able to deploy agents into production and have them improve as they go and have the, uh, this, these experiments be monitorable and traceable and replayable so that we can kind of treat, um, model optimization very much as a science and make this science accessible to people who have a very wide variety of, uh, use cases they want to deploy agents for, which don't a-all live in the training distributions of the big models.

And so, we want to be able to do this on top of the best and biggest open models in the world and make this accessible. Uh, and so, how do you manufacture signal? There's a bunch of techniques that we've found very useful.

Um, one is grounding. And so, grounding roughly means that you have some source material. Uh, and in machine learning generally, you want to have some notion of supervision. There's something you're learning from. Um, and in messy situations, we don't necessarily always have clean supervision, but we can get, uh, pretty re-reliable supervision if we kind of are careful about the techniques we use.

And so, grounding is one where you have some source material and the ability to do an A/B test of like with and without is a very useful way of creating this kind of capability gap where a model will do better if it has something in context.

And this gap is something we can exploit to create signal that we can then learn from. Judges are also really useful where we're relying on the fact that LLMs are already really powerful general reasoners for many things. And if we, uh, assign the if we set these judges up in theright way, then they can spend compute to make decisions about whether an action was good or bad.

Uh, we also want to be scaling search. So, search in many ways is something we can apply at many different layers of the pipeline, both in terms of creating tasks as well as the world, as well as the criteria for which we want to be giving judges for answering questions about the quality of a rollout.

And so, for source material, um, one very useful version of this, especially for this continual learning goal, is production traces themselves. And so, what we found is super helpful is taking existing traces from a deployed agent and treating these as the source material where, uh, we don't necessarily know upfront what the distribution of tasks is, but as an agent is deployed, you start collecting more and more examples of, let's say, user prompts or, uh, s uh, calls from an orchestrator agent down into a sub-agent, and this starts becoming the distribution.

We don't have labels yet, but it tells us at least what we want to look for. Um, and so, this is one ve-very useful category, especially for other things like, uh, search or for code. Uh, you have doc corpora corpora of documents.

You have repos that are also very useful for kind of anchoring your, um, your, uh, your learning as well that are and so, uh, taking these sources, these raw materials as places to kind of search for tasks from is a very useful way of starting to kind of create this environment out of nothing.

Well, it's not nothing. It's something from the real world. And because you have the real world, you want to use that, the real world, your production environment, your agent traces as the source from which you want to learn, even if you don't have supervision yet.

And so, one thing you have to do here is get tasks. And so, documents are actually a pretty easy version of this where you can just sample documents. You can have models generate question-answer pairs grounded in the documents.

You can verify that those answer those questions are answerable with other models that are still grounded in the documents. And then the actual task at hand involves throwing away the initial search. And so, you kind of get to work backwards.

And so, this general principle of like working backwards is starting from the solution or something that's close to the solution and ha where your real task is like further upstream. This is a very useful way of kind of having a you can verify the easy problem and then learn on the hard problem.

And so, anything where you can move backwards like this is super useful for kind of getting supervision for free. Um, in code, you can have, uh, you can use, uh, real-world PRs, the diffs, the descriptions, the test cases, uh, removing different pieces of fi different files to be able to have models start learning over code bases because you can kind of take something that is a completed artifact and start breaking it down into smaller pieces and then have replaying these pieces of getting to an end state that you know is reachable, uh, the a-attest that you train on.

And so, this idea of wanting to know that an end state is reachable and that you can then take steps back, throw away the solution, and then learn to find it again, uh, can be applied more generally beyond code as well.

Uh, and so, w-we talk about world simulators broadly as the, the sort of thing we might want to do in messier environments, which are not just, uh, production, uh, like which are not just, uh, doc search or code.

And so, a lot of the ones that we've been working on at Prime Intellect are related to things like tool use and web applications where we don't actually have full control ability of the backend state. There are some MCP tools or CLI tools or websites or applications where we can't actually program them yet.

And so, what we want to do is learn to simulate them. And so, we found that using combinations of, uh, universal backend infrastructure and test time scaling and search and kind of iterating between the simulator and the real, uh, and the real we can kind of ground in these production traces, this then allows us to create really high-fidelity simulators.

And so, we found that these simulators are actually really great for RL because you can one, if you have production data, you can make your simulator better and better over time, but also you have full control ability over the backend.

And so, you can actually do this reverse engineering where you get to kind of plant the answer. You can start from the end and work backwards, uh, and so that you, you have this verifiability baked into the simulator even if you don't have it in the real-world production deployment because you don't know in advance if a task was solvable at the time that you are being asked it.

Um, and in terms of doing this, a very useful thing is scaling judges. And so, a lot of times we will have a model that does something and it will make mistakes along the way, and it's easier to tell what went wrong in hindsight.

And so, the fact that you've already seen the chain of events after and you can look backwards and say, "Okay, the model made a mistake here. This thing doesn't feel quiteright." Or we asked seven different models and they all kind of agree this thing is wrong.

This is a very useful way of kind of, uh, spending compute to do search to then extract rubrics. These rubric questions are, are, are very effective at like kind of, uh, distilling down the, the search into something that we can then, uh, use to more cheaply audit and kind of also ground once we have these rubrics as a, a way of saying, "Okay, we need tasks that target these kinds of, uh, failure modes as well."

Um, all of this is under the umbrella of scaling search with test time compute. And so, we can scale search for mining traces by if we have offline production traces, we can just look at them more and think about them more and have more models play with them.

We can calibrate difficulty. So, RL to have the advantage gap that we mentioned, uh, needs to have a, a separation between what one model will do once and what a model what, uh, a collection of rollouts will do.

And so, you want tasks that are not too easy, not too hard, and you want to be searching for these and iterating on generating more of them. And so, this is another area where you can spend compute to, um, refine the difficulty of your, your task distributions, your task sets.

Um, for simulators, when you're building, uh, web applications or tools, uh, that need to simulate complex behavior, you can spend compute on searching these and then you can have agents refine the implementations of them. And this allows for increasingly high-fidelity environments.

You can do this on verification both at, uh, train time, uh, as well as, uh, offline when you're kind of creating these rubrics. You can do things like red teaming with adversarial prompt optimization to, to kind of explore for backdoors.

Then you can look for traces and spend compute mining these traces for understanding was this a reward a reward hack or was this actually kind of in the spirit of the task? And I think these things can kind of feel like reward hacking can kind of sneak up on you if you're not careful for it.

But in many cases, uh, the basic simple things actually work quite well where if the reward hacks are the sorts of things where a human can look at them and be like, "Oh, yeah, that's a reward hack." Judges are often quite good at doing this as well.

They just don't necessarily, uh, if tell if you tell the model not to do this, it won't necessarily do it in the rollout. But in hindsight, you can reflect on this and spend compute to kind of, uh, especially if you are collecting these over time and you are building up your corpus of examples of reward hacks, you can understand the sorts of things that go wrong, um, and, uh, address this by kind of again, spending inference compute on refining your implementations, refining your rewards, as well as validating these by training.

And so, we find that in many cases you can do a lot upfront, but also, also there are things that don't show up until you actually like start doing RL. And so, part of this is folding in, uh, training experiments themselves into the process of environment design where you can you can do, uh, small runs with individual models on like one environment and, and see what happens.

And you can understand the, the behavior changes. You can have metrics that log the, the types of tool calls that are being done that are like judges asking, uh, questions about the, the traces to understand, uh, how behavioral patterns are changing.

And all of these, uh, are very useful ways of kind of getting something from nothing and, uh, using compute as the thing that allows you to, uh, refine your understanding. And ultimately what you want is to surface the most important pieces up to the human, the, the highest level of, uh, questions about what is actually going on, what is the goal so that the all of this is deferring to the human for the most important pieces of like actually, uh, employing expert judgment to say this is good, this is bad, this is what I want, this is not what I want.

And, uh, these are all the ways that we kind of gain confidence in the environments. Um, I know we're running a little short on time, so I wanted to recap a couple blogs that we've, uh, put out recently that are kind of demonstrating pieces of this.

We have a, a blog called General Agent, um, which is, uh, demonstrating this for tool use, this, uh, online loop of generating, uh, solving and synthesizing new tasks and gating based on this pass rate, which then we train on and we see, uh, great uplift on, uh, popular benchmarks for tool use.

Um, additionally, beyond just RL, we found that it's quite important to think about cases where there's information in the world that RL alone will not, uh, explore. And so, there's this great work echo, uh, from some researchers that, uh, we are friends with and have been collaborating with.

Uh, and then we did our own kind of, uh, deep dive into this as well to look at what happens when you have an agent that is not just training with reinforcement learning, but is also getting supervised learning signal from the environment itself, which then allows the model to understand things like having a native world model of the environment, understanding what to expect because it has a, uh, a likelihood model of the tokens that the environment itself will generate.

And these are the sorts of things that, uh, in many cases allow the model itself to kind of more adeptly navigate the world and not just like refine its skill, but like get new information into its weights over time as well.

Um, and so, all of this is in spirit of making post-training easier, making continual learning easier, giving people the ability to, uh, create agents and not worry too much about having to, to fuss with the research pieces and fine-tune all the, the small details.

Today we still do, but we're kind of seeing paths forward of how we start automating this, uh, more and more by like having environments as the anchor, which we can then spend compute on refining. Uh, we can kind of use compute to mine, uh, the data we have from the real world to refine the, the signals where the humans are kind of just, uh, in the same way that with coding agents, we're kind of going to higher levels of abstraction.

We can do this with environment and reward design as well. Um, and all of this is what allows us to ultimately close the loop where models are then able to, uh, stay within the guardrails we give them. They go find the issues in production, and then they turn these back into new tasks that can then be trained on for getting better in the real world.

Um, we work hands-on with startup enterprises to help them train their models. We are also hiring quite a lot. Um, if you want to get in touch for either of these, find me after the talk. Thanks a bunch.