Data and Environment Curation for Post-Training LLMs — Mahesh Sathiamoorthy, Bespoke Labs

Hey everyone. Um, today I'll be talking about data and environment curation for post-training LLMs, and I'm Mahesh Sathiamoorthy. I'm co-founder and CEO at Bespoke Labs, and previously I was a researcher and engineer at Google DeepMind. So, very briefly, I will tell you a little bit about Bespoke, and after that the talk will be mostly around open-source work we have done.

So, Bespoke is an applied data research lab with a mission to help enterprises and frontier labs access high-quality data and RL environments for their post-training needs. So, very briefly, what we do and what we have done is that last year we put out something called Curator, which is a tool for curating synthetic data for post-training with, basically, SFT.

Andright after that, actually, DeepSeek landed and we started an effort to curate reasoning data, and that's how we started something called Bespoke Stratos, which eventually formed into the project called OpenThoughts, which some of you hopefully know about. And we have also been co-contributors to Terminal Bench.

Um, you know, these days we do a lot of research and build and ship RL environments, so I was actually looking forward to the previous talk from Nick, who is also, you know, doing something similar. And the other thing we do is we do a lot of post-training and help enterprises to get their own custom models,right?

That's the name of, that's how we ended up with the Bespoke title for the company.

The other thing I want to kind of mention is there is this, you know, in our industry there are a lot of people who create data, create RL environments, and then there are the researchers who consume this. But I feel like there is this slight mismatch, and it's kind of beneficial for someone to kind of go do both at the same time.

And in fact, as you're curating data, you want to put yourself in the shoes of the researcher to see what does it take to, you know, actually move the metrics on the models. So that's one of the motivations of how we kind of think about it.

The other thing I want to kind of talk about is, you know, how, you know, AI has evolved,right? So early on we used to think about and evaluate models on what they know. For example, this was a very popular benchmark on testing LLMs on various kinds of STEM, humanities, and all that knowledge.

And these days we have all these benchmarks that test

how agents are able to do things. We have moved on from knowing to doing,right? So that's the idea of agents, obviously. And one of the key principles, or one of the key things about agents, is that they are autonomous.

And there are, as I was saying, there are many benchmarks, including Swebench, Terminal Bench, and so on. But ultimately, for many people, what they care about is, are these agents autonomous for long durations of time? Nick had, sorry, Ross had a great talk on long horizon,right?

So that's the goal, is eventually we make these agents autonomous for maybe a few hours, or, you know, a few days, or a few weeks. And what is it that's blocking the autonomy of agents? It's basically reliability,right? So at some point something falls apart.

Like either they called the wrong tool, or they made a mistake, and you know, what not,right? And what's one lever to improve reliability? There are, of course, many. Obviously, you can prompt your way to improving the agent's reliability, or you can update the harness, you know, the tools and whatnot.

But post-training is a very powerful tool to improve reliability, or maybe even pre-train good models,right? So if you think of Frontier Labs, this is one of their primary mechanisms of improving agents over to get better capabilities in

various domains, or, you know, for better benchmark numbers, or better autonomy for longer and longer durations. And for post-training, one of the popular techniques, as you know, is reinforcement learning. And that's kind of something, you know, a lot of you are excited about, is the notion of RL environments.

But ultimately, for post-training, be it SFT or reinforcement learning, data is the bottleneck,right? So when I talk about data, RLNs are also something I'm calling it as data. It's just the data is now in a very different shape.

Again, here, you know, compute is kind of well-defined. Models, you know, good sort of models exist, and the infrastructure to post-train, for example, there are various providers like Fireworks, Tinker, or SlimeWorld, and whatnot. So all of those are somewhat well-defined.

Most of the places where people struggle, especially enterprises, is that they don't have access to good quality data and RLNs. And this obviously also applies to Frontier Labs, where they have all this infra set up and they are, you know, needing good quality RLNs,right?

Beyond, so that's one of the, this is kind of how we are thinking about why to invest time in, you know, doing data research and RLN research. And as a side note, one of the other, there are many other benefits of post-training.

For example, you can reduce latency or improve cost, throughput, and whatnot. And I'll give one concrete example of a post-training work we did with one of the enterprises. So in this talk, I will mostly cover some of the work we have done in the open-source community.

So we did some work on curating reasoning data for reasoning models, and for, you know, curating trajectories and environments for agents. And recently we had an engagement with post-training, which I'll very briefly talk about, and some tools on data curation.

So OpenThoughts is

a reasoning dataset as well as a paper,right? So we started this effort last year. As I was saying, this, after DeepSeek came out, we realized that there is a lack of very high-quality reasoning data in the community. Obviously, the labs have access to good data, but outside we didn't have access to data,right?

So we at Bespoke started this effort called Bespoke Stratos, and then we realized that this is actually quite useful. So we joined together with various folks in Stanford, UC Berkeley, UW, and so on, to create this consortium called OpenThoughts.

And we did a lot of work on basically identifying the curation recipe. And we also published this as a paper in iCLEAR of this year. And this is the main figure of the paper. So what it shows is, like, we figured out a curation recipe, and it shows the scaling law,right?

So again, this is last year when Amy and LiveCode mentioned these were some of the popular benchmarks. What we showed is that with this recipe, if you keep, you know, scaling up the dataset size, the, you know, it's a scalable recipe,right?

The metrics also improve. It's actually very widely used as well. For example, this is Microsoft CSO tweeting about the work, and this Alex is my co-founder. He's a chief scientist and also a professor at UC Berkeley. And this is John Schulman talking about OpenThoughts, that he has, he and his colleagues have been using it internally at Thinking Machines,right?

And some of their blog posts also reference this. So I'll talk about how we did the curation for OpenThoughts. This is the pipeline that we used. So you start with curate, you start with a bunch of source questions,right?

So there are various datasets out there that have the prompt response, and we choose with the prompts. We start with the prompts. These are various sources we have. And then

if you look at the paper, so if you look at this graph, for any given data point, say if there are 10,000 samples that you want, the question is then how do you choose the questions from all these different datasets so that you have 10,000 for the data point?

So then there is the aspect around how do you mix these questions. So you can use various methods. So the paper talks about, for example, using LLMs to check for whether this is a good question, hardness of a question, and so on.

And then you want to filter questions and generate the answers. Again, this is all, like, driven by LLMs,right? So this is the curation recipe we did for creating this reasoning dataset. And the answer generation is using teacher models.

So you can take other reasoning data, reasoning models such as DeepSeek or Qwen-based models, or even Gemini and whatnot. And then you can also filter the answers once you have the answers for these questions. And then you can also, you know, given a question, generate multiple answers or a single answer.

So these are various knobs in the curation recipe. And the systematic way of doing this is, like, you run ablations and figure out which, you know, in each of these stages what works, and you kind of proceed to the next.

So after doing all of this, you get the final recipe,right? So this, you can read this paper. It has lots and lots of, you know, information about how we did the curation. But here are some of the learnings that, you know, some of them are quite counterintuitive.

And some of this was also covered in last year's AI Engineer Conference. For example, sampling multiple answers per question works pretty well. This is something that we, it's kind of counterintuitive. So as an example, something else we could have done is we could have had more, many more questions, and then just answered them exactly once, versus taking one question and answering them 16 times.

I think the reasoning is probably that it gives, like, a variety of how reasoning is done. So during fine-tuning, we also use the reasoning traces,right? So I think the diversity helps there. And the other thing we saw is, like, the stronger teachers are not always the best, stronger models are not always the better teachers.

And there were a few other counterintuitive aspects around, like, you know, synthetic question generation or question answering working, whereas answer filtering and other aspects not working very well. And after the OpenThoughts work, which was around data curation for reasoning models, such as, you know, DeepSeek kind of models, we moved on to OpenThoughts agents, which is very similar.

But how do you curate these, the data and RL environments for training agents now,right? Not reasoning models. We have a very similar figure here. Again, we want to establish scaling laws. So as you increase the dataset size, we want to make sure that the curation recipe actually works.

And again, I'm not going to go into details here, but very similarly, there are various ways of choosing different sources. For example, Stack Exchange and whatnot. How do you mix the tasks? How do you filter? Generating the rollouts, choosing the teacher, and so on.

And again, these are some of the lessons, learnings.

As an example, even here we saw that stronger models are not necessarily the best teachers,right? So we found out some of the, I think, Qwen models were better than, for example,

Claude models, I think. And sampling multiple answers, again, helped in this case. Synthetic rewriting and task augmentation is something we thought would work, but it didn't very work, work very well. And the other thing is, like, in this whole process of building this OpenThoughts agent, SFT still contributed a lot to the gains.

RL was kind of, you know, it's very compute-intensive. And for the last few percentages, it really helped. But, you know, in many of the situations, for example, in enterprises, SFT actually works pretty well,right? And here is one concrete example I wanted to share on

actually deploying something to production,right, by post-training. So we have seen a lot of people talk about post-training, but in enterprise settings, we haven't seen a lot of successes, at least haven't seen that. Here is a very concrete example of, with Intuit, there is this app called Credit Karma, which if you install, there is a place where you can, the app gives you a reasoning as to why a credit card has been recommended.

And this you can prompt a model to do this, but one of the reasons, one of the places where it fails is that the, you know, it's not always compliant. So you have to have a long list of rules to make sure the responses are compliant, and that actually blows up the latency.

So answer here is, like, you want to curate data and post-train,right? Seems kind of straightforward, but one of the things that we ran into is the dataset can be quite imbalanced, and lots and lots of places, for example, you will have 0% APR, and the model after fine-tuning can kind of hallucinate these numbers.

So this again kind of ties back to what Ross talked about some time back with respect to tags. And we kind of, we created this specific curation recipe where instead of just having these questions, the prompt response pairs in plain language, we added these tags, which helped the model to focus on, you know, the kind of form rather than the specific numbers itself.

And that gave a big boost. And we saw that

the overall, the compliance metrics improved, the latency improved, the throughput improved, and eventually, you know, they are able to own the model,right? As frontier models improve, they don't need to kind of go and

update it. And also as we see now, the frontier models are also getting more and more expensive. And, you know, this kind of gives them a very good way for owning the model and also lowering the costs. I think with that, I want to briefly touch upon,

you know, Curator, the tooling that we had built last year, which is for curating reasoning data. So

what it does is you can basically, you know, specify the, you can either go with, say, a Hugging Face dataset where you have various prompts, or in many situations you may have collected logs and you want to get the responses and fine-tune a model.

So this Curator kind of makes it pretty easy to do that. And it comes with the integration with, you know, Tinker and Fireworks. And this is again the tool that we used originally for curating OpenThoughts. And here is a very, very detailed diagram of what we are building today.

But this again connects back to what Ross was talking about, where he was talking about algorithms, environments, and compute,right? So it feels like, you know, we are kind of converging on something very similar. So if you think about the stack that is needed to, say, not just curate these RL environments, but to post-train models, one of the things you need is obviously handle on, like, how do you build these RL environments?

How do you measure the quality? How do you track the different versions and so on? So that's one of the layers. And below that, you want various infrastructure to

use sandboxes,right? To spin up the rollouts, to spin up the sandboxes to generate rollouts. And especially if you have long horizon rollouts, then maybe at some points you need to do a checkpointing, and then you need to be able to snapshot or roll back to something else,right?

So that's the other, the lower level, you know, compute and orchestration. And at the top, I have been giving examples on post-training. So there is all this layer around, like, how do you do SFT? How do you do RL?

And so on. But there is also this method called JEPA, which is around, which is on prompt optimization. I don't know if you guys have heard of it, but you can use LLMs itself to kind of optimize the prompts based on reflection.

So that also works pretty well for updating the system prompts and also the harnesses. So this is kind of, I feel like, you know, the new architecture or the new reference stack for how, at least we are building and how many others are building the stack on how to build the RLNs and then also post-train agents.

I think with that, I'll end the talk and, you know, happy to take questions offline.