# The Base Model Is Dead — Varun Singh, Arcee AI AI Engineer · 2026-07-31 Varun Singh, pre-training lead at Arcee AI, argues the base model is dead: it no longer just mirrors web text but must carry the prior that reinforcement learning builds on. He traces how web text fell from 85% of GPT-3's mix to 15% in MAI Thinking 1, with code and STEM dominating, and how Nemotron 3 Ultra pulls SFT-style Q&A data back into pre-training. Synthetic rephrasing, as used in Arcee's Trinity Large and Kimi K2, upsamples information to teach task shapes early. He warns that without post-training-flavored data early, MoE load balancing can break when SFT distributions differ, citing MAI's cranking of the balancing coefficient. He frames training as supervised learning vs RL, noting RL compute now rivals or exceeds pre-training, as with Compose 2.5, and argues the base model's job is to provide atomic skills for RL to compose. ## Questions this episode answers ### What does Varun Singh mean by "the base model is dead"? Varun Singh argues the traditional base model, trained mostly on web text to mirror human knowledge, is outdated. GPT-3 used 85% web data, while MAI Thinking 1 cut it to 15%. With reasoning models like o1, RL now dramatically improves performance, so base models must incorporate instruction and synthetic data early to build priors for reasoning and agentic behavior, as seen in Nemotron 3 Ultra pulling SFT data into pre-training. [3:51](https://aiengineer.podhood.com/d56d90bb-4b31-442d-aaab-ad5aaf449f77?t=231000) ### Why is synthetic data being used in pre-training now? Synthetic data, often through rephrasing, up-samples seed data so models see information in multiple ways, yielding cleaner, higher-quality tokens shaped like instruct or agentic tasks. Varun Singh notes this early exposure helps models learn task representations from the start. Large models like Kimi K2 have used synthetic data across the entire pre-training dataset to improve quality and relevance. [9:42](https://aiengineer.podhood.com/d56d90bb-4b31-442d-aaab-ad5aaf449f77?t=582000) ### How has the role of the base model changed with the rise of reinforcement learning? With reasoning models like o1 and R1, RL is no longer a minor refinement but can dramatically boost capabilities. Varun Singh points out that RL now accounts for a huge share of compute—Cursor allocated far more RL compute than supervised learning. Consequently, the base model's job shifts from storing world knowledge to providing atomic skills that RL can compose, making supervised pre-training a preparation stage for RL. [13:20](https://aiengineer.podhood.com/d56d90bb-4b31-442d-aaab-ad5aaf449f77?t=800000) ## Key moments - **[0:00] Old Mirror** - [0:13] "The base model is dead, but not really." - [2:16] GPT-3's training data was 85% web text and Wikipedia. - **[3:51] Reasoning Shift** - [4:11] OpenAI's o1 and DeepSeek R1 made RL the main driver of model capabilities, not just a cherry on top. - **[6:18] Data Mix Shakeup** - [6:49] MAI Thinking 1 reduced web text to just 15% of its training data. - [8:05] Nemotron 3 Ultra pulled supervised fine-tuning chat data back into the pre-training phase. - **[9:42] Synthetic Data** - [9:42] Synthetic data upsampling via rephrasing exposes language models to the same information in multiple ways. - [11:56] MAI Thinking 1 fixed MoE load imbalance in SFT by significantly increasing the load balancing coefficient. - **[13:30] RL Dominance** - [14:45] Base models must acquire atomic skills that reinforcement learning can later compose during exploration. - **[15:55] Redefined Base** - [16:33] Base models have shifted from encoding general human knowledge to providing priors for reasoning and agentic behavior. ## Speakers - **Varun Singh** (guest) ## Topics Training Data Curation, Reinforcement Learning ## Mentioned Arcee AI (company), DeepSeek (company), OpenAI (company), Xiaomi (company), Zyphra (company), AlphaGo (product), Claude Code (product), Cursor (product), GPT-3 (product), Kimi K2 (product), Llama (product), MAI Thinking (product), Nemotron 3 Ultra (product), O1 (product), R1 (product), Trinity Large (product), ZyA1 (product) ## Transcript ### Old Mirror **Varun Singh** [0:13] Hi everyone, my name is Varun, I'm the pre-training lead at Arcee AI. And the talk I'm going to be giving today is called "The Base Model Is Dead," but not really. Um, the idea of the base model that we have kind of is like built on this idea of, like, training on super large-scale web text and the base model kind of being a reflection of, like, the whole knowledge of, like, the human internet. Um, you can see in, like, these I've taken these from a bunch of different papers on, like, the entire LLM training process. Um, our own model, Arcee Trinity Large Thinking, the process looked kind of like the simplified diagram on the left. I've taken the top one from GLM 4.5, the bottom one from GLM 5. Um, all these have a pre-training phase. And, uh, pre-training is like the stage where the model accumulates world knowledge, builds useful representations, all through next token prediction on, um, web text. Um, I've got a simplified, uh, transformer diagram, um, decoder-only transformer, and, um, a screenshot from the GPT-3 paper that talks about how, um, language models can, uh, learn how to do in-context learning through, um, unsupervised or self-supervised or, um, some some would even just call it supervised, um, learning on, uh, through next token prediction. Um, the way that, uh, older base models were trained was, like I said, mostly on, uh, things that reflected the entirety of human knowledge. Uh, so Common Crawl, uh, which is like a commonly available web scrape, uh, made up most of the training data set for GPT-3. Um, Webtex2, another web scrape data set. Um, some sources from, like, books as well. Um, and Wikipedia as, like, a high-quality, uh, representation of human knowledge. Um, you can see that, uh, web text alone here, including, like, Wikipedia, makes up, like, roughly 85% of the whole training mix. Um, looking at, uh, the bottom with LLaMA 3, um, web text still kind of makes up a majority of, of the, um, model's training data, with, like, 50% of the tokens corresponding to general knowledge. Back then, post-training was mostly shaping the model to use, um, the parts to, like, surface, uh, the knowledge that it accumulates in accumulated in pre-training, uh, in, like, a chat interface. So mostly allowing the model to adapt to a chat template, to the question-answer format, um, and be useful in an interaction that way. Um, RL was mostly just a cherry on top, um, shaping the, you know, flavor of the interactions more than conferring extra, um, knowledge or quality onto the base model itself. Now, in, in, like, this realm of, of, of how language models used to be, pre-training, um, and the base model kind of defined how good you were able to get a, a model. Um, it was, like, the bulk of the compute, uh, budget, and it was, um, the, the, like, core of the training process. Um, however, uh, this kind of changed a lot last year when, um, OpenAI I guess 2024, actually. ### Reasoning Shift **Varun Singh** [4:11] OpenAI released o1, um, pioneering reasoning models, and DeepSeek, uh, also released R1 in January 2025, um, allowing the whole world to know how to build these types of language models. And now we have this new, uh, new, um, use for reinforcement learning, which is no longer a cherry on top, but it can dramatically improve the performance of, of the model on various different tasks. Um, the, the famous graphs from o1 there talking about, uh, AIMEE performance, um, competitive math contest. Um, and then even later in the year, we, we saw Claude Code, um, coming to being as a way for developers to easily, um, kind of use language models in a in a terminal to build out applications as models got stronger and stronger on things like function calling. Um, and then people realized you could, uh, you know, RL this end-to-end. Um, and now models could learn how to interact with software environments and build software and, uh, perform really useful work. And so, um, the question then becomes, like, is your standard base model still, uh, what the best, uh, what would be the best, um, prior for the for the this large-scale reinforcement learning phase that, uh, ReasonOS and, um, Agentic models now use. And we can kind of see, like, in the in a few open research papers what the trend is where the trend is going. Um, and interestingly enough, the, um, it seems like not super clear yet. Um, I have my opinions on, like, synthetic data being the way forward, but I'm I've got, like, two contrasting, uh, perspectives here kind of in the slide. ### Data Mix Shakeup **Varun Singh** [6:18] The top image is from the MAI, uh, Thinking 1 paper, where they make a make a really large point to not use any synthetic data or any, uh, data from any other language model. Um, and they really try to, you know, filter their web scripts for this as well, um, in order of, uh, to kind of adhere to, like, the previous, um, paradigm of, like, uh, using human knowledge as a way to bootstrap, uh, model representations and, like, uh, capabilities. Um, but I would say that this is also like, even, even though they stuck with no synthetic data, the data mix that they've chosen here is still, um, totally different from what you'd expect in, like, a, um, in a in a classical, uh, language model. And, I mean, the, the main reason for that is that web text, which used to make, like, up to 85% of the training data in GPT-3, is now all the way down at 15%. And that I mean, that just shows that, uh, the value of, like, web text contributing to, like, the downstream, um, performance of, like, the models on RL and stuff is kind of, uh, it-it's still important, but taking a backseat to things like code and STEM abilities as the models kind of gain more real-world use cases related to, to those. Um, the other approach, uh, is to bring, um, post-training data and large-scale synthetic data back, uh, through pull it back through the process into the pre-training phase. Um, the, the bottom chart I've taken from Nemotron 3 Ultra, um, where they reveal their, um, the data recipe. And I'm not sure how readable it is, but these top three, um, on the left, uh, pie chart, the top three on the kind ofright,right side of it, uh, they're all labeled SFT, uh, with, with SFT as a prefix. And that's the type of question-and-answer kind of chat data set that you'd expect to see only in post-training. But by pulling it back into the process, they're able to, like, get the model to learn, um, the shape of these conversations and what kind of tasks they might be expected to do downstream, um, from the very beginning of the pre-training process. Um, and, uh, this follows, like, uh, a similar, um, trend in, like, diminishing, uh, the amount of web text used in the model. Um, yeah. Um, it's really interesting to see the, the Nemotron series lean so heavily into synthetic data, but, uh, MAI Thinking 1 kind of lean in the opposite direction. Um, I've, I've, I've just got this slide here as, like, a con uh, easy contrast that, uh, people can see on the, like, amount of web text and the amount of books and stuff, um, being less of a percentage here. And GPT-3 didn't even used to have any specific code data sets, but now code is, like, the dominating, um, data data subset that we have in, uh, pre-training recipes. Um, so I've mentioned synthetic data, but what is actually, uh, like, how is synthetic data used? There's a lot of, uh, talk around synthetic data that, you know, blindly tossing it into a model can cause the model to collapse and, uh, and performance to tank. ### Synthetic Data **Varun Singh** [9:58] But there's been a lot of work and, uh, even a, like, a large-scale, you know, example of this, uh, turning out really well. Um, so in our own in our in our own, uh, model, Trinity Large, we, uh, had a large amount of, uh, web, uh, web-scale synthetic data, um, mostly through rephrasing, where you take a seed data item and you sort of upsample it in the mix by, uh, generating synthetic rephrases of the same information. So the model sees the same information in, like, multiple ways. Um, the bottom two, uh, screenshots are from Kimi K2, um, an even larger-scale model that, uh, broadly used this, um, across the whole pre-training data set. Um, the top, uh, topright, um, screenshot is from a paper that, uh, resulted in the data set Swallow Code and Swallow Math, which are early examples of this. But the trend seems to be that, um, synthetic data not only allows you to get more and more tokens, uh, but also, you know, clean up tokens, get higher-quality tokens, and have, um, tokens that are shaped more like instruct or agentic tasks all the way back in pre-training and, uh, allowing the model to, like, learn those task representations from the very beginning. Um, another reason that it's, uh, beneficial to add post-training data early in pre-training is now with MoEs. Um, one of the biggest, uh, pain points in training in MoE is dealing with load balancing, um, where experts can specialize over the course of training and, um, they uh, and load balancing objectives aim to, uh, achieve broadly equal utilization of the experts, um, in a given batch or sequence, depending on the objective. Um, without, uh, post-training data in, uh, early in pre-training and with an MoE, one really easy pitfall, uh, that we can fall into is this is kind of illustrated in the MAI Thinking 1 report, which is that the data distribution that the model sees, uh, in post-training is really, really different, uh, compared to what it sees in pre-training. And this can cause massive imbalances. And, um, MAI overcame it by, uh, really cranking up the load balancing coefficient during the SFT stages. Um, but, I mean, ideally, you don't want to mess with the balance that far into training, and the model should learn stable representations from really early on. Another, uh, interesting thing that, um, is changing in base models now is that you there's this whole advent of mid-training, uh, which is exposing the model to the distribution that it would see during post-training and RL and add a longer context so for things like agentic traces to be allowed into the mix and, uh, to kind of help prepare the model that way. Um, a lot of models, though, are training with much longer context in pre-training, and there's no reason that these data sets can't be pulled back into the mix to allow for more stable representations from the very beginning. Um, I think, uh, a better a better way to understand the current, uh, phase of LLM training, uh, isn't so much, like, pre-training, mid-training, post-training, RL. Uh, it all gets a bit muddy that way. But there's two broad paradigms that are, like that really help build, uh, LLM today, and that's supervised learning, uh, through next token prediction, and RL. ### RL Dominance **Varun Singh** [13:45] Um, and RL is becoming more and more important. Um, the bottom, uh, thing is a screenshot from an interview, um, with, uh, the head of, uh, Xiaomi's Mimo Labs, uh, where she talks about how they allocate compute, uh, between research, pre-training, and post-training. And pre-training and post-training in the final model have a roughly equal compute allocation. Um, Compose at 2.5 takes us to the extreme where, um, Cursor really, uh, sank much, much more RL compute into the model than the model had ever seen in supervised learning. But with RL dominating such a massive amount of the compute budget, uh, it makes sense to view supervised learning as a way specifically to prepare the model for, uh, to build useful representations for R for RL instead of it being the bulk of, like, um, what the model would be used for, like, uh, previously. Um, there's been some some work on, uh, how supervised learning affects RL. Um, I really like this one paper where the main takeaways are basically that, uh, the base model needs to have some exposure to, like, uh, like the atomic skills that it would need to compose during RL, and, um, the model can learn to extrapolate from there during RL given, like, the environment has a sufficient level of difficulty. Um, I had to put in the classic AlphaGo graph there where RL eventually overtakes supervised learning. Uh, it's unclear if we'll see something like that for language models because, of course, you know, uh, human language is such an insane distribution to have to, like, learn through reinforcement learning alone. Um, but it's definitely possible that we might see diminished supervised learning and more and more RL, uh, which makes this kind of thinking of a base model as, um, atomic skills for RL more and more valuable. Another thing that some labs are doing, um, is kind of introducing novel data, uh, during supervised learning. And by novel, I mean something that the model really wouldn't have seen the shape of before. Uh, easy example is, you know, reasoning traces. They don't really look like, uh, like a ton of what, um, humans output. And, um, another interesting thing is, like, training for test-time compute, um, schemes, um, by kind of warming the model up to them during SFT, um, or even pre-training itself. ### Redefined Base **Varun Singh** [16:20] Um, these screenshots were taken from, uh, Zyphra's ZyA1 paper. Um, and I think that they're, uh, very interesting ways of thinking about how data can affect, um, like, uh, the skills needed to explore well in RL. Um, in conclusion, um, base models have kind of moved from general, uh, human knowledge and world priors to reasoning and agentic behavior priors. Um, of course, that's kind of, uh, reductive in the in a way that reasoning reasoners and agents are, like, the main way we see, uh, bots, uh, the main way we see these chatbots used now. But if a new paradigm were to take off, like, a new way of interacting with the models, um, it makes sense to, like, think of a base model as building a prior for that instead of, um, just building off, like, a massive, uh, scrape of web text. And yeah, thanks for thanks for listening. Thanks for your time. And yeah. --- This library is powered by PodHood (https://podhood.com), the podcast website platform.