# The Messy Reality of Scale: Synthetic Data and Pre-Training — Marah Abdin & Robert McHardy, poolside AI Engineer · 2026-07-26 Marah Abdin and Robert McHardy from poolside detail their synthetic data pipeline and pre-training tribulations at scale, culminating in a new 118-billion-parameter model for agentic coding that outperforms competitors. Marah describes using synthetic data to rephrase content and fill gaps, with a configurable pipeline (Hive) involving agents, orchestrators, and supervisors, covering rephrasing, multistage workflows, cross-domain porting, and multi-turn chats. Robert recounts failures like broken GPUs causing data corruption, a BF16 accumulation bug that stalled training, and a race condition in FP8 kernels silently corrupting 0.5% of gradients, all caught by model replica hashing. Their Laguna S model (118B total, 8B active) trained on 30 trillion tokens across 4,000 GPUs beats GLM 4.5 Air and other models on coding benchmarks like BigCodeBench and SpeedBench agentless, while remaining competitive on general knowledge. ## Questions this episode answers ### How does poolside's synthetic data pipeline work for pre-training? Marah Abdin explains poolside's pipeline has six modular components: seeds, metadata, generator function, and supplementary filters. They use rephrasing to reduce token repetition, multistage pipelines for complex generation (e.g., building novels chapter by chapter), cross-domain porting (converting math problems to code), and multi-turn roles with agents. An orchestrator dynamically manages agents, and a supervisor polices the entire process. They settled on a 13% synthetic data mix and built a 6 trillion token corpus. [1:50](https://aiengineer.podhood.com/d1d72449-6d87-47bc-978a-53057827386c?t=110000) ### What are the early results of poolside's upcoming 118B parameter model Laguna S? Robert McHardy previewed Laguna S, a 118B total parameter model with 8B active parameters trained on 30T tokens on 4,000 GPUs. On coding benchmarks like LiveCodeBench and BigCodeBench, it outperforms their previous XS2 and GLM 4.5 Air. It is competitive on general evals like BBH but lags on knowledge tasks like MMLU Pro due to a deliberate data focus on agentic coding. Post-training was still ongoing. [15:07](https://aiengineer.podhood.com/d1d72449-6d87-47bc-978a-53057827386c?t=907000) ### How does poolside ensure training correctness at scale using model replica hashes? Robert McHardy describes that they run multiple replicas of the same model and compute a hash over weights, which should be identical. Any mismatch triggers a training crash. This caught a broken GPU causing silent data corruption and a numerical precision bug. However, it misses race conditions because replicas don't process the same data in real runs, so they're developing a hash checker for forward/backward passes. [10:01](https://aiengineer.podhood.com/d1d72449-6d87-47bc-978a-53057827386c?t=601000) ## Key moments - **[0:00] Intro** - [0:33] Poolside released open weight models Laguna M and Laguna XS on Hugging Face, says Marah Abdin - **[1:50] Why Synthetic Data** - [2:12] Synthetic data complements organic data by exposing implicit rationale, planning, and structure, explains Marah Abdin - **[3:49] Rephrasing** - [3:49] Marah Abdin: Poolside uses multi-mode rephrasing of high-quality seeds to reduce token repetition and saturation during pre-training - **[5:01] Pipeline Components** - **[6:37] Pipeline Shapes** - [6:37] Marah Abdin describes a multistage pipeline that generates a novel step-by-step: first setting, characters, plot, then chapters - **[7:54] Hive** - [9:03] Poolside pre-training runs periodic model weight hash checks; any mismatch across replicas crashes training, says Robert McHardy - **[9:07] Trust Nothing** - **[10:39] Scale Failures** - [10:41] Robert McHardy recounts that a single broken GPU caused silent data corruption, spiky loss curves, and huge gradient norms - [11:56] Robert McHardy: BF16 tensor-parallel accumulation caused activations to grow, flattening loss at 50k steps; switching to FP32 restored convergence - **[12:45] Laguna S Scaling** - **[13:59] Race Condition** - [13:59] Robert McHardy reveals a race condition in DeepGem FP8 kernels that silently corrupted 0.5% of gradients, a blind spot for hash checks - **[15:15] Early Results** - [15:29] Robert McHardy shows Laguna S base model beating GLM 4.5 Air, Nemo 23Super, and DeepSeek V4 Flash Max on coding evals - **[16:50] Outro** - [16:50] Robert McHardy confirms the training recipe held as it scaled from XS to Laguna S, validating data, architecture, and numerical fixes ## Speakers - **Marah Abdin** (guest) - **Robert McHardy** (guest) ## Topics Training Data Curation, Coding Agents, AI Infrastructure ## Mentioned Hugging Face (company), Poolside (company), DeepGem (product), DeepSeek V4 Flash Max (product), GLM 4.5 Air (product), Laguna M (product), Laguna S (product), Laguna XS (product), Nemo 23 Super (product) ## Transcript ### Intro **Marah Abdin** [0:13] Hi, everybody. Thanks for coming to our talk. My name is Marah Abdin. I'm from poolside, I'm the synthetic lead for our data team, and today my colleague Robert and I will be talking a little bit about some of the challenges that we've seen as we scale our models over here. Particularly, if you haven't heard, we've switched from recently from releasing our models towards enterprise to also releasing towards, you know, everybody. We've actually put out two open weight models. There were Hugging Face a few weeks ago, we have Laguna M, Laguna XS. We also put out a tech report, which has a ton of detail if you're interested. As you can see here by the Laguna M point one and XS point two, this is actually because we have switched out quite a few things between those two models, and so a big flavor of this talk is going to be about kind of how did we transition from one to two. And in fact, we've continued to do so, and now we actually have a newer version, and Robert will give a sneak peek about a soon-to-be-released model. Okay, so I will be particularly talking about the synthetic data part of things. So there's three things that we did on the data side to kind of resolve some of the issues that we've seen with scale. One is that we implemented an automixer that basically just gives us the chance to do a cheaper sweep on clusters of our sets before moving on to more expensive experiments, and then we improved, just rethought our sampling of web data for higher recall. ### Why Synthetic Data **Marah Abdin** [1:50] And then the third one is that we relied a lot more on synthetic data, in a few forms, which I will go into. Okay, so before we kind of go into what does that mean and what have we done, et cetera, why would we kind of, it's sometimes it's fair at least to ask why synthetic data, and the thing is that at least at poolside we don't see it as a way to replace organic data. I don't see it so in the current state of the world at least, but it is a way to kind of complement it. And the thing is that organic data has a lot in it that is basically kind of implicitly hidden. A lot of things that could teach the model are not very presented in the most optimal way sometimes, and so synthetic data gives us a track to extract some of those features and then project them on some new planes. And this is how we get to expose implicit rationale, implicit planning, implicit structure, and a way for us to fill gaps and regularize not only how we present the tokens, but also how we are teaching the model. For XS point two in particular, we settled on 13% of the mix. This is only pre-training stages before post-training, and since then we've just been continuously generating more data in a bunch of directions, and now we have a 6 trillion token corpus that's continuously growing. Okay, so yeah, so kind of what I just said is that we saw some, I guess, limitations switching from Laguna M to point one to our point two models, and so one of those things is that we basically started on data. This is really not a crazy kind of problem. We very intuitively started from a place on a smaller scale where we were basically focusing on quality versus quantity, maybe a little too much, because eventually when we started scaling our models, we had to scale our training budget, and with that came some limitations because we started hitting repetition, like non-optimal repetition on some of our higher quality data, which saturated the model a little too early. ### Rephrasing **Marah Abdin** [3:49] So one of the ways that we've, particularly for this token uniqueness problem, we relied on, which is a very common form of synthetic data, rephrasing, which you know, you've just heard it like Beyond Web, for example. It's become pretty trendy these days. And you can see here that, you know, this is a, take this ablation, this is an ablation result, so take the numbers with a grain of salt, but what persists fairly consistently is the diff between using the orange would be just the seeds with repetition, and then the green would be replacing some of those repeated tokens with higher, like with multi mode rewrites, or at least, yeah, all of them or at least reducing the repetition. And so for rephrasing in particular, we did do the, you know, what everyone's doing with the, you know, generic kind of multi mode, very scalable pipeline, but we also took it a step far, and we did two other specialized pipelines. One to go from raw code to code and text, and one to go specifically for stem data, just because this is a very cheap, scalable pipeline. So it kind of, you kind of have to rely very heavily on the seed, and we push a little further on that for the stem documents. Okay, so, okay, so if you kind of think of everything as a kind of a modular way, you can think of every synthetic data pipeline as composed of the same six components. And so you have your seeds, your primary inputs, your metadata, your secondary inputs, your generator function, which can be an agent with tools or an LLM, you know, with some prompt template, and then some supplementary functions like filters and validators and so on. ### Pipeline Components **Marah Abdin** [5:26] And really you can compose just about all pipelines from like very simple to very expensive pipelines like this. And on that note, and kind of we have covered quite a bit of wide scope on the axis of complexity, and you kind of can think of it if at one end you have like the cheap, scalable pipelines that have used smaller models and can get away with it because they're seed heavy, examples of phrasing. And then on the other end you have more complex pipelines with a little more orchestration in the workflows. This is reserved when we're building on something that's worth it, educational data, but really it's, this is how we're not blocked or limited by whatever teacher model can do, and this is how we can be ambitious in our synthetic data, because the rule of thumb is if a task is too hard for your model, then your model will start to fall on its bias, lose correctness, lose diversity, so break down the task, make it simpler. And yeah, I will give some examples of kind of shapes rather than just like something more concrete about how did we use this modularity, and one shape is the form rewriting, it's just rephrasing, we already talked about this. Multistage pipelines, multistage workflows, basically this is what I also just said, you take a step and you break it down into multiple steps, you can aggregate the process and slowly build up the generation. An example of this where if you wanted to generate a novel, for example, you could generate one chapter at a time, but you could also, you know, take it a little slowly, one first like generate, you know, the setting, the character names, the character styles, the plot, you know, some twists, and then from there go into generating chapters one by one, you will absolutely get a better novel. ### Pipeline Shapes **Marah Abdin** [7:14] Okay, third is cross-domain porting, which really is just like moving from one mode to another. An example would be like translating code. Another example would be something we did, which is take our math problems and convert them to code. The last one is multi-term role. And by that all I mean is that instead of having kind of a very one, like very senior or linear, or even non-linear kind of view of things, you have more of an iteration. This encapsulates pretty much everything. And by that, like an example of that would be multi-turn chats when you have two agents talking to each other, or a task evolution pipeline where, you know, you have a judge and an evolver going back and forth for some K amount of time, and so on. ### Hive **Marah Abdin** [7:54] Okay, so lastly, forwarding off to Robert, I do want to kind of just mention that we, because of the modularity of the way we think about this, we can implement an infrastructure that's pretty configurable. So this is how we present Hive. Hive is basically a way for us to easily construct generations where now you have a queue of agents that you define. Each one has, you know, its prompt, its parameters, its model, et cetera, inputs, outputs, but it also has when you can configure when it enters the queue, when it exits, how many frequencies come in, and then we have orchestration in the middle between agents, and with this orchestrators are really useful because they give you more flexibility and kind of presenting a hierarchy between the LLMs that are generating is very, this is how you police them basically, but also give them some form of creativity and dynamically change the instructions per the next agent, or choose which agent goes next, which agent skipped, and so on. Lastly, you have the supervisor, which basically is someone who polices the orchestrator and has more of a global view. Cool, okay, that's it for me on synthetic data. Hope you learned something interesting. Handing over to Robert for pre-trainings. **Robert McHardy** [9:03] Allright, thank you, Marah. Because I will talk a little bit more on the actual pre-training side rather than just data. I liked in the previous talk, the speaker made a point that we should treat different data mixes holistically, different training stages. I want to make the same point that we should treat data and implementation of your training code base, correctness of it, and so on, also holistically. ### Trust Nothing **Robert McHardy** [9:28] If you've got data that sucks, you can't train a good model. If you've got a training code base that sucks, you also can't. So I specifically focus on architecture work and distributed training and so on, and the way we look at things in my team is we don't trust anything. There are so many things that can go wrong when you scale models to billions of parameters, to hundreds of billions of parameters, train on thousands of GPUs, and so on, and I want to show you some of the learnings that we got from training Laguna M dot one, and yeah, some of the surprising things that happen at scale. So one thing we do is we've got these model replica hashtags. So essentially when we train a model, we've got multiple replicas of the same model,right, distributed data parallel, and we know there's an invariant. The weights should always be the same across all of these replicas. That's something you can verify,right? You can calculate the hash over the weights, and you know that should always be the same across all replicas. So we do that in training and periodically compare them. If all of these hashes are identical, then we know we can continue training. If they're not identical, we know something has gone seriously wrong because such will never happen, and we crush the training. And I will give you some examples now of things that we've not shared before publicly like this, so I hope they're interesting. So first example here of shit that happens at scale are broken GPUs. On the left-hand side, we've got two loss curves, and on theright-hand side, the corresponding gradient norms that we observe during training. And you can see that these loss curves look quite different,right? ### Scale Failures **Robert McHardy** [10:53] Like the purple one has got quite some bumps, looks a bit spiky, the gradient norms are huge for that run, and there's actually no difference in terms of model configuration, training data, training implementation between these runs. They're exactly the same run, just in one of them we were got unlucky and we had a broken GPU included. That broken GPU caused silent data corruption and therefore made the training behave the way it did, and that is one of those cases that you can catch with these hashtags because you know this computation should be the same across all replicas, but it wasn't. Which brings me to the next instance of shit that happens at scale, and in this case, exploding gradients. Again, we're looking at two different loss curves than the corresponding gradient norm curves. The purple run is our initial training run for Laguna M dot one. We're a bit further into training here, around 50,000 steps or so, and you can see it stops converging,right? Like it just flattens out. And the reason here was that during training, the activations grew and grewright before the LM hit, the unembedding, and we have to perform some sort of accumulation here because we use tensor parallel for the unembedding, and that accumulation was performed in BF16 by default, and because of the growing scale that we observed in the activations, there wasn't enough numerical precision available anymore to do this accurately, and hence the model just couldn't learn anymore. And this is also a very dramatic point for this to happen because from there on it really like back propagates into the full model trunk. The orange curve is essentially just adding a fix on that. So we took the checkpoint from the purple curve, we moved that accumulation into FP32, and then from there on the model started converging again. The gradient norm, as you can see, actually started decreasing. Before then, we had an increasing trend, and this is also something you can only observe at scale, and that will break your model if you're not careful about it. So as Marah said, we took all of these insights on data, on numerics, and so on, and we turned them from M dot one into XS dot two. That's why we say it's a newer generation model. This included increasing diversity, reducing data repetitions, all of these numerical things I just mentioned, and just adding more observability and checks on that side, as well as generally optimizing the training and the architecture. ### Laguna S Scaling **Robert McHardy** [13:07] And XS dot two, if you look at it, it's open weights,right? So you can download it and use it for free. It's one of the most competitive models for its size and for coding specifically. That's why we focus on agentic coding. So we're pretty happy with it. However, you can say that the model with 33 billion parameters is pretty small, like you wouldn't probably observe any issues anyways training it. So what was important to us was to scale this,right? And this is where Laguna S comes in. This model is not public yet, so this is a preview. As I said, we treated XS as a testbed in a sense, and with Laguna S, we scaled this to a model that's 118 billion total parameters and 8B active parameters. Again, we trained it on 30 trillion tokens on 4,000 GPUs. So the scale was sufficient to not only test whether all the improvements we made on the data and architecture side hold, but also if any of these numerical issues come up again. ### Race Condition **Robert McHardy** [13:59] And of course, something happened. In this case, it doesn't actually have anything to do with scale, so it was just unfortunate. In this case, we had a race condition because we added FP8 training based on DeepGem FP8 kernels that are also like open source. We noticed these because we hit illegal memory accesses as well as NANS in the gradients, which after a while of debugging, we traced back to those kernels. There's also an unobservable effect that you wouldn't know about if you don't know that there's an issue. In our case, we noticed about 0.5% of the gradient gets silently corrupted, essentially replaced by random values. We do have a fix available that's in a PRright now. It's not been merged into DeepGem yet, but it's public on that QR code if anyone is interested. And it's also an interesting point because it's a blind jack in a blind spot in the hashtags. In real training runs, you don't have any redundancy where you have the same model weights and the same data, so you can never check if forward and backward actually behave the same across different model replicas. So you can also never check if there's a race condition in that. That's something that we're working onright now to essentially have a hash checker that can also do that at least as a dry run. And I want to end on some early results from this new model and demonstrating how it performs against some open weight models and also against our previous models. So first, I want to caveat this with these are base model evals,right? They are partly indicative of how the final model will look, but also not perfectly,right? There's still post-training happening. Not all of these will translate one to one to the final model. ### Early Results **Robert McHardy** [15:29] But if we look at them specifically on the coding part of the evals, so for instance, multiple E, live code bench, big code bench, Laguna S is not only stronger than XS dot two, which is our previous smaller model that performed very well, but also than the much larger M dot one, and it's also much better than GLM 4.5 Air, which is admittedly a bit older, Nemo 23 Super, which is quite recent, and then DeepSeek V4 Flash Max, which is quite recent and a fair bit larger. We can see it's competitive on big bench hard, for instance. It doesn't achieve the top eval result compared to these models, but it's quite close. We also see it's quite close in eval plus, and quite importantly for us, it does very well on speed bench agentless multilingual, which we use to sort of proxy agentic performance during pre-training, and in that case, it performs much better than all the other models we tested here. I also want to point out that, of course, there are, like, it's not the strongest model in the world,right? Like, for instance, MMLU Pro, knowledge benchmark, is something we don't care about that much compared to coding because we want to build the strongest agentic coding models. So here, like compared to Nemo Tron and DeepSeek, we have to say that they perform much better, and this mainly comes down due to data,right? It's a data gap that we could plug if we wanted to, but I think the point is all of the things we've found before were included in a recipe. ### Outro **Robert McHardy** [16:50] The recipe held, it scaled, and we will continue scaling it from here. So this model will also be available sometime in the future, relatively soon. Again, open weights, so all of you can download it and use it for free. And with that, I want to thank everyone for attending our talk. I added also a link to our careers page and our Twitter page if you want to check it out, and yeah, thank you very much. --- This library is powered by PodHood (https://podhood.com), the podcast website platform.