Taking Reinforcement Learning Cross Datacenter — Nan Jiang, Modal

Alright, cool. Hi everyone, uh, hope you all have a good time at the conference. I'm Nan from Modal. At Modal we spend a lot of time thinking about GPU capacity, like where it exists, how do we make it elastic, and what kind of workload can we actually use it.

Today I want to talk about one place where everything became, like, gets really interesting: the RL post training. A lot of RL discussionright now is about algorithm and the environments: Sandbox, PPO, gRPO, like to-call, maybe low precision training, maybe deterministic kernels.

Um, but when you run those experiments at a scale, the problem became more physical. Where are the GPUs? Are they in the same region? Do they have fast fabric? Can we get themright now? Maybe the default shape of RL compute is too restrictive.

Maybe some of the work we usually force in one cluster, uh, like can actually run on scattered, autoscaled capacity. So can we do RL across the globe? So this is what my talk is mainly about. So to make this more concrete, let's start with the RL loop itself.

So in the standard RL post training loop, we can see there's one trainer, and the trainer updates the policy, rollout worker, or maybe people call it sampler, use those policies to generate trajectories. The environment will be returning the reward and observations.

Those trajectories will go back to the trainer for the next updates. The important error here is the weight sync. In the default setup, trainer and the rollout will be living in the same cluster, and the weight sync will be super fast with RDMA.

But that also couples the rollout fleet to the trainer cluster. If the rollout needs to have more capacity, maybe more nodes during runs, um, you are normally limited by the fixed size during your trainer, maybe your trainer cluster.

So the next question is: what kind of compute shape did we actually force everything into? On the left side is the cathedral, uh, one region, one fast interconnect, uh, many GPUs wired together. This is theright shape for the trainer.

On theright is the sprawl. This is where a lot of, like, useful, usable compute actually lives. Different providers, different regions, different price, and different availabilities. There's still a lot of capacity out there, but it's not one perfect RDMA island.

This is the mismatch. Available compute is distributed, but the default RL loop asks for one tightly coupled cluster. And that cluster is exactly the hard part to get. RL wants all four of these at the same time: enough GPU, same region, fast fabric, and available now.

Any of these, like, is manageable, but all four of them, they are pretty hard to get at the same time. RDMA capacity is not elastic in the way that inference capacity is elastic. You cannot assume you can grow the trainer cluster, uh, halfway through a run just because rollout wants more nodes for a trainer.

So if the whole RL loop has no live, like, has to live inside the one cluster, rollout inherits the hardest part: capacity constraint in here. So that leads to the key question. Does the whole RL loop actually need that, like, this kind of shape?

So let's dive into this. Training is one tightly coupled job. Every step has collectives, all reduce, and model parallel communication. That part, it actually wants one fast fabric RDMA connected. Rollout is a fleet of serving jobs. It generates trajectories, call environments, or maybe tools, and they will be sending back data back to the trainer.

So across rollout jobs, there's no global all reduce. So the thing I want to move here is not backpropagation. Backpropagation should stay in the cluster. The rollout fleet is the one that can leave. More precisely, the movable unit is the rollout serving island, a coherent endpoint, or maybe a local group of endpoints, or that they can be serving one policy version.

Inside the island, a large model may still be having, like, local parallelism. They can do PDD segregation. They can have, like, local serving constraints. So across islands, the dependence is much lighterright now. Policy version in, and the trajectory and the metadata out.

So once we define the unit that way, the architecture is much, much more natural. Once we define the movable unit, the architecture is very straightforward in this case. We just have a trainer staying in the RDMA cluster, and that's where the backpropagation and the collectives go.

The rollout side will be fanning out across the sprawl. Each rollout island will be can be a single engine, or maybe a local serving group, depending on the model and the serving topology there. Across island, there is no global all reduce.

That's the most important thing there. The global interface is very simple. The trainer sends policy weight version out, and the rollout sends the trajectory and the metadata back. At this point, the architecture depends on one remaining link: the weight update.

So if we want to if we want to send the full parameter, like, full checkpoint from disk, or maybe through the network, then everything is, like, meaningless and, like, it will be breaking immediately. So, uh, after this aggregation, the things that we will be discussing about are the size of, like, go the size of the full parameters go through the disk.

Naively, that means shipping up full checkpoints every time rollout needs a new weight version. At this scale, the checkpoint is very huge. So a k-means scale NVFP4 checkpoint, you have, like, 500 gigabytes. Normally it takes a minute, or maybe normally it takes multiple minutes to hours to just do the weight sync.

So moving that over to monitoring might not be the smartest choice because, like, uh, when you're doing async, maybe even fully async training, you still want weight updates latency to be as slow as possible, like, within seconds. So the problem here is not whether rollout can leave the cluster.

The problem is, like, the full checkpoint is a wrong unit of synchronization. So the next question is: can we keep the exact same served version there, but send a much smaller object? So this is the bet. What if less than 1% of rollout visible weights got changed from one version to another one?

By rollout visible weights, I mean the weights in the served rollout checkpoint, uh, not the FP32 optimizer state, not the Adam, like, moments. Like, the weights are the rollout engine which we actually use to serve, maybe, let's say, the FP8 or maybe NVFP4 format.

If that's true, we do not need to ship the entire full parameter over the network. We just need to ship the change in the server view, the precision that got differenced. The important part here is, like, it's still bitwise reconstruction.

The rollout engine gets the same served version you would have gotten async to the full checkpoint there. So if this works, then the link shrinks from hundreds of gigabytes to maybe hundreds of megabytes. And this is something smart enough.

We can just send it across, like, the network. Soright now, we need to justify the less than 1%, like, claim. Why would this rollout visible weights barely change? Uh, now we get we will get into this mechanism. So it's kind of small Adam, like, step meets finite precision.

We need two prerequisites. The first one, two ingredients. The ingredient one is the precision. The optimizer may keep very high precision master weights, but the next forward pass reads the BF16 visible view. That view has finite resolution. Around the value of magnitude theta, uh, BF16 spacing is roughly theta over 128.

Uh, that spacing is people call it oop, basically the unit in the last piece. Basically, it's the distance between the adjacent representable BF16 value. But an update only needs to cross the nearest surrounding boundary to be viable, to be visible.

That boundary is about half of the oop. So roughly, like, it's theta over 256. For weight around 1, the BF16 oop is around 0.0078, and the nearest surrounding boundary is about 0.0039. If the optimizer nudged the master weight by something, like, smaller than that, uh, the BF16 visible value will run back.

So you will not see any change from the weight perspective, rollout weight perspective. So that's the floor. The second primitive there is we call push. So for Adam, or maybe Adam absorption, here we just, like, we ignore the weight decay turn.

The per sorry, the per-parameter updates is the learning rate times the normalized direction. The raw gradient can be dense and can have very different magnitudes across parameters. And Adam, like, divides by running, uh, gradient statistics. So the per-weight push is usually on the order of learning rates.

The per-parameter pass, I said there, they prove a bound. The Adam step is at most B times the learning rates. So you do not need to actually remember the exact, like, bound there. Like, the important note here is the Adam makes the push small and very controlled.

So at RL post training learning rates, the push is very, very tiny. So that's the push. Combining these two primitives, now we have to we have a whole, like, better picture. A served value changed only if the push cleared the floor.

The push is the Adam step, roughly the learning rates. The floor is the nearest BF16 rounding boundary, roughly theta over 256. Take theta equal 1, the BF16 boundary is about 0.0039. A typical RL Adam step here is around 3 millionths.

So the update is more than 1,000 smaller than the boundary, so the BF visible value will not change. This is not saying the master weight

is frozen forever. It is saying the value that rollout engine would serve does not change on this part. So the whole mechanism is pushing is push versus floor. Let's visualize this to have better understanding. The x-axis is the weight magnitude, and the y-axis is the update magnitude.

First, we look at the red line. The red line is the BF16 visible boundary. It's, like, theta over 256. And now we look at the green bound. This is the Adam push. It sits roughly around the learning rate and with a conservative upper bound.

So now we ask, like, where the most point fits. For most of the weights, the red floor is above the green push. Those updates exist in the master weights, but they are not visible in the served BF16 view at this step.

Small weights on the left can move. Large weights on theright, they will just stay the same. They will be absorbed. This is the Adam absorption. This is why the served update becomes very sparse. In this case, the object will be shipped as just a diff.

Uh, it's just a diff, not the entire FP32 optimizer state. We first look at the rollout view. The weight cast, like, will project it to the D type that the rollout engine will be serving. Then we will be comparing the version T-1 and the version T in this view.

The patch is the patch is the change in the precision, plus, like, replacement bits and also, like, some metadata. There are multiple lossless encoding. People can do selective overwrite. They can people can also do XOR. The important part is, like, there are bit equivalents, bit-level equivalents.

So it's not a floating point addition. So there's no additive delta drift. If a rollout engine applied the patch correctly, it reconstructs the same served version bitwise. So everything so far we explained is about the full parameter, which is hard part of RL.

So in full parameter reinforcement learning, the optimizer updates the whole model. But the rollout view patch is sparse, as the thing we just explained. Lower is small for a different purpose, for a different reason. The base model is frozen, and the adapter is small enough by construction.

So you do not need to so you do not need to have the push versus floor arguments here. So full parameter delta is small by absorption, and the lower updates are small by construction. Let's dive into detail about the paper itself.

So the paper, they mention more stats. I will be showing here. The measurement is not gradient sparsity. It's not optimizer state sparsity. They cast weights to BF16, compare constructive version stillness, and they compare the version bitwise. And they count what did not change over time.

Across model family, the result is around 99% of the time it's been identical per step. It also it also survives stillness. Even when the rollout lags, the change set remains very small. The important part is not only the number.

It is the patch is lossless. Change index plus the replacement value reconstructs the exact same version. So a common misconception there is, like, the word it works because RL gradients are sparse. They are not. The paper reports the gradients are dense.

About 99% of parameter gets non-zero gradients. The FP32 master update is also dense. It's just small. The main thing is, like, the rollout weight change is just 1% from the perspective of the rollout engine. So far, we mostly talk about BF16.

But the rollout often serves even lower precision, such as, like, MF4, F8, and NVFP4. And we can see many, many model providers doing this in their rollout. Uh, this is not the training precision. The training is just, like, in the normal BF16, although people can do QAT on that.

So for fixed-style format, the visibility floor is roughly theta over 2 to the mantisa plus 1. So as you can see, the FP4 will be higher, and F8 also will be between BF16 and FP4, which means in even lower precision, there will be less weight change.

So plane floors are easy to reason about. So each element has its own rounding ceiling. One value crossed the floor, one value they just flipped and changed. Group scales such as Int4, they are they are a bit different.

This is the region where many low-precision serving systems are moving towardsright now. For Int4, each weight is quantized against a shared group scale. And we can apply the same rationale, and also we can observe a similar thing for NVFP4.

It's hierarchical scales, and we can see there are different encoding and displaying mechanism for NVFP4. So this is from one internal run. So here's the model we serve, like, GLM 4.7 Air in FP8. And we can see in the beginning, there are only, like, 0.15% of weights got changed in the first step where the learning rate is high.

And after we have more training step, like, when the Adam is going relatively stable, and you can see the entire curve goes stable when you got only 0.05% weight change during each step. So we can see this pattern showing more generally.

We have different research. Uh, we saw different research across RL sync have a similar conclusion. And we saw different model providers, such as Cursor, Composer 2, MAI, they were using Adam in their post-training. At this point, assuming we can produce exact rollout weights version cheaply, the next question is, how do we do this in practice?

So from sparse delta to RL across globe, how do we do this with elastic rollout engines and also explicit stillness? This is the whole shape. The trainer is saying the RDMN cluster, after it got updated, it published a mutable rollout weights version to a shared bulletin board.

Rollout engine live outside of the training cluster, which means they don't need to be RDMN connected with the trainer. They can be in different regions or different providers. This is also request lane. You can see a request does not just say, "Give me the completion."

You will also say which version you will be sending requests to and which version you will be accepting. And the response will come back with the version and also, like, exact same information as if they are in the same cluster as the trainer.

They will be returning tokens, logprob, like, router replay information, and many more metadata. The trainer writes a mutable version to the broad after-optimizer state, uh, after-optimizer step. Engine pulls a version and materializes it locally in the checkpoint layout so they can just, like, serve directly.

The artifact defines the version. The engine chooses how to load and shard it. It does not change it does not choose a different server version. Since it will be displaying in a Hugging Face HF, it will be saved in Tensor format, which is accepted widely by many rollout engines, such as SGNN and VLM.

So we can support any compatible backend, attention backend, MOE backend, different parallelism, compatible serving D type, and any compatible GPUs there. We can talk more about the sidecar itself. The sidecar is basically what makes a normal rollout engine version aware.

If the version is already at acceptable commit version, the sidecar just proxies the request. If the engine is behind, but they can catch up, the sidecar just applies the missing transition. If they cannot get there, the sidecar just simply returns not ready.

So this will be supporting elastic rollout, and any idle GPU can just be used with design with this design to support this aggregated rollout. So this is more like the system latency analyst. In cluster, weight sync is fast because they have RDMN.

Uh, a full checkpoint across regions through network is pretty slow. And but if we use the delta if we use exactly what we described previously, we can decrease the number of, like, transfer size from, like, 500 gigabytes to 500 megabytes.

So you will be, like, extremely fast in seconds. So everything above was very general protocol. Stitch is one of very concrete implementation for model that we implementing everything above. So on the trainer side, Stitch publish, uh, what defines the rollout weights version.

And on the contract side, you will be pulling out and record everything on the bulletin board. And the only rollout side, you will be pulling the latest weights and start doing weight sync across different regions and different providers.

So Stitch itself is very framework agnostic about trainer and engine and also transport. It's very async first and also agent first agentic first. By doing this, we can have rollout engines, like, auto-scale globally. Each one self-sync its weights, serve accepted version, and return rollout metadata.

That means scattered inference capacity became one elastic rollout fleet. Instead of being limited by the trainer cluster, rollout can be the global pool. So inference capacity can now become RL capacity.

Last section, we have some, uh, ongoing explorations. So we can see a lot of model providers, such as Moonshot and also DeepSeek V4. They have, like they are adopting Muon in their post-training. Um, does the sparsity still hold for Muon?

Because a lot of things we discussed previously only for Adam. Second question is async RL at the scale. Right now, we can use the compute across the globe. Then how, like, how scalable is the fully async RL? This is a very open-ended question there.

And, uh, third question is, like, does it generalize past RL? Because, like, we have pretraining, mid-training, and SFT. Like, do we have can we apply the same paradigm there? Last but not least, we are working on some very hard problem.

And can work with us. You can check the link there, model.jobs. Thank you.