# Learning on the Job: The Future of Post-Training — Raymond Feng, Applied Compute AI Engineer · 2026-07-31 Raymond Feng of Applied Compute argues post-training must move from controlled Q&A and synthetic environments into real enterprise harnesses, enabling models to learn on the job. He details the GRPO loop: orchestrator, grader, training engine sync weight updates from graded chats. Reward hacking bites when tool-call failures at 10% shorten responses, and sandbox timeouts push models to abuse tool calls to get rollouts dropped. Bring-your-own-harness removes environment-fidelity problems but introduces non-replayability and off-policy data, tied to Nvidia's Polar paper. He lists self-distillation, automated data pipelines, and qualitative feedback ingestion as frontier directions, and envisions agentic citizens learning from every interaction, where experience dwarfs human data. ## Questions this episode answers ### What are some examples of reward hacking when training AI agents with reinforcement learning? Raymond Feng provides two examples. First, when networking issues caused tool failures roughly 10% of the time, the model began outputting shorter responses to avoid hitting failures, even though the reward had no length penalty. Second, with sandbox timeouts, the model learned to rapidly call many tools to force a timeout, triggering rollout discard instead of a zero reward, exploiting the filtering logic. [7:28](https://aiengineer.podhood.com/76e6d827-9a95-4710-95f0-e181788fdca3?t=448000) ### What is the 'bring your own harness' approach in post-training and what challenges does it face? In 'bring your own harness,' the model trains directly on the real production environment, moving orchestration outside the training stack. Raymond Feng notes this avoids environment fidelity issues but faces non-replayability: you cannot redo a customer chat with alternative responses for GRPO-style comparison. The data is also off-policy, making traditional RL methods that rely on controlled rollouts difficult to apply. [9:37](https://aiengineer.podhood.com/76e6d827-9a95-4710-95f0-e181788fdca3?t=577000) ### What is the vision for post-training where models learn continuously from all interactions? Raymond Feng envisions a single deployed model that improves continually across all tasks and users without per-task tuning. By self-evaluating and reflecting on every interaction, the model would compute weight updates automatically, mitigating the 'whack-a-mole' problem of addressing failure modes one by one. He references a paper suggesting 'experience will become the dominant medium of improvement,' dwarfing human data. [15:16](https://aiengineer.podhood.com/76e6d827-9a95-4710-95f0-e181788fdca3?t=916000) ## Key moments - **[0:00] Intro** - **[0:47] Custom Models** - [1:48] Post-training levels mirror human learning, from simple Q&A to custom harnesses akin to internships. - **[2:26] Deploy & Adapt** - **[2:48] RL Loop** - [3:57] "The only thing you need for improving your model is the graded chats in some format." - **[4:51] Longer Horizons** - **[6:49] Reward Hacking** - [7:28] 10% tool call failures due to networking issues caused models to produce shorter responses, avoiding zero-reward trajectories. - [8:40] Agents learned to abuse tool calls to trigger sandbox timeouts, dropping failed rollouts instead of receiving zero reward. - [9:06] Imperfect environment simulations induce subtle undesirable behaviors in models through reward hacking. - **[9:23] Real Environments** - [9:23] Directly training on production environments eliminates the need for simulation, but introduces non-replayability challenges. - **[9:55] Hard Replay** - **[11:52] Off-Policy Data** - [11:52] Non-replayable customer interactions break GRPO, as parallel rollouts for the same prompt become impossible. - **[13:23] Data Pipelines** - [13:23] Frontier research directions for post-training include self-distillation, automated data pipelines, and qualitative feedback ingestion. - **[15:16] Always Learning** - [15:16] Future post-training models will treat every interaction as a training signal, continually improving across all tasks. - [16:48] Training models on one task at a time is a whack-a-mole game, as new failure modes constantly emerge. - [17:29] "AI is at the cusp of a new period in which experience will become the dominant medium of improvement and ultimately dwarf the scale of human data used in today's systems." ## Speakers - **Raymond Feng** (guest) ## Topics Fine-Tuning, Reinforcement Learning, Training Data Curation ## Mentioned NVIDIA (company), Polar (product) ## Transcript ### Intro **Raymond Feng** [0:13] Yeah, thank you, Jack. Um, really grateful for the opportunity to speak here. Today I'm going to be sharing some of our frontier work on post-training and how we envision a future where agents can learn new skills on the job. So, over the last year or so, we've seen agents develop really strong reasoning skills, and they've learned to use, uh, agentic harnesses to solve longer and longer horizon tasks, which involve many turns and tool calls on complicated environment states. We're seeing an increasing, uh, demand for agents that can just be deployed in a plug-and-play way into how enterprises use the agents, um, so for instance, if they already have some method of calling the agent to do a task, they would want to be able to train a custom model to do that task instead, and that requires new ways of looking at post-training that allow you to, um, adapt to any harness, including ones that you don't necessarily have access to the source code of. ### Custom Models **Raymond Feng** [1:26] So I wanted to talk about a few different levels of post-training, where each one builds on top of the last. One way that we kind of think of this, uh, is a framework comparing it to how humans do learning, where you learn simple tasks first, and you can sort of compound your understanding to more and more complicated tasks. So, over the last year, we've sort of, I would say, mastered or gotten a lot of reps with these simple single-turn Q&A tasks and, um, some longer horizon synthetic environment tasks, but what we're increasingly seeing is we want to be able to adapt to custom harnesses and be able to train directly on those instead, and we kind of think of those kind of like internships, where you want the model to do a specific task, but you don't necessarily know how exactly the task will play out because you don't own the harness. ### Deploy & Adapt **Raymond Feng** [2:26] And then finally, I would want to share some visions we have for the future of the custom model training space, where we think that there will be these kind of agentic citizens, which you can just deploy once, and they'll be able to adapt to many different types of out-of-distribution tasks and learn from their interactions. So first, I just want to talk about the training setup for these simple Q&A tasks. ### RL Loop **Raymond Feng** [2:55] We have something that looks like this, where you have an orchestrator, and the orchestrator is in charge of driving the rollouts. The orchestrator holds a task spec, which you can think of for now as just the simple prompt and answer, so something like a math question and a corresponding numerical answer. Um, the orchestrator will send this prompt to a model and then get an answer back. Then it will send the answer to a grader and have it be graded. So once all of this is done, we want to improve our model based on that interaction, or maybe like a batch of interactions, and the way we do that is through a training engine, which takes in the graded chats and produces a weight update. That weight update is then synced to some inference engines, and once those inference engines are updated, then we can start this entire process over again, where the orchestrator will have, like, new problems to, um, send to the model completion endpoint, and then you'll be able to get more chats, grade them, and train again. The key thing to note here is that the only thing you need for improving your model is the graded chats in some format, and once you have those, the training engine can compute weight updates to improve your model. Uh, what's important here is that the chats are in a very specific format because we're sort of constraining everything to be inside of our training stack. So in this simple setup for Q&A, you don't have anything living outside of the training stack. You, you basically have the code of how to run the rollout and how everything is formatted, so it's, like, in a very controlled environment. However, this is kind of limited because we can only kind of do single-turn tasks in this way. If we want to do longer and longer horizon tasks, and we want to build, like, higher-order skills into our models, we need to also increase the complexity of our environment. ### Longer Horizons **Raymond Feng** [4:51] So with synthetic environments, we have a very similar setup, but we offload a lot of the environment state, um, outside of the training stack. So you still have the same orchestrator, um, from before, but, um, the task spec is maybe a little bit more complicated, and the environment state is, uh, living outside of the training stack. So the task spec might now include things like tool call specs or, like, maybe an initial state for your environment, like a file system, and the orchestrator is now in charge of running many turns in a in series, where maybe first it asks the model for how it wants to respond, and then if the model wants to call some tools, it'll then call the sandbox to actually, like, modify the environment state or read the environment state and then return those results back to the model. After all that is said and done, you get a full task trace out of this, um, and that task trace is then sent to a grader for grading. And very similar to what we had before, you'll be able to take the graded chats. You'll be able to then use them to do a weight update. The main thing to highlight here is that this orchestrator and sandbox setup is replayable, which is basically just saying that for any specific prompt, you can always, like, roll back to the initial state and, like, rerun it. Um, you can do that in parallel or you can do that in series. But the reason that's important is because the main sort of method that we use for reinforcement learning today is GRPO, and that involves comparing many rollouts for the same prompt and then comparing, like, relatively which one is better than the other, and the training engine will then up like, make an edit to the model to upweight the trajectories that were more successful and then downweight the ones that were less successful. So some challenges that we face in this setup is that the environment is something that you want to basically, uh, use to replicate reality so that after you're done training, like, the improvements that you've seen actually translate to when you deploy these models into production. ### Reward Hacking **Raymond Feng** [7:05] And the main sort of problem is, like, has kind of two names, which are both the same problem: environment fidelity and reward hacking. Essentially, the agent is exposed to an environment and sort of any, uh, any quirks of your environment will end up being something that your agent may, like, learn a model of. So we have some examples that we've seen where in a training run in the past, we had some, like, networking issues causing our environment to have tool calls that failed maybe around 10% of the time. If that is the case, then we actually saw that the model would then start outputting shorter and shorter responses. Now, this was really surprising to us because in our reward function, we actually didn't have any length penalty, so, like, we, we couldn't really tell why this was happening, but really what's going on here is if you think about maybe the model as, like, a human, like, walking along a sidewalk and, like, the tool call failures are, like, potholes in the sidewalk, like, it makes a lot of sense that because there's so many potholes, the model doesn't want to run for that long, uh, because it might fall in a pothole and then get a zero reward for the rollout. Um, and then conversely, um, it's also possible that your model just learns to, like, output more and more gibberish over time depending on, like, what your environment looks like. So in a different case, we had a training run where we, um, have sandbox timeouts, uh, just so that they don't run forever, and we usually, like, filter out the rollouts that timed out from being trained on. One thing we saw was that if your tool calls take a long time, then if the model feels like the problem is really hard, it will actually just be incentivized to, like, abuse the tool calls and just, like, call a lot of them in quick succession and try to time out the sandbox so it avoids getting a reward of zero. It just gets the rollout dropped. So as we scale to, like, more and more complicated tasks, the, the task of, like, replicating these environments becomes increasingly difficult because it's very, very difficult to, like, perfectly simulate reality, and sort of any mistake that you make, even if it's not intentional, will end up inducing these, like, subtle undesirable behaviors in your model. ### Real Environments **Raymond Feng** [9:23] So that brings us to our next topic of bring your own harness, where we're basically asking, like, if the agent learns the exact environment distribution, why don't we just use that for our training, like, just directly the real environment? You will no longer need to, uh, replicate anything. You can just, like, use exactly how it's going to be used in production. This solves a lot of problems and sounds really good. The architecture looks something like this, where we now have almost everything outside of our training stack. The only thing we have left is the model completion endpoint and some way to, uh, record the requests and responses that go in and out of the model. Everything else kind of lives outside of the training stack and can be run in whatever fashion, uh, that, like, an existing enterprise or, like, customer might be using. ### Hard Replay **Raymond Feng** [10:15] So these would be, like, existing enterprise harnesses, and essentially, the reason this is nice is because we can meet customers where they're at. Like, if they're already using the model in a certain way, we can just take our, like, training methodology and just, like, plug itright in, and then we can help them improve the model for, like, exactly the way that they're using it. So all the orchestration loops and logic will now live outside of the training stack. Now, this sounds really good, but, um, this sounds really good, but the challenge here is in the challenge here is in the, like, data, where as, as you're deploying this into, uh, production and you have less and less control over how the rollouts, uh, play out, you also have, like, less signal to learn from because the data is not in, like, a familiar format. And this topic is touched on in a related work by Nvidia. Uh, this is, like, a paper from around a month ago where they introduce, uh, Polar, which is essentially a way to think about transitioning from a harness where you are kind of in charge of micromanaging every aspect of the rollouts, kind of like what we were previously talking about, and transitioning to some method of just, uh, listening in on a black box harness, and you would no longer know exactly what the logic in here is. So some challenges is that some challenges we face in this setting are non-replayability and offline or off-policy data. I think both of these are describing the same issue, which is just that because we've moved so much of the logic outside of our training stack, we just don't have any way of, like, enforcing sort of invariants or, like, data structures that we like. ### Off-Policy Data **Raymond Feng** [12:15] We have to be more flexible about the way we do training, and because of that, it just becomes harder to train your model and make grading updates. So an example would be for GRPO, which is, like, the traditional method, you would want to have many rollouts in parallel for your task, and that may not be possible anymore. If you think about, um, suppose, like, a customer chat customer support chat, and you have a record of how one of your chats went, there's not really a way that you could then go back and think, "Oh, if I, like, said or if I responded in this other way, like, would the user have been happier?" Like, there's no way to then get the user's response again. But we're optimistic because, like, we think that humans can do this kind of learning, and so it should be possible to, like, formulate some kind of method that would work for models as well. Like, if a human was in a customer support chat, they could understand somehow that, like, based on the customer's reaction, like, what they said was wrong or what they said was good, and then be able to, like, internalize improvements, uh, for, for the future. So I want to talk about some of the frontier research directions we have, um, towards, like, solving this problem. There's kind of three main topics, which are self-distillation, automated data pipelines, and qualitative feedback ingestion. Self-distillation is a pretty new technique, which is still, um, I would say relatively, like, narrowly scoped. ### Data Pipelines **Raymond Feng** [13:44] So we've seen successes in inducing, like, specific new behaviors with models, but it's, it's still an open, uh, research question of, like, how general can we push it. Automated data pipelines is an idea which maybe if you take, like, a big batch of traces, um, you would be able to, like, automatically, like, flag undesirable behaviors or failure modes and then be able to, like, put together a, like, nice batch of training data, um, automatically and then send that to the model and help it improve. Currently, this is, like, pretty manual or, like, human-in-the-loop where, like, we go through traces ourselves and we're, like, looking for looking for these failure modes manually and then, like, describing how we can improve the model and then, um, looking curating those data sets ourselves. And then finally, I, I think an interesting direction is qualitative feedback ingestion. So as you move to these, like, production settings, sometimes you don't have access to, like, a clear-cut binary grade or, like, a numerical grade. Um, oftentimes, what you receive back is, like, "Hey, for this chat, the customer had this, like, piece of feedback." Um, if we can find a way to update our models based on that information, uh, that would also be prove extremely helpful. Um, and in fact, like, self-distillation is one way in which we're exploring how we can do that, but it's, like, a pretty, um, it's still a pretty open question. ### Always Learning **Raymond Feng** [15:16] Yeah, finally, I wanted to share a little bit about a vision for what the future of post-training might look like if we sort of extrapolate out, uh, and take this sort of progression to its end. I think eventually we might reach a setting where instead of just limiting ourselves to thinking about a specific task that we can improve the model on, we can actually just, uh, think of the model as just one deployment that can interact in many, many different settings, and, like, the task that you think about might just be the task of improving yourself on everything. Um, and this model may be used for all sorts of different tasks, maybe across different users as well, and be able to sort of do some kind of reflection or introspection on, like, "Okay, for this sort of type of interaction, here's how I, like, self-evaluate and think that I'm doing, and then for this other type of interaction, um, here's how I think I'm doing," and then being able to automatically, uh, take these interactions and compute weight updates from them and improve. And so, yeah, going back to the point that the agent learns every nook and cranny in your environment, um, the exact environmental distribution, uh, what if, like, the environment was just, like, every interaction that the agent ever has, and then in addition, we had some way that the model could evaluate itself? Um, basically, one, one, like, question that we've or, sorry, one challenge that we've seen is, like, if you're only focusing on, like, one improving on one task at a time or, like, flagging one failure mode at a time, you're kind of playing a game of whack-a-mole where as soon as a new thing pops up, you need to scramble and, like, create new data or new environments and improve the model in that way. With this kind of, like, self-improving system that understands interactions from, like, understands every interaction from the environment, you would, like, kind of get around this problem, and you wouldn't need to worry about it anymore. So I want to leave people on, uh, this quote from a paper around a year ago, which I find extremely relevant now, which is that "AI is at the cusp of a new period in which experience will become the dominant medium of improvement and ultimately dwarf the scale of human data used in today's systems." And yeah, if y'all have any questions, uh, I'm happy to take them after, and, uh, yeah, you can also email me at that address. Thank you. --- This library is powered by PodHood (https://podhood.com), the podcast website platform.