Understanding the Role of Context Management in RL Post-Training

Imagine an agent's context window as a bar that fills up while it works. Every tool call, every search result, and every reasoning step fills it a little more. At first, there's plenty of space. But over time, tool call outputs pile up, older reasoning steps collect dust, and eventually its bar hits capacity. This is the model's maximum context length, and all models have one.

Because of how transformer architecture works, every word has to be weighed against every other word that came before it. There's no skipping this; it's fundamental to how the attention mechanism works. Each new word costs more to add than the last, since it has to look back over an ever-longer history, and summed across the whole conversation. That adds up fast: the total accumulated cost increases quadratically until the bar is full and there's no space left to fill.

At this point there are only two options: stop, or find a way to empty the bar and keep going. That second option is what people mean by "context management", and it turns out to be a lot harder than it sounds.

Think of it like an employee handing off a shift. They can't relay every single thing they did, so they carefully decide what's worth passing on, so that the next employee can succeed. That's what compressing the bar back down asks of the model.

Done well, the briefing covers everything essential:

• The current plan
• Accumulated knowledge
• Incomplete tasks
• What's already been tried
• Why choices were made

Done poorly, the incoming employee inherits a shift they don't fully understand, and can end up going down the wrong path on poor information, like a game of broken telephone.

But a successful handoff only happens if someone gets the chance to prepare it. Anthropic's own agent traces show how fast the bar fills before anyone even gets to make that call. In one example, a research agent's context climbed to over 335K tokens across just five turns, with file-read results alone eating 96% of that total. On a smaller 200K-token model, that run didn't even get the chance to forget on its own terms and simply hit a wall mid-task.

This post is about why this management problem is a crucial part of how agents get trained through reinforcement learning (RL), and why it's reshaping which tools, like web search, actually belong in an agent's loop.

Why RL post-training makes the problem worse

First, a definition. What is reinforcement learning post-training?

RL post-training is the phase of AI model development where a pre-trained model learns from trial, error, and feedback loops to improve its reasoning, safety, and usability.

Instead of predicting the next token in a fixed dataset, the model generates full responses or, in agentic settings, entire trajectories of tool calls and reasoning steps. And each attempt gets judged by an automated check, a test result, or a rating from another model. The training algorithm then nudges the policy to make high-reward behavior more likely and low-reward behavior less likely, repeating this over many rounds.

Importantly, during RL post-training, an agent generates very long sequences of actions, tool calls, and observations that get scored and used to update the policy. Every one of those rollouts fills the context bar independently, many times, across thousands of training episodes. If the context management strategy is bad, it isn't just one conversation that suffers. It's the entire training process itself.

This shows up directly in the research. A study training the popular open-source Qwen3-4B model across four different environments (GSM8K, HotpotQA, ALFWorld, and WebShop) tested three context strategies side by side: appending everything, keeping only a sliding window of recent context, and using an LLM to summarize older context.

The conclusion? The choice of context management strategy had a large impact on how well the agent performed after training. On the GSM8K example specifically, the 'sliding-window' approach beat both 'append-only' and 'LLM summarization', and used fewer tokens too. But this is just one setup, with one task, one reward signal, and one quality bar. Winning strategies shift once you change any of those variables. So, the real takeaway is that how you manage context shapes what and how the model learns, and that shape isn't fixed.

An overview of context management strategies

Context management isn't one technique; it's a family, and most agents combine several.

No single strategy wins outright. Most production agents combine two or three, like masking recent observations while summarizing older turns.

Why web search and retrieval tools are central to solving this problem

The best way to manage a model's context window is to not fill it in the first place.

Instead, keep information outside the context window and pull it in only when needed, like how people use a search engine instead of memorizing a library. As agents get more capable, this trend is growing: teams increasingly let an agent steer itself toward relevant information rather than pre-loading everything into the prompt.

A web search tool is the perfect example of this strategy in action. Precise, relevant results mean less noise per step, a slower fill rate, less aggressive compaction, and fewer lost facts. This is why the API layer matters as much as the model itself. Tools like Tavily return clean, LLM-ready results instead of raw HTML, which directly shapes how fast that bar fills. Tavily also chunks content and exposes controls over what comes back, so an agent can pull the specific part of a page it needs instead of the whole document, keeping noise out of the bar before it ever arrives.

NVIDIA addressed this problem while training Nemotron 3 Ultra on Tavily-backed search data. The way agents handle search is noisy by nature: a generic search step often returns whole pages of verbose, broadly-scoped content well beyond what's actually relevant to the task, and every one of those bloated results piles straight into the history. It's a bit like preparing for a big essay by burying yourself in every potentially relevant book and article before you've chosen a direction.

Without explicit context management, a model can exhaust its window before finishing a question that needs several rounds of search. Their prior model, Nemotron 3 Super, was trained on search data with no context-management supervision, and NVIDIA found that gap costly enough to fix. For Nemotron 3 Ultra, they trained a dedicated search-specialized teacher on trajectories that practice operating under a finite context budget, using strategies like discarding stale search results and summary-based compression.

Good retrieval slows the fill rate

The issue here was never how good the search results were. Even with highly relevant, clean results, an agent working through a genuinely complex problem may still need to follow a long trajectory and read a large number of documents before it arrives at an answer, and that accumulation alone can push context toward its limit. Good retrieval doesn't eliminate the need to manage what happens once the window starts filling, but it does slow the process. The context problem sits downstream of retrieval. (For more on what training-grade search infrastructure looks like, see our post on the shift from search at inference to search in training.)

Summarization with different models

Going back to the analogy of an employee briefing the next shift, what the outgoing employee chooses to pass on, the current plan, what's been tried and ruled out, and the facts that took real effort to obtain, is what the incoming employee has to work with. Whatever isn't handed over might as well not have happened. Summarization for an agent works the same way: the model doesn't start with living memory of what it just did, it starts with whatever the previous version of itself decided was worth writing down. That's a strategic call, and it's why who does the summarizing, and how well, should be taken seriously.

The model doing the summarizing doesn't always need to be the same model doing the reasoning, and increasingly, it shouldn't be. Microsoft's own Agent Framework documentation recommends using a smaller, cheaper model for the summarization step specifically to cut costs while keeping summary quality intact. But off-the-shelf summarization often isn't good enough at demanding compression ratios, which is why some teams train a dedicated summarizer aligned to what the agent actually needs, not what reads well to a human. This turns "good" from a vague quality judgment into something testable: a summary counts as sufficient only if the agent, using it alone, reaches the same decision it would have reached with the full history.

The most advanced approach goes a step further and trains the summarizer alongside the task policy, under the same reinforcement learning reward. That means the summarizer learns to keep information because doing so improves the actual outcome, not because the summary merely looks thorough, while keeping context size flat or even smaller than baseline methods.

Open problems and the takeaway

Managing context is messy work. Summarization is lossy and unpredictable, and can keep different details across identical runs, so the same agent doing the same task twice can end up with two different versions of its own past. Agents are often blind to their own context state too, with no reliable read on how full the bar is or what's about to get dropped. The fix for this is not more training. Simply giving an agent visibility into its own context, with no extra training required, can greatly outperform several standard strategies on long-horizon benchmarks. Sometimes the gap is that the agent doesn't know what it’s about to lose.

There's also an unresolved architectural question underneath all of this: where should this logic live, inside the model, or outside?

• The first approach keeps a full, untouched log of everything the agent did, and hands the context management decisions to a separate runtime layer that sits outside the model. The raw record is never lost, and the compaction strategy can be swapped out or upgraded independently, without touching the model at all.
• The second approach trains the model itself to make that call, which is more efficient at run time, but ties the behavior to the specific model that learned it. Change the model, and that behavior doesn't carry over. It has to be retrained from scratch.

What began as a fix for running out of room has grown into a design decision that shapes what an agent can learn in the first place. Strategy changes performance, not just serving cost: search determines how fast the bar fills, summarization determines what survives once it does, whether that's a carefully budgeted step or a dedicated summarizer trained directly into the reward. The best agents aren't the ones that remember the most. They're the ones that know what to forget.