Context Window Management#

Context window management is the discipline of fitting a long, multi-turn agent conversation inside the LLM's token limit without losing the information needed to answer the next turn well. Even on a 1M-token model, raw chat plus tool outputs plus retrieved documents plus a long system prompt will saturate the window faster than first-time builders expect, so every production agent ships with a token budget allocator and at least one compression strategy.

flowchart LR
  Turn([New user turn]) --> Budget[Token<br/>budget allocator]
  Budget --> Sys[System +<br/>persistent rules]
  Budget --> Hist[Conversation<br/>compressed]
  Budget --> RAG[Retrieved<br/>chunks]
  Budget --> User[Current<br/>user query]
  Sys --> LLM
  Hist --> LLM
  RAG --> LLM
  User --> LLM
  LLM --> Resp([Response])

  classDef client fill:#dbeafe,stroke:#1e40af,stroke-width:1px,color:#0f172a;
  classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
  classDef compute fill:#d1fae5,stroke:#065f46,stroke-width:1px,color:#0f172a;
  classDef datastore fill:#fee2e2,stroke:#991b1b,stroke-width:1px,color:#0f172a;
  class Turn,Resp client;
  class Budget,Sys,Hist,User service;
  class RAG datastore;
  class LLM compute;

Why the window is finite even at 1M tokens#

Three forces work against just "stuffing everything in":

Cost. Tokens are billed per call. A 200k-token prompt at GPT-4 class pricing is 50x more expensive than a 4k-token one, and you pay it on every turn.
Latency. Time-to-first-token grows roughly linearly with prompt size on transformer inference. A 1M-token prompt can take 30 seconds before the first response byte arrives.
Attention degradation. The lost-in-the-middle effect (Liu et al., 2023) shows that recall accuracy is U-shaped: the model attends well to the start and end of the prompt and weakly to the middle. So a 1M-token window does not give you 1M tokens of usable signal.

The practical rule is to budget like a memory-constrained system, not like you have infinite RAM.

The five strategies#

1. Truncation (sliding window)#

Drop the oldest turns until the prompt fits. Cheapest and simplest; works when only the recent context matters (short customer-support chats, code completion, single-task tools). Fails when an early turn carried critical state ("my user ID is 42").

2. Rolling summarization#

Periodically replace the oldest N turns with an LLM-generated 200-500 token summary. The active prompt becomes [system] + [summary so far] + [recent turns] + [current query]. This is the default for general chat agents because it keeps cost low while retaining gist.

3. Hierarchical / tree compression#

Recursive summaries: the first compression collapses 20 turns into a paragraph, the next compression collapses 10 such paragraphs into a chapter summary, and so on. Used in agents that may run for hours or days (research agents, coding agents working through a backlog).

4. Retrieval-pruned context#

Treat the conversation history itself as a corpus. Embed each turn, store in a vector index, and at every new turn run a similarity search to pull the top-K relevant past turns. The active prompt only contains turns the model is likely to need. This is RAG against your own chat history.

5. Tool-call output truncation#

Tool outputs (search results, file reads, SQL responses) often dominate token usage. Truncate, summarize, or replace tool outputs with a reference ID after they have been consumed. A 50KB log dump becomes "log excerpt stored as ref://log-7821, 18 errors of type X."

flowchart TB
  Raw[Raw history<br/>+ tool outputs]
  Raw --> Trunc[1. Truncate<br/>oldest first]
  Raw --> Sum[2. Rolling<br/>summary]
  Raw --> Tree[3. Hierarchical<br/>tree of summaries]
  Raw --> Retr[4. Retrieve<br/>top-K relevant turns]
  Raw --> Tool[5. Truncate<br/>tool outputs]
  Trunc --> Prompt[Final prompt]
  Sum --> Prompt
  Tree --> Prompt
  Retr --> Prompt
  Tool --> Prompt

  classDef datastore fill:#fee2e2,stroke:#991b1b,stroke-width:1px,color:#0f172a;
  classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
  classDef compute fill:#d1fae5,stroke:#065f46,stroke-width:1px,color:#0f172a;
  class Raw datastore;
  class Trunc,Sum,Tree,Retr,Tool service;
  class Prompt compute;

Real systems combine all five. A production coding agent will: keep the system prompt persistent, summarize old turns hourly, truncate tool outputs immediately, retrieve only the most relevant past file reads, and slide off raw history older than the last summary.

Where this differs from agent memory#

Agent Memory is about persistence across sessions: facts stored in a vector DB so the assistant remembers your preferences next week. Context window management is the within-conversation problem of choosing what to actually include in the prompt for the next single LLM call. A real agent runs both: memory provides candidate content, the context manager decides what fits.

Context window management is the production discipline of fitting a long multi-turn agent conversation into the model's token limit without losing answer quality. Modern LLMs advertise 200k and even 1M token windows, but on real systems the active prompt is shared with system instructions, tool definitions, retrieved documents, and tool outputs, and every additional token costs money and latency. This guide covers the prompt budget allocator, the five core compression strategies, when to recompute summaries, and how to measure quality loss.

flowchart TB
  subgraph In[Inputs]
    Sys[System prompt<br/>+ persistent rules]
    Hist[Conversation history]
    Tools[Tool definitions<br/>+ tool outputs]
    Docs[Retrieved RAG chunks]
    Query[Current user turn]
  end
  In --> Alloc[Token budget<br/>allocator]
  Alloc --> Comp[Compression<br/>passes]
  Comp --> Prompt[Final prompt<br/>under limit]
  Prompt --> LLM
  LLM --> Resp[Response]
  LLM -. headroom .-> Resp

  classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
  classDef compute fill:#d1fae5,stroke:#065f46,stroke-width:1px,color:#0f172a;
  classDef datastore fill:#fee2e2,stroke:#991b1b,stroke-width:1px,color:#0f172a;
  class Sys,Tools,Query,Alloc,Comp,Prompt service;
  class LLM compute;
  class Hist,Docs datastore;
  class Resp service;

Why finite-window thinking applies even at 1M tokens#

Three independent forces make "just stuff everything in" the wrong default:

Cost scales linearly with prompt tokens. At GPT-4 class input pricing of roughly $2.50 per million tokens, every 100k tokens added to a prompt costs $0.25 per call. A coding agent that runs 500 calls per session is paying $125 per session just to carry around context the user has forgotten about. Anthropic's Claude and OpenAI's GPT-4 both surface prompt caching to mitigate this, but cache hits still pay the read price and require careful prompt structure.

Time-to-first-token grows with prompt size. Transformer prefill is roughly O(N) in prompt length on optimized inference; on a 1M-token prompt time-to-first-token can exceed 30 seconds. Interactive agents target sub-second responses, so a multi-megabyte prompt is a non-starter regardless of the model's stated window.

Attention degrades in the middle of long prompts. Liu et al.'s "Lost in the Middle" (2023) showed that LLMs attend strongly to the start and end of a long context and weakly to the middle. The exact U-curve depends on the model, but the qualitative effect persists in current frontier models. Stuffing a 200k context with relevant docs does not give you 200k tokens of usable recall: it gives you something closer to the first 20k and the last 20k.

The practical conclusion: treat the context window as a constrained resource and explicitly budget it.

The prompt budget allocator#

A budget allocator divides the model window into named slices, each with a hard cap and a priority. A typical layout for a 200k-token model:

Slice	Budget	Priority	Notes
System prompt + rules	4k	P0	Never dropped. Cache-friendly.
Tool definitions	2k	P0	Required for tool-calling models.
Retrieved RAG chunks	16k	P1	Top-K, reranked, capped.
Conversation summary	4k	P1	Rolling summary of older turns.
Recent raw turns	24k	P2	Sliding window of last M turns.
Tool outputs (active)	8k	P2	Truncated; older outputs replaced by refs.
Current user query	4k	P0	Never dropped.
Response headroom	8k	P0	Reserved for the assistant reply.
Safety margin	4k	P0	Tokenizer drift, retries.

The allocator runs every turn: count tokens in each slice, and if the total exceeds the limit, compress or drop slices in reverse priority order. P0 slices are inviolable; the agent should fail loudly rather than truncate a system prompt or the user's actual question.

flowchart LR
  In[Raw slices] --> Count[Token count<br/>per slice]
  Count --> Over{Over<br/>limit}
  Over -- no --> Build[Assemble<br/>prompt]
  Over -- yes --> Drop[Drop / compress<br/>by priority]
  Drop --> Count
  Build --> Out[Final prompt]

  classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
  classDef compute fill:#d1fae5,stroke:#065f46,stroke-width:1px,color:#0f172a;
  class In,Count,Build,Out,Drop service;
  class Over compute;

The five compression strategies#

1. Sliding window truncation#

Keep the most recent M turns, drop older ones. Simplest and cheapest; runs in O(1) per turn. Works when the conversation has no long-range dependencies: customer-support chats, code completion within a file, single-task tools. Fails when an early turn carried persistent state ("my user ID is 42" said in turn 1, needed in turn 50).

Mitigation: always pair sliding-window with a small "pinned" slice that carries hand-extracted persistent facts from the dropped turns.

2. Rolling summarization#

Periodically collapse the oldest N turns into a summary. The active prompt becomes [system] + [summary so far] + [recent raw turns] + [current query]. Two triggers in production:

Every N turns (predictable, but recomputes when nothing changed).
Threshold-based: when history crosses 60 to 75 percent of the window, summarize. Cheaper because it runs less often, but spiky cost.

Hybrid: threshold trigger with a max-age cap, so the summary never goes stale even on long quiet sessions.

The summary itself is generated by an LLM call. A typical prompt: "Summarize the following conversation in under 400 tokens. Preserve names, IDs, decisions, and unresolved questions. Drop chit-chat." Smaller, cheaper models (Haiku, GPT-4o-mini, Gemini Flash) work fine for this; you do not need the frontier model to summarize a chat.

3. Hierarchical / tree compression#

Recursive summarization for very long sessions. Build a tree:

flowchart TB
  Root[Session summary<br/>~500 tokens]
  Root --> S1[Section summary 1<br/>~1k tokens]
  Root --> S2[Section summary 2<br/>~1k tokens]
  S1 --> P1[Paragraph 1<br/>~300 tokens]
  S1 --> P2[Paragraph 2<br/>~300 tokens]
  S2 --> P3[Paragraph 3<br/>~300 tokens]
  P1 --> T1[Raw turns 1-20]
  P2 --> T2[Raw turns 21-40]
  P3 --> T3[Raw turns 41-60]

  classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
  classDef datastore fill:#fee2e2,stroke:#991b1b,stroke-width:1px,color:#0f172a;
  class Root,S1,S2,P1,P2,P3 service;
  class T1,T2,T3 datastore;

The active prompt carries: root summary + the section the user is currently in + the recent paragraph + last few raw turns. Older content stays present but at lower resolution. This is the pattern used by long-running coding agents and research assistants.

Recompute the tree bottom-up: only summarize a level when the level below it has filled its slot.

4. Retrieval-pruned context (RAG over chat history)#

Treat past turns as a corpus. Embed each turn (or each message pair), store in a vector index, and on every new turn run a similarity search to pull the top-K relevant past turns. Active prompt: [system] + [retrieved past turns] + [last few raw turns] + [current query].

This is mechanically identical to RAG over documents (see chunking strategies and rag patterns) with two operational differences:

The index updates every turn, so use an embedding model with cheap incremental writes (pgvector, Chroma, Qdrant).
Recency matters: rerank candidates by score = similarity + alpha * recency so a stale match does not crowd out a fresh one.

Retrieval is the right move when the conversation has occasional callbacks to distant past turns ("you mentioned a config option earlier, what was it"). It is overkill for short linear chats.

5. Tool-call output truncation#

Tool outputs dominate token usage in agentic systems. A single grep over a large codebase, a SQL query against a busy table, or a web search returning 10 articles can each emit tens of thousands of tokens. Strategies, in order of cost:

Hard truncate the output to N tokens with a marker [...truncated, full output stored at ref://abc]. The agent can re-fetch if needed.
Summarize the output before injecting it. A 10k-token search result becomes a 500-token summary plus the top three URLs.
Replace the output with a reference ID after a few turns. The model sees only the metadata; the raw bytes live in external storage.

The agent loop should also dedupe: if the same tool was called with the same args twice, do not repeat the output in the prompt.

When to recompute the summary#

The two production triggers:

Trigger	Pros	Cons
Every N turns	Predictable cost, summary always fresh	Recomputes even when content has not changed
Threshold-based (e.g. 70 percent of window)	Cheap when sessions are short	Spiky latency on the turn that crosses the threshold
Hybrid (max-age + threshold)	Best of both	More state to track

In benchmarks at OpenAI and Anthropic, threshold-based triggers at 60 to 75 percent of context dominate. Hybrid is used in long-running agents where a 4-hour quiet session might otherwise carry a stale summary.

Measuring quality loss#

Compression always loses information. The question is whether the loss is acceptable for the cost saved. The standard eval setup:

Build a held-out test set of multi-turn conversations with known "right" answers for the final turn (gold answer or rubric).
Run two pipelines side by side: full_context (no compression) and compressed_context (your real allocator).
Score both with an LLM judge or human raters (see LLM Evals).
Report: pass rate delta, token cost ratio, p95 latency ratio.

A healthy production allocator hits 90 to 95 percent of full-context quality at 10 to 20 percent of the token cost. Anything below 85 percent on multi-turn callbacks usually means the strategy is missing context: add retrieval, or pin specific entities (user IDs, recent decisions).

Targeted micro-evals also help: "needle in a haystack" tests where a key fact is placed early in a long conversation, then queried at the end. They catch lost-in-the-middle regressions when you change models or prompt structure.

A reference token-budget allocator#

from dataclasses import dataclass, field
from typing import Callable
import tiktoken

enc = tiktoken.encoding_for_model("gpt-4o")

def n_tokens(text: str) -> int:
    return len(enc.encode(text))

@dataclass
class Slice:
    name: str
    content: str
    priority: int            # 0 = never drop, higher = drop first
    max_tokens: int          # hard cap for this slice
    compress: Callable[[str, int], str] | None = None  # optional shrink fn

@dataclass
class Budget:
    model_window: int = 200_000
    response_headroom: int = 8_000
    safety_margin: int = 4_000

    @property
    def usable(self) -> int:
        return self.model_window - self.response_headroom - self.safety_margin

def fit(slices: list[Slice], budget: Budget) -> str:
    # 1. Cap each slice at its own max.
    for s in slices:
        if n_tokens(s.content) > s.max_tokens and s.compress:
            s.content = s.compress(s.content, s.max_tokens)
        elif n_tokens(s.content) > s.max_tokens:
            s.content = _hard_truncate(s.content, s.max_tokens)

    # 2. If total still over usable budget, drop or compress by priority.
    while _total(slices) > budget.usable:
        victim = _pick_victim(slices)
        if victim is None:
            raise RuntimeError("Cannot fit P0 slices under budget; refuse the call.")
        target = max(0, victim.max_tokens // 2)
        if victim.compress and target > 0:
            victim.content = victim.compress(victim.content, target)
            victim.max_tokens = target
        else:
            victim.content = ""

    return _render(slices)

def _total(slices: list[Slice]) -> int:
    return sum(n_tokens(s.content) for s in slices)

def _pick_victim(slices: list[Slice]) -> Slice | None:
    droppable = [s for s in slices if s.priority > 0 and s.content]
    if not droppable:
        return None
    return max(droppable, key=lambda s: (s.priority, n_tokens(s.content)))

def _hard_truncate(text: str, n: int) -> str:
    toks = enc.encode(text)
    return enc.decode(toks[:n]) + " [...truncated]"

def _render(slices: list[Slice]) -> str:
    return "\n\n".join(f"### {s.name}\n{s.content}" for s in slices if s.content)

Wire it up per turn:

def build_prompt(state) -> str:
    slices = [
        Slice("system", state.system_prompt, priority=0, max_tokens=4_000),
        Slice("tools",  state.tool_defs,     priority=0, max_tokens=2_000),
        Slice("rag",    state.rag_chunks,    priority=1, max_tokens=16_000,
              compress=summarize_chunks),
        Slice("summary",state.summary,       priority=1, max_tokens=4_000,
              compress=resummarize),
        Slice("recent", state.recent_turns,  priority=2, max_tokens=24_000,
              compress=drop_oldest),
        Slice("tools_out", state.tool_outputs, priority=2, max_tokens=8_000,
              compress=summarize_tool_output),
        Slice("query",  state.user_query,    priority=0, max_tokens=4_000),
    ]
    return fit(slices, Budget())

The shape of this code is what every production agent ships: explicit slices, explicit caps, explicit priorities, explicit fallbacks. The strategies it composes (truncation, summarization, retrieval, tool-output trimming) are the same five strategies covered above.

Production failure modes#

Summary drift. Each round of resummarization loses fidelity. After 10 levels, the summary describes a different conversation than the one that happened. Mitigation: keep a raw transcript in cold storage; reload and resummarize from raw when drift is suspected, or test summary fidelity with periodic evals.
Pinned-fact rot. "Pin" a fact like the user ID and the agent will dutifully carry it forever, even after the user logs out. Pinned slices need lifecycles too.
Retrieval staleness. A query about "the file we changed yesterday" should not pull a 6-month-old turn that happened to mention the same filename. Rerank by recency.
Tool-output amnesia. Replacing tool outputs with refs is great, until the model wants to re-read the original. Keep the ref store hot for the session lifetime, not just one turn.
Cache invalidation. Most prompt caching (Anthropic, OpenAI) requires the prefix to be byte-identical across calls. A summary that changes every turn breaks the cache for everything after it. Put cache-friendly content (system, tools) before cache-breaking content (history, query).

FAQ#

What is context window management in an LLM agent?#

It is the set of strategies that keep the prompt sent to the model under its token limit while preserving the information needed to answer well. Common moves are truncation, rolling summaries, hierarchical compression, and retrieval over the conversation itself.

Why do we need context window management if models support 200k or 1M tokens?#

Large windows cost more per call, increase latency, and suffer from the lost-in-the-middle effect where attention degrades for content in the middle of a long prompt. A smaller, well-curated prompt is usually cheaper and more accurate.

What is the lost-in-the-middle problem?#

Liu et al. 2023 showed that LLMs recall information placed at the start and end of a long context much better than information in the middle. Packing 200k tokens does not give you 200k tokens of usable recall.

When should an agent recompute its rolling summary?#

Either every N user turns or when the conversation exceeds a token threshold, whichever comes first. Production systems usually use a threshold of 60 to 75 percent of the model context window so headroom remains for the response.

How is context window management different from agent memory?#

Agent memory persists state across sessions in external stores like vector databases. Context window management is the within-conversation problem of fitting the active prompt under the model's token limit on this single call.

How do you allocate a prompt budget for an LLM agent?#

Split the model context into reserved slices: system prompt and rules, retrieved chunks, conversation history (compressed), the current user query, and headroom for the response. Each slice gets a hard cap, and the allocator drops or compresses lower-priority slices first when the total exceeds the limit.

What is hierarchical context compression?#

Recursive summarization. Recent turns are summarized into a paragraph, several paragraphs are summarized into a chapter, several chapters into a section, and so on. The active prompt carries one entry from each level, so older content stays present but at lower resolution.

How is retrieval over conversation history different from RAG over documents?#

Mechanically the same: embed each turn, store in a vector index, search at query time. The difference is the data source (chat history vs documents) and the freshness requirement: chat history is appended every turn, so the index needs incremental updates.

How do you measure quality loss from context compression?#

Run paired evals: send the same query through the full-context pipeline and the compressed-context pipeline. Score answers with an LLM judge or human raters. A 5 to 10 percent quality drop for a 5x cost reduction is usually acceptable; a 30 percent drop is not.

What is the fastest way to fit a long chat in the LLM context window?#

Sliding window truncation: drop the oldest turns until the prompt fits. Pair it with a small pinned-facts slice so persistent IDs and decisions are not lost.

What is the lost-in-the-middle problem in one sentence?#

LLMs recall information at the start and end of a long prompt much better than information placed in the middle, so packing the context window does not give you proportional recall.

What is a healthy prompt budget split for a 200k token model?#

Roughly 4k system, 2k tool defs, 16k retrieved chunks, 4k rolling summary, 24k recent turns, 8k tool outputs, 4k user query, 8k response headroom, 4k safety margin.

When should the rolling summary be recomputed?#

When conversation history crosses 60 to 75 percent of the context window, or every N turns, whichever happens first.

How do you know if compression is hurting quality?#

Run paired evals: same query through full-context and compressed-context pipelines, score with an LLM judge or humans, watch for pass-rate drops above 5 to 10 percent.

Agent Memory: persistent state across sessions, the upstream of context management
RAG Patterns: retrieval-augmented generation, used to prune chat history too
Chunking Strategies: how to slice content before it enters the prompt
LLM Evals: measuring quality loss from compression
Semantic Cache for LLMs: reduces context cost by caching whole responses

Quick reference#

TL;DR#

Even on 200k or 1M token models, raw context is expensive, slow, and degrades attention in the middle. Production agents budget the prompt explicitly and compress older content using truncation, summarization, hierarchical trees, retrieval, and tool-output trimming.

Why bother with a 1M token model#

Cost scales linearly: 100k tokens per call adds ~$0.25 at GPT-4 input pricing.
Latency: prefill is O(N); 1M-token prompts have 20 to 30 second time-to-first-token.
Lost in the middle (Liu et al., 2023): recall is U-shaped; mid-prompt content is mostly ignored.

The five strategies#

#	Strategy	Cost	When to use
1	Sliding window truncation	O(1)	Short linear chats, code completion
2	Rolling summarization	One LLM call per recompute	General chat agents
3	Hierarchical / tree compression	LLM calls per level	Long-running coding / research agents
4	Retrieval over chat history	Embed per turn + vector search	Conversations with distant callbacks
5	Tool-output truncation / ref-ID	Truncate or one summary call	Any agent that calls tools

Real systems combine all five. A coding agent usually: summarizes older turns, slides off raw history past the summary, retrieves relevant past file reads, and replaces big tool outputs with reference IDs.

Prompt budget slices (200k window)#

Slice	Budget	Priority
System prompt + rules	4k	P0 never drop
Tool definitions	2k	P0
Retrieved RAG chunks	16k	P1
Rolling summary	4k	P1
Recent raw turns	24k	P2
Tool outputs (active)	8k	P2
Current user query	4k	P0
Response headroom	8k	P0 reserved
Safety margin	4k	P0

The allocator drops or compresses in reverse priority. P0 inviolable; the agent fails loudly if P0 alone exceeds the window.

Summary recompute triggers#

Every N turns: predictable, recomputes when nothing changed.
Threshold (60 to 75 percent of window): cheap; spiky latency on the threshold-crossing turn.
Hybrid (threshold + max-age): production default for long sessions.

Use a cheap model (Haiku, GPT-4o-mini, Gemini Flash) for the summary call. You do not need the frontier model to compress a chat.

Hierarchical compression layout#

Session summary  (~500 tok)
  Section 1 summary (~1k)
    Paragraph 1 (~300)  -- raw turns 1-20
    Paragraph 2 (~300)  -- raw turns 21-40
  Section 2 summary (~1k)
    Paragraph 3 (~300)  -- raw turns 41-60

Active prompt carries: root + current section + current paragraph + last few raw turns.

Retrieval over chat history#

Embed each turn (or message pair) on write.
pgvector / Chroma / Qdrant for cheap incremental writes.
Score by similarity + alpha * recency to avoid stale matches.
Top-K = 3 to 8 for chat callbacks. Higher K bloats context.

Tool-output strategies#

Hard truncate with [...truncated, ref://abc].
Summarize the output before injecting.
Replace with reference ID after a few turns; rehydrate on demand.
Dedupe identical calls.

Cache-friendly ordering#

Most providers (Anthropic, OpenAI) cache on byte-identical prefix. Order slices so cache-stable content comes first: 1. System prompt 2. Tool definitions 3. Persistent rules 4. Retrieved chunks (cache by chunk-set hash if reused) 5. Rolling summary (breaks cache when it changes) 6. Recent turns 7. User query (always new)

Measuring quality loss#

Paired evals: full vs compressed pipeline, same queries.
LLM judge or human raters score answers.
Track: pass-rate delta, token cost ratio, p95 latency.
Healthy target: 90 to 95 percent of full-context quality at 10 to 20 percent of the cost.
Run "needle in a haystack" tests when changing model or prompt structure.

Watch-outs#

Summary drift: 10 layers of resummarization will silently change the story. Keep raw transcripts cold; periodically reload.
Pinned-fact rot: pinned IDs survive after they should not. Give pins a TTL.
Retrieval staleness: re-rank by recency, not just similarity.
Tool-output amnesia: keep ref store hot for the session.
Cache invalidation: changing summary breaks every cached call after it. Order matters.

Quick reference (interview cheat)#

A 200k window does not give 200k of usable signal (lost in the middle).
Always allocate a budget; never just concatenate.
Default compression for chat: rolling summary at 70 percent threshold.
Default for long-running agents: hierarchical tree.
Default for callback-heavy chats: retrieval over history.
Always truncate or summarize tool outputs.

Refs#

Liu et al., "Lost in the Middle: How Language Models Use Long Contexts" (2023)
Anthropic, "Prompt caching" documentation
OpenAI, "Prompt caching" cookbook
Packer et al., "MemGPT" (2023): context as virtual memory
Lilian Weng, "LLM Powered Autonomous Agents" (2023)

Context Window Management#

Why the window is finite even at 1M tokens#

The five strategies#

1. Truncation (sliding window)#

2. Rolling summarization#

3. Hierarchical / tree compression#

4. Retrieval-pruned context#

5. Tool-call output truncation#

Where this differs from agent memory#

Why finite-window thinking applies even at 1M tokens#

The prompt budget allocator#

The five compression strategies#

1. Sliding window truncation#

2. Rolling summarization#

3. Hierarchical / tree compression#

4. Retrieval-pruned context (RAG over chat history)#

5. Tool-call output truncation#

When to recompute the summary#

Measuring quality loss#

A reference token-budget allocator#

Production failure modes#

FAQ#

What is context window management in an LLM agent?#

Why do we need context window management if models support 200k or 1M tokens?#

What is the lost-in-the-middle problem?#

When should an agent recompute its rolling summary?#

How is context window management different from agent memory?#

How do you allocate a prompt budget for an LLM agent?#

What is hierarchical context compression?#

How is retrieval over conversation history different from RAG over documents?#

How do you measure quality loss from context compression?#

What is the fastest way to fit a long chat in the LLM context window?#

What is the lost-in-the-middle problem in one sentence?#

What is a healthy prompt budget split for a 200k token model?#

When should the rolling summary be recomputed?#

How do you know if compression is hurting quality?#

Related Topics#

Quick reference#

TL;DR#

Why bother with a 1M token model#

The five strategies#

Prompt budget slices (200k window)#

Summary recompute triggers#

Hierarchical compression layout#

Retrieval over chat history#

Tool-output strategies#

Cache-friendly ordering#

Measuring quality loss#

Watch-outs#

Quick reference (interview cheat)#

Refs#