Overview
A context window is the maximum amount of text (measured in tokens) a language model can process in a single interaction. It includes everything — system prompts, conversation history, retrieved documents, and the model’s response. Context window management is a core challenge in AI application design because exceeding the window causes information loss, while using it inefficiently wastes capacity on irrelevant content. Modern models offer windows from 8K to 200K+ tokens, but attention quality typically degrades with length, making strategic content placement essential even when window size is sufficient.
Key Concepts
- Tokens — The atomic units of text processing. Roughly 4 characters or 0.75 words per token in English. Different models use different tokenizers.
- Context engineering — Deliberately managing what enters the window: prioritizing high-value information, summarizing verbose content, and structuring placement for attention.
- Lost-in-the-middle — Research showing models attend more strongly to information at the beginning and end of the context, with weaker attention in the middle.
- Sliding window — Dropping older messages as new ones arrive. Simple but loses potentially important early context.
- Summarization — Compressing earlier conversation into summaries to preserve key information while freeing token space.
- RAG integration — Retrieval-augmented generation injects relevant documents into the context window on demand rather than stuffing everything upfront.
- Token budgeting — Allocating portions of the window to different purposes (system prompt, history, retrieved docs, response space).
Trade-offs
- Larger windows enable more context but increase latency and cost
- Summarization preserves space but loses detail and nuance
- Stuffing context improves recall but degrades attention quality
- Token counting is model-specific and adds implementation complexity
Related
- Prompt Engineering operates within context window constraints
- RAG manages context by dynamically retrieving relevant information
- Agent Patterns must account for context window limitations in multi-step reasoning