Skip to main content
AI & Agents

Context Window

A context window is the maximum amount of text a language model can consider at one time, measured in tokens and shared by the system prompt, conversation history, retrieved documents and tool results. Anything outside the window is invisible to the model, so what goes in — and what gets evicted — is a core design decision.

A token is roughly three-quarters of an English word, and window sizes have grown from 4,096 tokens in early GPT-3 to 128,000 in GPT-4-class models and a million or more in Gemini-class models. The growth is real and useful, but the window is shared space: verbose tool output, a long system prompt and retrieved passages all draw from the same budget.

A larger window is not the same as a better answer. Models attend unevenly to long contexts — content in the middle is recalled measurably worse than content at the beginning or end, the "lost in the middle" effect documented in 2023 and still visible in practice. Cost and latency also scale with input length, so stuffing an entire knowledge base into the window is slower and more expensive than retrieving the three passages that matter.

Production systems therefore treat context as a budget to be managed. The system prompt and tool schemas are reserved first, retrieval injects only ranked passages, conversation history is summarised once it ages, and tool results are truncated or compacted before being returned to the model. Multi-step agents fill windows faster than chat does, because every action adds an observation, and context exhaustion mid-task is one of the commonest agent failures.

Eviction strategy deserves as much design attention as inclusion. A sliding window over raw history preserves recency but forgets commitments made earlier in the conversation. Periodic summarisation by a smaller model compresses history at the cost of losing exact wording. Retrieval over past turns brings back what is relevant to the current question and nothing else. Most production systems combine all three, and token counts are tracked before every call rather than discovered from an API error after one.

A growing share of agent design is deciding what should not live in the window at all. Long-term facts belong in an external memory store, large documents in a retrieval index, and bulky tool output behind a reference the model can page through on demand. The window is for what the model must reason over right now; treating it as storage is how systems become slow, expensive and unreliable simultaneously, and the fix is architectural rather than a bigger model.

Codazz builds this in production — LLM Integration.

FAQ

Context Window
FAQ.

Common questions about context window.

Ask Us Anything

Depending on the provider and framework, the API either rejects the request or silently truncates the oldest content — and silent truncation is worse, because the model then answers without information it appears to have. Production systems manage the window explicitly: summarising history, capping tool output and tracking token counts per request.

No. Retrieval is cheaper per query, faster, easier to permission per user, and produces answers with citations to specific sources. Long context is excellent for one-off analysis of large documents; for a corpus that changes and has access rules, retrieval remains the correct architecture.

Input tokens are billed on every call, so context is a recurring operating cost rather than a one-time design choice. A 2,000-token system prompt costs that on every request regardless of the user's message, and a chatty agent loop can multiply a short question into tens of thousands of input tokens. Trimming boilerplate, caching stable prompt prefixes where the provider supports it, and capping tool output are the levers that show up on the invoice.

Related terms

RAG (Retrieval-Augmented Generation)

RAG is a technique where an AI system retrieves relevant passages from your own documents at query time and generates an answer grounded in them, with citations.

Agent Memory

Agent memory is the set of mechanisms an AI agent uses to retain information beyond a single model call: short-term memory held in the context window for the current task, and long-term memory written to an external store and retrieved when relevant.

Embedding

An embedding is a list of numbers representing the meaning of a piece of content, produced by a model so that semantically similar items sit close together in the resulting space.

Browse the full glossary →