A context window is the maximum quantity of text a large language model can hold in view during a single inference pass, measured in tokens rather than words or characters. It is a hard architectural ceiling covering everything the model sees at once: the system prompt, conversation history, any retrieved documents, tool definitions and their outputs, plus the response being generated. Exceed it and something must be dropped. Frontier models now advertise windows of a million tokens or more, but the practical constraint has shifted—cost and accuracy degrade well before the stated limit is reached, which makes deciding what to exclude a more important engineering skill than finding a model that can technically fit everything.
For English prose, roughly 0.75 words per token—so 1,000 tokens is about 750 words, and a 128,000-token window holds something on the order of a 250-page book. Code, JSON, and non-Latin scripts tokenize far less efficiently, sometimes at more than double the token count for the same visible content. Always measure with the tokenizer of the model you are actually serving rather than estimating from character counts.
The request fails outright, or the framework silently truncates it—usually from the middle or the oldest turns. Silent truncation is the more dangerous outcome, because the model still answers fluently while missing the very instruction or document that mattered. Production systems should measure token counts before dispatch and fail loudly or summarize deliberately, never leave the trimming decision to a default.
No. Two effects work against it. Attention cost scales quadratically with sequence length, so a long prompt is materially more expensive and slower on every call. And models exhibit "lost in the middle" behavior—recall is strongest at the beginning and end of a long context and weakest for material buried in the centre. Feeding a model fifty documents when three are relevant typically produces a worse answer than retrieval would, at many times the price.
Input tokens are billed on every single call, and in agentic systems the accumulated history is resent with each step—so a conversation that grows unchecked pays for the same prefix repeatedly. Prompt caching, aggressive context pruning between turns, and summarizing older history into compact state are the standard levers, and they often cut inference spend more than switching to a cheaper model would.
Shakudo's AI Gateway gives teams a single control point for routing requests across models with different window sizes and price points, so a long-context model is used only when the payload genuinely requires it. Combined with vector database retrieval running inside your own infrastructure, this keeps prompts small and targeted—improving answer quality and reducing token spend at the same time, without proprietary context ever leaving your environment.
