What is the Context Window?

The context window is the maximum number of tokens —fragments of text between a word and a character— that a language model can process in a single interaction. It acts as the model's short-term memory: everything inside the window the model "sees"; everything outside does not exist to it. Understanding this limit is the difference between a coherent answer and a silent failure.

Pro Tip: When your prompt or conversation approaches the window limit, the model starts "forgetting" the earliest parts. If you work with long documents or extended chats, monitor token usage like you monitor your laptop battery.

What is it

Every language model has a maximum token limit it can handle at once. That limit is its context window. It includes both the message you send (prompt) and the response the model generates.

Common window sizes as of 2026:

Note: "Tokens" are not exact words. A token can be a whole word (dog → 1 token), a syllable (learning → 2 tokens: learn + ing), or even a single character. As a rule of thumb, 100 tokens ≈ 75 English words.

Mental model

Imagine a work desk. The context window is the desk size. You can place documents, instructions, examples, and the ongoing response on it. Everything that fits on the desk, the model considers. Everything that does not fit is left out and the model cannot use it.

If you overload the desk —put more tokens than it can hold— the model does not complain. It simply drops the oldest content to make room. This is called "lost-in-the-middle" and is the most common source of weird errors in LLM applications.

Three types of content compete for space in the window:

1. System instructions: the base prompt defining the role and rules.
2. External context: retrieved documents (RAG), conversation history, examples (few-shot).
3. Generated response: the text the model is producing.

Warning: If your system prompt is 2K tokens, you add RAG with 50K tokens, and expect a 10K token response, you need at least 62K tokens of window. Always sum everything before choosing a model.

How it's used

Managing the context window is not something you "install" — it is administered through design decisions for each interaction.

1. Know your model's limits

Before building any application, check the window size of the model you will use. Do not assume all models have 128K. Smaller or older models may have only 4K or 8K.

2. Calculate token usage

Use a tokenizer (the same as the model's) to measure how much space you occupy.

# Example with tiktoken (OpenAI)
import tiktoken
 
encoding = tiktoken.encoding_for_model("gpt-4o")
tokens = encoding.encode("Your text here")
print(len(tokens))

3. Define a management strategy

Three common strategies:

• Truncation: cut the oldest content when you exceed the limit. Simple but loses information.
• Summarization: periodic summaries of history to compress information. More accurate but adds latency and cost.
• Sliding window: keep only the last N tokens of conversation. A balance between simplicity and retention.

4. Monitor in production

In real applications, log token usage per interaction. When a query approaches 80% of the limit, trigger alerts or automatically apply your management strategy.

Pro Tip: For long conversations, use "cumulative summarization": every N messages, ask the model to summarize the conversation up to that point in 100 tokens, and use that summary as context instead of the full history. This stretches your effective reach without increasing token count.

When to use it / when not to

Pay attention to the context window when:

• Building chatbots with long history (customer support, tutors, assistants)
• Processing large documents with RAG (each retrieved chunk takes space in the window)
• Using few-shot prompting with many examples (each example adds tokens)
• Accuracy matters and you cannot afford the model "forgetting" early instructions

You do not need to obsess when:

• The interaction is a single short question and answer (fits any window)
• Using models with very large windows (1M+ tokens) and your content is moderate
• The risk of the model forgetting something has no serious consequences (personal prototypes, experimentation)

Pro Tip: When migrating from one model to another (e.g., GPT-4o's 128K to Gemini's 1M), do not assume everything works the same. Models with giant windows often have lower retrieval accuracy in the middle of the window ("needle in a haystack" problem). Always test with your real data.