AI context windows, explained.

Everything a model "knows" during a conversation is either baked into its weights at training time or present in the window right now. There is no third place. That single fact explains most of the practical behavior people find surprising: why chatbots forget, why long conversations drift, and why pasting a 200-page PDF doesn't work as well as expected.

Tokens, not pages

Windows are measured in tokens — subword units averaging roughly ¾ of an English word. A 128K-token window is on the order of 300 pages; a 1M-token window, several novels. Sounds infinite. It isn't, for three reasons.

Why bigger windows didn't end the problem

1. Attention degrades over long inputs

Models attend unevenly across very long contexts. Information buried mid-window is recalled less reliably than information near the start or end — the well-documented "lost in the middle" effect. Stuffing the window trades retrieval precision for the illusion of completeness: the fact is technically in context and practically invisible.

2. Cost and latency scale with input

You pay per token, per request. Carrying 400K tokens of "maybe relevant" material on every turn multiplies cost and slows responses — for context the model mostly ignores. Efficient systems send the relevant few thousand tokens, not the available few hundred thousand.

3. The window still resets

This is the limit no window size fixes: when the session ends, the window empties. A 10M-token window that resets is still amnesia — just roomier. Persistence requires a store outside the window entirely.

The working-memory model

The productive mental model comes from computing's memory hierarchy: the context window is RAM, not disk.

• Window (RAM) — fast, expensive, volatile. Holds what's needed for the current task.
• Memory store (disk) — durable facts that survive sessions: decisions, preferences, state. Loaded into RAM on demand via semantic recall.
• Knowledge corpus (disk) — indexed documents, from which retrieval loads only the relevant chunks per query (how retrieval works).

Nobody solves RAM limits by buying enough RAM to hold their whole hard drive. You page in what you need. Retrieval and memory are exactly that paging mechanism for AI — which is why they matter more, not less, as windows grow: the bigger the window, the more tempting the anti-pattern of filling it.

Practical implications

For everyday AI use

• Don't re-paste stable context every session — store it once (memory) and let recall inject it when relevant.
• Don't paste whole documents when you need answers from them — index them and let retrieval pick chunks (a knowledge base).
• Long conversations drift because early turns effectively fade; important conclusions should graduate from the window to the store: "remember this decision."

For agents and automation

• Each run gets a fresh window; anything worth keeping must be written out before exit (agent memory patterns).
• Tool outputs can flood a window fast; summarize-then-store beats carry-everything.

How the pieces fit

A well-engineered session composes its window per request: standing instructions (small), recalled memories relevant to the question (smaller), retrieved chunks from the corpus (targeted), and the live conversation. That composition discipline is context engineering — and services like VeelIQ Context exist to make the store-and-retrieve side of it automatic: the assistant calls recall and search_knowledge over MCP and pages in precisely what the moment needs.

Key takeaways

• The window bounds a single request; it is working memory, not storage.
• Bigger windows help but don't repeal the three limits: attention degradation, token economics, and session reset.
• Durable AI systems pair the window with external memory and retrieval — RAM plus disk, paged on demand.
• The skill isn't filling the window; it's composing it.

Related reading