There is a persistent misconception in how people talk about large language models: that they "remember" previous interactions, that they "learn" from a conversation, that they "know" something because you told them last session. None of this is accurate. Understanding why requires a clear look at the mechanism.
What Actually Happens When You Send a Message
Every time you send a message to an LLM, the model receives a single block of text. That block contains everything: your system prompt, the full conversation history, any retrieved documents, tool call results, and your latest message. The model processes this block from scratch, produces a response, and discards all state. Nothing persists.
The next message? Same process. The model receives another block — this time with the previous response appended — and processes it again from scratch.
This is the context window: a fixed-size buffer that holds everything the model can "see" at any given moment. It is not a database. It is not a memory system. It is a sliding document that gets rebuilt and re-read with every inference call.
The Difference Between Context and Memory
Human memory is associative, lossy, and persistent. You store experiences across time, reconstruct them imperfectly, and access them without re-reading every prior experience in sequence.
The context window is none of these things. It is:
- Complete: the model sees everything in the window, not a compressed summary
- Non-persistent: nothing survives past the current inference
- Bounded: there is a hard token limit, and exceeding it causes truncation or rejection
- Order-sensitive: position in the window affects how much attention a piece of information receives
This distinction matters practically. When you design a system that relies on an LLM, you are not designing a system with memory. You are designing a document assembly pipeline that constructs the right block of text before each inference call.
Token Budget as a First-Class Constraint
I cross-referenced three sources — Anthropic's documentation, observed behavior in production pipelines, and published research on attention degradation — and found consistent agreement on one point: token budget is a first-class engineering constraint, not an afterthought.
Every element you add to a context window has a cost:
- System prompt: 500–2000 tokens is typical; complex CLAUDE.md files can exceed 5000
- Conversation history: grows linearly with turn count
- Retrieved documents: each file or chunk adds directly to the count
- Tool definitions: each MCP server or function definition adds overhead
- Tool call results: can be substantial if the tool returns large payloads
A model with a 200k token context window sounds spacious until you account for all of these. In practice, usable space for retrieved knowledge is often 30–50% of the nominal limit.
What This Means for Knowledge Base Design
If context is a document, not a memory, then knowledge base design becomes document design. The question is not "how much can I store?" but "what should be in the document at inference time?"
Several principles follow from this:
Prioritize relevance over completeness. A knowledge base that injects 40 files into every request is not more informed than one that injects 5 well-chosen files. It is noisier. The model has to process everything in the window regardless of relevance, and irrelevant content competes with relevant content for attention.
Front-load the most important information. Research on attention patterns in transformer models consistently shows that information at the beginning and end of the context receives more weight than information in the middle. If there is something the model must not miss, it should not be buried at line 3000 of a system prompt.
Treat conversation history as a liability. Long conversations accumulate context. After 20–30 turns, a significant portion of the window may be occupied by early exchanges that are no longer relevant. Systems that do not manage history will degrade over time within a session.
Write for re-reading, not for recall. Because the model re-reads every document on every call, documents written with redundancy, headers, and explicit structure outperform dense prose. The model is not recalling something it "learned" — it is reading it again right now.
The Architecture Implication
Once you internalize that the context window is a document, not a memory, the architecture of intelligent systems changes. You stop asking "does the agent know this?" and start asking "is this in the document the agent will read?"
That reframe changes what you build. You build document assembly systems. You build handover documents for session continuity. You build retrieval pipelines that construct the right context block. You build compression routines for long conversation histories.
The model is not a participant with a past. It is a reader, and you are the author of what it reads.
Comments 0