Tokens, Context Windows, and Why Prompts Matter
Every model has a finite short-term memory measured in tokens. Understanding it explains why prompts get "forgotten," why long chats drift, and how to keep answers sharp.
A token is the unit an LLM reads and writes. It's usually a word fragment — "learning" might be one token, "tokenization" a few. As a rough guide, 100 tokens is about 75 English words.
The context window
The context window is the maximum number of tokens the model can consider at once — your prompt, the conversation so far, and its own reply all count against it. It is the model's entire short-term memory. Nothing outside the window exists to the model.
- Too much filler pushes important instructions out of the window.
- In long chats, early messages get truncated or summarized away.
- Retrieval and tools let you pull in only the relevant context on demand.
Practical prompting
Because attention is spread across everything in the window, signal-to- noise matters. Put the most important instructions where the model will weigh them, be specific about the output you want, and give concrete examples rather than long abstract descriptions.
Weak: "Summarize this."
Strong: "Summarize this in 3 bullet points for a non-technical
reader. Each bullet under 15 words."