Temperature is just how much you let step 3 stray from the highest-probability choice. Zero means always pick the top token.
What LLMs actually do
T'Chaka Dev· 9 pages· 5 min read
A language model does exactly one thing: given a sequence of tokens, it produces a probability distribution over what the next token might be. Every capability people find impressive is that single operation, run in a loop.
Tokens
A model never sees characters or words. Text is split into tokens — common chunks, roughly three-quarters of a word each — and each token is mapped to a number.
DIAGRAM a sentence split into labeled token boxes, each with its integer id underneath
This is why models are strange about spelling. They are not looking at letters.
The prediction loop
- Read the entire context.
- Produce a probability for every token in the vocabulary.
- Sample one.
- Append it to the context and go back to step 1.
DIAGRAM the four-step loop drawn as a cycle, with the context growing by one box each pass
Context
The context window is everything the model can see at once — your prompt, the conversation, any documents you pasted, and its own output so far. Nothing outside that window exists as far as the model is concerned.
What this explains
Once the loop is clear, most model behaviour stops being mysterious:
- Hallucination — a plausible next token is not a true one
- Cost scaling with length — the loop runs once per token
- Prompt sensitivity — the prompt is literally the starting context
- Cutoffs mid-sentence — the loop hit a token limit, not a thought