Back to all articles

How LLMs Work: What Happens Between Your Prompt and Its Reply

A plain-English, diagram-led walkthrough of what a large language model does between the moment you hit send and the moment its reply appears.

7 min read
How LLMs Work: What Happens Between Your Prompt and Its Reply

TL;DR: your prompt gets chopped into tokens by a BPE tokenizer, each token becomes a list of numbers, and the whole prompt is run through the transformer once in a parallel prefill pass that caches every token’s key/value vectors. From there the model decodes one new token at a time — reusing that cache, scoring the full vocabulary, sampling one token, and streaming it to you — until a stop token shows up. That prefill-then-decode loop is the entire reply.

Ask ten people “how does ChatGPT work?” and most will say some version of “it’s trained on the internet and it knows stuff.” True, but it skips the one detail that actually explains everything else about how these models behave: an LLM never writes a sentence. It writes one token, looks at what it just wrote, and writes the next one. Over and over, until it decides to stop.

Once that clicks, a lot of “weird” LLM behavior stops being weird — why it can trail off mid-thought, why it sometimes states a wrong fact as if it were obviously true, why raising “temperature” makes it more creative and less predictable. This post walks the entire round trip, from your keystrokes to its last word, with a diagram for every stage.

One distinction worth making upfront: everything below is inference — running an already-trained, frozen model on your one prompt. Training is a separate, one-time process (weeks of compute over huge datasets) that happens long before you ever type anything, and it’s what produces the billions of fixed weights that inference just reads from. Your conversation never changes those weights; it only flows through them.

The big picture

Before the details, here’s the whole trip in four stops: your words get split into pieces, those pieces become numbers, a stack of math turns the numbers into a prediction, and the prediction gets turned back into words you read.

Four stages of an LLM request: you write a prompt, it's split into tokens, the transformer predicts one token at a time, and you read the streamed response

Everything below is “stage 2” and “stage 3” zoomed in, because that’s where the actual intelligence lives.

Step 1: Tokenization — chopping your words into pieces

An LLM doesn’t read letters or whole words. It reads tokens — chunks of text that are sometimes a whole word, sometimes half a word, sometimes a single punctuation mark. A tokenizer, trained once ahead of time, holds a fixed vocabulary (commonly 30,000–100,000 entries) and a set of rules for splitting any input into pieces from that vocabulary.

For example, a common word like "the" is usually one token, but a rarer or longer word gets split:

"unbelievable" → ["un", "believ", "able"]

This is why LLMs can handle made-up words, typos, and languages they’ve barely seen: if a chunk isn’t in the vocabulary as a whole word, it just gets broken into smaller, familiar pieces. Each token is then swapped for a plain integer ID — “the” might be 464, “cat” might be 3797 — because everything from here on is pure arithmetic, not text.

The splitting rule itself is usually learned ahead of time with an algorithm called byte-pair encoding (BPE): start from individual bytes, then repeatedly merge whichever adjacent pair shows up most often in a huge training corpus, until the vocabulary reaches its target size. OpenAI’s tiktoken, Google’s SentencePiece, and Hugging Face’s tokenizers library are the common frameworks that implement this (or close variants like WordPiece) — different model families each ship their own trained vocabulary, which is why the same text can split into a different number of tokens depending on which model you ask.

Step 2: Embeddings — turning IDs into meaning

A token ID like 3797 is just an index; it carries no meaning on its own. So each ID is looked up in an embedding table and swapped for a vector — a list of a few thousand numbers that represents that token’s meaning, learned during training. Tokens used in similar contexts end up with similar vectors: “cat” and “dog” sit closer together in that number space than “cat” and “spreadsheet” do.

One more thing gets added at this stage: position. “The dog bit the man” and “The man bit the dog” use identical tokens in a different order, so the model adds positional information to each vector before anything else happens — otherwise it would have no idea which word came first.

Step 3: Inside the transformer — attention and feed-forward layers

This is the part people mean when they say “the model.” A modern LLM stacks dozens of identical blocks on top of each other, and every block does two things to every token’s vector:

  • Self-attention: each token’s vector looks at every other token’s vector in the input and asks, “how relevant are you to me?” This is how the model resolves things like pronouns — in “the trophy didn’t fit in the suitcase because it was too big,” attention is what lets the model figure out that “it” more likely refers to the trophy.
  • Feed-forward layers: a smaller, per-token calculation that refines what attention just gathered, adding the nonlinear processing that lets the model represent more than simple averages of its inputs.

The transformer stack from bottom to top: token embeddings, self-attention, feed-forward layers, and output probabilities

Stack this block 30, 60, or 100+ times, and each repetition builds a richer, more context-aware representation of the text. By the final layer, the last token’s vector encodes not just what it is, but what it means given everything that came before it in the whole conversation.

Step 4: Predicting the next token

At the very top of the stack, that final vector gets multiplied against the embedding table in reverse: instead of token → vector, it’s vector → a raw score for every single token in the vocabulary. Tens of thousands of scores, one per possible next token.

Those raw scores (called logits) aren’t probabilities yet — they’re just numbers, and some are negative. A function called softmax squashes all of them into a tidy probability distribution that adds up to 100%. The result might look like:

" sat"    → 41%
" ran"    → 22%
" jumped" → 9%
" is"     → 3%
...

Notice this is a prediction over the entire vocabulary, not a single guess — the model always considers every option before one gets picked.

Step 5: Sampling — how a token actually gets chosen

With probabilities in hand, the model needs to pick one. The simplest approach, greedy decoding, always takes the highest-scoring token — reliable, but it can feel stiff and repetitive. In practice, most chat models sample instead, and two settings control how:

  • Temperature reshapes the probabilities before picking. Near 0, it behaves almost like greedy decoding (safe, predictable, a bit flat). Higher values flatten the distribution, giving lower-probability tokens a real chance — more variety, more risk of an odd choice.
  • Top-p (nucleus sampling) throws away the long tail of unlikely tokens first, then samples only from the smaller set that makes up, say, the top 90% of probability mass — so the model never picks something wildly implausible, even at higher temperature.

This is the step that explains why the same prompt can get a different reply every time: picking a token is a weighted dice roll, not a lookup.

Step 6: Prefill, then decode — one token at a time, streamed

Here’s the detail that ties the whole thing together: the model does not produce a sentence in one pass, and it does not treat your whole prompt the same way twice. There are two distinct phases:

  • Prefill: the very first forward pass runs over every token of your prompt at once, in parallel — that’s the fast part, GPUs are good at it. While doing this, the model saves each prompt token’s attention key and value vectors in memory. This saved memory is the KV cache, and building it is why there’s often a short pause before anything appears, sometimes called time to first token (TTFT).
  • Decode: every token after that is generated one at a time. Each decode step only has to compute attention for the one new token, reusing the already-cached keys and values instead of recomputing attention over the whole conversation from scratch. That token gets appended to the input, and the entire stack — embed, attend, predict, sample — runs again to produce the next token. This repeated, cache-reusing step is called autoregressive decoding.

As each decode step finishes, its token is immediately detokenized back into text and streamed to your screen — that’s why a chat reply appears to type itself: it genuinely is being generated and sent one piece at a time, not played back from a finished answer.

Sequence diagram: you send a prompt to the decoder, which sends tokens to the model, gets back next-token probabilities, samples and repeats, then sends you the finished reply

The loop keeps going until one of two things happens: the model generates a special stop token that means “I’m done,” or a maximum length limit is hit and generation is cut off regardless. There’s no separate “wrap up the answer” step — “knowing when to stop” is just another token prediction, like any other.

The context window — how much the model can see at once

Every model has a hard limit on how many tokens it can hold in one go — your prompt, the whole conversation history, and every token it has generated so far, all added together. This is the context window: commonly somewhere between 8,000 and a few hundred thousand tokens, depending on the model.

The limit isn’t arbitrary. Self-attention (step 3) compares every token to every other token, and the KV cache (step 6) keeps growing as the conversation grows — both get more expensive, in compute and memory, the longer the context gets. The window is set wherever that cost stops being practical for a given model.

A useful mental model: the context window is the model’s entire field of view for this one request. Nothing before it and nothing outside it exists as far as that forward pass is concerned — there’s no separate long-term memory being consulted. If a long conversation starts to “forget” something you said earlier, it’s usually because that message has scrolled outside the window, not because the model chose to ignore it.

Putting it all together

That’s the entire path from your prompt to its reply: six steps, repeated as many times as the response has tokens. Here’s the whole loop condensed into something you can hold in your head.

Six-step cheat sheet: tokenize, embed, attend, predict, sample, repeat

A short way to say it out loud: chop it, number it, attend to it, score it, pick one, and do it again. Once you can recite those six words, you already understand more about how an LLM works than the architecture diagram alone would teach you — because every quirk people notice (confident wrong answers, varied replies, mid-sentence cutoffs) traces back to one of these six steps.

If you’re deploying one of these models instead of just using it, see how to deploy an AI agent with Amazon Bedrock for the operational side: packaging, IAM, and invoking it from your own code.

Frequently asked questions

Is an LLM looking up answers from a database?

No. It has no lookup step at response time. Everything it "knows" is baked into billions of numeric weights during training, and it reconstructs an answer by predicting one token at a time from those weights.

Why do LLMs sometimes make things up (hallucinate)?

Because the model is doing next-token prediction, not fact-checking. Each token only has to be statistically plausible given what came before, so a fluent, confident-sounding sentence can still be wrong.

What does "temperature" actually control?

How much randomness is allowed when picking the next token. Temperature 0 always takes the single highest-probability token (deterministic); higher values let lower-probability tokens win sometimes, making replies more varied.

Why does ChatGPT's reply appear word by word instead of all at once?

Because that's how it's generated, not a typing animation. The model produces exactly one token, sends it to you, then uses it (plus everything before it) to produce the next one.

Why is there a short pause before the first word, then a steady stream after that?

The pause is "time to first token" — the one-time cost of the prefill phase, where the whole prompt is processed in one parallel pass. After that, each new token reuses the cached KV values from everything before it, so decode runs at a steady, much cheaper pace per token.

Is this post describing training or inference?

Inference. Training happens once, offline, over weeks and adjusts billions of weights by learning from huge datasets. Inference is everything this post covers — running that already-trained, now-frozen model on one prompt to produce one reply. The weights don't change between requests.

What happens when a conversation goes past the context window?

Something has to give. Most chat apps quietly drop or summarize the oldest messages so the most recent ones still fit; raw APIs instead return an error if the prompt alone is already too long. Either way, tokens outside the window are invisible to the model — it isn't choosing to forget them, it genuinely cannot see them.

Written by

Emdadul Islam

I build web, mobile and backend products, and write up what I learn along the way: the bugs, the trade-offs and the fixes that held up in production. More about me →

Next article

Write fast, reliable Go integration tests against a real PostgreSQL database with Testcontainers: setup, isolation, parallel tests and GitHub Actions CI.