Autoregressive Generation

Autoregressive generation is how every decoder-only LLM produces text. The model predicts a probability distribution over the next token given everything seen so far, samples one token from that distribution, appends it to the sequence, and repeats. Stop on an end-of-sequence token or a length limit. It’s a structurally simple loop — and the main reason LLM latency, cost, and throughput look the way they do.

The loop

Given a prompt , the model generates output as follows:

1. Run the prompt through the model. The final layer produces logits over the vocabulary at position .
2. Convert to a probability distribution: .
3. Sample from this distribution (or take its argmax for greedy decoding).
4. Append to the sequence.
5. Run the model again on the extended sequence to get logits at the new last position.
6. Sample , append, repeat.

Stop when the model produces an end-of-sequence token or a maximum length is reached.

Why it’s the bottleneck

Reading the prompt and generating each new token are computationally very different:

• Prefill processes all prompt tokens in parallel — a few large matrix multiplies. GPUs love this.
• Decode generates each new token sequentially — token requires token ‘s output. No parallelism within a request.

This is why a 1000-token prompt with a 50-token answer takes much less time than a 50-token prompt with a 1000-token answer, even though the total work looks similar. Per-token decode latency is roughly constant; total latency scales with output length.

The KV cache, briefly

Naive autoregressive generation re-runs the full transformer over the entire sequence at every step — work to generate tokens. The KV cache avoids this by storing the keys and values from previous positions; each new step only computes , , for the new token and reads cached , for prior positions. This drops generation to total work — the reason inference is even feasible.

Sampling strategies

The choice of how to sample from the next-token distribution shapes the output:

• Greedy. Always pick the argmax. Deterministic, but often dull or repetitive.
• Temperature. Scale logits by before softmax. sharpens (more confident, less diverse); flattens.
• Top-k. Restrict to the highest-probability tokens before sampling.
• Top-p (nucleus). Restrict to the smallest set of tokens whose probabilities sum to at least .
• Beam search. Maintain candidate sequences, expand each, keep the best by joint probability. Better for translation and code; rarely used for open-ended chat.

For RAG and retrieval-adjacent uses, low-temperature or greedy decoding is typical — you want determinism, not creative variation.

Implications for production

Per-token cost is the structural reality of LLMs. It’s why output-length budgeting matters in prompts, why a reranker that surfaces the right 5K tokens out of 1M wins (the LLM emits a short, on-target answer instead of plowing through everything), and why specialized small models — single score or vector in one parallel forward pass, no loop — are so much faster than asking an LLM to do the same task.