Also known as: retrieval-augmented generation, retrieval augmented generation
RAG is the pattern of retrieving relevant documents and feeding them into an LLM as context, so the LLM can answer with grounded, citeable information instead of guessing from its training data.
RAG (Retrieval-Augmented Generation) describes any system where an LLM’s response is conditioned on documents fetched at query time. Rather than rely solely on what the LLM memorized during training (which is stale, lossy, and uncitable), RAG augments the model’s context with fresh, source-attributable text retrieved from a corpus you control.
The canonical pipeline:
Raw user input enters the system, optionally with conversation history.
Typically hybrid search or dense embeddings over a vector index returns N candidate documents.
A cross-encoder reorders the candidate set so the actually-relevant docs are at the top.
The top-K reranked documents are formatted into the LLM’s prompt, often with citations.
The LLM produces an answer grounded in those documents.
The response is checked against the retrieved context.
The famous failure mode of RAG is hallucination, but hallucination is downstream of retrieval failure. If step 2 returns the wrong documents, the LLM will dutifully synthesize an answer from the wrong context — and sound confident. That’s why retrieval quality (Recall@K) and reranking quality (NDCG@10) matter so much for end-to-end RAG quality, even though the user never sees those metrics directly.
A common production rule of thumb: a 1-point NDCG@10 improvement on the reranker maps to a measurable lift in user-perceived answer quality. The signal compounds.
Frame the system as a pipeline. End-to-end answer quality is bounded by
Hallucination in RAG is almost always a retrieval bug. Fix the retrieval — the generator is rarely the bottleneck.
Almost always retrieval quality, not the LLM. A 1-point NDCG@10 lift on the reranker compounds into measurable answer-quality lift; a bigger generator on top of bad context just produces confidently-wrong answers faster.
Two levers: shrink the prompt with a cross-encoder reranker so only top-K passages survive, then compress further if you're still pushing the model's effective context window. Long, redundant context is the main driver of [context rot](/concepts/context-rot/) failures.
Aggregation queries ('how many', 'what's the total'), tasks that require executing code or hitting an API, and questions whose answer is implicit across thousands of documents. For those you want structured query, tool-use, or summarization pipelines — not similarity search.
Posts on the ZeroEntropy blog that reference rag (retrieval-augmented generation).
