Also known as: confabulation, ungrounded generation, factual error
Hallucination is when an LLM generates a confident-sounding statement that's factually wrong or unsupported by the input. It's the load-bearing failure mode of LLMs in production.
Hallucination is when a large language model generates content that’s plausible-sounding but factually wrong, fabricated, or unsupported by the input. The LLM doesn’t know it’s hallucinating — its tone is identical to when it’s correct, which is what makes the failure mode dangerous in production.
Pre-training optimizes the model to predict likely next tokens given everything seen so far. The objective rewards plausibility, not factual accuracy. Several specific failure modes:
The bias toward plausibility is structural. Pre-training data overwhelmingly contains confident, declarative text; very little of it is “I don’t know” or “I’m uncertain about X”. The model learns the distribution, which is the problem. RLHF/DPO can push toward more cautious outputs but at the cost of helpfulness — over-aligned models refuse too often.
Calibration improvements (the model’s stated confidence matching its accuracy) help downstream systems decide when to trust the output. But calibration is hard to achieve via training alone; it usually requires explicit supervision signal that doesn’t exist at scale.
RAG changes what the model is hallucinating from. Without retrieval, the model draws on parametric memory — every fact in its training data plus every plausible-looking interpolation — which is a vast space, and most failures are confident memory recalls of slightly-wrong facts. With retrieval, the bulk of the model’s evidence is grounded in the prompt, which sharply reduces the prior on confabulating from training memory. But two failure modes survive: (1) the retrieved context doesn’t contain the answer, and the model falls back on parametric memory anyway; (2) the retrieved context contains the answer but the model synthesizes a claim that goes beyond what the context says (over-generalization, aggregation drift, hedge inversion). The first is fixed by better retrieval; the second is fixed by faithfulness checking on top. Neither is a model-level intervention — both are architectural.
The dominant pattern in production AI is to never trust LLM-generated facts at all. Instead:
Hallucination is the central reliability problem in production LLM applications, not an edge case. The whole architecture of modern RAG — retrieve, rerank, ground, verify — is defense in depth against this single failure mode.
The architecture of modern RAG is the architecture of defense-in-depth against hallucination. Retrieval sharpens the prior, reranker raises the supporting passage, faithfulness checks verify the output. Each layer assumes the others won’t be perfect.
It reduces it dramatically but doesn't eliminate it. The model can still synthesize claims that go beyond what the retrieved context actually says, especially when retrieval is incomplete. RAG turns 'hallucinating from training memory' into 'hallucinating from a smaller, grounded prompt' — better, but the failure mode persists. Citation extraction and answer-grounding checks are the next layer of defense.
Pre-training rewards plausibility, not truth. The model is trained to produce text that looks like its training distribution; saying 'I don't know' is statistically rare in the training data, so the model is biased toward generating something that sounds right. Calibration tuning helps, but the underlying bias survives.
The robust pattern is to verify each claim in the LLM's output against the retrieved context — typically with a small fine-tuned model that does sentence-level entailment ('does this claim follow from the retrieved doc?'). Outputs that fail entailment get flagged or regenerated. Specialized faithfulness models are increasingly the right tool for this; a frontier LLM doing the same job is slow and expensive.
