Also known as: LLM judge, model-graded eval, LLM rater
Using a frontier LLM to score outputs — relevance, faithfulness, answer quality — at scale where human raters can't keep up. Powerful for graded labels, but introduces position bias, verbosity bias, model bias.
LLM-as-judge is the technique of using a frontier language model — Claude, GPT, Gemini — as an automated rater. Given a query, a candidate output, and a rubric, the LLM emits a score (or a pairwise preference). Repeat across thousands of examples, average, and you have an evaluation signal where you couldn’t afford humans.
The technique is now load-bearing infrastructure for retrieval and RAG evals: the graded relevance labels behind ZeroEntropy’s MTEB reannotation, the faithfulness scores in modern RAG benchmarks, and the pointwise targets behind zELO reranker training all come from LLM judges.
Three properties make LLMs surprisingly good raters:
These aren’t speculative — they’re measurable and large enough to flip rankings. Mitigations:
A judge rubric that produces stable scores has three properties: explicit category definitions, anchor examples, and a forced output format. The TREC graded-relevance scale is a textbook example — “0: not relevant; 1: related but not directly useful; 2: relevant but partial; 3: highly relevant and complete” — each level pinned to a worked example. Forced output format means the prompt requests JSON with a single score field plus a rationale, so parsing is robust and you can audit disagreement cases. The biggest failure mode is the tempting open-ended rubric (“rate quality from 1 to 10”) — agreement collapses because the judge has no shared meaning of “8 vs 9,” and small prompt edits swing the average score by 1+ points.
Pairwise judges are more reliable but produce
This is exactly what zELO does for reranker training, and it’s what produces the calibrated score-calibration that downstream thresholding relies on.
The right mental model: it’s not a replacement for human evaluation — it’s a force multiplier that lets you scale to corpus sizes where human eval was impossible. Sanity-check with humans on a sample, always.
LLM-as-judge is human-in-the-loop with the loop closed by a model.
Position bias (preferring whichever answer comes first or last), verbosity bias (longer answers look better), self-preference bias (a model rates its own outputs higher), and consistency drift (the same prompt scored differently across calls). All of these are measurable, and most are mitigable with rubric design and randomized order.
Pairwise. LLMs are more consistent at 'is A better than B?' than at 'is A a 7 out of 10?'. The downside is O(N²) comparisons. The standard fix: collect pairwise preferences, fit a Thurstone or Bradley-Terry model to recover continuous scores, then train a pointwise model against those scores. zELO does exactly this.
On well-defined tasks (relevance, faithfulness, factual correctness), Cohen's kappa with humans is typically 0.6-0.8 — comparable to human-human agreement on the same task. On open-ended quality judgments it's lower, ~0.4-0.6, and the variance across LLM judges is enough to matter. Always sample human-judge a subset to measure agreement.
