Graded Relevance LLM Judge

A graded relevance LLM judge is an LLM-as-judge configured to emit a graded score per (query, document) pair instead of a binary relevant/not-relevant verdict. The standard scale is 0-3:

• 0 — Irrelevant. Document does not address the query in any meaningful way.
• 1 — Marginal. Document is on-topic but does not answer the query.
• 2 — Relevant. Document partially answers the query or provides directly useful context.
• 3 — Highly relevant. Document fully and directly answers the query.

This rubric is borrowed from the TREC graded-relevance tradition and operationalized via a frontier LLM (Claude, GPT, Gemini) prompted with the rubric, anchor examples, and a forced-format output. Run it across a benchmark and you get graded labels at human-comparable agreement, in hours instead of months.

The binary-relevance ceiling

Most retrieval benchmarks ship with binary labels — a (query, document) pair is either in the qrels file or it is not. MTEB ’s retrieval portion is binary. BEIR ’s majority of subsets are binary. Even MS MARCO ’s training data is overwhelmingly binary.

Binary labels reward “did you find a relevant doc” but cannot see the difference between “the most relevant doc is at position 1” and “the most relevant doc is at position 9, with eight unrelated docs above it.” Both score 1.0 on recall@10 . Both score 1.0 on hit@10. Two models with very different production behavior produce the same number.

The ceiling is not just theoretical. As models converge near the top of binary leaderboards, the per-model differences shrink to noise, and the leaderboard becomes uninformative. Every additional point of recall@10 is harder to win and means less in production. Graded labels reset the headroom by giving the metric something finer-grained to measure.

What graded relevance unlocks

The single most important consequence: graded labels make NDCG@K meaningfully different from recall@K. Under binary labels, NDCG@K and recall@K are highly correlated — both reward “find a relevant doc, place it high.” Under graded labels, NDCG@K rewards ordering by grade: a model that places a 3 above a 2 above a 1 scores higher than a model that retrieves the same three docs but orders them 2-1-3.

How to build the judge (rubric, CoT, calibration)

Three properties separate a graded-relevance judge that produces stable evaluations from one that does not:

Rubric design. Each grade has an explicit definition and a worked example. “0: irrelevant; 1: marginal; 2: relevant; 3: highly relevant” is the skeleton; the body is a per-grade anchor passage from a domain similar to the eval set. Without anchors, judges drift toward the middle of the scale.

Chain-of-thought before the grade. The prompt asks the judge to write a 1-3 sentence rationale enumerating which facets of the query the document addresses, then emit the integer grade. This is not cosmetic. Empirically, CoT-before-grade lifts inter-judge Cohen’s kappa from ~0.55 to ~0.75 on retrieval relevance — into the range of human-human agreement on the same task. The rationale forces the judge to do the reasoning that a 2-vs-3 distinction requires.

Forced output format. JSON with rationale and grade fields, parsed with a strict schema. Free-form output drops 5-15% of judgments to parse failures and silently biases the eval toward whichever queries had parseable judge outputs.

Multi-judge ensembling. Three frontier LLMs (Claude + GPT + Gemini) judging the same pair, with per-rater calibration via Thurstone or Beta fits, is meaningfully more robust than any single judge. Disagreements are diagnostic — pairs where the three judges split 0/2/3 are usually genuinely ambiguous and worth removing from the eval set.

What we found when we re-ran MTEB this way

ZeroEntropy reannotated 28 retrieval datasets from MTEB using exactly this methodology — three independent LLM judges, 0-3 scale, CoT before grade, per-rater calibration. The result was diagnostic in three specific ways:

• Models everyone assumed were great rose to the top. The reannotation rewarded models with well-ordered top-10 lists, regardless of leaderboard position.
• Models everyone suspected were leaderboard-tuned fell. Models that retrieved the relevant doc but ordered the top-10 poorly took visible NDCG hits under graded labels.
• Sibling-model gaps widened. Small/mid/large lineups in the same provider’s family — including ours, Voyage’s, OpenAI’s — that looked near-tied under binary labels showed clear ordering gaps under graded labels. The bigger models were bigger; binary was just blind to it. In effect-size terms, within-family Cohen’s widened by roughly - across most subsets — the formal signature of “the test got better,” in the sense developed in eval set quality .

The methodology, not the result, is the point. Any benchmark with binary qrels can be re-annotated with graded labels via a properly-built LLM judge, and most binary leaderboards become more informative — sometimes meaningfully different — under that treatment. The MTEB reannotation is the worked example.

Caveats — judge bias, contamination, cost

Graded-relevance LLM judges inherit every bias of the underlying LLM-as-judge stack, plus a few specific to the graded setting:

• Length bias amplifies on graded scales. Judges grade longer documents 0.3-0.5 grades higher on average, holding actual content constant. Mitigation: truncate all documents to a fixed token budget, or normalize via a length-matched control.
• Position bias does not apply directly (there is only one document per judgment), but anchor-prompt bias does — the order in which the 0/1/2/3 anchor examples appear subtly biases the judge. Mitigation: shuffle anchor order across runs.
• Contamination of the judge model. If the judge LLM was trained on the eval queries (e.g., the judge has seen MS MARCO during pretraining), grades skew toward what the judge “remembers” as relevant rather than what the document actually says. Use a held-out judge or a domain the judge demonstrably has not seen.
• Cost. A 28-dataset graded reannotation with three frontier judges per pair runs into the hundreds of thousands of API calls. Distillation into a smaller judge model is the practical move at production scale — same logic as zELO ’s pairwise teacher.

The right mental model: graded-relevance judging is a higher-resolution measurement instrument with the same systematic-error footprint as binary judging, plus a few new pitfalls specific to the rubric. The resolution gain dominates the new pitfalls when the benchmark is built carefully — which is why every modern reranker leaderboard is moving in this direction.