Calibration-Discrimination Analysis

You have two scoring systems on the same set of items — say a first-pass retriever’s similarity and a reranker’s relevance score, or a model’s predicted relevance and a graded-relevance judge’s label. The standard summary is Pearson or Spearman correlation. That gives you one number. The technique here gives you a per-item residual, lets you stratify those residuals by any known subset, and turns the gap between subset distributions into a Youden-J-style discrimination statistic.

The mental picture: regress one score on the other. The fitted line is the global agreement. The signed perpendicular distance from each point to the line is its disagreement. If a known-positive subset has systematically larger residuals on the agreement-positive side, the two scorers are jointly identifying positives, and the size of the gap tells you how cleanly.

The Setup — Two Scoring Systems, One Ground-Truth Subset

Concretely, you have:

• A set of items — typically (query, document) pairs.
• Two scores per item: (e.g., retriever similarity) and (e.g., reranker score).
• A subset of ground-truth positives — items you know are correct, from labeled qrels or a held-out gold set.

Fit a linear regression on the full set. For each item, compute the signed residual . Positive residuals mean ” scored higher than would predict”; negative residuals mean ” scored lower.”

Residuals as Agreement Signal

Two scoring systems that perfectly agree (modulo a linear rescaling) have residuals concentrated tightly around zero. Two systems with weak agreement have residuals spread wide. The interesting case is when residuals are spread and structured — when subsets of items reliably sit above or below the regression line.

For our purposes, “structured” has a specific meaning: the ground-truth-positive subset should have residuals systematically above zero. The intuition: a positive item is one both scorers are likely to score high on, and “both high” lifts the residual above the regression-implied baseline. A non-positive item that one scorer happened to over-score and the other appropriately under-scored has a residual driven by single-scorer noise, not joint agreement.

If you plot two histograms — residuals over vs residuals over the complement — they should be displaced. The displacement is the discrimination signal.

Discrimination Via Residual-Distribution Divergence

For a candidate threshold , define:

This is exactly the gap, at threshold , between the GT-positive residual survival function and the general residual survival function. The argmax is the Youden-J statistic for the residual distribution:

is the maximum vertical gap between the two ROC-style curves traced out as sweeps. It is a single number on that measures how cleanly the two scorers’ joint agreement separates ground-truth positives from the rest. is the threshold that achieves the gap — the operationally useful number, because it’s the cutoff you’d actually pick.

means the two scorers’ agreement carries no information about positivity. means the agreement perfectly separates from its complement. Real systems land in the middle, and the absolute number is comparable across model versions in a way correlation alone is not.

What This Catches That Raw Correlation Doesn’t

A high Pearson correlation between two scoring systems is a necessary condition for them to be useful together. It is not sufficient. Two scorers can be highly correlated globally and still systematically disagree on the cases that matter — the positives.

This is the same logic behind the standard calibration diagnostics — model scores can have right rank order and wrong scale — applied jointly to two scorers instead of one scorer plus ground truth.

Worked Example — Index Score vs Reranker Score

Take a retrieval task with (query, document) pairs. For each, you have from a dense-retrieval bi-encoder and from a cross-encoder reranker. You have a graded-relevance -judged subset of items with grade .

The procedure:

1. Fit the regression. on all 10K pairs. Pearson might be 0.6 — the two scorers globally agree.
2. Compute residuals. for every .
3. Stratify. Split residuals into and .
4. Sweep . Compute over a fine grid of candidate thresholds. Take the argmax.

A typical result on a well-behaved retriever-reranker pair: , — meaning the threshold “GT positives have residuals above 0.05” separates them from non-positives with a 40-percentage-point survival-function gap. That is more discrimination signal than either score alone provides on the same set, and it gives you both a quality summary () and an operating threshold ().

Where This Connects To Standard Tools

This technique is essentially AUC computed in residual space. Same shape as the standard ROC analysis — sweep a threshold, plot two survival functions against each other, take the maximum vertical gap — but the coordinate is residual, not raw score. The advantages over raw-score AUC are exactly the advantages of looking at calibration leftovers over raw outputs: you’ve subtracted the global agreement, and the remaining signal is purely about where the two scorers disagree.

It is most useful as a diagnostic: comparing across model versions, across datasets, or across reranker-retriever pairs tells you whether your stack’s joint discrimination is improving in a way correlation alone can hide. It is less useful as a deployed threshold — the is overfit to the specific GT subset you used and will drift across domains the same way a raw-score threshold does.

Summary

Calibration-discrimination analysis is the right tool when you have two scoring systems and a known-positive subset and you want to know how cleanly the two scorers jointly identify positives. Fit a regression, take residuals, stratify by GT, compute the Youden-J of the residual distribution. The result is a single comparable number () and an operating threshold () that together summarize what global correlation cannot.