Eval Set Quality

Eval set quality is the degree to which a benchmark can resolve real differences between models. A benchmark on which sibling architectures at several parameter counts (e.g., 0.6B, 1.7B, 4B, 32B) all score within of each other has failed to resolve the scaling curve for that family, irrespective of which metric is reported. The diagnostic tools for benchmark quality — item difficulty and discrimination, internal consistency, inter-rater agreement, paired effect size — were developed in classical psychometrics through the early-to-mid 20th century (Spearman 1904, Cronbach 1951, Cohen 1960) and apply without modification to ML benchmarks.

Six axes characterize where an eval set can fail:

1. Item-level pathologies. Items at ceiling, floor, or with weak item-by-total correlation. Diagnosed by CTT .
2. Aggregate unreliability. Items do not internally agree on what they measure. Diagnosed by Cronbach’s α .
3. Label noise. Gold labels are inconsistent across annotators. Diagnosed by Cohen’s κ .
4. Insufficient discriminative power. The benchmark does not separate known-different models. Diagnosed by within-family Cohen’s .
5. Pool incompleteness. For retrieval evals, qrels omit relevant documents not retrieved by any system in the original pool. Mitigated by bpref or condensed metrics.
6. Contamination. Test data has leaked into the pretraining corpus. Detected by n-gram or perplexity heuristics; not fully eliminable.

Item-level pathologies (CTT)

For each item in the benchmark, two statistics summarize its behavior:

• Item difficulty — the fraction (or mean grade) of test-takers who answer correctly.
• Item discrimination — the point-biserial correlation between the item score and the test-taker’s total score.

The categories of pathological items, with conventional action thresholds:

A benchmark with more than 30% of items in any of these categories requires triage: drop or re-annotate the affected items and recompute everything downstream (α, model rankings, leaderboard order). Rankings often change visibly after the bad items are removed.

Aggregate reliability (Cronbach’s α)

After CTT-based pruning, internal consistency of the trimmed test is measured by Cronbach’s α :

Conventional thresholds: excellent, good, acceptable, problematic. α should be reported alongside (the number of items), because the Spearman-Brown relation causes α to increase with test length even when added items are pure noise; comparing α across benchmarks of different sizes without is unsound.

α also assumes tau-equivalence — every item measures the same underlying construct. Multi-domain benchmarks (e.g. MTEB , whose subsets span math, biology, code, and general knowledge) violate this assumption directly. The corresponding correction is to compute α within each subdomain and report the per-subdomain distribution. A single pooled α on a heterogeneous suite is a length artifact rather than a reliability measure.

Label noise (Cohen’s κ)

A test with internally-consistent items still fails if its gold labels are inconsistent across annotators. The standard inter-rater agreement statistic is Cohen’s κ :

where is observed agreement and is agreement expected by chance. For ordinal labels (e.g. graded relevance on a 0-3 scale), the appropriate variant is weighted κ with quadratic weights, which correctly penalizes large disagreements more than small ones.

The conventional thresholds: indicates label noise dominates; is the floor for a usable benchmark; is considered gold-standard. LLM judges with chain-of-thought before grading, anchor examples, and forced output schema typically reach on retrieval relevance — comparable to reported human-human ceilings in the same task.

Discriminative power (within-family Cohen’s )

Statistical significance becomes trivial at large : with thousands of items, almost any non-zero metric difference clears . The relevant question for benchmark quality is therefore whether the benchmark resolves models that are independently known to differ.

The standard experiment is the sibling-family discrimination test. Given a model family at multiple parameter counts (for example 0.6B / 1.7B / 4B / 32B of one architecture), compute paired Cohen’s between adjacent siblings on the benchmark:

where is the mean per-query (or per-item) metric difference and is its standard deviation. The conventional thresholds, following Cohen (1988):

Pool quality (retrieval-specific)

For retrieval benchmarks such as BEIR and TREC, gold relevance labels are produced by pooling: a fixed set of contributor systems is run on each query, the union of their top- candidates is judged, and any document not retrieved by any contributor is implicitly labeled non-relevant. The procedure is sound when the pool is large and diverse enough that essentially all relevant documents appear in it.

Two failure modes arise. First, when a new system retrieves relevant documents not present in the original pool, those documents default to non-relevant and the new system is penalized for finding genuinely relevant material. Second, when the pool itself is small — older TREC tracks pooled hundreds of contributors, whereas modern benchmarks often pool five to ten — the share of unjudged-but-relevant documents grows with the number of systems being compared.

The standard mitigation is bpref (binary preference), which scores only judged documents (counting the fraction of judged-relevant documents ranked above judged-non-relevant ones) and ignores unjudged documents in both the numerator and denominator. NDCG computed on a condensed list — unjudged documents removed before scoring — is a closely related approach.

Contamination

If test data leaked into the pretraining corpus, evaluating that test measures the model’s memory rather than its capability. Contamination is the only axis on this list that admits no clean statistical test; it is mitigated by operational discipline.

A clean evaluation on a privately-curated held-out corpus is more informative than any number of ablations on public benchmarks.

Layering across MTEB and BEIR

The diagnostics above are not uniform across the constituents of a large benchmark suite. Within MTEB ’s 50+ datasets, some subsets (e.g. ArguAna, TREC-COVID) satisfy every axis: well-spread , healthy , sufficient κ, and clear within-family . Others fail several at once — certain FiQA and CQADupStack subsets are concentrated near ceiling for frontier embedders ( between adjacent sibling sizes), and several Touché-2020 splits have low original-annotation κ with metric differences that fail to survive a paired bootstrap.

The implication is that a single suite-level composite (mean across all subsets) loses signal. The corresponding correction is to weight by per-dataset reliability and to report the unweighted variant separately, with the per-axis diagnostic numbers attached. The same pattern applies to BEIR , MS MARCO , and other large retrieval suites.