Rubric-Calibrated Preferences: Cross-Query Calibration of LLM Judgments via Item Response Theory

Across the 14 rerankers the authors evaluated, nDCG cannot tell them apart on 45.6% of NanoBEIR queries and 30.7% of BRIGHT queries: either every reranker scores 1 (saturation), every reranker scores 0 (floor), or more than half of the pairs among the 14 rerankers get the same score (compression). The root cause is the human relevance labels (qrels): they are expensive and sparse, noisy, and limited to a few discrete grades, and the closer the rerankers are to each other, the worse this gets. The Cohere authors propose RCP (Rubric-Calibrated Preferences). It first has an LLM read 10 documents at a time and score them, combining many such rounds into a fine-grained ranking of all documents under a query. It then has the LLM answer five yes/no questions about each document, and uses item response theory (IRT) on those answers to convert every query's scores onto one shared ruler, yielding RCP-nDCG with continuous gains. Checked against blind ratings from 46 external annotators, RCP gains rank a useful document above a useless one with probability 0.910, against 0.651 for the original qrels; on NanoBEIR, the share of reranker pairs that can be significantly separated rises from 34.1% to 63.5%.

1. The problem: relative judgments share no scale, and absolute judgments are too coarse

Using an LLM as a relevance judge allows labeling at scale with quality close to human. But the LLM's two ways of judging each have a weakness. The first is relative judgment, meaning pairwise or listwise comparison. It can separate very close documents under the same query, but scores from different queries share no common scale, so they cannot become graded labels, support a shared threshold, or be aggregated across queries. The Bradley–Terry (BT) model looks only at score differences between two documents, so adding the same constant to every document of a query changes nothing. Scores from two queries therefore share no common zero point and cannot be compared directly. The judge is also more decisive on some queries and spreads their scores further apart, so the size of the scores differs between queries too. The second is absolute judgment, meaning assigning grades directly. It can be compared across queries, but the grades are too coarse, and many documents under the same query end up tied.

The question the authors want to answer is: can a fixed set of yes/no criteria put within-query LLM preferences on a scale shared by all queries, with results that agree with human judgment? They test this in reranker evaluation, because there is an independent human reference there, and because scores obtained this way can replace sparse, discrete, noisy qrels.

2. Method: the tournament sets the order, the rubric sets the scale

IRT was originally used for exams: from examinees' answers to the same set of questions, it places all examinees on one ability scale. RCP borrows this idea, treating documents as examinees and the rubric's criteria as questions. The pipeline has two stages plus a calibration step.

The first stage uses a tournament to set the order within each query. Each query has 150 candidate documents. The judge LLM reads only 10 of them per call, and the 10 documents shown together in one call are called a "window". The judge gives each document in the window a score from $-5$ to $+5$, and the prompt tells it to treat this score as a logit: the larger the gap between two documents, the more certain the judge is that the first is more relevant. The same document can get a different score in a different window, so raw scores are not comparable across windows. The authors therefore use only gaps within a window: each pair's gap becomes a preference probability $p_{i{i}^{\prime }}=\sigma (s_i-s_{i^{\prime }})$, giving 45 pairs from 10 documents, and the BT model combines the pairwise preferences from all windows into one fitted score per document.

Windows are scheduled in three rounds, each solving one problem. The first round groups documents at random so that all 150 are seen a few times and get a rough score. The second round puts documents with similar rough scores in the same window, because the judge's scores are influenced by the other documents in the window: a middling document that keeps landing next to much stronger ones looks worse by contrast and gets marked down. These two rounds use 80 windows, and each window is shown again with its 10 documents in reverse order, for 160 calls. The third round is adaptive: it spends the remaining calls where ranks are still uncertain and worth settling. It runs in 7 batches of 8 windows. Before each batch the BT scores are refit on all comparisons so far. Every pair of adjacent ranks in the current order then gets a "worth comparing again" score, the product of three factors. The score is higher the closer BT's predicted probability of their order is to one half (a coin flip), higher the nearer the pair is to the top (weighted by nDCG's position discount), and higher the fewer windows the two have shared. The batch then greedily picks 8 windows, each made of 10 consecutively ranked documents, taking the stretch whose scores add up to the most; once a stretch is picked its scores are discounted so the next window does not pick the same stretch. This round adds 56 calls, so each query takes 216 judge calls in total (160 + 56), and each document appears in about 14 windows on average.

The second stage has the judge answer absolute criteria document by document. The judge again reads 10 documents at a time, but no longer compares them; instead it answers five yes/no questions for each document separately, each harder than the last. Each query gets 100 such windows. The first 50 spread appearances evenly across documents, and the last 50 group documents with similar preliminary estimates, so each document is asked about roughly 6.7 times on average. The five questions are:

(Parameters are fitted values on NanoBEIR with Qwen3.5-397B as the judge. The higher the difficulty, the more relevant a document must be to pass the criterion; the higher the discrimination, the better the criterion separates slightly more relevant documents from slightly less relevant ones.)

The final step uses these answers to align all queries onto one scale. BT scores and the ability values of the IRT two-parameter model (2PL) are both on a logit scale, so the authors apply a linear transform to the BT score and treat it as the document's "ability", the IRT score for how relevant the document is. The formula below gives the probability that document $i$ passes criterion $c$ under query $j$:
$$P(Y_{ijc}=1)=\sigma ({\gamma }_c({\tau }_j{\hat{\theta }}_i^{\mathrm{BT}}+{\alpha }_j-{\beta }_c))$$
Each criterion's discrimination ${\gamma }_c$ and difficulty ${\beta }_c$ are shared across all queries. Each query has its own offset ${\alpha }_j$, which fills in the constant BT is missing, and its own scale ${\tau }_j>0$, which makes "one logit of ability difference" mean the same amount of relevance difference on every query. During fitting, the BT scores stay fixed. The second-stage answers of all queries are pooled, and the shared ${\gamma }_c$ and ${\beta }_c$ are estimated together with each query's own ${\alpha }_j$ and ${\tau }_j$, so that the predicted pass rates match the actual answers as closely as possible. Because ${\tau }_j>0$, the order within each query is guaranteed to match the first stage (Proposition 3.1); calibration changes only how different queries compare.

With a unified ability value, RCP-nDCG can be defined: replace nDCG's discrete gains with the discrimination-weighted "probability of passing each criterion":
$$g(\tilde{\theta })=\frac{{\sum }_c{\gamma }_c\sigma ({\gamma }_c(\tilde{\theta }-{\beta }_c))}{{\sum }_c{\gamma }_c}$$
$g$ rises smoothly from 0 to 1 and climbs fastest near the criteria's difficulties, because once ability crosses a criterion's difficulty, the chance of passing it rises sharply. The ideal ranking that nDCG needs is simply the order from the first stage. On NanoBEIR, 79.3% of candidate documents have ability below C1's difficulty, meaning they do not even count as "topically relevant"; only about 15.6% fall between the difficulties of C1 and C4, where $g$ rises from 0.22 to 0.69, so the top candidates are separated mainly along this stretch.

A hypothetical example shows what calibration changes. The top document of query A passes C1 through C3, while the top document of query B passes only C1. BT scores mean something only within their own query, and both documents hold their query's top score, so BT alone cannot say which is more relevant. After calibration, A's top document lands at a higher ability value and gets a higher gain.

3. Setup and calibration quality

The evaluation covers four groups of data: 13 NanoBEIR datasets and 12 BRIGHT tasks (both with binary qrels), TREC-DL 2019 and 2020 (graded 0–3 by NIST assessors), and 8 ViDoRe v3 corpora (three grades). To compare reranking alone, each query's candidate pool is fixed to the top 150 from RRF fusion of three retrievers (Cohere Embed v4, Octen-Embedding-8B, BM25), and positives the retrievers missed replace the lowest-ranked candidates. The 14 rerankers come from six families: Cohere, Contextual AI, Qwen3 Embedding, Voyage AI, Jina AI and ZeroEntropy. The main judge is Qwen3.5-397B, and the second judge, gpt-oss-120b, is used to check whether conclusions change with the judge.

The human reference works as follows. The authors hired 46 paid external annotators, each with at least a bachelor's degree in the subject they labeled. Each comparison is one query plus the top five from each of two rerankers; annotators rate the combined documents from both lists in random order, without seeing which list a document came from, the qrels, or any metric scores. The final analysis covers 311 comparisons and 7,080 human ratings, with three annotators per comparison.

The calibration itself is reliable, with three main pieces of evidence. First, once a document's calibrated ability is known, whether it passes one criterion adds almost nothing to predicting whether it passes another (the mean absolute correlation between answers drops from 0.60 raw to 0.123), so the "questions are independent" assumption IRT requires roughly holds. Second, criterion parameters fitted on other datasets and moved to a held-out dataset predict answers with 60.6% lower cross-entropy than "guess each criterion's average pass rate", showing the parameters transfer. Third, the per-query scale ${\tau }_j$ varies about threefold between the 5th and 95th percentiles, which shows that queries really are on different scales and the scale term is necessary.

Calibration also brings scores closer to people. ECE (expected calibration error) measures the average gap between predicted probabilities and observed frequencies, with 0 meaning perfect agreement. Taking a sigmoid of raw BT scores without query scaling gives an ECE of 0.371 for predicting answers; after 2PL calibration it falls to 0.017. The gain $g$ can also be read as the probability that one annotator rates the document Useful or better. Tested against the 7,080 human ratings, its ECE is 0.054, which shows this reading largely holds. The Spearman correlation between per-query average scores and annotators' average ratings rises from 0.538 for raw BT to 0.795 after calibration.

4. Main results: better separation, and the right separation

First, are the labels themselves right? Over the 2,268 documents rated blind by people, RCP gains rank a useful document (human average ≥2) above a useless one with probability 0.910, against only 0.651 for binary qrels, because about half of such document pairs share the same qrel label. Qrels disagree with people in both directions: annotators found 27.8% of documents without a positive qrel actually useful, and 37.5% of documents with a positive qrel actually useless.

Next, can it tell rerankers apart? The authors use paired t-tests to count how many of the 91 reranker pairs are significantly separated: on NanoBEIR, 34.1% with qrel-nDCG@10 and 63.5% with RCP-nDCG@10. BRIGHT, ViDoRe v3 and both years of TREC-DL also show more separation, and switching the judge to gpt-oss-120b does not change the conclusion. The authors also built a rubric-only control, Count-nDCG: a document's gain is simply the share of criteria it passes across all its second-stage answers, with no tournament ranking and no IRT calibration. It separates about as many pairs as RCP-nDCG, so the extra separating power comes mainly from the rubric. Count-nDCG, however, leaves many documents of a query tied, and the tournament keeps ordering them. The authors built each query's ideal top 10 under both methods and asked three LLM judges from different model families to pick the better list blind; they chose the RCP-nDCG list on 80.6–98.8% of queries, so the finer differences come from the tournament.

Separating more is not enough; it has to separate correctly. After an annotator grades the documents of both lists, their own grades give each list an nDCG@5, and the list with the higher score is that annotator's pick; the list most annotators pick is the comparison's verdict. Among the 185 comparisons whose verdict matches exactly one metric (including cases where the other metric ties and picks neither list), the metric that matches is RCP-nDCG 72.4% of the time. Chance is 52.7% rather than one half. Suppose annotators picked at random: when the two metrics pick different lists, each matches the verdict half the time, but when only one metric names a winner and the other ties, the comparison can only count for the one that named a winner. qrel-nDCG ties more often, so these comparisons mostly count for RCP-nDCG, which puts chance slightly above one half. When the RCP-nDCG gap reaches 0.20 or more, annotators agree with it 96.8% of the time; with the same gap, qrel-nDCG gets only 53.1%. On TREC-DL, NIST grades significantly separate only 14 reranker pairs in 2019 and 37 in 2020; RCP-nDCG agrees with NIST on all of them while also separating many more pairs.

The biggest rank change involves ZeroEntropy's rerankers. Under qrel-nDCG@10, Qwen3-Reranker-8B and Contextual v2 6B lead and ZeroEntropy's three models are at the bottom; under RCP-nDCG@10, zerank-2 and zerank-1 rise from 14th and 13th to 1st and 2nd. This raises a concern: zerank is trained on pairwise preferences from several LLMs, so RCP might simply be rewarding a signal from the same source. The authors check how people choose: in the 66 comparisons where RCP-nDCG favors zerank and qrel-nDCG favors the opponent, human annotators, who use no LLM at all, chose zerank 78.8% of the time. Removing every comparison involving zerank lowers RCP-nDCG's agreement rate from 72.4% to 68.1%, still clearly above chance. So zerank's rise is largely real, but the bias concern is not fully ruled out.

5. Is the rubric itself stable?

The authors tested the original rubric and 15 rewritten variants on TREC-DL. As long as the rubric does not ignore the query entirely or crowd all criteria at one end of the difficulty range, not a single significantly separated reranker pair flips. This shows RCP is not very sensitive to the rubric's wording, but the rubric must cover a range of difficulty from lenient to strict. The authors also caution that these five criteria were designed for general retrieval, and other applications may need different criteria.

6. Limitations and practical value

The authors acknowledge several limitations. RCP gains and human ratings both judge each document on its own and ignore redundancy between documents, which matters especially for multi-hop tasks like HotpotQA. Both measure "appears relevant", not factual correctness. The questions given to the annotators were deliberately modeled on the rubric, so agreement between humans and RCP is not a fully independent check. Relevance is hard to judge consistently in the first place: only 77.8% of pairs of annotators rate within one grade of each other. An LLM judge may share preferences with the rerankers being judged, so models trained on RCP signals should be checked against other labels. There is also a conflict of interest to keep in mind: all the authors are from Cohere, Cohere paid for the human labeling, and two of the 14 rerankers and one of the three retrievers are Cohere products.

For teams working on retrieval and RAG, the value of this work is that it moves evaluation from "depending on sparse human labels" to "label the candidate pool once with an LLM, then evaluate any reranker at no further cost". The code, all judge scores and the 7,080 human ratings are open source; once the pool is labeled, any model that reorders these candidates can be scored directly without calling the judge again. The authors recommend that benchmarks report RCP-nDCG and qrel-nDCG together rather than replacing the latter with the former. Two points matter in practice. When the RCP-nDCG gap is below 0.02, agreement with people is only 47.2%, close to chance, so such gaps should be treated as ties. And if your own reranker was itself distilled from LLM preferences, keep a human-rated set for cross-checking.