Better Retrievers, Worse Labels: How Incomplete Relevance Judgments Distort Abstention Evaluation in Conversational RAG

This paper finds that a common rule for scoring refusals in RAG evaluation gets worse as the retriever gets better. The rule says: if none of the documents judged relevant for a turn appears in the retrieved context, the turn counts as unanswerable, so refusing is right and answering is wrong. On RECOR (a multi-turn conversational retrieval benchmark built from StackExchange and BRIGHT questions), across four retrievers from weakest to strongest, the share of contexts the rule calls unanswerable that actually contain the answer rises from 28% to 72%. The paper proposes no new model; teams that evaluate RAG systems, and especially teams that score refusals, should read it because it shows that current scoring rewards wrong refusals. The core idea in one sentence: whether a refusal is right depends on whether the retrieved text contains the answer, not on whether it hit the judged list.

What problem is it solving?

End-to-end conversational RAG evaluation lets each system answer from its own retrieval. That means someone has to decide, for each retrieved context, whether it is enough to answer, before a refusal can be scored as right or wrong. Judging every context by hand is too expensive, so benchmarks borrow the relevance judgments (qrels): if none of a turn's judged relevant documents is in the top-5, the context is treated as insufficient.

The trouble is that relevance judgments are never complete. Annotators only looked at some documents, and strong retrievers are exactly the ones that find useful documents nobody looked at. An illustrative example:

Figure 1 of the paper gives a real case. A user asks why glucose meters detect glucose but not sucrose. One retrieved document says the meter relies on the enzyme glucose oxidase and responds only to D-glucose, which is the answer. Yet that document is a negative in the qrels; the judged relevant document contains the same passage but was not retrieved. Under the rule, the model that refuses scores a point and the model that answers correctly loses one.

How the authors show the rule degrading

They first re-run the two official BM25 baselines of RETECO (a SemEval-2027 conversational retrieval task that uses RECOR as its data) and match the published nDCG@10 to four decimals, so the starting point is trustworthy. Then they build a four-step ladder of retrievers from weak to strong. The two weakest are BM25 on the current turn only (nDCG@10 0.183) and the official BM25 with dialogue history (0.438). The third is Fusion (0.550): an LLM expands the query into three versions (for example by first writing a hypothetical answer), and each version runs through BM25. Each run's scores are linearly rescaled to 0–1 within that run (top score 1, lowest 0) and summed per document. The strongest, Hybrid (0.591), adds a dense retriever (reason-embed-qwen3-4b) to the Fusion mix.

Every top-5 context of every retriever then gets a content label. A judge model reads the dialogue, the last question, the reference answer and the five documents, and says whether the documents support the core of the reference answer: yes, partly, or no. The primary judge is phi-4; Qwen3-14B, from a different family, labels everything again, and four small models each label it too. To guard against a lenient judge, a model of another family (Claude Opus 5.5) audits 250 contexts blind, without knowing the retriever or the rule's label, in two independent runs plus an adjudicating third.

With content labels in hand, the rule's precision as an unanswerability detector can be computed. Precision compares only the yes and no labels and leaves out partly.

Results: the stronger the retriever, the less the rule can be trusted

A false miss is a context the rule calls unanswerable while the judge finds it sufficient. The denominator is every context the rule calls unanswerable, and the numerator counts only those the judge clearly labels yes. From the weakest retriever to the strongest, their share rises from 28% to 72%, and counting partly as well it rises from 39% to 92%. On strong retrievers, most contexts the rule calls unanswerable can in fact be answered.

Part of the precision drop comes from the base rate: on strong retrievers, few contexts are truly insufficient according to the content judge (386 falling to 42), so false misses come to dominate what the rule calls unanswerable. But the authors also report Youden's J (recall minus false-alarm rate), which does not depend on the base rate, and it too falls from 0.52 to 0.15. The precision drop is not only because strong retrievers leave fewer truly unanswerable contexts; the rule's own ability to separate answerable from unanswerable contexts is getting worse.

One might suspect lenient judges. The authors run several checks. The first uses no judge at all: if a retrieved document is a near-verbatim copy of a judged relevant document, it is relevant by the qrels' own standard and only carries a different ID. This class alone makes up 16.4% of the contexts the rule calls unanswerable on Fusion, a lower bound on how often the rule is wrong. The second is a negative control: each turn is paired with the retrieval of another dialogue, and phi-4 calls only 4.1% of those sufficient, so it does not wave everything through.

The most convincing check lets the answering model speak. Qwen3-4B answers 62% to 67% of false-miss contexts correctly, only 23% to 30% of truly insufficient ones, and 1% of the same questions with no documents. The correct answers come from the retrieved text, not from the model's memory. False misses really do contain the answer.

Where does the answer come from? On Fusion, 41% of false misses rest on a hard negative the benchmark mined for the same question and placed in the corpus as a distractor; 14% on a near-copy of a judged document; 19% on the judged document of another turn of the same dialogue; and 13% on a neighbouring chunk of the judged page. So most false misses get their answer from documents tied to the same dialogue. But such documents already make up over 70% of all documents in those contexts, so this shows where the answers sit, not that distractors generally contain them.

Swapping the label changes the conclusions

The rule does not just shift numbers; it changes who looks good.

First, the readers' ability to tell answerable from unanswerable is understated. Each reader states answerable or unanswerable on its first line, and the authors take the probability the model assigns to the answerable label at the first generated token as its score for the context, then compute AUROC from that score. On Hybrid, the rule puts all three readers near chance (0.50 to 0.54), while under the content label Qwen3-4B reaches 0.72.

Second, the best detector changes. Under the rule, the best unanswerability detector is the retriever's own top-1 score, which never reads the documents. The reason is plain: the rule records whether the retriever found the judged document, and the retriever's confidence predicts exactly that. Under the content label, the reader's own judgment ranks first at every retriever, though its lead is not significant on Fusion and Hybrid.

Third, the authors list every comparison an end-to-end study would usually report: refusal rates and AUROC across retrievers and across readers, how much refusal rates differ between answerable and unanswerable contexts, whether AUROC is significantly above chance, and so on, 258 in total. Switching to the content label changes the direction or significance of 75 of them, and 13 flip direction outright. Swapping one content judge for the other changes only 25, none by direction. The comparisons are uncorrected for multiple testing, and the authors say the count indicates scale, not significance.

On MTRAG (another multi-turn RAG benchmark, whose relevance labels come from human annotators while they chatted with a retrieval system), the rule's precision also falls from 0.72 to 0.50. MTRAG has no mined negatives and uses human labels, so the problem is not specific to how RECOR was built.

Limitations as I see them

First, all content labels on RECOR come from LLM judges, and every judge sees the reference answer, so they may share a leniency toward on-topic documents. Most false misses sit in passages no human ever saw, which have no human check, and item-level agreement is only moderate (κ 0.38–0.49). The small judges are more lenient still (one calls 30.7% of negative-control contexts sufficient), so the authors use them only to confirm the trend.

Second, the retriever ladder has only four steps, and even the strongest, Hybrid, reaches only 0.591 nDCG@10 on RECOR. Whether stronger retrievers make things worse can only be extrapolated from this data.

Third, the query expansions behind Fusion and Hybrid came from a closed model. Using just one of them (the LLM first writes a hypothetical answer passage that BM25 then searches with), the closed model's expansions reach 0.546 nDCG@10 while open-model regenerations reach only 0.453, so reproducing those two steps requires the authors' released expansions.

How a search team could try this

1. On your own RAG evaluation set, sample one or two hundred retrieved contexts that missed every judged document and check by hand whether they contain the answer. This is the cheapest step and directly gives the current rule's precision.
2. If precision is clearly below 0.7 (in the paper only the weakest, turn-only BM25 reaches that level), add a content label for each retrieved context, using two judge models from different families, and first run a negative control with other questions' retrievals to make sure the judges do not pass everything.
3. When comparing retrievers, score refusals only against content labels and report refusal rates per retriever.
4. Check the set's hard negatives for answers and relabel those that contain one; otherwise they keep penalising the better retriever.

Verdict

Whether a refusal is right can only be read from the retrieved text itself. Neither comparing refusals across retrievers nor choosing an unanswerability detector should rely on a label derived from relevance judgments; if it must be used, audit its precision on a sample for each retriever.