RAGCompass: Generalizable Retriever Routing via Reranking

Valentin Six, Mufei Li, Dongqi Fu, Yinglong Xia, Hong Li, Pan Li
24 minread4.40scoreNo. 111-06 AI Search issue
Contents · 6 sections1The candidates are the passage sets retrieved for this query2Train selection against the answers the passages support3Which queries support the abstract's 28% improvement?4What happens when components change?5What is excluded from the 3.9% overhead?6Start with answer supervision, not a more elaborate comparison head
Rel 5Nov 4Prac 4Evid 4Time 5

Different retrievers can return passages with different value for answering the same question. RAGCompass (a router selecting among retrieved passage sets) runs the candidate retrievers first, then uses a trained reranker to compare their results and select one set for the answering model. Across all test queries, answer F1 improves by 2.28 percentage points over the best fixed retriever. The abstract's 28% relative improvement applies to a filtered population, not all queries. This is most relevant to systems already considering multiple retrieval routes for answer quality, rather than a pre-retrieval router that avoids running the other candidates.

The candidates are the passage sets retrieved for this query

A conventional question-answering pipeline fixes one retriever, then gives its passages to a reader. Retrieval methods favor different matching signals. Lexical matching may locate a direct answer for one question, while another method finds more complete material for another. Similar average scores do not imply equally useful results on each query.

Consider an illustrative question asking where a writer was born. One passage set contains a biography stating the birthplace; another mainly discusses the writer's books. Both concern the writer, but topical relevance alone does not determine how well they support this answer. A passage set here means all candidate passages returned by one retriever for the current query.

In the evaluated configuration, RAGCompass runs five retrievers, each returning five passages. The candidates come from dense retrieval, BM25, SPLADE-v3, ColBERTv2, and HippoRAG2. The router compares these five sets and selects one. It neither merges all passages into a single context nor chooses a retriever before retrieval happens.

RAGCompass Figure 2, cropped from the original PDF: the query first reaches the retrievers. Each passage set passes separately through a shared reranker; a small comparison network reads their representations jointly and selects one set for the reader. The dashed branch generates answers and supervision for every set during training only.
RAGCompass Figure 2, cropped from the original PDF: the query first reaches the retrievers. Each passage set passes separately through a shared reranker; a small comparison network reads their representations jointly and selects one set for the reader. The dashed branch generates answers and supervision for every set during training only.

This distinguishes it from routers using query features or retrieval scores to predict a route's quality. RAGCompass already has the current results and evaluates their usefulness for answering. A newly added retriever can supply another passage set without first requiring a learned identity label.

Train selection against the answers the passages support

Training has access to reference answers. For each query, the authors give every retriever's passage set separately to a fixed reader, generate an answer from each, and compare it with the reference using answer F1. The paper calls this value utility: the answering outcome supported by that set.

Here q is the query, D_i the ith retriever's set, A the reader, and a* the reference answer. In the illustrative birthplace question, a set about the writer but missing the birthplace can still yield a poor answer. Supervision therefore comes from the answer produced from each set rather than only from relevance labels on passages.

The scorer uses Qwen3-Reranker-0.6B. Each forward pass receives the query and one passage set. The five sets have separate passes through shared parameters. Under a routing instruction, the logit for yes supplies the set's score.

Training divides each route's answer F1 by temperature τ and applies softmax across the candidate sets to obtain target weights u. A softmax over model scores produces selection probabilities p. The objective minimizes their KL divergence:

Higher-answer-F1 sets receive higher target weights. Lower temperature concentrates those weights more strongly on high-F1 sets. The model learns how to distribute selection probability across the sets.

The scorer alone can select the highest-scoring set. The full model adds a two-layer Transformer comparison network that jointly processes the sets' hidden representations and outputs final scores. It can compare representations before compressing each to a scalar. The authors first train the scorer, then freeze it and train the comparison network. Neither component has parameters dedicated to a particular retriever.

Training and use have different costs. Label construction requires the reader to answer from every route. At inference, the router reads only the query and retrieved passages. It has no reference answer and does not generate five answers to choose among. The reader generates once, after one set has been selected.

Hoot saysAnswers teach scoring; inference does not trial-answerClose ✕
Learn scores from answers; select before answering online
Each route answers the same question separately and is compared with the reference. The scorer learns their relative utility; the reference is not an online router input.Each retrievedpassage setFixed readeranswers each setReference answersupplies F1Train shared scorer
Each route answers the same question separately and is compared with the reference. The scorer learns their relative utility; the reference is not an online router input.Each retrievedpassage setFixed readeranswers each setReference answersupplies F1Train shared scorer
1/2 · Training: answers supply selection labelsEach route answers the same question separately and is compared with the reference. The scorer learns their relative utility; the reference is not an online router input.
The scorer predicts selection scores from the query and passages. Only the chosen set reaches the reader; no reference or per-route generated answers are read.Each retrievedpassage setShared modelpredicts scoresSelect one setReader generatesone answer
The scorer predicts selection scores from the query and passages. Only the chosen set reaches the reader; no reference or per-route generated answers are read.Each retrievedpassage setShared modelpredicts scoresSelect one setReader generatesone answer
2/2 · Inference: select before generatingThe scorer predicts selection scores from the query and passages. Only the chosen set reaches the reader; no reference or per-route generated answers are read.
1/2

Training labels measure each route's actual answer quality. At inference the model predicts which set is useful, without the true answer or reader trial generations. All candidate retrievers still run.

SourcePaper Sections 3.1–3.2, Figure 2, Appendix K

Which queries support the abstract's 28% improvement?

Experiments cover Natural Questions, MuSiQue, HotpotQA, and 2WikiMultihopQA, including single-hop and multi-hop questions. The experiments use a fixed Llama-3.1-8B-Instruct reader. The main table includes only queries where the best and second-best retrievers' answer F1 differ by at least 0.05, so route selection has a measurable distinction to learn.

That filtering depends on reference answers and per-route generations. It selects offline training and evaluation examples; it is not an online requirement for users to supply the answer. The selected population is 18.7% of the unfiltered test set. Table 2 compares methods on that same subset.

ConfigurationAnswer F1
Best fixed retriever0.2782
RRF result fusion0.3002
MoR, the static-weight fusion variant used here0.3203
Fine-tuned scorer only0.3412
Scorer plus comparison network0.3552
Oracle choosing the best outcome after answering0.7508

The complete model gains 7.70 percentage points over the fixed retriever, reported as a 28% relative improvement. The scorer supplies most of the gain; the comparison network adds 1.4 percentage points. The oracle needs to know how well every set's answer turns out. It measures remaining selection opportunity rather than providing a deployable method.

Paper Figure 3: the upper row groups queries by population share; the lower row groups the remaining F1 gap between the router and oracle. The main reported population is only 18.7% of queries. Many other queries are near-ties across routes or yield no correct answer when the fixed reader answers separately from each route.
Paper Figure 3: the upper row groups queries by population share; the lower row groups the remaining F1 gap between the router and oracle. The main reported population is only 18.7% of queries. Many other queries are near-ties across routes or yield no correct answer when the fixed reader answers separately from each route.

Across all test queries, the improvement is 2.28 percentage points. The best fixed comparator also changes from SPLADE-v3 to HippoRAG2, so this is not simply the main-table gain rescaled by population share. Deployment value depends on the complete query distribution, not only on its most distinguishable cases.

A separate experiment tests supervision more directly. With examples, comparison network, and evaluation held fixed, replacing answer utility with gold-passage nDCG reduces answer F1 from 0.3552 to 0.3341. This supports training routing for the downstream answer. Comparisons with query-only models also change architecture and capacity, so their entire gap cannot be attributed solely to reading passages.

What happens when components change?

Shared content scoring makes new retrievers' outputs valid model inputs, but accepting an input and choosing accurately are different properties. The authors remove one retriever and all its labels from training, then restore it among test candidates to evaluate selection under an unseen retrieval mechanism.

The identity-free full model loses 0.16–1.17 F1 percentage points relative to training with all five routes. Adding dedicated retriever-identity embeddings substantially worsens some held-out settings. This supports shared content representations in the tested setting, not a quality guarantee for arbitrary new retrievers.

The corpus-shift experiment excludes one dataset's queries and corpus from router training and evaluates on it. Retention divides held-out gain over the fixed retriever by the corresponding in-distribution gain. HotpotQA retains 103%, 2Wiki 84%, and MuSiQue 49%; Natural Questions starts with negligible gain. Gain retention varies substantially across these corpora.

Reader transfer tests one replacement: Qwen2.5-7B-Instruct answers instead of the Llama used for labels. The router retains 71%±2% of its advantage, measured against each reader's own best fixed retriever. This swap preserves part of the gain but does not isolate evidence quality from evidence–reader fit. Remeasuring selection quality after a reader upgrade remains important. The three query-only routing baselines are author reimplementations with different or smaller configurations, while MoR uses static weights; comparisons apply to those versions.

What is excluded from the 3.9% overhead?

On one L40S and one batch setting, routing takes 0.222 seconds per query, against 5.687 seconds for one retriever's retrieve-and-answer pipeline: about 3.9%. This accounts for the decision computation that reads the passages and selects a set. It excludes the additional retrieval cost of expanding from one retriever to five.

The retrievers can run concurrently, and final answer generation still happens once. Concurrency does not eliminate compute, index, or service-resource costs. A system already retrieving from several sources has a different incremental cost from one currently running a single retriever. Label construction also requires the reader to answer separately from every candidate set, creating a substantial offline expense.

An engineering comparison should therefore measure multi-route retrieval, reranker forward passes, and final generation together. Adding only 0.222 seconds to an existing system does not establish the complete cost after adoption. The 3.9% figure is the decision-overhead ratio for the paper's device and setting.

Start with answer supervision, not a more elaborate comparison head

A team already using multiple retrieval routes can begin with queries that have reference answers. Let a fixed reader answer separately from each route's results and check whether answer quality actually differs. Train a shared scorer from those labels and compare it with the best fixed retriever and existing fusion over all queries. The scorer provides most of the main-table improvement, so a more elaborate comparison network can be evaluated later.

Testing can also hold out new retrievers or corpora, while recording quality and cost by request type. These are suggestions derived from the paper, not results established in other systems. RAGCompass's practical argument is that, once multiple result sets are available, examining their usefulness for answering offers a more adaptable selection signal than assigning each retriever a fixed identity.

DimensionAssessment and reason
Problem importanceHigh: retrieved sets can differ in answer value for the same query.
Method noveltyModerate: answer-supervised shared reranking selects passage sets.
EvidenceModerate: filtered-population gains exceed full-query gains; transfer varies by dataset.
Engineering costModerately high: multi-route retrieval, labels, and training; online answer generation happens once.
Worth trying?Conditional: strongest fit when existing multi-route results offer enough full-distribution answer gain to cover the extra cost.

How did this read for you?

One tap, no sign-in; tap again to undo. Only anonymous counts are kept, never who you are.

Discussion

Add experiment details, reproduction notes or a different take. Requires a GitHub sign-in.

Loading the discussion…