LTRR: Learning To Rank Retrievers for LLMs

RAG systems usually connect to one fixed retriever, but different questions suit different retrievers, and for some questions it is better not to retrieve at all and let the model answer on its own. This paper treats choosing a retriever as a ranking problem. For each training question it records how much each retriever improves answer quality over no retrieval, then trains a ranking model to rank retrievers, with no retrieval as one of the candidates. On six retrievers built from sparse and dense retrieval with different rerankers, a pairwise XGBoost router lifts the answer correctness score from 0.5776 for the best single retriever to 0.5900, while always picking the right one would reach 0.7948. The routing gain is real but still far from the ceiling, and which answer-quality metric is used for training labels matters a lot.

RAG teams should read this for two ideas they can use directly: label with "how much better than no retrieval" rather than a retrieval metric, and rank "no retrieval" together with the retrievers. The first aligns the router with generation quality; the second merges "whether to retrieve" and "where to retrieve" into one decision. Keep in mind that the gain is only about 2%, and the gap between the learned router and the best training-free rule is not statistically significant. Its features also require querying all six retrievers first, so it saves no retrieval cost.

What problem does it solve?

A single RAG system receives questions of very different styles. The three questions below (illustrative) each suit a different approach:

Most systems connect one retriever and send every question down the same path. Existing routing methods have two problems. Many choose by retrieval metrics, such as which retriever has higher recall, but for an LLM higher recall does not mean a better answer. Many also rely on corpus statistics, such as similarity between the question and the corpus, while in real systems a retriever is often just a search interface with no access to the corpus behind it.

The paper sets up a scenario close to that reality: the router cannot see inside the retrievers, only the question and what each retriever returns, and the goal is a better final answer.

Overall pipeline

The two rerankers have different goals. Score regularization computes pairwise embedding similarity among the top-k documents, keeps each document's m most similar neighbors, and replaces each score with a weighted average of neighbor scores, repeated a few times, so a cluster of similar documents rises together; the goal is better retrieval quality. Stochastic reranking uses the original scores as sampling weights and draws documents one by one without replacement: the first position is drawn from all documents by weight, the second from the rest, until the list is full. This yields one complete ranking; 50 such rankings are generated and one is picked at random. High-scoring documents usually stay near the top, but low-scoring ones get a chance to appear; the goal is fairer exposure across documents, which earlier work found can sometimes improve answers. The six retrievers share one corpus and differ only in how they retrieve.

Core mechanism: where labels come from and how features are computed

First the labels. For each training question the generator answers once with no retrieval and once with each of the six retrievers' results. Each answer is scored by an answer-quality metric, and the no-retrieval score is subtracted to give each retriever's "gain". A gain can be negative, meaning the retriever hurt.

For example, a question scores 0.40 with no retrieval, and the retrievers do as follows (illustrative):

Normalization maps the worst option for this question to 0 and the best to 1. This is needed because difficulty varies a lot: some questions score 0.2 however they are answered, others score 0.8 no matter what, and only after normalization can labels from different questions be trained together.

The authors compared three training styles and pairwise training did best. They attribute the stability of pairwise XGBoost to the fact that routing only takes the top choice, so order matters more than values.

Next the features. The router has no access to retriever internals, so features come in two kinds. Pre-retrieval features depend only on the question: the question embedding (E5, reduced to 32 dimensions with PCA), question length and question type (keyword style or natural language). Post-retrieval features need each retriever's top-k results first:

"No retrieval" has no post-retrieval features. Tree models like XGBoost fill them with the training median, and neural models learn a dedicated vector.

The most important point here is cost: computing post-retrieval features requires querying all six retrievers before deciding which result to use. So the router saves no retrieval cost; what it saves is the mistake of handing the generator the wrong material. When it picks no retrieval, the only saving is shorter context at generation time.

Results

There are five test splits. In the balanced split both training and test contain all five question types; the other four each hold out one type for testing only, to see how the router does on question types it has not seen. The table below uses the answer correctness metric (AC) for training and evaluation. AC compares the generated answer with the reference answer and combines factual overlap and semantic similarity into a continuous score from 0 to 1, so the numbers are mean quality scores, not accuracy:

Three things stand out. First, pairwise XGBoost beats the best single retriever by 0.0124 on the balanced split, about 2%, and the gain is significant there and on unseen complex and open-ended questions, so learned routing does help. Second, it is slightly below the single retriever on multi-aspect questions, so it does not win on every type. Third, pairwise SVMRank drops to 0.5329 on the balanced split, worse than a single retriever, so a badly chosen model can be worse than no routing.

Test-set size also matters when reading these numbers. The balanced split has about 2,000 test questions, while the four unseen splits have only about 400 to 800; comparison has just 385. On sets this small a gap of about 0.01 is easily lost in random noise, which is why the authors ran significance tests on every comparison, and why many numerical gains did not count as significant.

The ceiling row says the most. Always picking right reaches 0.7948, about 0.22 above the best single retriever, while the best router captures only about 6% of that gap. Retrievers are highly complementary, but the current features cannot yet tell which question needs which retriever.

Against training-free rules, learned routers are usually better on paper, but the authors acknowledge the gap is not significant after multiple-comparison correction. The simplest MaxSim rule, picking the retriever whose most relevant result is most similar to the question, already captures most of the gain.

Another finding concerns the label metric. When trained with BEM, another answer-quality metric that uses a BERT classifier to judge whether the generated and reference answers are equivalent for the question and outputs that probability, no router significantly beats the single retriever: on the balanced split the best single retriever scores 0.3599 and pairwise XGBoost only 0.3626. The authors explain that AC agrees better with human judgment, so the learned signal is cleaner. In practice, switching the scoring metric may make the same method show no effect.

Limitations as I see them

First, it does not save retrieval cost. Post-retrieval features require querying every retriever, which with six retrievers means six times the retrieval cost, and then only one result set is used. The paper's discussion says routing can "activate only necessary retrievers", but their implementation cannot do that. Whether pre-retrieval features alone would keep the roughly 2% gain is outside the experiments.

Second, the data and setting are narrow. All questions are synthesized by DataMorgana, there is one 15-million-page web corpus, one 10B generator, and the retrievers are just two base methods plus rerankers. In real systems different retrievers often sit on different sources, such as internal docs, web search and databases, with much larger differences; routing gains could be larger, or the features could fail entirely.

Third, the gain is small and sensitive to choices. The 0.0124 balanced-split gain needs a significance test to confirm, disappears with BEM labels, and turns negative with pairwise SVMRank. A team following this would likely spend real effort tuning metrics and models before reproducing a positive result.

How a RAG team could try it

1. Compute the ceiling first: sample a few hundred production questions, run each retriever and no retrieval, score answers with an answer-quality metric, and see how much always picking right would add. If even the ceiling adds only a point or two, routing is not worth it.
2. Check how often no retrieval wins: count the questions where not retrieving gives the best answer. If the share is not small, start with only the binary decision of whether to retrieve, which is cheaper and less risky.
3. Try training-free rules first, such as choosing the retriever by MaxSim. In this paper it already captures most of the gain and needs no training data.
4. If rules are not enough, build labels as the paper does: gain over no retrieval, normalized within each question, and train a tree model with pairwise ranking. Train once with each of two answer-quality metrics and check that conclusions agree.
5. Do the math before launch: post-retrieval features require querying every retriever, so weigh latency and cost against the gain; if the cost is too high, try a version with pre-retrieval features only.

Bottom line

Putting "no retrieval" among the candidates and labeling by "how much better than no retrieval" is the most reusable design here, but the measured gain is only about 2%, far below the always-right ceiling.