Contents · 7 sections
1Three ways to score, and what each costs2With many candidates, the off-the-shelf model gets confused3How Curator is trained4Walking through it: London and the Thames5Results6What this means for people building RAG and retrieval7LimitationsSHARE
"The Thames flows through London" looks irrelevant on its own, so why must it stay? How Curator picks RAG evidence in one passThe question: "Which river flows through the city that hosted the 2012 Summer Olympics?" Retrieval returns two passages. One is about the Olympics and contains the sentence "The 2012 Summer Olympics were held in London". The other is about London and contains "The Thames flows through London".
Both sentences must stay, or the answer is incomplete. But judged one at a time, "The Thames flows through London" mentions neither the Olympics nor 2012, and looks unrelated to the question.
This paper, from AWS (Amazon Web Services) AI Labs, tackles sentence-level evidence selection for RAG: within a given token budget, keep only the sentences or table cells from retrieved passages that the answer needs, and pass them to the generator. There are three existing approaches. A reranker scores each sentence paired with the question alone, never seeing the other sentences, so the Thames sentence gets dropped. A second approach reads all candidates again when scoring each one, which is the most accurate but processes the same context K times. The third uses a class of "decider models": they score predefined options directly without generating text, and newer ones can produce many judgments in a single forward pass. Used for evidence selection, they read all candidates once and give each a score. The catch is that as candidates grow, an off-the-shelf decider has a harder time telling evidence from the rest. Curator fine-tunes this decider on evidence-selection examples with varying numbers of candidates and varying evidence density. On QAMPARI it still takes about 0.9 seconds per question, and it retrieves 9% to 16% more evidence than per-sentence scoring with full context, which takes 20 seconds per question.
Train a decider specifically to "judge many sentences at once" and it selects better than per-sentence judgment at the cost of one forward pass; the advantage is largest when the token budget is tight.
Three ways to score, and what each costs
First, what is being compared. Every method selects the same way: give each candidate a score, add candidates from highest to lowest until the token budget is full. Only how the scores are produced differs.
Per-sentence scoring looks only at the question and the current sentence, with no view of other candidates. Per-sentence scoring with full context reads all candidates once to score sentence 1, then again to score sentence 2, so K candidates mean K full reads. Joint scoring reads once and every candidate's score comes out together.
The paper compares the three with the same off-the-shelf decider. On a 200-question QAMPARI sample, per-sentence scoring with full context has the highest recall, 0.504; joint scoring gets 0.460 and single-sentence scoring 0.428. But per-sentence with full context takes about 20 seconds per question, joint scoring under 1 second. This shows the off-the-shelf model loses accuracy when judging all candidates at once.

With many candidates, the off-the-shelf model gets confused
Joint scoring is cheap, but is it accurate? The authors run a controlled experiment: each candidate set holds exactly 3 true evidence sentences and the rest are distractors, with the total K raised from 8 up to 128 or 256, and they measure how well the model separates evidence. The metric is AUROC: the probability that the model scores a random evidence sentence above a random distractor.
The larger K, the lower the AUROC. On FEVEROUS it falls from about 0.89 at K = 8 to about 0.77 at K = 128; on QAMPARI from about 0.79 to about 0.68 at K = 256. So the more sentences judged at once, the worse each judgment gets.

Is it just that the input gets longer? The authors run a diagnostic. On 15 QAMPARI questions, going from 8 to 128 candidates drops AUROC from 0.800 to 0.652. But adding 14,000 irrelevant tokens to the context at K = 128, without adding sentences to judge, moves AUROC from 0.652 to 0.690. This diagnostic suggests that the number of simultaneous judgments may matter more than length alone.
How Curator is trained
The base is Perplexity's pplx-decider-flash-v1.2, a decider built on Qwen3.5-4B. Each candidate sentence has a judgment position in the input, where the model outputs two logits, "evidence" and "not evidence"; their difference is the sentence's score.
Training data comes from QAMPARI, HotpotQA and 2WikiMultiHopQA, which mark the sentences that support each answer. The authors split passages into candidate sentences, mark the evidence, and build training examples three ways, with K from 8 to 128. The first samples K sentences at random, keeping the natural sparsity, sometimes with no evidence at all. The second fixes 3 evidence sentences and fills the rest with distractors, so evidence gets sparser as K grows. The third packs in as much evidence as possible, up to K − 1 sentences, forcing many "yes" answers in one pass.
With many candidates, evidence is a tiny fraction, and because distractors dominate, an ordinary loss leans toward them. Three evidence sentences are 37.5% of 8 candidates but only 2.3% of 128. The authors switch to a class-balanced loss: average the loss over evidence sentences and over distractors separately, then add the two, so a hundred-odd distractors cannot drown out the signal from 3 evidence sentences. Hoot says
Use the body's setting: K = 128 candidates, 3 evidence sentences and 125 distractors. Early in training the model cannot yet spot evidence, so each evidence sentence has loss 0.7; most distractors are already judged right, so each has loss 0.1. These two loss values are illustrative.
number of distractors → share under the ordinary average → averaging per class → evidence's share → samples with no evidence
- number of distractors → ordinary averageThe ordinary loss adds up the losses of all 128 sentences and averages them, so every sentence weighs the same. The 3 evidence sentences total 2.1, the 125 distractors 12.5.3 × 0.7 = 2.1; 125 × 0.1 = 12.5
- ordinary average → evidence's shareEvidence makes up only about 14% of the total loss, even though each evidence sentence is judged far worse. Most of the total loss comes from distractors that are already right.2.1 ÷ (2.1 + 12.5) ≈ 14%
- per-class average → evidence's shareThe class-balanced loss averages within evidence, giving 0.7, then within distractors, giving 0.1, and adds them. Evidence's share rises to about 88%.0.7 ÷ (0.7 + 0.1) ≈ 88%
- per-class average → weight of one evidence sentenceUnder the ordinary average each evidence sentence weighs 1/128; under the class-balanced loss it weighs 1/3. So the larger K, the further apart the two losses are on evidence.128 ÷ 3 ≈ 43 times
- sample with no evidence → only one classRandomly sampled examples sometimes contain no evidence at all. Equation (4) sums only over the classes present, so only the distractor average remains.loss = distractor average 0.1

Source
paper section 3.3, Eq. (4): the class-balanced loss averages within each class present and adds the averages; K = 128, 3 evidence sentences, 37.5% and 2.3% are from the same section; per-sentence losses 0.7 and 0.1 are illustrativeFine-tuning touches very few parameters: LoRA on the 8 full-attention layers plus the judgment output layer, about 3.8 million parameters out of 4.54 billion. In addition, 40% of training examples come from general classification, selection and scoring tasks, with distillation toward the original model so it keeps its general skills. On 19,631 examples across 7 general benchmarks, Curator's macro average is 0.8810 against 0.8733 for the original, with no statistically detectable difference on any of them.
Walking through it: London and the Thames
We use the example from the paper's Figure 1. The two retrieved passages are split into 6 candidate sentences; token counts, scores and the budget are illustrative.
Step 1, split. The Olympics passage has three sentences: "Danny Boyle directed the opening ceremony", "The 2012 Summer Olympics were held in London", "204 nations took part". The London passage has three: "London has many museums and landmarks", "The Thames flows through London", "It is one of Britain's main rivers". Assume about 8 tokens each.
Step 2, one forward pass. The question and the 6 candidates go into the model as one input, and each of the 6 judgment positions gives a pair of logits. Say "Olympics in London" scores 3.0, "The Thames flows through London" scores 2.0, and the other 4 sit around −1.0, meaning the model treats the first two as evidence.
Step 3, why the Thames sentence gets 2.0. When it is scored, it sits in the same input as "Olympics in London", so the model can follow the "London" link. Scored on its own with only the question, it might get −0.5, below "London has many museums".
Step 4, fill the budget. Set the budget to 20 tokens. In score order, "Olympics in London" goes in first, using 8; then "the Thames", 16 in total; the next one would make 24, over budget, so stop. The generator receives exactly these two sentences, which cover both hops of the answer.
Step 5, the cost. With 6 candidates, per-sentence scoring with full context runs 6 times and reads all 6 sentences each time, 36 sentence reads; joint scoring runs once and reads 6, so the savings grow with the number of candidates. Real QAMPARI questions have far more candidates; the paper measures 293,296 input tokens and 20.065 seconds per question for per-sentence scoring with full context, against 11,411 tokens and 0.925 seconds for Curator.
The off-the-shelf decider goes through the same five steps. Its problem is step 2: when K goes from 6 to 128, the scores stop separating well. Curator changes exactly the scores this step produces.
Results
First, QAMPARI, which was used in training. The paper uses 855 development questions that also served for checkpoint selection, so the numbers lean optimistic.
| Method | Recall (budget 512) | Recall (budget 2048) | Seconds per question | Input tokens per question |
|---|---|---|---|---|
| Off-the-shelf decider, joint | 0.2763 | 0.5791 | 0.909 | 11,411 |
| Off-the-shelf decider, per-sentence with full context | 0.3166 | 0.6242 | 20.065 | 293,296 |
| Qwen3-Reranker, per-sentence | 0.1874 | 0.5230 | 0.688 | 6,664 |
| Curator | 0.3663 | 0.6784 | 0.925 | 11,411 |
(Numbers from the paper's Table 2)
This shows Curator costs almost the same as the original joint scoring yet beats per-sentence scoring with full context, which costs over 20 times more. The tighter the budget, the bigger the gap: relative to the original model, recall is 17.1% higher at budget 2048 and 32.6% higher at budget 512.
Next, tasks never seen in training. On the fact-checking dataset FEVEROUS at budget 512, recall rises from the original model's 57.0% to 84.7%, against 74.6% for per-sentence reranking, which shows the skill carries over to other tasks. On the financial dataset MultiHiertt and the legal-contract dataset CUAD at budget 128, Curator's recall beats the strongest baseline by 10.7% and 14.4%; at budget 2048, by only 1.5% and 2.8%.
Finally, answer quality. With a 120B model answering, on MuSiQue at budget 128, Curator's context brings accuracy to 48.7%, against 39.7% for per-sentence reranking, 40.3% for the original model and only 19.3% for whole-passage reranking. At budget 2048, all methods land between 56.6% and 56.8%, so the value of selecting sentences lies mainly in tight contexts.
Figure 3 reruns the controlled experiment from section 2. The original model's AUROC falls with K, while Curator stays flat or rises slightly. On QAMPARI it holds up to K = 256, although training used K of at most 128.

The ablations point the same way. Fine-tuning on single-sentence examples alone improves discrimination, but it still drops as candidates grow; switching to joint training, even with small candidate sets only, largely removes the drop; mixing in different K values does better on large candidate sets.
What this means for people building RAG and retrieval
When context is tight, it pays to replace whole passages with selected sentences, and the selector must see the other candidates.
Budgets of 128 to 512 tokens are where selecting sentences and sending passages differ most: on MuSiQue at budget 128, whole-passage reranking answers fewer than half as many questions correctly as sentence selection. If you build agents, context must also hold tool outputs and reasoning, and the budget left for evidence often falls in this range.
If you use an off-the-shelf multi-judgment decider, first test how candidate count affects it: fix a few evidence sentences, add distractors step by step, and watch whether AUROC drops. If it drops a lot, follow Curator's recipe with your own labels and fine-tune jointly: varied K, varied evidence density, a class-balanced loss, and some general tasks mixed in to prevent forgetting.
Limitations
The main QAMPARI numbers come from a development set that was also used for checkpoint selection. Downstream answering uses a single 120B model, whose identity is not named.
The paper's limitations section is empty. DecisionRAG, a concurrent work that also uses decider models for retrieval, is not compared directly; the authors say its setting differs.
In Figure 3, the candidate pools for different K are sampled separately, and the authors themselves say these results cannot attribute the gains entirely to the number of judgments. When candidates do not fit in one context, they must be scored in groups and merged; the cost of that case is not reported separately.
How did this read for you?
Hoot made a note. Thanks for the feedback.
One tap, no sign-in; tap again to undo. Only anonymous counts are kept, never who you are.
Related papersMore from Paper Daily
Retrieval · 2026-10-29Tessera: Structure-Completed Evidence Units for Web Retrieval-Augmented GenerationAsk Wikipedia who produced Willy Wonka and the Chocolate Factory, and the answer sits in the producer row of the page's infobox. Text extraction drops the two names in that cell, the reader model sees only the director, and it answers wrong. Tracing answer evidence from page source to reader input over all of English Wikipedia, this paper finds such loss is common: for 7.6% of questions the evide…
Retrieval · 2026-11-16AntEmbed: Dual-Mode Multimodal Embedding Model for Efficient and Fine-Grained RetrievalMultimodal retrieval has two kinds of embeddings. Discriminative embeddings encode the raw input directly, which is fast but misses fine-grained meaning. Generative embeddings first have the model reason about the query intent and distinguishing evidence and then encode, which is more accurate but adds a reasoning pass to every query. AntEmbed, from Soochow University and Ant Group, lets one mode…
Agentic · 2026-10-23A Small Model Asks Better: Fine-Tuning Query Clarification for Retrieval-Augmented GenerationIBM builds ClariEdge, a ~3,900-sample multi-turn RAG clarification set that specially includes hard negatives that "look ambiguous but the referent is already fixed by context," then LoRA-tunes small Granite models to decide whether to clarify and to list candidates. Granite-4.1-3B LoRA reaches 96.6% judgment accuracy versus 79.4% for prompt-based GPT-4o and 73.1% for GPT-5.6 Sol (which tends to…
Retrieval · 2026-10-25Learning Marginal Viewpoint Utility for Budgeted Evidence Selection in RAGAsk a question about election participation, and a complete answer should cover three kinds of people: those who voted, those who did not, and those who were not eligible. If the context holds only voters' answers, it represents only one viewpoint, however accurate and relevant each one is. LMVU is a linear ranker with only 18 coefficients. At each step it computes novelty and similar features fr…
Retrieval · 2026-11-13PCG-RAG: Progressive Compositional Grounding for Retrieval-Augmented Generation in Complex Web Question AnsweringComplex web QA often requires first working out who or what several clues describe, then looking up an attribute of it, and a common mistake is to treat a name as settled after only one clue matches. The paper's example is a London power station converted into an art museum: a system that settles on Battersea from London and power station alone retrieves an architect who actually belongs to a dif…
Retrieval · 2026-11-12OpticalRAG: Pixel-Space Compression for Token-Efficient Retrieval-Augmented GenerationTo save tokens, RAG systems usually delete words or summarize the text, and the harder they compress, the more detail they lose. OpticalRAG goes the other way: it renders documents as page images, encodes them into visual tokens with a frozen vision encoder, and does both retrieval and compression on those visual tokens. A text query passes through a light projection into the same visual feature…
Retrieval · 2026-11-11Lightfoot: Graph Retrieval-Augmented Generation with Exact, Model-Free Document DeletionGraphRAG in production has two costs that papers rarely count: what it takes to build the graph, and how painful it is to delete a document. When a contract expires or a user asks for erasure, a document and everything derived from it must go, yet entity summaries that blend several documents are hard to take apart. Lightfoot requires every record in the graph to be computed only from single-pass…
Retrieval · 2026-11-05From Logs to Faults: Reference-Free Fault Localization for Retrieval-Augmented Generation PipelinesRAGLog reads only the API logs each run of a RAG pipeline leaves behind, with no reference answers and no code changes, and points out which stage of the run went wrong. This matters because a pipeline can go wrong in the middle and still give the right final answer. Across natural runs of five open-source pipelines, 37% had at least one faulty stage, and more than a third of those still answered…
Discussion
Add experiment details, reproduction notes or a different take. Requires a GitHub sign-in.
Loading the discussion…