Contents · 8 sections
1What problem does it address?2How the metrics are defined3What the metric adds beyond recall4How the packer is designed5Results6Limitations7How a RAG team can use this8Conclusion and assessmentSHARE
This paper is for engineers building retrieval-augmented generation (RAG) systems who must decide which retrieved passages go into a limited context window. The core idea fits in one sentence: teams usually judge retrieval by recall@k, but the model reads the context after it has been trimmed and packed; under a tight budget the two drift apart, and the quantity that matters is whether the gold answer appears in the context the model actually reads. The author calls this answer-in-context. On HotpotQA, even when the top 5 retrieved documents contain every gold document, 27% of questions lose the answer during packing, and their exact-match rate is 0.11 against 0.50 for questions where the answer survives. Building on this, the author designs a context packer that optimizes for answer coverage. It reliably beats relevance-ordered truncation and LLMLingua-2 compression, but against a hand-tuned heuristic it ties in 12 of 14 comparisons.
What problem does it address?
A RAG pipeline has three steps: a retriever returns documents, a packer selects pieces of them to fit a token budget, and a reader model generates the answer. Recall is measured after the first step and only asks whether gold documents were retrieved; the model reads the output of the second step.
When retrieved relevant content exceeds what the budget can hold, the packer must choose, and recall stops describing what the model sees. Multi-hop questions suffer most, because the answer depends on linking facts from several documents. An illustrative example: the question is "In which country was the director of film X born?". Of 5 retrieved documents, the first two both describe film X and the third is the director's biography, which says "born in Paris, France". Recall is perfect. But if the budget only fits two passages, a relevance-ordered packer keeps the two overlapping film passages and drops the biography, so the model cannot answer.
The study uses three multi-hop QA datasets (HotpotQA, MuSiQue, 2WikiMultiHopQA) and two single-hop ones. The setting is deliberately tight: the main budget is 160 tokens, the main reader is Qwen2.5-3B, and everything runs on two T4 GPUs.
How the metrics are defined
All three quantities are computed on the packed context that is handed to the model, which is what separates them from recall:
| Metric | How it is computed | Value in the example |
|---|---|---|
| answer-in-context | 1 if the normalized gold answer appears as a contiguous span in the context | Document 3 was dropped, so 0 |
| gold-document coverage | Share of gold documents that contribute at least one passage | One of two gold documents, so 0.5 |
| gold-token density | Share of context tokens that come from gold documents | Depends on passage lengths |
Why not just use recall? For extractive answers, a model almost never answers correctly if the answer is absent from its context. Answer-in-context is a necessary condition for a correct answer; recall is not.
For long free-form answers without an exact span, the author adds a graded version: slide a window twice the answer length over the context, measure what fraction of answer tokens it covers, and take the maximum. When the full answer appears, the window covers every answer token and the score is 1, matching the binary version; when only part appears, the binary version gives 0 while the graded one returns the covered fraction, so long answers still get a useful signal.
What the metric adds beyond recall
Pooling every packing-policy-by-question pair on HotpotQA, answer-in-context correlates with F1 at 0.50, compared with 0.31 for recall and 0.33 for gold coverage. Mean F1 is 0.596 when the answer is in context and 0.123 when it is not.

A natural objection is that the metric is just recall in disguise, or nearly the same as getting the answer right. The author runs two checks.
The first is incremental explanatory power. Recall alone explains 8.5% of the variance in F1; adding answer-in-context raises that to 25.7%, an extra 17 points, which shows it carries information recall lacks. On the other two multi-hop datasets the gain is larger, 23 and 27 points. The correlation between answer-in-context and recall is only 0.24 to 0.41, so it is far from a relabeled recall.
The second is two interventions. On 2WikiMultiHopQA, the author's packer places more gold documents than the heuristic (coverage up 0.054), yet answer-in-context does not move (-0.007) and neither does F1 (-0.008, not significant). The answer usually sits in the document the heuristic already ranks first; the extra documents are bridge evidence. Conversely, LLMLingua-2 compresses at token granularity, breaks answer spans apart, cuts answer-in-context by 0.243, and accuracy falls with it. When coverage changes but answer presence does not, quality stays flat; when answer presence changes, quality follows. That points to answer presence as the variable doing the work.
How the packer is designed
Since the goal is to get the answer into context, the author frames passage selection as budgeted submodular maximization. A submodular function has diminishing returns: the more you have already selected, the less each additional passage adds. The objective is a weighted sum of four terms:
| Term | Role | Weight |
|---|---|---|
| Relevance | Sum of per-passage relevance scores, which come from candidate generation and are taken as input | 1.0 |
| Query coverage | How many distinct query content words are covered; repeats earn nothing | 0.5 |
| Representativeness | How much of the candidate pool the selection represents, with a cap so near-duplicates cannot inflate it | 0.4 |
| Source diversity | Square root of each document's relevance mass, summed, which spreads picks across documents | 0.3 |
The representativeness term works like this: for each candidate, sum its similarity to the selected passages, cap that at 0.3 times its total similarity to all candidates, then sum over candidates. The cap means a candidate stops adding value once it is represented enough, so packing several near-duplicate passages cannot push this term up without limit.
The solver is the standard cost-scaled greedy: at each step, among passages that still fit, add the one with the highest marginal gain per token. For example, passage A has 40 tokens and adds 0.4 (0.01 per token); passage B has 20 tokens and adds 0.3 (0.015 per token), so B goes first. A final check returns the single best passage instead if it beats the whole greedy set.
There are four baselines. Naive packing concatenates by relevance until the budget is full, which is the default in most RAG pipelines. MMR (maximal marginal relevance) is the classic redundancy-aware reranker. LLMLingua-2 compresses the same candidates to the same budget. The strongest baseline is the focused heuristic, the best packer from the author's earlier project. As described, it picks passages greedily, first by how many uncovered query words a passage adds, then, on ties, by preferring passages from documents not yet used; it checks the budget only after each addition and truncates when it overflows. The author treats it as a rough hand-built approximation of the same objective: it captures query coverage and source diversity but has no representativeness term, no per-token scaling, and no budget awareness while choosing. For a fair comparison, both methods use the same candidate passages and the same per-passage relevance scores, so the selection rule is the only difference.
Results
HotpotQA, budget 160, 3B reader, averaged over three seeds (Table 3 of the paper):
| Packing policy | F1 | Exact match | Tokens used |
|---|---|---|---|
| LLMLingua-2 compression | 0.326 | 0.233 | 129.6 |
| Naive packing | 0.400 | 0.306 | 151.1 |
| MMR | 0.410 | 0.313 | 151.7 |
| Focused heuristic | 0.429 | 0.331 | 152.1 |
| Submodular packer | 0.451 | 0.359 | 145.5 |
| Best policy per question (headroom reference) | 0.601 | 0.487 | 141.5 |
The submodular packer is the best fixed policy on every seed and uses fewer tokens. It beats naive packing by 0.051 F1 and LLMLingua-2 by 0.125. Compression loses so much because token-level deletion breaks answer spans: even naive packing cut to a 128-token budget still scores 0.368 F1, above compression's 0.326. A per-question breakdown attributes 81% of the packer's gain over the heuristic to 37 questions where it newly placed the answer in context, because the number of questions with every gold document in context rose from 256 to 289.
With two other 3B readers (Falcon3-3B and Phi-3.5-mini), the advantage over naive packing stays significant (+0.029 and +0.048), but the edge over the heuristic disappears (-0.004 and +0.002).

As readers get larger, the edge over the heuristic is absorbed: a bigger model can find the answer in the heuristic's context on its own, so denser gold content stops buying accuracy. Budgets show a similar pattern. Across 96, 128, 160 and 224 tokens, the edge over the heuristic is significant only at 160, while the lead over naive packing holds at every budget.
Limitations
First, the edge over the hand-tuned heuristic mostly does not hold up. Of 14 comparisons, 1 is significantly positive, 1 significantly negative, and 12 are ties. With no true effect, 14 tests would be expected to produce one or two significant results by chance, and the author concedes that the single positive result is only a hypothesis, and that the conditions under which the packer helps were inferred after the fact from the same experiments.
Second, the setting is far from deployment. A 160-token budget holds two or three passages, while real RAG budgets are usually thousands of tokens; relevance is lexical, and the embedding model is only bge-small; the cross-family check runs only at 3B, and 32B uses a single seed. At a 224-token budget the edge over the heuristic is already not significant.
Third, answer-in-context is lexical matching. Paraphrased answers count as absent, and the author notes that a small part of the 27% lost-answer group falls into this category. The graded version helps with long answers but is still token overlap and does not measure faithfulness to the source.
Fourth, this is single-author work and every conclusion comes from one experimental pipeline; reproducing it means rebuilding its passage segmentation and the focused heuristic baseline. Also, picking the best policy per question reaches 0.60 F1 against 0.45 for the best fixed policy, so different questions favor different packing. But the submodular packer is already best or tied-best on 79% of questions, and on the rest, which policy wins is only known after seeing the reader's answers. That information is unavailable at inference time, so the author reports 0.60 as headroom rather than a deployable method.
How a RAG team can use this
- Add a diagnostic first: next to recall, log answer-in-context, that is, whether the gold answer appears in the final context passed to the model. Watch the share of questions with perfect recall but a dropped answer; in the paper's HotpotQA setting it is 27%.
- If you currently truncate by relevance or squeeze the budget with LLMLingua-style compression, switching to passage-level selection is likely to give a steady gain; in the paper this held for every reader and budget.
- If you already have a well-tuned heuristic that covers query words and spreads across sources, do not expect the submodular packer to add much.
- For multi-hop tasks, check retrieval first: on MuSiQue only 18% of questions have every gold document retrieved, and no packing strategy can fix that; iterative multi-hop retrieval should come first.
Conclusion and assessment
The most valuable part of this paper is the diagnosis rather than the method: in budget-limited RAG, recall does not mean the model saw the answer, answer-in-context explains final quality better, and two interventions support it as the key intermediate variable.
| Dimension | Assessment |
|---|---|
| Importance | Medium-high. The trade-off between context budget and retrieved evidence is an often-ignored part of RAG engineering |
| Novelty | Medium-low. Submodular greedy is textbook; the new part is tying the objective to the diagnostic |
| Evidence | Strong for the metric; the packer's edge over a strong heuristic is essentially a tie, which the author reports honestly |
| Engineering cost | Low. The diagnostic is string matching and the packer is a simple greedy |
| Worth trying | Add the diagnostic now; try the packer when replacing naive truncation or compression |
| Best fit | RAG systems with tight context budgets, multi-hop QA, and smaller readers |
How did this read for you?
Hoot made a note. Thanks for the feedback.
One tap, no sign-in; tap again to undo. Only anonymous counts are kept, never who you are.
Discussion
Add experiment details, reproduction notes or a different take. Requires a GitHub sign-in.
Loading the discussion…