Contents · 6 sections
1Why the table still ranks lower once the caption matches2Walk-through: how this table rises from rank 4 to rank 23How much recall drops if graph propagation is removed4How much more accurate the answers become once retrieval improves5What this means for people who do long-document QA6LimitationsSHARE
Ask which model has the highest F1, and the caption ranks first while the table of numbers ranks fourth: how LMEG-Doc connects the evidence
You ask, in a long technical report: "Which model has the highest F1?" Retrieval quickly ranks a caption first, because the caption talks about an F1 comparison. The table that actually lists each model's scores is mostly numbers, shares little wording with the question, and ranks only fourth.
The answer is in the table; what retrieval sees is the caption. Another question is harder: the prose says "see the table later," but the table is several pages away. If you only rank by how much this passage or this page looks like the question, the relevant figures and tables fall down the list.
This paper, from Capital One and the University of Southern California, is about multimodal retrieval over long documents: when useful evidence is scattered across pages and modalities, how to bring it back together. Existing text retrievers such as BM25 and BGE-M3 were built for lexical or semantic matching on text. Applied to documents, they usually first recognize the words on a page or parse them into passages, then score each fragment on its own. Visual retrievers such as ColPali treat a whole page as an image and score patches against the question, which helps keep layout and figures, but still rank by the similarity of a single page or region. LMEG-Doc first links document elements into an evidence graph from layout and context, then uses a vision-language model to add cross-page relations, and finally propagates multi-view retrieval scores over the graph. It needs no task-specific training. On the multimodal document retrieval benchmark MMDocIR, the share of annotated regions covered by the top 10 layout elements (L@10) rises from 62.0% for the strongest published baseline to 70.7%. When the retrieved evidence is given to the open-weight model Qwen3.8-27B, answer accuracy on DocScope rises by 7.6% over the same model answering without that evidence package, and on the financial-report benchmark FinRAGBench-V by 15.8%.
These gains appear when more than one piece of evidence has to be found; if you take only the single most similar page, it does not beat a retriever already fine-tuned on MMDocIR.
Why the table still ranks lower once the caption matches
The table sits next to the caption, so why does it still fall to fourth? Existing methods give each page or each layout element its own similarity score against the question, then rank by that score. A caption that contains words such as "F1" gets a high score. A table of model scores does not necessarily repeat the question's words, so its score is low. Connected evidence is split apart, and at answering time the system has the caption but not the table.
LMEG-Doc first cuts the document into elements with positions, then links those elements into an evidence graph that does not depend on the question and can be reused. Parsing uses Nemotron-Parse v1.2, which splits each page into headings, paragraphs, captions, figures, and tables, and records the page number, bounding box, type, and text. Inside a page it first adds four kinds of edges: reading-order edges that chain successive elements, caption edges that join a figure or table to its caption, prose-context edges that join a visual element to surrounding paragraphs, and spatial edges that join nearby elements.
Beyond pairwise edges come groups. A visual group holds a figure or a table together with its caption and explanation; a section group holds a heading and the elements in that section. The group itself is also a node, joined to its members by membership edges. That way, if one member directly matches the question, retrieval has a chance to bring the others back with it.
Are in-page edges enough for a table several pages away? They are not. The paper adds cross-page relation refinement (CRR). It first proposes cross-page candidates from lexical and semantic similarity, visual similarity, citations in the prose, shared numbers, and whether a table looks like a continuation, and a candidate must include at least one figure or table. A vision-language model then looks at the corresponding image regions, text, page numbers, and types, judges whether a relation exists, and gives a type and a confidence. Figure 1 states that a cross-page edge is added to the graph only if confidence is at least 0.8. In the example, a citation edge at 0.86 and a continuation edge at 0.92 are kept, and a "no relation" score of 0.31 is dropped. The graph is built before any question is asked, and retrieval does not call a large model again. Hoot says

Walk-through: how this table rises from rank 4 to rank 2
The walk-through follows paper Figure 2. The question is: "Which model has the highest F1?" In the document, the caption and the table on page 12 are already joined by a caption edge in the same visual group.
Step 1. Four-view scoring. BM25 checks lexical overlap, BGE-M3 checks textual semantic closeness, SigLIP 2 compares the question with a cropped region, and ColPali interacts the question with image patches on the original page. For each element, its bounding box selects the patches in that region; each question token takes the maximum dot product among those patches, and those scores are summed to give the element's ColPali score. The four scores are standardized separately, then added with weights, giving each element an initial score. The caption matches the question, so its initial score is 0.80; the table is mostly numbers, so its initial score is only 0.12, and it ranks 4th.
Step 2. Propagation over the graph. Ranking by initial scores alone would leave this table behind. The system treats high-scoring elements as seeds and uses personalized PageRank to send scores along the edges: HippoRAG, for example, uses it on knowledge graphs to pass relevance from known-relevant nodes to their neighbors, and it is used here on the evidence graph. The caption is the seed, with score 0.80; the table receives this score and rises from 0.12 to 0.58.
Step 3. Reciprocal rank fusion first merges the initial ranking and the propagated ranking, then intra-group refinement is applied. RRF adds reciprocal weights from the two lists: the closer to the top, the more is added, so elements that already matched still keep a place in the fused ranking. The fusion scores marked in the figure are: caption 0.78, table 0.58, and a prose passage 0.34. Then members of the same visual group are promoted to the highest score in the group, so the figure, the table, and the caption move forward together. The table's rank rises from 4 to 2.
Step 4. Pages are scored next, using the numbers in the figure. A page's global score is ColPali's whole-page similarity; its local score is the sum of the propagated scores of its elements. The two are first normalized, then averaged with equal weight. In Figure 2, page 12 has local score 1.00 and whole-page score 0.80; half and half is 0.5×1.00+0.5×0.80=0.90, and it ranks first. Page 4 has the highest whole-page score, 1.00, but its in-page propagated score is only 0.15, so the blend is 0.5×0.15+0.5×1.00=0.575, written as 0.57 in the figure, and it ranks lower. Page 13 is 0.5×0.70+0.5×0.35=0.525, written as 0.52.
Step 5. The evidence is packed. The layout evidence that remains is the caption and table on page 12, plus a prose passage on page 4; the pages are 12, 4, and 13. The answering model gets both the caption and the table of numbers, without changing how it reasons.

What this example can show is: on the same page, when the caption matches and the table scores low, an edge on the graph can pass the score over and bring the table back. It cannot show how useful cross-page edges are, because this table's lift happens inside page 12. Cross-page refinement has to be read from the multi-page results later.
How much recall drops if graph propagation is removed
If only graph propagation is removed, L@10 on all MMDocIR questions falls from 70.7% to 68.5%, a drop of 2.2 points. Removing intra-group refinement lowers it to 69.6%, a drop of 1.1 points.
Removing cross-page refinement drops all questions only from 70.7% to 70.5%; but on questions whose evidence spans multiple pages, L@10 falls from 54.8% to 52.9%. In-page relations contribute most of the overall gain; cross-page edges are more useful when evidence is spread across pages.
LMEG-Doc is not trained on the MMDocIR task. When page recall looks only at the first page, it is 56.7%, against 57.0% for Col-Phi3, which was trained on this benchmark.
When the first five pages are included, it is 85.7%, 3.5 points above Col-Phi3's 82.2%; L@10 rises from 62.0% for the strongest published baseline, ColPali, to 70.7%. Recall is therefore higher on the first five pages and on the first ten layout elements.
| Method | P@1 | P@5 | L@10 |
|---|---|---|---|
| ColPali | 53.0 | 80.8 | 62.0 |
| Col-Phi3 (trained on MMDocIR) | 57.0 | 82.2 | 61.1 |
| LMEG-Doc (no task-specific training) | 56.7 | 85.7 | 70.7 |
(Numbers from paper Table 2.)
On DocScope and FinRAGBench-V, the controls are BM25 and BGE-M3 for text retrieval, and ColPali, which ranks whole pages only. LMEG-Doc is highest at the first 3 pages, the first 5 pages, and layout L@5 and L@10. On the financial-report benchmark, recall of the first five pages is 3.5 points above ColPali, and L@10 is 52.0%. Looking only at the first page, it is still below ColPali: 40.2% vs 43.4% on DocScope, and 57.9% vs 61.8% on FinRAGBench-V. ColPali has no layout metric on these two benchmarks, because it ranks whole pages only.
Split by evidence type, the full method raises L@10 on figure questions by 7.4 points over four-view fusion, from 70.2% to 77.6%. Building layout relations into the retrieval pipeline helps bring figure evidence back.
On questions that need evidence from different modalities at once, page P@10 rises by 6.3% and L@10 by 3.0%. On single-page and multi-page questions, L@10 rises by 3.2% and 3.3%.
How much more accurate the answers become once retrieval improves
Retrieved pages and layout elements are packed into an evidence package and plugged into an existing QA pipeline, without changing the answering model. In Table 4, with Qwen3.8-27B answering and no LMEG-Doc evidence package, accuracy is 71.6% on DocScope; the corresponding FinRAGBench-V figure is 68.7%. After the LMEG-Doc evidence package is added, the scores rise to 79.2% and 84.5%, gains of 7.6 and 15.8 points.
With the same answering model, if only the evidence from four-view fusion is added, DocScope is 77.9% and FinRAGBench-V is 82.7%. Full LMEG-Doc is another 1.3 and 1.8 points above four-view fusion, so building relations into retrieval still helps answering.

The 79.2% also exceeds the proprietary models in the same table: Gemini 3.1 Pro at 78.9% and GPT-6.1 Sol at 78.4%. The answering model here is the open-weight Qwen3.8-27B, not fine-tuned further for this task.
Split again by whether the evidence crosses pages. Relative to four-view fusion, answer accuracy on multi-page questions rises from 73.6% to 75.3% on DocScope, a gain of 1.7%, and from 83.3% to 88.3% on FinRAGBench-V, a gain of 5.0%. This shows that full relation-aware retrieval helps on multi-page QA. Single-page questions also improve, but by less. The F1-table example earlier happens inside one page; the isolated contribution of cross-page edges can be read only from the earlier CRR ablation on multi-page recall, not pulled out of this QA comparison on its own.

What this means for people who do long-document QA
There is no need to change the answering model, or to fine-tune the retriever on layout labels first; the evidence package can be plugged into an existing QA pipeline. Do not spend the retrieval budget on the first page only.
If you are doing QA over papers, technical reports, or financial disclosures, start with a set of questions where "the caption matches, but the answer is in the figure or the table," and see where those figures and tables currently rank. If figures and tables often rank late, ranking by single-fragment similarity alone may miss this complementary evidence; an evidence graph offers another path to bring it back. The graph can be built before any question is asked and reused across questions, with scores propagated over it at retrieval time.
Limitations
The FinRAGBench-V QA numbers come from 539 questions in the English subset, on financial reports rendered as page images, with right and wrong judged by the model Qwen3.6-35B-A3B. Cross-page refinement adds little on all questions; its use has to be read on the subset whose evidence spans multiple pages, with the exact drops given in the ablation above.
How did this read for you?
Hoot made a note. Thanks for the feedback.
One tap, no sign-in; tap again to undo. Only anonymous counts are kept, never who you are.
Related papersMore from Paper Daily
Retrieval · 2026-11-11Yuanbao-Reranker: Document-Grounded Synthesis and Preference Distillation for Text RerankingRerankers fine-tuned pointwise on relevant and not-relevant labels have a gap: every negative for a query carries the same label, so the model never learns which negatives sit closer to the positive. Tencent's Yuanbao Search pairs each positive with three negatives of different difficulty and adds a distillation loss: the teacher's score gap between two documents acts as a lower bound, and the st…
Retrieval · 2026-11-12DocFS: File Systems as Agentic Reasoning Environments for Complex Multimodal Document Question AnsweringWhen a VLM agent answers questions over long documents that mix text, figures and tables, the hard part is often how the document is laid out for it, not the model. DocFS takes a plain route: it turns the document into a file system, with sections as nested directories and text, figures, tables and equations stored as separate files, plus index files and relation files that record which text goes…
Retrieval · 2026-11-11Lightfoot: Graph Retrieval-Augmented Generation with Exact, Model-Free Document DeletionGraphRAG in production has two costs that papers rarely count: what it takes to build the graph, and how painful it is to delete a document. When a contract expires or a user asks for erasure, a document and everything derived from it must go, yet entity summaries that blend several documents are hard to take apart. Lightfoot requires every record in the graph to be computed only from single-pass…
Retrieval · 2026-10-24Chunk Recomputation under Real Document Revisions: Measuring Update Cost and Retrieval Quality of Fixed, Recursive, and Content-Defined ChunkingChunking schemes for RAG are almost always compared on one static corpus, yet the documents in a knowledge base keep changing. Each chunk is embedded on its own, so if an edit moves a chunk boundary, the chunks on either side become new embedding inputs even when not a single word in them changed. The authors take adjacent revisions of Chinese and English Wikipedia articles and allow an old vecto…
Retrieval · 2026-11-16CURATOR: Scalable Fine-grained Evidence Selection for Retrieval-Augmented Generation using Decision ModelsRAG usually stuffs whole retrieved passages into the LLM, though often only a few sentences in them can answer the question. Picking those sentences is not easy: a sentence can look irrelevant on its own, and whether to keep it may depend on other retrieved text. Judging each sentence with the full context is too expensive. Decision models can score many candidate sentences in one pass, with each…
Retrieval · 2026-10-30When Is Additional Retrieval Worthwhile? A Cost-Aware Study of Evidence Gain and Answer Reliability in RAGIs adding a second retrieval pass to RAG worth it? This paper first gets evidence with vector retrieval plus reranking, then has a small linear model read the scores left by that pass, such as how high the top-1 similarity is and how far it sits above the second result. From those it predicts whether switching to hybrid BM25-plus-vector retrieval would find more evidence, and it switches only whe…
Retrieval · 2026-10-29Tessera: Structure-Completed Evidence Units for Web Retrieval-Augmented GenerationAsk Wikipedia who produced Willy Wonka and the Chocolate Factory, and the answer sits in the producer row of the page's infobox. Text extraction drops the two names in that cell, the reader model sees only the director, and it answers wrong. Tracing answer evidence from page source to reader input over all of English Wikipedia, this paper finds such loss is common: for 7.6% of questions the evide…
Retrieval · 2026-10-25When Agentic Retrieval Fails Silently: Framework and Production Evidence from a Creative AI AssistantIn Adobe Express's AI assistant, retrieval metrics that assume the user saw a ranked list and acted on it stop working. The chat entry point has a reformulation rate of 44.4%, higher than the template gallery search served by the same retrieval system, at 33.1%. The reason is that reformulation means different things on the two sides. In the gallery, a user reformulates because the results were i…
Discussion
Add experiment details, reproduction notes or a different take. Requires a GitHub sign-in.
Loading the discussion…
