Retriever-Free Retrieval: Navigating Documents with Decision Models

Most RAG systems still retrieve with BM25 or vector similarity, and whatever evidence retrieval misses, no amount of generator capability can recover. DecisionRAG drops the retriever altogether and lets a decision model browse the corpus the way a person browses a library, from topics to pages to passages. A decision model is a kind of LLM interface that generates no tokens: given a context and a set of yes/no questions, one forward pass returns the probability of "yes" for each question. Candidates within each request share a context and are judged together; oversized levels are split across parallel requests whose scores are pooled for screening, and only the most promising candidates are expanded. On 2.68 million Wikipedia paragraphs, its Hit@1 is 17.4 points higher than Qwen3-Embedding-8B retrieval plus an 8B reranker, and retrieval takes about 1.7 seconds per query.

Teams building RAG or document search should read this paper. It replaces the retrieve-then-rerank pipeline with level-by-level pruning along the corpus's own table of contents, and it uses a clean control to show that the gain comes from navigation itself. The same decision model reranking the top 100 passages from vector retrieval scores 7 Hit@1 points lower than navigating the hierarchy. The core idea in one sentence: the corpus structure narrows the search, and the decision model compares the candidates at each level side by side to decide which branches to follow. I think it is still some distance from production, though: it depends on a commercial decision-model API that costs 2.33 cents per query on the large corpus, more than GPT-5 reranking, and the number of candidates kept per level is tuned separately for each dataset.

What problem is it actually solving?

The paper runs one question through the whole text: "Who is the main character in Christopher Nolan's film from the year 2000?" The answer sits in the Wikipedia page for Memento, in the passage describing its protagonist, Leonard Shelby. The difficulty is that the question names neither the film nor the character.

The root problem is that the retriever is far weaker than the generator that follows. A reranker can improve the order but can only choose among what retrieval returned; agentic retrieval lets an LLM walk the table of contents step by step and is capable enough, but generating a string of tokens at every step is too slow. DecisionRAG wants an LLM's judgment at retrieval-level speed.

The pipeline: walking down the table of contents

Taking Natural Questions, a corpus of 2.68 million paragraphs, the pipeline is:

Wikipedia has no level above the page, so the authors cluster pages into 1,000 groups with k-means over e5 embeddings of titles and first sentences, and let Qwen3-8B write a one-sentence summary of each cluster from its page titles. The other three datasets use their existing help-center, page and heading structure. The key division of labor: the corpus structure sets how many candidates each level must look at, and the decision model decides which branches to follow. The parts are unpacked below.

Why score candidates in one shared context?

A decision model is used in two ways. One is parallel yes/no questions: the context holds the query and the descriptions of every candidate at this level, each candidate gets the question "is this candidate worth expanding for this query", and one forward pass gives the probability of "yes" for each:
$$p_i^{\mathrm{bin}}=P(\mathrm{Yes}\mid s,Q_i)$$
Here $s$ is the shared context and $Q_i$ is the question about candidate $i$. Each question is normalized only over "yes" and "no", so the probabilities across candidates need not sum to 1, and the number of candidates is limited only by context length. The other is a multiple-choice question: the candidates become options, and the probabilities are normalized across options and sum to 1. Jev supports at most 255 options, so multiple choice is used only for the final reranking.

How does sharing a context help? Asked alone, "is the Gladiator page worth reading", the page overlaps the query a lot and the model might give 0.6 (illustrative); asked in the same context as Memento, the model can see a clearly better option, and the same question drops to 0.2 (illustrative). Shared context improves comparability across candidate scores. Screening prioritizes recall, so scores from separate requests can be pooled. For the final ranking, comparing candidates in one shared request is more helpful.

The paper tests this on a fixed pool of 200 candidates made of the gold paragraph plus hard negatives from BM25, SPLADE-v3 and e5. With each candidate in its own request, Hit@1 is 47.8%; with 50 candidates per request it rises to 54.3%, and larger requests add little. Comparability thus comes mainly from a few dozen rivals in the same request, so a large level can be split into parallel requests of about 50 candidates with almost no loss. The same holds in the full pipeline: on MultiDoc2Dial, scoring the passages of all selected pages together rather than page by page raises R@1 from 36.7% to 44.4%.

Why latency barely grows with the corpus

Candidates within a level are judged in parallel, so latency depends on the number of levels, not on the total number of passages. For a roughly balanced hierarchy with at most $b$ children per node, $N$ passages need about $\lceil {\log }_{b}N\rceil$ levels; each level adds one sequential decision, giving the latency estimate:
$$T(N)\approx t_{\mathrm{req}}\lceil {\log }_{b}N\rceil$$
$t_{\mathrm{req}}$ is the latency of one request, about 0.2 seconds in the paper. Working it out (illustrative): with $b=100$, a million passages need 3 levels, about 0.6 seconds; the paper's own example is a billion passages needing only 5 levels, about one second. This assumes the server can handle all requests of a level at once. Cost, by contrast, grows with the number of candidates judged, at most $k\cdot b$ per level, where $k$ is the number of nodes kept at the level above.

The risk of pruning is that it cannot be undone: a cluster dropped at the first level takes all its pages with it. The authors measure this cost on Natural Questions by varying the share of clusters kept at the first level. Keeping 15% of clusters gives Recall@10 of 86.2%, slightly above the 84.5% from scoring all 108,626 page titles directly, at a fifth of the cost. Below 5%, the cluster holding the gold page is often dropped, and at 1% Recall@10 falls to 63.7%. The budget should therefore lean toward recall, keeping more at the first level.

Results: beating retrieve-then-rerank without training

The main experiment compares methods with no task-specific training. The table lists each dataset's top-rank metric: MultiDoc2Dial is multi-turn dialogue over government help centers, measured by R@1; ConditionalQA is scenario-based questions over UK government guidance and Natural Questions is real search questions over Wikipedia, both measured by Hit@1.

Against the 8B reranker, DecisionRAG is clearly ahead on all three datasets, by 17.4 points on Natural Questions. Against GPT-5 reranking it wins some and loses some: 65.4 versus 60.8 on Natural Questions, 69.7 versus 73.4 on ConditionalQA. Adding the top 20 passages from vector retrieval to the last level helps a bit on all three, which says the two approaches find complementary evidence.

Does the gain come from navigation or from the decision model itself? The authors let Jev rerank the top 100 passages from vector retrieval, scored exactly as DecisionRAG's last level scores them. On Natural Questions, this reranking reaches Hit@1 of 59.4%, versus 66.4% for navigation plus vector candidates. Interestingly, the vector top 100 contains the gold passage more often, 98.4% versus 94.0%; but when both contain it, navigation ranks it first 70.1% of the time versus 61.9%. During navigation, the model gives candidate paragraphs binary scores in a shared context. The paragraphs are grouped by page and listed in reading order. The authors suggest that this may help relevance judgments. The subsequent multiple-choice request uses only the top 5 passages after binary scoring as options, and its context also contains only those 5 passages.

Against generative navigation, the authors keep the hierarchy and budgets fixed and switch to GPT-5 writing out, level by level, the numbers of the candidates to keep. On Natural Questions, GPT-5 reaches Hit@1 of 59.0% against DecisionRAG's 65.0%, while taking 250.8 seconds per query against 1.73. At this step, generating tokens buys no better judgment.

Retrieval quality also carries into the final answers. The authors have the same Qwen3.5-27B answer from each method's top 5 passages. On MTRAG (multi-turn dialogue over IBM Cloud technical documentation), the share of answers judged correct rises from 39.4% with Qwen3-Reranker to 44.1% with DecisionRAG, meaning the extra correct evidence does turn into more correct answers. Training Qwen3-8B on MultiDoc2Dial data as a dedicated decision model lifts R@1 to 60.7, 5.7 points above the best published task-trained retriever the paper compares with.

Where I think it falls short

First, the cost rests on a commercial API. The main experiments use TypeSafe AI's Jev, at 2.33 cents per query on Natural Questions, above the 1.40 cents for GPT-5 reranking; the Qwen3-Reranker running locally on one A6000 has no API fee. For a search system with tens of millions of queries a day that unit price is hard to accept; it suits low-volume, high-value enterprise document QA better.

Second, hierarchy and budgets are tuned per dataset. How many candidates to keep at each level was chosen on a separate design set for each dataset, for example 10 pages on MultiDoc2Dial and only 3 on ConditionalQA. That means a new corpus needs retuning. A budget that is too tight is costly: keeping only 1% of clusters at the first level drops Recall@10 to 63.7%.

Third, the latency figures have a precondition. The analytical latency model assumes the server can process all requests at a level in parallel. The measured latency on Natural Questions is roughly 1.7 seconds. Measurements process one query at a time, with at most eight concurrent requests within each query. The roughly 17,500 page titles are split into requests capped at 20k tokens.

How a recommendation or search team could try it

1. Start on a document collection that already has a table-of-contents structure, such as a help center, product documentation or a knowledge base. No clustering is needed, levels and node descriptions already exist, and it is the easiest place to see whether navigation itself helps.
2. Run the reranking control before building navigation. With the same scoring model, rerank the top 100 results of your current retrieval and also navigate the structure, then compare top-rank hit rates; if navigation cannot beat reranking, the architecture change is not worth it.
3. Keep the first-level budget generous, and add the top 20 vector-retrieval results to the last level. In the paper this costs almost nothing extra and recovers evidence that pruning drops.
4. Log the number of requests, the concurrency and the cost per query from the start. Whether it can go live depends on unit price and concurrency, not on the paper's median latency.

Bottom line

The key to using an LLM at the retrieval step is letting it compare candidates in one shared context without generating tokens; DecisionRAG shows this works on small and mid-sized document collections, while large-scale search must wait for decision models to get cheaper.