DeFT-RAG: Incrementally Decomposing Multi-Hop Questions into Self-Contained Sub-Questions with Reusable Answers

Teams building RAG and question-answering search should read this one, because it cleanly separates the many jobs an LLM is asked to do in iterative retrieval, and once those jobs are split, accuracy, context length and reusability all improve. Multi-hop questions need evidence found link by link through intermediate entities that the question never names. Graph-based RAG starts from the entities the question mentions and mostly reaches only the first link. On three-hop questions it recovers the full evidence chain for less than a tenth of them. Iterative methods can search step by step, but the LLM must plan, order, rewrite and keep track at once, and each task fails in its own way. DeFT-RAG has the LLM write one sub-question at a time that names every entity it asks about, and adds each answered sub-question with its answer to a QA table by fixed rules, instead of the LLM keeping track. Yes-or-no decisions, such as whether a passage is relevant or whether the answers so far are enough, go to a small fine-tuned model. On MuSiQue with gpt-4o-mini, accuracy rises from the strongest baseline's 0.577 to 0.629. With a cache shared across 1,000 MuSiQue questions, 23.1% of sub-questions are answered from the cache and LLM calls drop by 12.9%, with no loss in accuracy. Letting the LLM do one small thing at a time is more reliable than having it plan the whole chain at once.

What problem is it actually solving?

The paper's example: "Where do greyhound buses leave from in the city where the performer of Darlings formed?" The correct chain is Darlings, then Kevin Drew (the album's artist), then Toronto (where he formed Broken Social Scene), then Toronto Coach Terminal. The paper calls Kevin Drew and Toronto bridge entities: the evidence chain passes through them, but the question never mentions them.

Searching with the question alone returns passages that share its words: about greyhound dogs, about a Greyhound coach model, about a person named Darling. Passages about Kevin Drew and Toronto rank low. On MuSiQue, vanilla RAG finds the full evidence chain for only 7% of three-hop questions.

The paper's Figure 1 shows one failure of each existing family:

Each failure has its own cause. Graph-based methods usually retrieve once per question, walking the graph from the entities the question names, and rarely get past the first bridge entity: most of them find the full chain for only 6% to 10% of three-hop questions. Iterative methods can find the next bridge entity as they answer, but they ask the LLM to do several things in one call. It must plan sub-questions in advance, decide which depends on which, rewrite the next sub-question when an answer arrives, and remember what is known in a growing prompt. The paper sorts the errors by these four jobs: planning, dependency, rewriting and tracking errors.

One more factor makes errors worse: most methods take a fixed top k passages. On a set of MuSiQue questions answerable in one step, the paper measured accuracy of 0.620 with the top 5 and 0.630 with the top 10, nearly unchanged, even though the share with all evidence present rose from 62% to 88%. Keeping only the needed passages from the top 10 lifted accuracy to 0.793. This shows that more passages do not help the LLM answer; removing irrelevant ones does.

The pipeline: the LLM does only three generation jobs

DeFT-RAG reduces the LLM's work to three jobs: write the next sub-question, answer a question from passages, and answer the original question from the QA table. Everything else goes either to rules or to a 4B yes-or-no model (the paper's decision model, called the decision model below).

How one question goes:

Steps 2 to 5 form one round, and each round writes one sub-question; at most 6 rounds run, and if it is still not enough after that, the system answers directly from the relevant passages among the top 20 retrieved for the original question. Now the three designs one by one.

Why write one "self-contained" sub-question at a time?

Self-contained means the sub-question names every entity it asks about, uses no reference like "that city" or "this performer", and does not depend on another sub-question's answer, so it can be searched and answered on its own. The paper's example: once the QA table holds "the Rhine ends in the North Sea", then "Where does the North Sea meet the Atlantic?" is self-contained, while "Where does that sea meet the Atlantic?" is not.

Walking through the Darlings question: with an empty QA table, the LLM writes "Who is the performer of Darlings?", and the answer Kevin Drew goes into the table. Next round, it sees Kevin Drew in the table and writes "In which city did Kevin Drew form his band?", and Toronto goes in. Then it writes "Where do greyhound buses leave from in Toronto?". Each sub-question is written fresh from the answers already known.

Why does this avoid the earlier errors? With no plan in advance, there is no need to guess a bridge entity before finding it. With no dependencies between sub-questions, there is no dependency order to get wrong and no other sub-question to rewrite when an answer arrives. The LLM does one thing at a time: phrase the question for the next missing piece of information.

Why not leave tracking and decisions to the LLM?

Tracking goes to the QA table. It is a two-column table, one sub-question and its answer per row, updated entirely by rules with no model call. Whether a sub-question was answered is read from the LLM's reply itself. If the LLM gives an answer from the kept passages, the pair is added. If it gives none, the sub-question is recorded, and the next prompt tells the LLM not to ask them again; a new sub-question with the same wording as an old one makes the LLM write another. Writing sub-questions, deciding whether enough is known, and giving the final answer all take the original question plus this table as input, with no past passages or reasoning. Compared with IRCoT, which keeps all passages and reasoning in one growing prompt, the table is short and every entity found is written down plainly, so "wrote Toronto in the reasoning, then lost it" cannot happen.

Yes-or-no decisions go to the decision model. It starts from a 4B model trained to output yes probabilities. It is then fine-tuned with LoRA on gold annotations from the training splits of the three datasets. A gold supporting passage is labeled relevant. A top 10 containing all supporting passages is labeled answerable directly. A QA table holding the answer of every hop is labeled enough (this kind uses only MuSiQue and 2Wiki, which have per-step answers). Each decision has its own threshold: 0.5 for relevance and for enough facts, and a stricter 0.7 for answering directly. The paper's reason for the stricter one is clear: a multi-hop question wrongly judged answerable in one step is almost always answered wrong, while a one-step question sent into the loop only costs a few more calls.

The most direct benefit of relevance selection is that the LLM reads fewer passages: each step retrieves the top 10 and keeps only 2.1 to 2.4 on average.

Why can answers be reused across questions?

Because sub-questions are self-contained, the same sub-question has the same meaning and answer no matter which original question it came from. The paper's example: "In which city is the headquarters of Sazerac Company located?" has the answer New Orleans, whichever of the company's whiskey brands the original question asked about.

The cache is checked in two layers. First, look for an old sub-question with exactly the same wording, ignoring case and punctuation. If none, take the 5 most similar old sub-questions and have the decision model judge each for whether it asks the same thing, with a strict threshold of 0.8, because a wrongly reused answer carries an error into another question. If one matches, its answer is used directly, with no retrieval and no LLM call.

How well does it work?

First the main results. Each dataset has 1,000 questions, and all methods share the same LLM and the same embedding model. The table shows LLM-judged accuracy (gpt-4o-mini answers, Qwen3.6 judges):

The largest gain is on MuSiQue, which has the longest evidence chains: 5.2 points above the runner-up Hi-Q, and 5.3 points with Qwen3.6 as the LLM, both significant. On 2Wiki it is on par with Self-Ask. This suggests that the longer the chain, the more it pays to take planning and tracking off the LLM.

The evidence coverage comparison says more. On three-hop MuSiQue questions, DeFT-RAG gives the LLM all the evidence for 46% of them, the highest of all methods, while the LLM reads only 390 words of passages per question on average, less than half of vanilla RAG's 870. Self-Ask covers more two-hop chains but reads 2,590 words per question, because it keeps all passages and reasoning in the prompt. So the gain does not come from stuffing in more passages but from finding them more precisely and keeping fewer.

Two ablations show what each part contributes. Turning off relevance selection and giving the top 5 passages instead lowers average accuracy over the three datasets by 3.8 points, and by 5.3 points on HotpotQA; the loss comes mostly from questions answered directly, which must find all evidence in one list of passages. Without fine-tuning the decision model, accuracy on the first 300 MuSiQue questions falls from 0.623 to 0.587, and fine-tuning helps the direct-answer decision the most.

The cache experiment: answering the 1,000 MuSiQue questions in order with one shared cache, 23.1% of sub-questions get their answer from the cache and LLM calls drop by 12.9%; accuracy is 0.642 against 0.629 without the cache, a difference that is not significant. This shows reusing intermediate answers across questions brought no visible errors.

Limitations as I see them

First, its latency is the highest of all methods. The median is 9.9 seconds per question, against 2.4 for vanilla RAG. The paper breaks it down: on MuSiQue each question makes about 7 decision-model calls at 1.6 to 1.8 seconds each, which take sixty to seventy percent of the answering time. Replacing it with a closed decision model over an API, at about 0.16 seconds per call, cuts the median to 5.0 seconds with accuracy within 1.2 points. So the bottleneck is how the decision model is served, but for online question answering it is still a hard cost.

Second, the LLM does not always keep sub-questions self-contained. On 2Wiki, some questions compare the ages of two films' directors, and the LLM writes nested sub-questions such as "what is the birth date of the director of this film" before the director is named. Such sub-questions go unanswered 38% of the time, against 4% for sub-questions asking a single relation, and 75 of 235 such questions hit the step limit or start repeating. This is the main reason it only ties Self-Ask on 2Wiki.

Third, the decision model is fine-tuned on training sets with gold evidence. The paper uses 48,036 training examples labeled from the three datasets' annotated supporting passages and from the per-step answers of MuSiQue and 2Wiki. Real products rarely have such labels, and the paper's experiments cover only these three labeled datasets, so moving to another domain means preparing new training data for the decision model.

How a question-answering search team could try it

Step one: test fixed top-k on your own data first. As in the paper's Table 1, take one-step questions and compare accuracy with the top 5, the top 10, and only the needed passages. If needed-only is clearly better, relevance selection is worth building.

Step two: replace tracking in your existing iterative retrieval with a rule-kept QA table, and have the LLM write one self-contained sub-question at a time. This only changes prompts and the outer control logic, with no model to train; watch multi-hop accuracy and LLM input tokens.

Step three: add the decision model. Validate the gain first with an off-the-shelf large model making the yes-or-no calls, then fine-tune a small model to cut cost; keep an eye on the number of decision calls and their latency, which take most of the answering time in the paper.

Step four: if your live questions share many intermediate facts, add the cache. Keep the reuse threshold strict and spot-check that reused answers really answer the same thing.

Verdict

The key lesson: in multi-hop question answering, giving the LLM fewer jobs helps more than giving it more passages.