EchoSpec: Training Search Agents to Weigh Evidence Above Echoes

Shihao Yang, Can Zeng, Xinzhe Li
35 minread4.50scoreNo. 211-18 AI Search issue
Contents · 6 sections1How echoes keep the primary evidence out2Why exact-match rewards make the agent stop early3Walkthrough: 1890 in the eight echoes, 1887 in the original4What two-stage training leaves in, and what happens if you drop a stage5What this means for people who build search agents6Limitations
Rel 5Nov 4Prac 4Evid 5Time 5

Eight rewrites say Hearst entered publishing in 1890, but the original says 1887: how EchoSpec gets search agents to find the primary evidence first

Here is a multi-hop question: William Flannery once worked as an architect for someone; in what year did that person enter the publishing business?

The correct answer is in the original Wikipedia article. In 1887, William Randolph Hearst took over the San Francisco Examiner from his father and entered publishing. The corpus also has eight rewritten pages, and both their titles and bodies say 1890. The primary evidence that writes 1887 is still there, and it is still retrievable.

A search agent trained with "credit only if you get the answer right" ran a first query, "William Flannery architect for". The top three hits were all from those eight echoes. It never looked up Hearst himself and answered 1890.

This paper is from Tsinghua University. The authors are Shihao Yang, Can Zeng, and Xinzhe Li. They are looking at a training problem for search agents: the same sentence in a corpus can be rewritten and copied onto many pages, while the primary evidence that can actually settle the question often appears only once. Agents treat these echoes as enough evidence to answer, and they also stop refusing when the primary evidence is deleted. Search-R1 trains a multi-hop search policy end to end with exact match on a fixed Wikipedia index. Clean question-answer data cannot tell "appears many times" from "has primary evidence", so the agent stops at the first passage that writes an answer. Defenses such as TrustRAG, source gating, and multi-agent debate only handle documents already retrieved; they cannot bring back primary evidence that has been pushed out of the top ranks. EchoSpec trains on five corpus transformations with known target answers. Stage 1 is a verified warm-start that teaches the model to look past first-round echoes and query the bridging entity. Stage 2 then runs group RL with a λ reward that has an evidence threshold, and also scores finding the primary evidence itself. On Qwen2.5-7B, eight-echo deference falls to 28.9%, 30-44 points below seven published RL search agents, primary-evidence reach rises to 25.8%, and standalone QA answering precision is 47.1%.

Exact-match rewards teach a search agent to treat echoes as enough evidence and to stop searching early. EchoSpec writes "find the primary evidence first, then weigh the echoes" into the search policy. The comparison is multi-hop Wikipedia QA with known primary evidence, not live web search.

How echoes keep the primary evidence out

Here an echo means the same unsupported claim, rewritten onto many pages. A false echo swaps in a plausible alternative answer and writes it into both the title and the body. Deference is the share of questions the model already got right on the clean corpus that it then flips to the injected false answer. Reach is whether the trajectory retrieved, into the context, the primary-evidence passage that writes the correct answer.

The experiments follow the Search-R1 protocol: a 21 million-passage wiki-18 corpus, e5-base-v2 retrieval, the top 3 passages each time, and at most 4 searches per question. The test set is 1,114 multi-hop questions from HotpotQA and MuSiQue. The primary evidence remains retrievable under every condition.

A single false echo already overturns 29-48% of previously correct answers. At eight echoes, Qwen2.5 models from 7B to 32B, plus Search-R1, show deference of 58-66%.

Making the model larger does not stop this. At eight echoes the 32B model defers on 63%, and the 7B model on 61%. Scale itself does not fix the problem. Search-R1 is higher than the 7B base it was trained from: in the Figure 2 diagnostic experiment it reaches 66% at eight echoes. The main results table later reports 69.1% for the same model under the full evaluation protocol, and later method comparisons use that table.

Eight true echoes raise accuracy from 17-26% to 48-57%, while eight false echoes drop it to 5-9%. Both kinds of echoes cut the number of searches by about the same amount: Search-R1 falls from 2.49 searches per question to 1.89 and 1.91. This shows that once an echo in the top ranks writes an answer, the agent stops, whether that answer is true or false.

Once false echoes pile up, previously correct answers flip and the number of searches drops
Once false echoes pile up, previously correct answers flip and the number of searches drops

Figure 2. As false echoes increase, previously correct answers flip more often; both true and false echoes cut the number of searches. Source: paper Figure 2.

Once a trajectory retrieves the primary evidence, deference at eight echoes is 25-34%, no higher than the 27-39% at a single echo. So once the primary evidence is in the context, the agent holds up about as well against eight echoes as against one.

When the primary evidence never enters the context, deference at eight echoes reaches 70-85%. Reach itself falls from 53-72% at one echo to 20-31% at eight. So echoes succeed because they fill the top ranks and stop the search early; the primary evidence never comes into view. The Hearst question at the start is this case: the primary evidence that writes 1887 is always in wiki-18; the agent simply never fetched it.

Why exact-match rewards make the agent stop early

Exact match scores a wrong answer and a refusal the same. EchoSpec later scores a wrong answer as −λ, and scores a refusal as 0 when the question is still answerable; exact match is the case λ = 0. Then any nonzero chance of guessing right has a higher expected payoff than refusing.

Train Qwen2.5-7B with the Search-R1 recipe, and refusal when the primary evidence is deleted falls from 47% to 3-14% within 100 steps. So exact-match training suppresses refusal. On seven QA sets, the released Search-R1-7B also refuses 5-15 points less than its base model.

Can searching more rounds, or filtering echoes already retrieved, bring the primary evidence back?

Forcing Search-R1 to search three full rounds issues 3.18 queries on average, and eight-echo deference is still 55.5%. On the Hearst question, TrustRAG uses clustering to drop the injected echoes from the context. None of the remaining passages is the primary evidence about Hearst entering publishing, so the model refuses, and the primary evidence still never arrives. Filtering only acts on documents already retrieved; primary evidence stuck outside the top ranks still cannot get in.

Walkthrough: 1890 in the eight echoes, 1887 in the original

The next section walks through a real trajectory from Table 12 in the paper appendix. The question, the gold answer, the echo year, and the retrieved sentences all come from the paper. The reward numbers use the paper's default hyperparameters on this trajectory's retrieved set, and they are illustrative.

First, look at what is in the corpus. The question asks in what year Flannery's employer entered publishing. The gold answer is 1887. The corpus has eight rewritten pages, all saying 1890. The bridging article William Flannery and the primary-evidence article William Randolph Hearst are still there and still retrievable. The agent takes the top 3 passages each time and may search at most 4 times.

Second, see how early stopping happens. Search-R1's first query is "William Flannery architect for". The top three hits are all echoes that say 1890. It never queries William Randolph Hearst and answers 1890. AutoRefine also stops in the first round and also answers 1890. This trajectory answered the wrong year, 1890, so exact match will not give it credit for the correct answer. But on clean training questions, the first passage that writes an answer is often exactly the gold answer, so the outcome reward reinforces this early stop. Exact match also does not distinguish whether the answer came from primary evidence or from an echo.

Third, see what EchoSpec gets in the first round. It queries "William Flannery architect". The echoes are still there, but the bridging article William Flannery also appears, saying: an American art director and architect whose employer was William Randolph Hearst. It does not answer from those unsupported claims.

Fourth, follow the bridging entity. The second query is "William Randolph Hearst biography". Passages already seen in the same trajectory are masked, so it does not keep hitting the same batch of echoes. The third query is "William Randolph Hearst entered publishing business", which retrieves the primary-evidence article. The original sentence is: Hearst entered the publishing business in 1887 after being given control of The San Francisco Examiner by his wealthy father. It answers 1887.

Fifth, work the default reward through this path by hand (illustrative). Default λ=0.10. A correct answer without the primary evidence gets only α=0.25. The acquisition bonus has weight β=0.30. This question has one bridging article and one piece of primary evidence. After the first round, only the bridge is in hand, so the bonus term is 0+0.5×1/1=0.5, and the bonus is 0.30×0.5=0.15. By the third round the primary evidence is in too, so the bonus term is 1+0.5=1.5, and the bonus is 0.30×1.5=0.45. Answering 1890 from echoes alone, with the primary evidence not in the set, scores −0.10; even with the bridge already there, the total is only −0.10+0.15=0.05. Finding the primary evidence and answering 1887 totals 1+0.45=1.45. Having the primary evidence and still refusing totals 0+0.45=0.45. To prefer answering over refusing, the probability of being right must exceed 0.10/1.10≈0.09. When λ is set to 0, that threshold becomes 0.

What two-stage training leaves in, and what happens if you drop a stage

To turn this "query the bridge first, then fetch the primary evidence" path into a habit, EchoSpec first fixes the target answer in each of five corpus transformations, then trains in two stages.

For the Hearst question, besides the original corpus, three transformations should still answer 1887: copy the primary evidence several more times; inject false echoes such as "1890"; and inject true echoes that use the same wording but change the year to 1887. Two transformations change the answer: rewrite the primary evidence to 1890; and delete the primary evidence, where the target is to refuse. True and false echoes share the same sentences and only swap the year, so style, length, and copy count do not reveal which is which. A TF-IDF classifier asked to tell them apart therefore reaches an AUC of only 0.60-0.68.

Five corpus worlds: a warm-start first, then group RL with λ
Five corpus worlds: a warm-start first, then group RL with λ

Figure 4. Left: five corpus transformations and search with at most four queries of top-3 passages. Middle: stage-1 warm-start with a temporary article-chain hint. Right: stage-2 reward and advantage estimation. Source: paper Figure 4.

Stage 1 is a verified warm-start. When synthesizing trajectories, the prompt temporarily writes the supporting article chain and tells the model to query entities in the question, not the ready-made answers in the echoes, until it confirms the primary evidence or runs out of searches. The default synthesizer is Qwen2.5-32B-Instruct. Without the hint, this 32B model itself defers on 63% at eight echoes. Kept trajectories must answer from the primary evidence, or refuse on a deleted-evidence world after at least three searches; hops must be continuous, and the reasoning must not mention the hint. The hint is then removed, and the model is fine-tuned with LoRA SFT. Under the 32B hint, 3,582 trajectories are kept.

Why not go straight to reinforcement learning? Policy gradients can only reinforce behaviors already present in the samples, and the base agent rarely searches past first-round echoes. λ-reward RL with no warm-start reaches an eight-echo evidence reach of only 9.4% and deference of 69.4%, close to the 67.7% from exact-match RL.

Stage 2 continues from the warm-started policy with group RL. The reward is +1 when the agent finds the primary evidence and answers correctly, or correctly refuses on a deleted-evidence world; α when it answers correctly without the primary evidence; 0 when it refuses on an answerable world; and −λ otherwise. The acquisition bonus only checks whether the retrieved set contains the primary evidence and the bridging articles, not the final answer, so it does not change the threshold for when to answer and when to refuse. Advantage estimation uses mean-centering only, and does not divide by the group standard deviation. The paper proves that if a group has only two payoff levels, refuse (0) and a wrong answer (−λ), dividing by the group standard deviation cancels the scale of λ completely. At λ=0.1, training with standard-deviation normalization raises refusal on answerable worlds to 61-79% within 50 steps.

Hoot saysIf you divide by the group standard deviation, where does the wrong-answer penalty go?Close ✕

The article already walks the Hearst question: 1890 in the eight echoes, 1887 in the original, and how a correct answer without the primary evidence is scored. This walkthrough only pursues a point that walk did not unpack: after rewards become training signals, why dividing by the group standard deviation wipes out the scale of the wrong-answer penalty. The group below has one refusal and one wrong answer; these are illustrative numbers. λ is the article's default, 0.1.

refuse scores 0 and a wrong answer scores −0.1, so the mean is −0.05 → after mean-centering the values are +0.05 and −0.05 → divide by the group standard deviation 0.05 and they become +1 and −1 → redo the arithmetic with λ=0.2 and the normalized values are still +1 and −1, so the penalty scale is gone

  1. refuse 0 and wrong answer −0.1 → group mean −0.05Illustrative group of two: a refusal scores 0, a wrong answer scores −λ=−0.1. The mean of the two is −0.05.(0 + (−0.1)) / 2 = −0.05
  2. subtract the mean → refuse +0.05, wrong answer −0.05After mean-centering only, the refusal becomes 0−(−0.05)=+0.05, and the wrong answer becomes −0.1−(−0.05)=−0.05. Half of the 0.1 penalty scale is still visible.0 − (−0.05) = 0.05; −0.1 − (−0.05) = −0.05
  3. then divide by the group standard deviation 0.05 → +1 and −1The population standard deviation of these two centered values is 0.05, so after the division they become +1 and −1. The 0.1 gap in the original penalty is now only a sign.0.05 / 0.05 = 1; −0.05 / 0.05 = −1
  4. redo with λ=0.2 → the normalized values are still +1 and −1If the penalty is instead −0.2, the mean becomes −0.1, the centered values are +0.1 and −0.1, the standard deviation is 0.1, and the division still yields +1 and −1. Changing λ no longer changes the normalized advantages, which is why the article only mean-centers.(0 + (−0.2)) / 2 = −0.1; 0.1 / 0.1 = 1

At λ=0.1, training with standard-deviation normalization raised the refusal rate on answerable worlds to 61–79% within 50 steps. The right side of Figure 4 shows stage-2 rewards and advantage estimation; it is used here only to mark that point, not to retell the five corpus worlds or to walk the Hearst retrieval path again.

Paper Figure 4: five corpus transformations and the search budget on the left, the stage-1 warm-start in the middle, and stage-2 rewards and advantage estimation on the right.
Paper Figure 4: the right side shows stage-2 rewards and advantage estimation. The walkthrough above only computes how dividing by the group standard deviation cancels the scale of λ when a group has only refuse and a wrong answer; it does not retell the five corpus worlds.
SourceThe article and paper: the advantage estimate only mean-centers and does not divide by the group standard deviation; when a group has only refuse (0) and a wrong answer (−λ), dividing by the group standard deviation fully cancels the scale of λ. The default λ is 0.1. At λ=0.1, training with standard-deviation normalization raised the refusal rate on answerable worlds to 61–79% within 50 steps. The right side of Figure 4 shows stage-2 rewards and advantage estimation.

Warm-start without stage 2 leaves deference at 47.2% and reach at 21.0%. After stage 2, deference falls to 28.9% and reach rises to 25.8%. Setting λ to 0, deference is still 29.3%, but refusal when the primary evidence is deleted falls to 20.6%, and QA answering precision falls to 42.3%, so the warm-start plants the evidence-gathering path in the policy, stage 2 keeps pushing deference down, and λ is what preserves refusal and answering precision.

MethodEight-echo deferencePrimary-evidence reachRefusal when primary evidence is deleted
Search-R169.1%13.3%0.2%
Exact-match RL67.7%15.9%1.4%
Warm-start only47.2%21.0%33.1%
λ-reward RL only69.4%9.4%32.5%
EchoSpec28.9%25.8%37.6%

On Qwen2.5-7B, EchoSpec's eight-echo accuracy is 16.7%, and standalone QA answering precision is 47.1%.

Switch to Qwen2.5-3B, and deference falls from 62.5% to 30.4%. Switch to Llama-3.1-8B, and it falls from 70.0% to 30.6%, with reach at 26.5% and answering precision at 55.4%, so this training points in the same direction on different backbones.

The frozen policy transfers zero-shot to unseen BGE and BM25 retrievers, with deference of 29.4% and 25.1%.

Higher reach goes with lower deference; EchoSpec keeps search depth as echoes grow
Higher reach goes with lower deference; EchoSpec keeps search depth as echoes grow

Figure 5. Left: higher evidence reach tracks lower eight-echo deference. Right: baselines search less as echoes grow, while EchoSpec keeps searching deeper. Source: paper Figure 5.

Reach and deference move in opposite directions. The share of questions with at least three searches rises from 49% to 61%. After turning off "mask passages already seen", reach falls 3.3 points and deference barely changes.

This comparison can show the following. Without stage 1, eight-echo reach stays at 9.4% and deference stays at 69.4%. Without stage 2, deference stays at 47.2%. λ is what preserves refusal and answering precision. What it cannot show is how much the evidence threshold α and the acquisition bonus each still add once a 32B warm-start is already in place. The single-seed results for those two terms on eight false echoes fall inside seed variation. When the 7B model synthesizes its own trajectories, the bonus lowers deference from 56.0% to 51.5%, but the 7B synthesis keeps only 2,537 trajectories, against 3,582 from the 32B model, so the trajectory count also differs.

What this means for people who build search agents

First check whether the search policy stops at the first passage that writes an answer. Filtering only the documents already retrieved will not bring back primary evidence that has not yet entered view.

If you train a search agent with exact match on clean QA data, do not look only at clean accuracy. Add a check: inject several rewrites of the same false claim into the corpus, and see whether already-correct answers flip; then delete the primary evidence, and see whether refusal collapses.

When writing queries, prefer the bridging entities in the question; do not take a ready-made answer string from the top-ranked passage and search it again. Independently, the method does not divide by the group standard deviation, so that the reward scale is preserved. The proof that λ is fully cancelled applies to the case where a group has only two payoff levels, refuse and a wrong answer.

Limitations

The tests use dense retrieval and a BM25 index over 21 million Wikipedia passages. Live web search also has dynamic rankings, pages altered to climb the ranks, and tampered sources. Building these training worlds requires multi-hop questions with known primary evidence.

How did this read for you?

One tap, no sign-in; tap again to undo. Only anonymous counts are kept, never who you are.

Discussion

Add experiment details, reproduction notes or a different take. Requires a GitHub sign-in.

Loading the discussion…