EMBERS: Empirically Motivated Robustness Testing of Deep Research Systems via Defect-Guided Fuzzing

Shuochuan Li, Zan Wang, Dong Wang, Yuhang Jiang, Huajin Cui, Chen Zhang, Gang Wang, Junjie Chen
31 minread4.30scoreNo. 611-05 AI Search issue
Contents · 8 sections1One question, reworded, and the answer goes wrong2First find the step where it goes wrong3How EMBERS works: rewrite questions the system already gets right4Results: in the same 24 hours, far more failures found5The traces from testing can also be used for training6Limitations as I see them7How a search team could try it8Verdict
Rel 5Nov 4Prac 4Evid 4Time 4

Deep Research systems split a question into subquestions, search the web repeatedly and then write an answer, so a small change in wording can push an early slip all the way through to a wrong answer. The authors analyzed 1,932 execution traces from three representative systems and found nine recurring kinds of defects, spread over subquestion writing, online search and final reasoning. From this they built EMBERS, a defect-guided fuzzing framework. It starts from questions a system already answers correctly, rewrites the question or an intermediate search input to target these defects, and checks that the rewrite still asks for the same thing. With the same 24-hour testing budget, it finds 3.21 times as many failing cases as the strongest baseline, DR-Arena. Fine-tuning on the execution traces produced during testing also makes the model's answers more consistent.

Teams building search agents or Deep Research products should read it. The most useful part is not "yet another testing framework" but the stage-by-stage defect table: nearly two thirds of the errors start at the subquestion-writing step, more than search and reasoning combined. Find the step where things go wrong first, then rewrite questions to hit that step's weakness, and you find far more failures. In this paper's comparison EMBERS finds 7.87 times as many failures as the paraphrasing baseline, because paraphrases mostly miss the places where a system is actually weak. One caution: all these failures are flagged by an automatic grader. In a human check of 384 rewrite results, the automatic grade was right 88.02% of the time.

One question, reworded, and the answer goes wrong

The paper observed this defect: while writing a subquestion, a system shortened "American Pie 4 story setting time" to "American Pie 4 story time", dropping the qualifier "setting", and could therefore find the release date instead of the in-story period. The table below is an illustration built on that defect; the original question, the rewrite and both outcomes are made up, to show what EMBERS tries to test.

Content (illustrative)
Original questionIn what year is the story of American Pie 4 set?
Subquestion the system writes"American Pie 4 story setting time" -> finds the in-story period, correct
Rewritten question"In American Pie 4, in what year do the things the characters go through happen?" (no longer says "story setting" directly; it still asks for the in-story year, and the answer is unchanged)
Subquestion the system writes"American Pie 4 year" -> finds the release date, wrong

Getting the original question right only shows that the system can get through with that one wording. With another wording that means the same thing, one early step can drift, and the later searches and reasoning follow the wrong direction further and further.

First find the step where it goes wrong

The authors collected 1,932 execution traces from three systems, DeepDiver, WebSailor and WebExplorer, on three benchmarks. They randomly sampled 384 of the wrong traces and, by hand, found in each one "the earliest error that was never corrected", assigning the failure to the stage where that error sits.

A Deep Research execution trace: the system writes subquestions, calls search and reads pages, then puts together the answer
A Deep Research execution trace: the system writes subquestions, calls search and reads pages, then puts together the answer
StageShare of failuresTypical defects
Writing subquestions65.6%ambiguous wording misread (18.8%), a qualifier dropped while splitting the question (18.5%), focusing on salient clues and missing the key one (17.4%), an entity given only by indirect description and not recognized (10.9%)
Online search24.2%page content unreadable (11.2%), irrelevant or colloquial words in the query (5.7%), pulled toward a popular related topic (4.7%), translation drift (2.6%)
Final reasoning10.2%all facts found, but the final counting, ordering or addition is wrong (10.2%)

The final-reasoning category is telling. Asked for "the sum of the year John Lennon was born and the year he died", a system found both years and then added up the digits of the years instead. The longer the research process, the easier it is for the system to forget what the question actually asked it to compute. The authors also note that these shares describe where failures came from in these systems, not how likely each stage is to fail.

How EMBERS works: rewrite questions the system already gets right

EMBERS turns the defect table into nine rewrite rules. The whole process is a loop: pick a question the system has answered correctly, rewrite it with one rule, run the system again, and see whether it still answers correctly.

EMBERS overview: seed questions from datasets go into a seed pool; each round picks a seed, picks a rule, and a rewriter that uses web search produces a new input; the system reruns, failures are recorded, and promising non-failing inputs go back into the seed pool
EMBERS overview: seed questions from datasets go into a seed pool; each round picks a seed, picks a rule, and a rewriter that uses web search produces a new input; the system reruns, failures are recorded, and promising non-failing inputs go back into the seed pool

Step one is picking seed questions. Each candidate question is run three independent times, and only questions answered correctly all three times are kept as "seeds". This first rules out questions the system already gets right only some of the time, so a later failure is more likely caused by the rewrite. Randomness cannot be ruled out completely, though. Each seed stores the question, the reference answer and the full execution trace of that run. The reference answer stays on the testing side and is used only for grading; on a rerun the system gets only the question, or the part of the trace before the rewrite point.

Step two is choosing where to rewrite, then how. There are two kinds of places: the original question, or a subquestion or search query in the middle of the trace (both called "subquestion" below). EMBERS first picks each kind with probability one half, then picks one applicable rule at random. The original question has five rules: wording that is easy to misread, an indirect description instead of a name, qualifiers made less explicit, true but irrelevant background added, and an extra counting or ordering step. The last rule keeps what the question finally asks for unchanged: for example, the system is first asked to put a few related events in time order, and still only answers what the original question asked (illustrative). A subquestion has four rules: mix in a popular related topic, mix in unrelated frequent words, make it colloquial, or use another translation or mixed Chinese-English. Picking the place first keeps the side with more rules from taking most of the testing budget. The authors wrote no rule for unreadable pages, since that is a property of the outside environment.

Step three is rewriting and verifying. A large model does the rewriting; the authors used DeepSeek-V3.2. It first finds which parts of the question can change and which conditions must stay, then chooses a direction. Before writing, it searches the web to verify that a new description really points to the same thing. For example, when an event name is replaced with an indirect description, a search first confirms that the description fits only that one event. After writing, it checks the rewrite against the original question once more to see whether it still asks the same thing, and records the rewrite with the evidence it found.

Step four is rerunning and grading. When the original question is rewritten, the system reruns from scratch. When a subquestion is rewritten, the system keeps all reasoning and search records before that point, the text of that one subquestion or search query in the trace is replaced with the rewritten version, and the system then generates the later searches and reasoning itself. Afterwards the same DeepSeek-V3.2 model, with a separate grading prompt, compares the system's final answer with the reference answer. This happens only during testing; the system under test never sees the reference answer while answering. A failing case is one where the original question was answered correctly and the rewrite is answered wrongly.

Step five is deciding which inputs stay for more rewriting. EMBERS groups inputs by "which original question they came from and which rule was used last". Within a group, an input is kept only if its trace is longer than any seen before in that group. Steps here count reasoning steps plus tool calls. The authors use trace length as a rough signal: in their earlier study, wrong traces averaged 19.86 steps and correct ones 15.19. A seed's chance of being picked keeps shrinking. Each seed's weight starts at 1: it scores 1 point for each generation of rewriting away from the original and 0.5 points each time it is picked, the weight falls exponentially with the total, and at 10 points it is down to 0.05. Each round draws seeds in proportion to their weights. This avoids circling on the same question and keeps rewrites from drifting too far from the original.

Hoot saysHow a seed's weight keeps fallingClose ✕
The more often a question is rewritten or picked, the less likely it is drawn again. At 10 points the weight drops from 1 to 0.05.
An exponential decay curve: weight 1 at 0 points, about 0.41 at 3 points, 0.05 at 10 points00.51036101210 ptsseed points (generations × 1 + times picked × 0.5)selection weightpaper settingnew seed: weight 1illustrative: 2 generations, picked 2 times10 points: down to 0.05
An exponential decay curve: weight 1 at 0 points, about 0.41 at 3 points, 0.05 at 10 points00.51036101210ptsselection weightpaper setting10.410.05
3.0 points → weight 0.41

The x-axis is a seed's total points: 1 per generation of rewriting away from the original, 0.5 each time it is picked. Illustrative: rewritten 2 generations and picked 2 times gives 3 points and a weight of about 0.41. This sends more of the testing budget to fresh questions and keeps rewrites from drifting far from the original.

Sourcepaper §4.3.4 seed probability update: P = P_init·e^{-α(w1·generation + w2·times picked)}, P_init = 1, P_finish = 0.05, max_times = 10, w1 = 1, w2 = 0.5

Results: in the same 24 hours, far more failures found

The experiments cover four setups crossing two frameworks (WebSailor, WebExplorer) with two models, times four datasets, for 16 settings. All three methods share the same seeds, the same grader and the same 24 hours.

MethodFailures found, 16 settings totalHumans: "still asks the same thing"Humans agree with the auto grade
RobustAlpacaEval (paraphrasing)1,06099.22%89.32%
DR-Arena (new questions built from the web)2,59528.13%58.59%
EMBERS8,34093.49%88.02%

EMBERS finds the most failures in 15 of the 16 settings. The count alone is not enough; the failures also have to be real. Only 28.13% of DR-Arena's questions still ask the original thing, so many of its "failures" are really changed questions. Paraphrasing barely changes meaning, but it also rarely makes the system fail. EMBERS sits between paraphrasing and DR-Arena: its changes are large enough to hit the system's weak spots, while more than nine in ten questions still ask the same thing.

Failures (orange), correct answers (blue) and failure rate (red line) for each of the nine rewrite rules
Failures (orange), correct answers (blue) and failure rate (red line) for each of the nine rewrite rules

By rule, the most effective are "make qualifiers less explicit" and "use an indirect description", with 65.95% and 55.14% of rewrites answered wrongly. Both target the subquestion-writing step, matching the defect table above. Among search rules, colloquial wording and alternative translations work better than mixing in unrelated words, which has the lowest failure rate, only 20.84%.

The ablation covers only 4 settings: the two frameworks paired with their own models, WebSailor with WebSailor-32B and WebExplorer with WebExplorer-8B, each on FRAMES (a multi-hop QA benchmark) and its Chinese version FRAMES-ZH. Removing any component lowers the failure count. Without web search during rewriting, failures drop 27.85% and 32.86% fewer test questions are produced. Replacing "pick the place, then the rule" with a random pick from all nine rules, about as many test questions are produced but failures drop 19.91%, so this step's job is sending the testing budget to places more likely to fail. Seed maintenance contributes least: failures drop 4.59%.

The traces from testing can also be used for training

The authors fine-tuned WebSailor-32B on execution traces EMBERS produced on OpenSeeker. First LoRA SFT on the correct traces, then a round of preference optimization with KTO, using correct and wrong traces as positive and negative examples. The evaluation sets, FRAMES and FRAMES-ZH, are not in the training data. Each question is run three times and counts as "consistently correct" only if all three are right.

Evaluation setConsistently correct beforeConsistently correct after
FRAMES48.54%54.49%
FRAMES-ZH60.43%69.57%

So the traces produced during testing are useful not only as bug reports but also as training data that makes the system steadier on the original questions. The wrong traces serve only as negatives in KTO. The authors themselves say they have not yet tested whether the fine-tuned model is also steadier on rewritten questions.

Limitations as I see them

First, the failure count depends on automatic grading. In the human check of 384 rewrite results, 88.02% of the grades were right, and the wrong grades were split roughly evenly between "correct" and "wrong". Rewriting and grading use the same model, so their biases may be correlated. So 8,340 is "suspected failures", and each one still needs human confirmation.

Second, rewrites still drift. Humans judged that 93.49% of the questions still ask the original thing and 88.80% keep every condition needed to pin down the answer. Among the problematic inputs the authors examined, 63.2% lost a clue needed to pin down the answer, 31.6% added a requirement that changed the reference answer too, and 5.1% did not follow the assigned rule.

Third, each rewritten question runs only once. Seeds must be right three times out of three, but a rewrite runs once. Execution is stochastic, so some "failures" may just be bad luck; the authors accepted this to test more questions within 24 hours.

How a search team could try it

  1. From production logs, pick questions the system answered correctly three times in a row, and keep the full execution traces.
  2. Sample one or two hundred wrong traces and mark by hand the earliest error as subquestion, search or reasoning, to see whether your system's distribution matches the paper's.
  3. Implement the two rules with the highest failure rates first: make qualifiers vague and replace entity names with indirect descriptions, using search before rewriting to verify the description points to only one thing.
  4. Check a sample of rewrites by hand to confirm they still ask the same thing, then see which kinds of questions the failures cluster on.
  5. Put failures that reproduce reliably into a regression test set, and only then consider them as training data.

Verdict

EMBERS's value is in first working out which step Deep Research systems usually get wrong, then building test questions aimed at those weak spots, instead of rewording questions blindly.

DimensionAssessment
ImportanceHigh. Deep Research chains are long, and one early slip can carry all the way through
NoveltyMedium-high. A stage-by-stage defect table turned directly into rewrite rules, with resumption from an intermediate step
EvidenceFairly strong. 16 settings, ablations and human checks, but failure counts depend on automatic grading
Engineering costMedium. Needs a system that saves traces and can resume from an intermediate step, plus a large rerun budget
Worth tryingYes. The first two rules and the defect-labeling method can be used directly for regression testing

How did this read for you?

One tap, no sign-in; tap again to undo. Only anonymous counts are kept, never who you are.

Discussion

Add experiment details, reproduction notes or a different take. Requires a GitHub sign-in.

Loading the discussion…