Let Agents Design How They Search: Automatic Interface Optimization for Agentic Search

When an LLM agent searches a document collection, the tools it can use, how the collection is pre-organized, and how the prompt teaches it to use those tools together make up its "search interface". Today's interfaces are hand-designed and generic across collections; DCI (direct corpus interaction), for example, simply gives the agent a terminal and lets it read raw documents with commands like grep and cat. SearchWeaver, from Korea University and KAIST, instead has a coding agent (Claude Code, running Claude Opus 5.5) design the interface. It reads the search agent's full execution logs and per-question scores, works out why searches failed, and revises three things at once: pre-built index files, executable search scripts, and usage instructions for the search agent. It then reruns the agent and checks the score, for 10 rounds. With GPT-5.4-nano as the search agent, accuracy on BrowseComp-Plus rises from DCI's 71.6% to 78.4%, and mean nDCG@10 over four BRIGHT subsets from 45.7 to 55.0; handing the optimized interfaces unchanged to Gemini 3.5 Flash-Lite also beats DCI. The conclusions largely hold, but part of the BRIGHT gain comes from a strong retrieval ranking that one baseline lacks, which the main text does not make obvious; it gets its own discussion below.

1. The problem: one interface does not fit every collection

Recent agentic search methods all work within a fixed interface. RAG hands the model only the passages a retriever returns; DCI lets the agent read raw documents with terminal commands; DR-DCI first pulls documents into a workspace via dense retrieval and then browses them; RISE (restricted interaction-space retrieval) fences off a restricted workspace and adds model-generated tables of contents and section anchors to documents. Which of these wins depends on the collection: DR-DCI beats DCI on BrowseComp-Plus (73.1% vs 71.6%) but loses on BRIGHT (42.1 vs 45.7); RISE scores 65.0% on BrowseComp-Plus but only 25.9 on BRIGHT. The authors conclude that interfaces should be designed for a specific collection and model, and that this design work can be handed to a coding agent.

The idea follows prompt optimization such as DSPy and harness optimization such as Meta-Harness. The difference is the target: prompt optimizers change only text, Meta-Harness changes the orchestration code around a model, and SearchWeaver changes how the collection the agent faces is organized and which operations it can call.

2. Method: a coding agent repeatedly revises the interface

2.1 What an interface consists of

A candidate interface has three parts. First, pre-built files: before any question arrives, the whole collection is processed once, for example to build a "title to file path" index reused by every question. Second, executable tools: Python scripts that can call the supplied BM25 and dense retrieval, restrict search to a subdirectory, or combine the agent's document judgments with a retrieval ranking. Third, usage instructions telling the search agent when to use which tool and how to check clues. The three must be designed together: a tool that "merges the agent's judgments into the ranking" needs instructions telling the agent when to read documents and how to pass its judgments to the tool.

A candidate may also replace the search agent's base system prompt and its context-management policy (which decides when earlier tool outputs get truncated or cleared). The search agent's model weights never change.

2.2 The optimization loop

Each round, the designer looks at the collection and all previous candidates and writes a new interface; the new interface is run on the optimization questions, returning per-question scores and full execution logs, including which commands were run, what tools returned, and the final answer. Gold document IDs for optimization questions are also given to the designer, helping it distinguish "never found" from "found but not recognized". Every candidate's results are kept for later reference whether or not it improved. After 10 rounds, the interface with the highest mean score on the optimization set is evaluated on the test questions.

Leakage safeguards are in the appendix: a candidate whose files contain optimization question text, question IDs or gold labels is rejected before evaluation, and test question and label files are made unreadable during optimization. The search agent gets at most 300 turns and 30 minutes per question.

3. Results

BrowseComp-Plus asks multi-clue questions over a fixed web corpus, hereafter BCP (multi-clue QA benchmark), judged by GPT-4.1, with 30 questions for optimization and 800 for testing. BRIGHT is reasoning-intensive document retrieval; four subsets are used (biology, earth science, economics, robotics), with 213 optimization and 210 test questions in total, scored by nDCG@10. Each collection gets its own optimized interface.

3.1 Main results

SearchWeaver leads on both benchmarks and all four subsets. With 800 BCP test questions, 6.8 points is about 54 questions, a reliable gap. Each BRIGHT subset has only about 50 test questions, so a few points on one subset deserve caution, but the roughly 13-point gains on biology and earth science are far beyond that noise.

3.2 What each collection's interface learned

The appendix shows the final interface for each BRIGHT subset. All four share a skeleton: the agent still searches and reads raw files with the shell, and a final tool merges the agent's judgments with a ready-made dense ranking into the submitted list. They differ in where retrieval enters and how much weight the agent's judgments get. Biology confines search to the question's topic directory; the agent labels documents it reads as encyclopedia, paper or web, and a merging tool ranks the submitted 20 from three candidate sources: documents labeled encyclopedia, the dense ranking inside the directory, and files similar to the first encyclopedia document. The appendix gives only these three sources and that paper and web documents count at half weight, not the merging formula or each source's share; in its example, the two encyclopedia documents rank 1 and 2, and places 3 to 5 go to two web pages and another chunk from the same source. Earth science searches like plain DCI and uses retrieval only for the final ranking. Economics starts from dense candidates, keeps the agent's order and fills remaining slots from the dense ranking. Robotics puts documents the agent grades as answering the question first.

None of these strategies was written in by hand, and they all differ, which is exactly the paper's point that interfaces should be tailored per collection. But they share one thing: all four BRIGHT interfaces end up relying on the supplied ReasonEmbed-Qwen3-8B ranking. The DCI baseline has no dense retrieval at all, the "agentic search (dense)" baseline uses the smaller ReasonEmbed-Qwen3-4B, and DR-DCI uses BGE-large. The paper does not report what that 8B ranking scores on its own without the agent, so the 9.3-point BRIGHT gain cannot be split from the main text into interface design versus stronger retrieval resources.

3.3 Is changing the prompt enough?

The next comparison partly answers the question from Section 3.2. Two variants: first, the same loop but only the instructions may change, with no generated files or tools and no retrieval resources; second, SearchWeaver's final interface with its generated files, tools and corresponding instructions removed, without re-optimization.

Changing only the prompt, with no retrieval resources at all, lifts the BRIGHT average from 45.7 to 52.0, about two thirds of the total gain. That means most of the benefit comes from "telling the agent how to search this collection", not from the extra ranking. Only the remaining 3 or so points need files, tools and retrieval resources, and they are concentrated in biology and earth science: removing files and tools costs those two subsets 9.7 and 11.5 points. Economics is slightly higher without them (38.8 vs 38.3), and prompt-only optimization scores best there (41.8). In other words, the toolset optimized for economics did not help; it was selected only because it had the highest mean on the optimization set, which shows that selecting over 10 rounds and a few dozen optimization questions brings in some noise.

3.4 Tool time

The paper stresses that SearchWeaver saves time over DCI: tool execution time per question (excluding model inference) falls from 42.4 s to 32.9 s on BCP and from 63.2 s to 36.6 s on BRIGHT. The appendix breaks it down by subset:

The saving comes mainly from biology, because it restricts search to one topic directory; the earth-science interface keeps DCI's whole-corpus search and saves little. Compared with other methods SearchWeaver is not fast: DR-DCI and RISE use a fraction of its tool time. It trades more tool time than those two for higher scores, so the authors' claim that it sits on the Pareto frontier among the evaluated methods is accurate, but "saves time" holds only relative to DCI. Also, only tool execution time is counted, not model inference time, which in multi-turn interaction may well be the larger share; the paper does not report it.

3.5 Does it transfer to another model?

Handing the interfaces optimized with GPT-5.4-nano unchanged to Gemini 3.5 Flash-Lite gives 73.9% BCP accuracy (DCI 60.9%) and 58.8 mean BRIGHT nDCG@10 (DCI 50.7), above DCI on all four subsets, with BRIGHT tool time per question falling from 23.1 s to 15.5 s. Working without re-optimization means a good part of what was optimized belongs to the collection itself, not to tricks for one model. Note, though, that with Gemini, RISE and DR-DCI reach 69.1% and 69.6% on BCP, both more than 8 points above DCI; gaps between methods shift a lot with the model, so a ranking under one model should not be taken as settled.

4. Limitations

The limitation the authors state is that experiments use fixed collections; when documents are added or changed, when the pre-built index and the tools that depend on it need updating, and whether that can be done incrementally, is not studied.

There are several more. First, optimization cost is not reported. Every round has Claude Opus 5.5 read full execution logs and then reruns the search agent over the optimization set; the call volume and cost over 10 rounds are the numbers anyone adopting the method most needs. Second, BCP optimization uses only 30 questions, and the paper does not analyze overfitting to them; the test gains show there is no severe overfitting, but that may not hold for other collections. Third, retrieval resources on BRIGHT are unequal, as Section 3.2 explains. Fourth, each collection was optimized only once, with no report of variation across seeds or designer models.

5. Practical takeaways

For teams building QA agents over an internal knowledge base, this gives a process to copy: prepare a small batch of questions with gold answers, and have a coding agent read the execution logs and revise the prompt, generate index files and write search scripts. The ablation says the cheapest first step is prompt-only optimization: BCP accuracy goes from 71.6% to 75.6%, BRIGHT gets two thirds of the gain, and no extra retrieval resources are needed. If the collection has a clear directory structure or decent dense retrieval, then let the designer build tools that restrict search to relevant directories and combine the agent's judgments with the retrieval ranking. If the collection changes often, you will need to design how the generated index is maintained.