Contents · 8 sections
1What problem does it solve?2The big picture3Why not just plug two trained components together?4How the mechanisms work5Results6Limitations7How a retrieval team could try this8VerdictSHARE
Worth reading for teams working on tool retrieval, query rewriting or dense retrieval, because it nails down a common effect: a separately trained rewriter and a separately trained encoder do worse when simply plugged together. This paper from SAP (the German enterprise software company) trains an LLM rewriter and a retrieval encoder in turns, each against the other. The core idea in one sentence: the rewriter turns the user's words into a description in "tool manual" style, the encoder is trained only on those descriptions, and the rewriter is then DPO-trained on the encoder's retrieval scores, for three rounds. On a ToolBench subset of about 10,000 tools, compared with the strongest single-component baseline (encoder fine-tuning only), NDCG@5 rises by 2.5 points on average for standard queries, 6.3 points for colloquial vague queries, and 8.0 points for cross-domain vague queries. The price is one extra 4B-model generation per query, taking per-query latency from about 12 ms to about 546 ms.
What problem does it solve?
Before an LLM agent calls a tool, it must pick a few candidates out of tens of thousands of APIs; this step is tool retrieval. The usual approach encodes queries and tool descriptions as vectors and takes the nearest ones. The difficulty is that users speak casually while the catalog is written as technical documentation.
An illustrative example shows the gap. A tool in the catalog is described as "returns the 7-day weather forecast for a city name, including precipitation probability". A standard query might be "use a weather API to get the forecast for Paris for the coming week", close in wording to the description; a vague query is "Going to Paris this weekend, should I bring an umbrella?", which shares no keyword at all.
Both existing routes have weaknesses. The first is fine-tuning the encoder on query-tool pairs. It is accurate when wording is close and falls hard when wording shifts: in the paper, fine-tuned BGE scores NDCG@5 of 84.2% on standard G1 queries and only 44.7% on colloquial paraphrases of the same queries. The second is HyDE (have an LLM write a hypothetical target document, then retrieve with that document's vector). The LLM understands that "umbrella" means checking the weather, but what it writes does not know the catalog's wording, and it hurts on standard queries.
G1, G2 and G3 are ToolBench's three difficulty tiers: G1 involves tools from a single domain, G2 crosses domains within one category, and G3 crosses both domains and categories, getting harder each step.
The big picture

| Step | What is trained | Data | Purpose |
|---|---|---|---|
| S1a encoder warmup | BGE-large (about 335M parameters) | Real query and tool-description pairs, InfoNCE | Get the strongest encoder-only baseline first |
| S1b rewriter warmup | Qwen3.5-4B, LoRA | The catalog itself, 5 renderings per tool | Teach the rewriter the catalog's wording |
| S2 generate descriptions | Nothing | The rewriter writes a tool description for each training query | Produce training data for the next step |
| S3 retrain encoder | Encoder | Only "generated description and its tool" pairs, no real queries | Adapt the encoder to the rewriter's output |
| S4 align rewriter | Rewriter, DPO | Good/bad description pairs picked by encoder scores | Make the rewriter write descriptions the encoder retrieves well |
S2 to S4 form one round, and the paper runs three. At inference only the rewriter's description is used for retrieval; the original query is not encoded at all.
Why not just plug two trained components together?
This is the paper's starting point. Intuitively, a trained encoder plus HyDE rewriting should get the best of both. In practice, connecting the vanilla Qwen rewriter to the S1a-trained encoder drops standard G1 NDCG@5 from 84.2% to 73.4%, and all six evaluation cells get worse.
The reason is that the encoder's similarity is calibrated to the inputs it saw in training. In S1a the query side holds users' own words; at inference it gets a technical description of dozens of words, whose position in vector space has drifted from where training put it, so nearest-neighbor search goes astray. This shows the rewriter and encoder must be tuned to each other's outputs together rather than trained separately and assembled.
The paper also compares two rewriting styles. "Query rewriting" has the LLM make the user's words more specific, so the encoder still receives a sentence in the user's voice; HyDE writes a tool description, so the text the encoder receives is written the same way as catalog documents. On cross-category vague G3 queries, the former reaches only 6.2% NDCG@5, below unrewritten vanilla BGE (8.3%); HyDE reaches 17.4%. This shows that in tool retrieval, writing "text that looks like a tool manual" is far more useful than restating the user more clearly, which is why CoHyDE takes the HyDE route.
How the mechanisms work
The first step is the S1b rewriter warmup, which has the rewriter memorize the catalog. Why? The vanilla LLM's descriptions don't match the catalog's wording. The authors render each tool in 5 ways (title only, plus API name, plus tool description, title plus API name plus API description, full record) and train the rewriter on next-token prediction, about 50K samples for 8 epochs. Without this step, standard G3 NDCG@5 drops from 60.1% to 50.5%, while G1 drops only 5.5 points. Cross-category tools share the fewest words with users' phrasing and depend most on the rewriter having learned the catalog's language first.
Encoder retraining uses the same 5 renderings: each training pair randomly draws one rendering for the tool. The encoder then gives similar vectors to long and short versions of the same tool, which makes it steadier when rewriter outputs vary in length.
In each round, S3 retrains the encoder only on generated descriptions. Why not mix in real queries? The authors tried half real queries and half generated descriptions; it brought no gain and slightly hurt vague queries. Their explanation is that real queries pull the encoder back toward the S1a state, which is exactly the state that fails on vague queries. So this step deliberately shifts the encoder's input distribution away from raw queries.
In each round, S4 runs DPO on the rewriter using retrieval scores. Why? S2's descriptions are only guaranteed to look like a tool manual, not to retrieve the right tool. For each training query the authors sample 4 descriptions at temperature 0.7, retrieve with the freshly retrained encoder, score by NDCG@5, take the best as the chosen sample and the worst as the rejected one, and run standard DPO (β=0.1).
NDCG@5 can be computed by hand. Illustration: a query has 2 correct tools, and the retrieved list hits at ranks 1 and 3. The actual score is 1/log₂2 + 1/log₂4 = 1 + 0.5 = 1.5; the ideal case puts both correct tools at ranks 1 and 2, scoring 1 + 1/log₂3 ≈ 1.63; NDCG@5 ≈ 1.5 / 1.63 ≈ 0.92. If the 4 descriptions score 0.92, 0.39, 0.39 and 0, the 0.92 one is chosen and the 0 one rejected; queries whose 4 scores are all equal are dropped because they can't tell good from bad.
S4 depends on the sampled descriptions differing. Early on the authors also tried an optional step: before the loop, a short SFT of the rewriter on about 2,754 "query to description" samples that the vanilla rewriter generated itself and that were cleaned, so its output style would look more like the catalog. The final pipeline does not use this step, but it revealed a pattern: when that SFT went from 100 steps to 1,000 or 5,000, the 4 descriptions sampled in S4 became very similar, the good-bad gap shrank, and the closed-loop gain fell.
Results
Main results (NDCG@5, %, excerpt of the paper's Table 1):
| Method | Std G1 | Std G2 | Std G3 | Vague G1 | Vague G2 | Vague G3 |
|---|---|---|---|---|---|---|
| BM25 | 49.0 | 31.0 | 17.3 | 17.8 | 10.3 | 3.4 |
| text-embedding-3-large | 61.6 | 36.3 | 26.7 | 34.8 | 18.1 | 19.8 |
| HyDE (vanilla Qwen with vanilla BGE) | 58.2 | 34.9 | 35.1 | 37.4 | 17.7 | 17.4 |
| Encoder fine-tuning only (S1a) | 84.2 | 71.7 | 57.1 | 44.7 | 30.7 | 14.9 |
| Vanilla Qwen with S1a encoder | 73.4 | 59.5 | 46.2 | 41.2 | 29.3 | 17.6 |
| CoHyDE round 3 | 86.8 | 73.6 | 60.1 | 49.4 | 38.7 | 21.1 |
Relative to S1a, CoHyDE gains 2.6, 1.9 and 3.0 points on standard queries and 4.7, 8.0 and 6.2 points on vague queries. This shows joint training costs essentially no standard-query precision and earns its gains on vague queries. In addition, the three rounds of encoder retraining use no real queries at all, only continuing from the S1a encoder and training on generated descriptions, and standard-query precision does not fall as a result.
The ablations support this division of labor. Connecting the trained rewriter to vanilla BGE drops standard queries by 23.6 to 35.5 points, the worst collapse. Conversely, pairing the jointly trained encoder with the vanilla rewriter costs only 0.5 points on standard G1 and is even 2.0 points higher on standard G2; but standard G3 drops 6.4 points and each vague tier drops 5.3 to 5.9 points. This shows the encoder carries precision on easy queries, while the rewriter's value is concentrated on hard and vague ones.

How is the number of rounds chosen? Gains from round 1 to 2 are generally larger than from round 2 to 3. The authors ran a fourth round with another random seed: standard G3 rose from 57.5% to 62.5%, but all three vague tiers fell by 0.9 to 1.8 points. In other words, training further tilts the system toward standard queries, and round 3 is where vague queries peak.

Against prior methods: Shao et al. only iteratively retrains the encoder and still uses the raw query at inference; CoHyDE leads it by 14.3 to 18.5 points on the three standard tiers. RaFe trains a query rewriter with feedback from an external reranker and keeps the encoder frozen; it trails CoHyDE by 6.4 to 6.9 points on standard queries, and on vague G3 it is 13.6% versus 21.1%. This shows that training both sides, with the encoder's own retrieval score as feedback, beats training only one.
Limitations
First, the size of the gains is comparable to the evaluation set's statistical error. Per the appendix, the 95% confidence half-width of NDCG@5 is about ±2 points on G1, ±3 on G2, and ±5 to 6 on G3, because G3 has only 100 evaluation queries. The authors' own conclusion: round 3's gain over S1 on vague queries clearly exceeds this band only on G2, and stays inside it on G1 and G3. Rerun with another seed, standard G3 at round 3 is 57.5%, just 0.4 points above S1a's 57.1%.
Second, serving cost. Because every query first has a 4B model generate a description, it takes 546 ms on average on one H200, versus about 12 ms for the encoder alone, so latency is about 45 times higher. Training isn't cheap either: reproducing the main result takes about 50 GPU-hours. The paper also does not compare against cross-encoder reranking or sparse-plus-dense hybrid retrieval, two common online alternatives with latency far below 500 ms.
Third, the evaluation is on the lenient side. The 10,000-tool subset keeps every gold tool of the evaluation queries and samples the rest proportionally from 46,980 tools, so there are far fewer distractors than in the full catalog. Vague queries were paraphrased by Claude and self-checked by Claude, with humans spot-checking only 50. Also, in Table 2 the six Recall@5 values of the "vanilla rewriter with jointly trained encoder" row are identical to those of the "vanilla Qwen with S1a encoder" row in Table 1, while the two rows' NDCG@5 values differ, so be careful when citing those numbers.
How a retrieval team could try this
- First confirm the problem exists: build a set of colloquial queries from online logs or manual paraphrases, compare against well-formed queries, and see how far the current encoder's NDCG@5 or Recall@5 drops. If it drops only a few points, this method isn't worth it.
- Do the two cheapest warmups: train the encoder with several renderings of the same document, train the rewriter with next-token prediction on the document collection itself, then try HyDE retrieval once. This already shows whether the rewriter's descriptions match the collection's wording.
- Then run one loop: have the rewriter generate descriptions, retrain the encoder on descriptions only, then sample 4 descriptions per query and run DPO on the new encoder's retrieval scores. Watch vague queries above all; standard queries not dropping counts as a pass.
- Handle latency before launch: call the rewriter only on queries where the encoder's confidence is low, or cache descriptions offline for frequent queries. The bar is P90 latency within budget while the recall gain on vague queries is kept.
Verdict
The paper's most useful finding: the input distributions of a HyDE-style rewriter and an encoder must be tuned together, since training each well and then assembling them is worse; after joint training, the gains concentrate on queries whose wording is far from the documents' wording.
| Dimension | Assessment |
|---|---|
| Importance | Medium-high. Tool retrieval and colloquial queries are a common bottleneck in deploying agents |
| Novelty | Medium. HyDE, InfoNCE and DPO are all off the shelf; the new part is training both sides and closing the loop with encoder scores |
| Evidence | Medium. Thorough ablations, two seeds and confidence intervals; but only on a 10,000-tool subset, and part of the vague-query gain lies within the error band |
| Engineering cost | Medium-high. One extra 4B-model generation per query, about 546 ms latency; about 50 GPU-hours to reproduce |
| Worth trying? | Worth trying where colloquial queries are common and a few hundred milliseconds is acceptable; latency-sensitive online retrieval must solve generation cost first |
| Fits | Agent tool retrieval, enterprise API catalog search, and search where user wording and document wording differ a lot |
How did this read for you?
Hoot made a note. Thanks for the feedback.
One tap, no sign-in; tap again to undo. Only anonymous counts are kept, never who you are.
Discussion
Add experiment details, reproduction notes or a different take. Requires a GitHub sign-in.
Loading the discussion…