ACARE: Agent-Context-Aware Reranker for Iterative Search

When a search agent works through a hard question, the sub-queries it sends are often just a few keywords. The goal and constraints of the original question are left out. A conventional reranker scores documents against that sub-query alone, so a document that matches the keywords but is useless for the actual task can rank near the top. ACARe, from NVIDIA and the University of Waterloo, gives the reranker the original question and the agent's current reasoning as well, so it judges whether a document helps the whole task. The most telling result is a control: without changing a single weight of Qwen3-Reranker-4B, simply adding the original question to its input lifts a small agent's answer accuracy from 48.2% to 54.8%. Fine-tuning on agent trajectories to get ACARe brings it to 57.6%. Across five agents, ACARe ranks better than the base reranker and needs fewer searches.

Teams building agentic retrieval or deep research systems should read this for a practical reason. The change lives entirely in the reranker input. Retrieval and the corpus stay untouched, and it works on top of third-party search APIs. The core idea in one sentence: the sub-query says what to find at this step, while the original question and reasoning say why, and the reranker reads both before judging relevance. In my view the most useful finding is that control. Most of the gain comes from feeding context to the reranker at all; the extra 1.7 points from dedicated training are not significant after correction.

What problem does it solve?

An illustrative example. A user asks which director's debut film won an award at the 1998 Berlin festival and later made a film set in space. The agent first wants the 1998 Berlin-winning debuts, so it sends the sub-query "1998 Berlin award film".

A sub-query-only reranker places the review and the travel guide above the interview, because "debut" and "space" are not in the sub-query. The agent then reads useless documents and has to search again with new keywords, which raises both search count and API cost.

Methods such as AgentIR put the reasoning into the first-stage retriever, but that requires indexing the whole corpus locally. ACARe sits in the second stage and only scores returned candidates. It therefore works with BM25, dense retrieval, and search APIs that return document text.

Pipeline: one reranker, four input modes

ACARe is fine-tuned from Qwen3-Reranker-4B and still scores documents one at a time. The input and one candidate are concatenated, and the score is the logit difference between the next tokens "yes" and "no". What changes is the input.

The reasoning is the text the agent wrote just before issuing this search, from the current turn only. Both fields are produced by the agent anyway, so no extra model call is needed. The only cost is a longer reranker input.

Why drop context fields at random during training?

Different agents expose different context. Open-weight models show their reasoning, while GPT, Gemini and Claude APIs usually do not. Training one model per case would be expensive to maintain. The authors drop the reasoning and the original question independently with probability 0.5 during training. When both fields exist, each of the four modes appears about a quarter of the time. A dropped field is removed from the input and the prompt switches to the matching template, so one checkpoint serves every mode.

The training objective is localized contrastive estimation (LCE). Each group has one positive and several retrieved negatives, and the loss is softmax cross-entropy over the group's scores with temperature 0.5. The intuition is to teach the model to pick the positive out of a group of hard negatives. A hand-worked illustration: in a group of 8, if the positive scores 2 and seven negatives score 0, the positive's probability is about 0.89 at temperature 0.5. If one negative also scores 2, it drops to about 0.45, and the loss pushes the two apart.

Where does the training data come from?

Labels must be per sub-query, not just per original question. The authors extend DR-Synth, the synthetic data released with AgentIR. Starting from questions with gold answers and supporting documents, an agent produces search trajectories. For each sub-query, a stronger labeling model ranks the retrieved candidates while seeing the original question, gold answer and supporting documents. The top document becomes the positive and the bottom seven become negatives. Together with the sub-query, its reasoning and the original question, they form one training instance. A separate judge removes trajectories whose final answer is wrong.

The authors then re-retrieve candidates for the existing sub-queries with Nemotron-3-Embed-8B and relabel them. They also run a new rollout in which the agent searches with this retriever and is shown the labeler's top documents. The pool grows from 5,238 instances in DR-Synth to 16,601. Re-mining exposes each sub-query to negatives from a different retriever, reducing dependence on any one retriever.

Results: better rankings, fewer searches

First, does fine-tuning help without any context? In one-shot retrieval, each original question is searched once and ACARe runs in M0, with exactly the base model's input. Across BrowseComp-Plus (BCP), BRIGHT and TRQA, with four first-stage retrievers, ACARe has the highest nDCG@10 everywhere. It beats the base reranker by 1.7 to 4.5 points. Training on agent sub-queries alone already makes the reranker better suited to keyword-style queries.

Second, how does it do in real multi-step search? The authors plug five agents into NVIDIA's Retrieval Bench; open-weight agents use M3 and proprietary ones use M2. Each search retrieves 100 candidates, reranks them and returns ten unseen documents. At the end the agent submits a final ranking of ten documents, and nDCG@10 is computed on that list.

ACARe beats the base reranker by 0.9 to 2.0 points for every agent, with fewer searches. Fewer searches follow from more useful top-ten results, so the agent rephrases less often. Both rerankers cut API cost relative to no reranking. ACARe's extra 12% saving over the base reranker is significant only for Nemotron-3 Ultra.

TRQA shows something worth noting. Its retriever, AgentIR-4B, already uses reasoning during retrieval. Adding the context-blind base reranker on top lowers nDCG@10 for four of five agents, whereas ACARe improves all five. A context-blind reranker can undo the order a context-aware retriever has produced.

Third, does the gain come from context or from training? The authors fix a small agent, Nemotron-3.5 Lightning, in the Piika harness and compare on 200 BCP questions.

The unchanged base model goes from 48.2% to 54.8% just by seeing the question, and to 55.9% with reasoning too. ACARe adds 1.7 more points on the same input, but that last step is not significant after correction. Most of the benefit comes from putting the question into the reranker input; training is a bonus.

Finally, small agents gain more. On BCP, the three small agents Luna, Flash-Lite and Haiku with ACARe close 73%, 74% and 51% of the nDCG@10 gap to their larger counterparts without reranking.

With the base reranker the same gaps close by only 46%, 46% and 26%. This means a good retrieval tool is a much cheaper upgrade than a bigger agent.

When both sides use ACARe, the small agents cost 8%, 28% and 38% of their larger peers, at 4.6, 13.4 and 15.5 points lower nDCG@10. That means the gap does not vanish, but the price drops a lot.

Limitations

First, the cost analysis counts only agent API spend, not the reranker. The authors assume a locally served 4B reranker is negligible, yet ACARe reads the question and reasoning for every candidate, with 100 candidates per search by default. In end-to-end latency both rerankers are slower than no reranking, and ACARe is only 7% to 8% faster than the base.

Second, the evidence for the training increment is thin. The context control uses one small agent on one dataset, and ACARe's 1.7-point lead over the context-fed base is not significant. On BRIGHT, whose corpus appeared in Nemotron-3-Embed-8B's synthetic training data, ACARe gains only 0.2 points over the base.

Third, the answer-generation study is small. The ClimbMix run has only 57 questions with standard deviations of 2 to 5 points, and Lightning starts at 10.2% accuracy. Read the direction, not the exact numbers.

How a search or recommendation team could try it

1. Start with a zero-training version: append the user's original question, then the agent's current reasoning, to the existing reranker input. The paper gets most of its gain from this step alone.
2. Track three numbers: top-ten nDCG or human relevance rate, searches per task, and end-to-end answer accuracy. Fewer searches is the clearest signal that the agent is getting more useful documents.
3. If it works, consider fine-tuning on your own agent trajectories, labeling per sub-query and randomly dropping context fields so one model serves agents with and without visible reasoning.
4. Watch input length and latency. Reasoning can be long, so cap it and compare reranking 50 versus 100 candidates.

Verdict

A sub-query cut off from its question cannot say what it is really looking for. Handing the question and reasoning to the reranker is a cheap change with a clear payoff, and the trained ACARe adds a little more.