SkillQR: Agentic Skill Discovery for Reason-Intensive Query Rewriting

In reasoning-intensive retrieval, user questions are often indirect, and feeding them straight to a dense retriever works poorly. The usual fix is to let an LLM rewrite the query first, typically with one instruction for every query, such as "generate relevant and detailed information". SkillQR argues that the right direction for a rewrite depends on the corpus being searched. The same question, "how to implement binary search", needs a very different rewrite when the target is theoretical computer science papers than when it is Python Q&A. SkillQR scans the corpus offline, distills a few rewriting "skills", and at query time picks a skill before rewriting, with no training data at all. On BRIGHT, with pseudo-relevance feedback removed to isolate the skills, average NDCG@10 rises from 32.1 with the generic instruction to 33.7, matching hand-written subject instructions. The 14B version spends 38.1 seconds per query on rewriting, versus 85.3 seconds for the iterative ThinkQE.

The paper is worth reading because it automates corpus-specific rewrite design, a job that used to require someone to read the corpus and hand-write prompts. The core idea in one sentence: mine a small set of rewriting skills from the corpus offline, then let the same LLM choose one at query time. In my view its value lies in being cheap and portable rather than in raw accuracy. It ties with RL-trained rewriters, holds up better on a medical corpus, and still sits far from the ceiling of always picking the right skill.

Why should rewrite strategy follow the corpus?

The authors start with a control on BRIGHT. The same 7B model rewrites and the same retriever searches; only the rewrite instruction changes. The generic instruction is used for every subject, while the human instruction gives coding and theorem subjects their own prompts. For the Pony language subject, for example, it asks for the relevant Pony syntax and language features with explanations.

On LeetCode the generic rewrite is worse than not rewriting at all, and only recovers once the prompt asks for the required data structures and algorithms. The same model retrieves much better once the writing direction is right. The catch is that hand-writing instructions requires knowing the corpus well and must be redone for every new corpus.

An illustrative scenario: an internal knowledge base holds code snippets, API docs and incident write-ups, and a user asks why a service dies five minutes after starting.

Pipeline: mine skills offline, pick one online

SkillQR splits the work into an offline and an online stage so the expensive corpus reading happens once. Offline, it runs over the corpus and outputs at most five skills. Online, the LLM picks one skill for each query and rewrites accordingly.

Offline: how are skills mined from the corpus?

Step one is to see what the corpus contains. The document embeddings already computed by the retriever are clustered, so no extra encoding is needed. The number of clusters is about the square root of the corpus size. For each cluster, the documents nearest the centroid go to a proposer LLM, which assigns a domain and a content type, such as "math / proofs".

Step two drafts skills. The proposer sees all cluster labels plus the generic instruction and is asked to expand it into the minimal skill set covering every cluster. Clusters in the same domain are merged, and only genuinely different knowledge types stay separate.

Step three has a judge LLM check each cluster. It samples documents from the cluster and decides whether any current skill could guide an LLM to write text resembling them. If so, the cluster is covered. If not, the judge names the missing knowledge type and says whether to broaden an existing skill or add a new one. A climate-dynamics cluster might be folded into an earth-science skill, while an interface-design cluster that matches nothing gets a new skill. The loop runs at most three rounds with at most five skills. The small cap keeps routing choices few, so routing errors reflect skill quality rather than an overloaded router.

Walking through the illustrative knowledge base: clustering yields "backend / error logs", "backend / config", and "frontend / interaction design". The proposer drafts two skills for symptoms and config keys. The judge marks the design cluster uncovered and suggests a new skill. The proposer adds one for interface elements and flows, every cluster is covered, and the loop stops.

This cost is paid once. All of BRIGHT is mined in about two hours on one H100, and the largest subject, LeetCode with 400k documents, takes 21 minutes.

Online: choose a skill, then rewrite

At query time a single LLM, 7B or 14B, does everything. It reads the query and a numbered list of skills, reasons about the genre and the knowledge needed, and selects one, preferring a domain skill over the generic one when it fits. The original query is used for a first retrieval, and the top five documents serve as pseudo-relevance feedback. They go into the rewrite prompt together with the chosen skill. The final query is the original query concatenated with the expansion, which keeps the user's wording and adds the vocabulary corpus documents actually use.

The paper's example is a physics question whose gold document defines simple harmonic motion. The generic rewrite reads like a worked solution and scores an NDCG@10 of 0. SkillQR picks a concept-explaining skill, so the rewrite lists and explains the concepts involved, closer to the definition entries in the corpus, and scores 43.1.

Results: no training, yet on par with trained rewriters

Besides BRIGHT, the authors test transfer on R2MED, a medical reasoning-retrieval benchmark. Baselines are ThinkQE (14B, three rounds of retrieval and rewriting), TongSearch (7B, RL on 30k examples), and Reasoner-Rewriter (7B, RL on 68k examples built from BRIGHT documents, plus human subject instructions).

Timings were measured with the Diver retriever. On BRIGHT, SkillQR is not significantly different from same-size rivals, yet it is faster. At 14B the saving comes from a single rewrite pass instead of three rounds. At 7B it comes from shorter outputs, because trained rewriters imitate long reasoning from their training data.

R2MED is the more interesting result. Both RL rewriters were trained with BRIGHT as the target, and on medical text Reasoner-Rewriter barely beats the original query. SkillQR 7B is significantly better than both on both retrievers. This means a strategy baked into weights breaks when the corpus changes, while re-mining skills for each new corpus adapts naturally.

Skill quality was also tested on its own. With pseudo-relevance feedback removed and the Diver retriever, mined skills average 33.7, the same as human instructions, against 32.1 for the generic one. Human instructions use the generic prompt for all StackExchange subjects, while SkillQR finds dedicated skills there, lifting robotics from 28.3 to 30.2.

The routing ceiling deserves attention. The authors rewrite every query with every skill and keep 722 queries where the choice matters. The LLM router reaches 42.9, while always picking the best skill would reach 53.4. That means the skills themselves are good enough and the bottleneck is selection.

Limitations

First, online latency is high. Rewriting takes 13.9 seconds per query at 7B and 38.1 seconds at 14B. That suits offline batches or deep research, not online search with millisecond budgets.

Second, it does not always beat the raw query. With Diver, SkillQR 7B scores 11.8 on LeetCode versus 13.4 for the original query. At 14B with Reason-Embed, its BRIGHT average of 38.5 still trails ThinkQE's 39.3.

Third, the routing gap is wide. With the LLM router at 42.9 against an oracle at 53.4, much of the potential depends on skill choice. Skills are mined from a static corpus, and refreshing them as the corpus evolves is left open.

How a search or recommendation team could try it

1. Check whether rewrite direction matters for your corpus. Rewrite a few hundred hard queries with a generic prompt and two or three hand-written domain prompts, and compare recall metrics. If the gap is small, skip this.
2. If the gap is clear, cluster existing document embeddings and run the proposer-judge loop to mine up to five skills, then review the skill text by hand.
3. Keep rewriting offline or nearline, for example by caching rewrites for frequent hard queries, or enable it only for agentic search and QA entry points that tolerate seconds of latency.
4. Monitor routing. Log which skill each query gets, sample skill-sensitive queries to compare skills, and add in-prompt examples if routing is clearly off.

Verdict

Rewrite strategy should come from the corpus, not be frozen into a model. SkillQR automates this with a light pipeline, ties with trained rewriters and travels better across corpora, but latency and routing are the two issues to solve before deployment.