A Small Model Asks Better: Fine-Tuning Query Clarification for Retrieval-Augmented Generation

In a multi-turn RAG conversation, two kinds of question should get a clarifying question before an answer. One is a question with an unclear referent: the assistant has just described two credit cards and the user asks "what rewards does this card have?" The other is subtler: the question reads as clear, but the retrieved documents support several answers. Ask "is the language that descended from Common Brittonic still spoken today?" and the documents mention Welsh, Breton and Cornish. IBM Research built ClariEdge, a dataset of about 3,900 examples that deliberately includes hard negatives: questions that look ambiguous but whose referent the earlier turns already fix. They use it to LoRA-tune 3B-scale Granite models so that one small model both decides whether to ask and, when it does, writes a clarification that lists the candidates. Granite-4.1-3B LoRA reaches 96.6% decision accuracy; the best prompted model, GPT-4o, gets 79.4%, and GPT-5.6 Sol only 73.1%. Retrained without hard negatives, accuracy on the hard-negative class falls from 87.3% to 35.4%, which shows that learning when not to ask comes mostly from those examples.

1. The problem: when to ask, and what to say

Existing datasets each cover only part of this. CLAMBER evaluates isolated single questions with no dialogue history and no retrieved documents. AmbigQA counts a question as ambiguous whenever retrieval surfaces several factual answers, yet for a question like "who controlled the Italian government in 1922?" today's models simply answer "Mussolini and the Fascist Party" with no need to ask. ClariQ and Abg-CoQA have dialogue but do not use retrieval results as evidence. A real RAG assistant has to judge using the dialogue and the documents together.

The hard part is on the other side. Many questions look ambiguous but the earlier turns have already settled them. If the user said "I work at the Northeast branch" and then asks "when does this branch's service open?", "this branch" is unambiguous. A model that asks about these too interrupts the user on every turn, which is worse than an occasional wrong answer. So the core of the task is not "find ambiguous questions" but drawing an accurate line between real and apparent ambiguity. The data design, the ablation and the cross-domain test in the paper all revolve around that line.

2. How the dataset is built

2.1 Three classes

Conversations come from four domains of MTRAG, IBM's earlier multi-turn RAG benchmark: ClapNQ (Wikipedia), Cloud (technical docs), FiQA (finance) and Govt (government sites). Examples fall into three classes:

Ambiguous examples have two sources. First, annotators were given an entity type and its instances (say, "park" with several state parks) and wrote a question that deliberately names none of them, such as "what are the park's opening hours?"; a script then appended it as the last turn of an existing multi-turn conversation. About 75% were attached to conversations on the same topic and about 25% to unrelated ones, simulating a user who suddenly switches topic. The authors then checked each result by hand and dropped cases where the earlier turns happened to name the entity and so removed the ambiguity. Second, Mistral-Large-3 generated questions over 793 conversations, targeting an entity type with several instances in the dialogue; for 228 longer conversations (more than 6 turns, several entity types) the model first wrote out its reasoning step by step, then checked whether the question really could not be resolved from the context, rewriting it if not, over several rounds. Human annotators then judged every generated question from both routes: only truly ambiguous ones became ambiguous examples, and the rest were labeled resolvable.

2.2 Hard negatives come from the generator's mistakes

A good share of the questions from the second route are not actually ambiguous. One appendix example: the assistant has just explained bots, and the user asks "what are their capabilities?"; by coreference "their" can only mean bots. Mistral-Large-3 produced this as an ambiguous question, which shows that the generator itself cannot tell where the line is. The authors reviewed every item: genuinely ambiguous ones went to Ambiguous, ones resolvable from context went to CLEAR-hard. The review also surfaced recurring kinds of apparent ambiguity: comparison questions ("which one has the best battery life?" compares all candidates rather than pointing to one), ordinary follow-ups, and terse keyword queries like "Price?". In effect, CLEAR-hard collects the generator's misjudgments and uses them as a training signal.

Note that hard negatives are built only for the setting without retrieved documents. The post-retrieval setting has no counterpart, so cases like "the documents seem to offer several answers but they can be merged into one reply" are not covered.

2.3 Reference clarifications

Each ambiguous example has a reference clarification written in one of three styles by number of candidates: with 2–3 candidates, list each with a one-line answer; with 4–8, list just the names; with too many or no enumerable candidates, simply ask "which X do you mean?". Mistral-Large-3 drafted them and humans corrected them. Two authors independently labeled 80 test examples (20 hard negatives, 40 random, 20 ambiguous) for ambiguity, reaching a Cohen's Kappa of 0.875; all 4 disagreements were in CLEAR-hard, again showing that even people find this line hard.

3. Model and deployment

The authors LoRA-tune Granite-4.0-micro, Granite-4.1-3B and GPT-OSS-20B on 3,509 training examples, with rank 32 and α 64 on both attention and MLP layers, on a single H100, for up to 15 epochs with early stopping on the validation set. The input is the dialogue history plus optional retrieved documents; the output is a JSON object: CLEAR (meaning no clarification needed) if the question is clear, otherwise one clarifying question.

In deployment the same LoRA is used in two places: before retrieval it sees only the dialogue and decides whether to ask first; after retrieval it sees dialogue plus documents and decides whether to ask before generating. That adds at most two small-model calls per turn. A 3B model keeps latency and cost low, and the pre-retrieval check can also save a pointless retrieval.

The baselines are seven prompted models: GPT-5.6 Sol, GPT-4o, GPT-OSS-120B, GPT-OSS-20B, Granite-4.1-30B, Granite-4.1-3B and Granite-4.0-micro. All receive the identical system prompt, which spells out the three clarification styles and the rule to use only information in the context. Because Granite-4.1-3B exists in both a prompted and a LoRA version, the gap between them can be attributed to training rather than model size.

4. Results

4.1 Whether to ask

(%) The most telling part of this table is not the overall score but the fact that prompted models err in two opposite directions. GPT-5.6 Sol and GPT-OSS-20B over-ask: above 90% on Ambiguous but only 48.1% and 58.2% on hard negatives, and they misjudge nearly 30% of ordinary clear questions too, so roughly one in four normal questions gets a clarification. GPT-4o and the small Granite models under-ask: GPT-4o scores 92.4% on hard negatives but only 48.1% on Ambiguous. Two frontier closed models leaning opposite ways shows that this bias does not shrink with scale; the prompt simply does not convey where the line is.

The post-retrieval results make the point sharper. After retrieval GPT-4o gets only 32.6% on Ambiguous: with several answers plainly in the documents, it usually just answers. The two Granite LoRAs reach 100% and 99.0% there. But the post-retrieval subset has no hard negatives, only ambiguous and ordinary clear questions, so those perfect scores come from an easier distribution and should not be compared directly with pre-retrieval ones. Pre-retrieval is the hard part: Granite-4.1-3B LoRA gets only 84.5% on Ambiguous there, because without documents the model has only linguistic cues to separate real ambiguity from a resolvable reference.

4.2 Are the listed candidates right?

Clarification quality is measured on candidates: precision is the share of generated candidates that are correct, recall the share of reference candidates covered. The judging is done by Llama-3.3-70B in four steps, each with its own prompt: whether the clarification is open-ended; extracting the options it lists; whether each option is relevant to the ambiguous entity; and whether it is in the reference list. An option counts as correct if it is in the reference, is supported by the retrieved documents, or fits general knowledge and the user's intent. These metrics are computed only on examples the model correctly flagged as ambiguous.

(%) Prompted models match the LoRAs on precision; nearly the whole gap is recall. The candidates they list are mostly right, but they often leave out equally valid options, especially after retrieval, where GPT-4o and GPT-OSS-120B both have recall around 64%. Read this table with its first column: GPT-4o's F1 rests on only 50 examples; for the other 54 it never asked. A high F1 on few flagged examples is of limited value to the user.

The metric itself is lenient in two ways. First, a candidate counts as correct if it fits general knowledge, even if absent from the reference. In an appendix example the model lists Cumbric, which the reference lacks, and still gets full precision. Second, for an open-ended reference ("which park do you mean?"), recall is full as long as the generated clarification also has an open-ended part. The first is reasonable; the second favors the "list a few, then add 'or something else?'" style.

4.3 Removing hard negatives

To isolate their effect, the authors trained another Granite-4.0-micro LoRA on the same data with one difference: all CLEAR-hard examples removed.

Without hard negatives the model asks about almost every question that looks ambiguous, so Ambiguous accuracy actually rises to 97.1% and correctly flagged examples go from 88 to 101, while hard-negative accuracy falls to 35.4%. Ordinary clear questions barely move (98.5% vs 97.5%), so the model has not become trigger-happy overall; it has lost judgment specifically on questions with pronouns and several entities. Clarification F1 also drops by 4.6 points; the authors' explanation is that a model that has learned the line also handles candidates more precisely when it does decide to ask.

4.4 Unseen domains

The authors also took two new domains from MTRAG-UN, banking and telecom, about 230 examples each, and built hard negatives and ambiguous examples with the same process. Granite-4.0-micro LoRA scores 92.6% and 90.2% overall, essentially level with its in-domain 92.4%, which suggests it learned to judge the line rather than the wording habits of the four training domains; the best prompted model reaches only 71.9% (GPT-OSS-120B, telecom). Both failure directions persist in the new domains, and over-asking gets worse: on banking, GPT-5.6 Sol scores only 29.2% on hard negatives and Granite-4.1-30B only 27.8%.

On clarification quality in banking, GPT-4o (89.82%) and Granite-4.1-30B (86.61%) beat the LoRA (81.56%) on F1, but they correctly flag only 42/79 and 60/79 ambiguous examples against the LoRA's 69/79. That means their higher F1 is computed on fewer, possibly easier examples, so F1 alone overrates a model that rarely asks.

5. Limitations and how to read the numbers

The authors acknowledge two limits: the model asks one clarifying question and does not run multi-turn clarification, and the data and evaluation are English-only. Several more points are worth noting.

First, the test set is small. There are only 104 ambiguous and 79 hard-negative test examples, so one example is worth 1 to 1.3 points; gaps of a few points between models should not be over-read, though the 10-to-30-point gaps between LoRAs and prompted models are reliable at this size. Second, hard negatives cover only the pre-retrieval setting, so the 100% post-retrieval accuracy is measured on a distribution without them. Third, clarification quality is scored entirely by one LLM, and the paper does not report how well that judge agrees with humans. What the paper does validate is the references: two authors each wrote a clarification for 20 ambiguous examples, word-overlap F1 was 0.774, and manual review found the two versions agreed on the entity and candidates and differed only in wording. So the references are reliable; whether the judge is equally reliable is untested. Fourth, the prompted baselines use a single prompt with no few-shot examples or targeted tuning, so part of the over- and under-asking bias might be reduced by a better prompt.

6. Practical takeaways

When building a multi-turn RAG assistant, "should I ask?" can be handed to a 3B LoRA called once before and once after retrieval, instead of relying on a large model with a prompt. The main lesson is about data: the training set must contain counterexamples that look ambiguous but are settled by context, and the easiest source of them is to have a large model generate ambiguous questions and then pick out, by hand, the ones it got wrong. If post-retrieval false ambiguity is common in your setting, for example several documents that really say the same thing, you will need to add that kind of hard negative yourself. ClariEdge is public on Hugging Face (ibm-research/ClariEdge).