Every Clause Counts: Instruction-Following Reranking by Multiplying Small, Answerable Questions

A search reranker can respect the requirements users attach to their queries without training any parameters: written for beginners, in a certain language, official sources only. DMReranker splits an instructed query into a topic and a few concrete conditions, lets a small model answer one narrow question per condition, and multiplies the probabilities that each condition is met into one score. Most rerankers were trained only to judge topical relevance, so they treat these requirements as a few extra words. The InfoSearch benchmark measures this with WISE (an instruction-sensitivity score): each query is ranked three times, with no requirement, with the requirement and with the opposite requirement, and a reranker earns points when documents that meet the requirement move up under the requirement and down under the opposite one. Moves in the right direction add points, moves in the wrong direction subtract them, and changes near the top count more, so the score can be negative. GPT-4o used as a reranker averages only 33.5 WISE and the best open reranker 14.5, while DMReranker reaches 50.9. If any requirement is clearly unmet, the total score is pushed far down, and the document drops out of the top.

Search ranking teams should read this paper, because it offers a cheap, decomposable method. The core idea in one sentence: one model splits the requirement into conditions, a small model answers "is this met?" for each candidate and each condition, and plain code multiplies the probabilities. On the same small model, asking split questions instead of one combined question raises WISE from 32.9 to 47.6. With requirements, its nDCG@10 is 78.3, above GPT-4o's 74.2, which shows that the multiplied condition factors do not hurt relevance.

What problem is it solving?

Take the example in Figure 1 of the paper. A user asks "How do I tune indexes in FooDB?" and adds "official docs only, for non-experts." There are two candidates. A is the chapter on indexes for new users in the official FooDB manual. B is a forum thread about B-tree internals and index tuning.

On topical relevance alone, B is slightly ahead and ranks first. But the user has already said no forum threads and nothing written for experts. DMReranker multiplies the three probabilities. B gets only 0.04 on "official docs," so its product is 0.011, and A moves to the top.

InfoSearch is the benchmark used for testing. It has six condition types: audience, keyword, format, language, length and source. Each query comes in three versions: the original with no requirement, an instructed version, and a reversed version that asks for the opposite (for example "forum posts only"). Evaluation uses the same human labels to check whether the ranking moves in the requested direction. These labels are used only for scoring; the reranker never sees them while ranking.

The pipeline: parse once, profile once, then ask per clause

The pipeline is below; only the final combine step does not call a model:

Candidates are the top 100 documents retrieved by e5-mistral, the same pool the published InfoSearch baselines rerank. So the gains in the tables come from how the reranking is done, not from retrieval.

Step one: split the requirement into conditions

The parser reads the query and outputs four things. The first is a topic sentence, the information need with every requirement removed; in the example, "how to tune indexes in FooDB". The second is attribute rules, which restrict a document attribute to allowed values, such as "source must be official docs". The third is other rules, each a yes/no question about the document, such as "Does the document mention FooDB 3?", together with whether "yes" or "no" keeps it. The fourth is date and version rules, such as "published after 2023".

Why the self-check? The same model looks at the query and the parse again and votes to keep or drop each rule. It may drop rules and rewrite the topic, but it cannot add rules. The authors' reason is that an invented rule can push the only relevant document out of the top ten, while a missed rule at worst leaves the ranking as good as a plain relevance reranker. Inventing is far more costly than missing.

Step two: each candidate answers a few narrow questions

The questions are answered by Clef-flash, a 9B judge model. It takes no free-form prompt, only the document and a set of typed questions, and returns a probability for each option. There are three question types: yes/no, choice among named options, and a score on an ordered scale.

The first kind of question is the document profile, which fills in six attributes for each document ahead of time: source, format, audience, length, language and status (whether it is outdated). Each attribute has fixed options; source, for example, has official docs, blog, forum Q&A, news, academic and other. The model returns a probability distribution for each attribute, and separately returns a confidence value saying how sure it is of that answer. These questions do not involve the query, so each document is profiled once and the result is shared by every query. If an attribute that a rule needs has confidence below 0.3, the 27B Clef model answers again, and only those low-confidence attributes are replaced.

The second kind is relevance. The model rates on four levels, 0 to 3, how fully the document answers the topic sentence, and the probability of level 2 or higher is the relevance score. The question uses the topic sentence with requirements removed, not the raw query. If "for non-experts" also appeared in the relevance question, the model might fold the condition into relevance and count it twice.

The third kind covers other rules plus date and version rules: one yes/no question each, taking the probability of the answer that keeps the document. "Most recent first" rules work differently. The judge sees two candidates, each cut to its first 80 words, and is asked which is more recent; each pair is asked in both orders and the answers are averaged. This gives, for every other candidate e, a probability that e is more recent than d. Multiplying it by e's relevance score gives the probability that e is both useful and more recent than d, and this value is subtracted from 1. The product of this value over all other candidates is d's recency factor: the probability that no other useful candidate is more recent. Hoot says▾

Hoot saysHow the recency factor is built: a candidate newer than d only pushes d down if it is also usefulClose ✕

Newer is not enough; it must also be useful: each rival's threat is P(newer than d) × its own relevance score, subtracted from 1; multiplying over all rivals gives the probability that no useful candidate is newer than d.

1. e1 is both newer and useful, so only 0.28 is kept; it alone does most of the pushing down.
2. e2 is just as new, but its relevance score is only 0.1, so 0.92 is kept and d is barely affected.
3. Why multiply by relevance: without it d keeps only 0.04, so an off-topic newer document could push a good document down.

Illustrative: only two rivals, both with a 0.8 probability of being newer than d, and relevance scores 0.9 and 0.1; all numbers are made up. Bottom right, for contrast: without the relevance weight, an irrelevant newer document would also push d far down. Source: paper §3, the "most recent" rule: pairwise who-is-newer questions asked in both orders and averaged, multiplied by the rival's relevance score, subtracted from 1, then multiplied across candidates; all numbers are illustrative.

Step three: multiply

The total score is:
$$s(d)=P_{\text{use}}(d)\cdot \prod _r{P}_{\text{keep}}(d,r)\cdot m(d)\cdot \prod _a{P}_{\text{allowed}}(d,a)$$
$P_{\text{use}}(d)$ is the relevance score of document $d$. $P_{\text{keep}}(d,r)$ is the probability that it passes yes/no rule $r$. $m(d)$ is the product of the date, version and "most recent" factors; most queries have none, so it is 1. $P_{\text{allowed}}(d,a)$ is the profile's probability mass on the options that rule $a$ allows.

The intuition: treat the requirements as independent conditions that must all hold, and the score is the probability that the document is useful and meets every requirement. There are no weights, thresholds or learned parameters. A clearly failed requirement gives a factor near 0 and the document drops out of the top. When everything is met, the score stays close to the relevance score.

A mental-math example (illustrative): the requirement is "English only" and the profile gives a document English 0.6, Chinese 0.3, other 0.1. Only English is allowed, so this factor is 0.6. If the rule allowed English and Chinese, it would be 0.9. An uncertain attribute only lowers the score somewhat; it does not delete the document.

Every question, answer distribution and factor is logged. Because combining is just multiplication, a factor can be dropped afterwards and the ranking recomputed with no new model calls. The ablations below were computed this way.

Experiments: where it beats GPT-4o

InfoSearch has 600 core queries, and the public release has 1,597 instructed queries. All rerankers reorder the same top-100 candidates, and baseline numbers are cited from the InfoSearch paper rather than rerun. Besides WISE, the table reports SICR (strict compliance ratio, which requires the matching document's own score to rise and fall with the requirement).

Across the six condition types, DMReranker's WISE leads GPT-4o on four, all of them about what kind of document something is. Audience is 52.1 against 7.4 and length 37.9 against 10.2, and format and source also lead clearly. GPT-4o scoring only 7.4 on audience shows that rerankers barely react to such conditions, while the profile questions ask about exactly these attributes. GPT-4o leads on keyword and language, 63.0 against 58.3 and 53.1 against 46.7. Both can be checked by reading the text, and GPT-4o compares candidates together in windows of 20.

With no requirement, the parser adds no rules, so the system reduces to the relevance score. Its nDCG@10 is then 82.9, close to GPT-4o's 83.8, which shows that asking a four-level question about the topic sentence with requirements removed is by itself a solid relevance reranker.

Is the gain from splitting, or from a strong judge?

To answer this, the authors built a single-question baseline on the same Clef-flash: for each candidate, one yes/no question, "Is this document useful for this query including its instruction?", ranked by the probability of yes. In a paired comparison on a fixed random subset of 300 queries, splitting raises WISE from 32.9 to 47.6. The 95% confidence interval of this 14.7-point gain is 7.3 to 22.2, which excludes 0. This shows that, on the same judge, splitting itself brings a gain. Per type, the gain concentrates on audience and format, the other types are within noise, and on source the single question leads by 10.5 points. With only 50 queries per type, the authors treat the average difference as the main result.

The ablations point the same way. With relevance only, WISE falls to -4.0. Without attribute rules, audience, language and source drop to around or below 0, while keyword is unchanged. Without yes/no rules, keyword falls from 58.3 to 4.1. The reason is that the parser keeps "must contain this term" in the topic sentence and turns "must not contain this term" into a yes/no rule. Format and length use both kinds of rules, so removing either one costs points. Attribute rules cover what kind of document it is, yes/no rules cover what it says.

Why the other metric, SICR, is lower

SICR's standard is stricter than WISE: it requires the matching document's own score to go up when the requirement is added and down under the reversed one. Every DMReranker factor lies between 0 and 1, so one more factor can only make a score smaller. The matching document's own score therefore rarely rises; it ranks higher because the other documents are pushed down. On language, 159 of 200 queries have an SICR of 0.

The authors give two pieces of supporting evidence. On those 300 queries, the single-question baseline has an SICR of 37.0 and DMReranker 20.3. The single question asks about the whole requirement directly, so a matching document's own score can rise. Replacing the product with a mean raises SICR from 19.2 to 33.0 while WISE falls from 50.9 to 47.4. This mean variant was tried after seeing test results, and the paper uses it only as evidence about the metric. There is one more point: InfoSearch does not say how the output of list-wise rerankers such as GPT-4o was turned into scores. If scores come from ranks, SICR only checks whether ranks move in the requested direction. Computed from ranks, DMReranker's SICR is 67.0. So SICR is comparable only between systems that form scores the same way, and the 19.2 against 32.1 gap is not necessarily a gap in behavior.

Limitations as I see them

First, the attribute list was written with the benchmark in mind. The six attributes map onto InfoSearch's six condition types, and the language options are only English, Chinese and other. A new domain needs a new list, and the authors did not measure how much a list written without the benchmark would lose.

Second, development touched the test set. The authors debugged the pipeline on 60 test queries each from format and source, and lowered the confidence threshold from 0.5 to 0.3 to save cost. Without these 120 queries, format WISE is 43.8 and source is 67.3, close to the full results, which suggests the tuning did not noticeably inflate the scores.

Third, there are many calls. From an empty cache, one ranking needs 145 judge calls on average, or 58 without the source-type pairwise comparisons, while GPT-4o needs 9 list-wise windows. Most of the calls come from "most recent first" date rules, which compare all 100 candidates pairwise and account for 60% of all calls. That factor brought no gain; removing it raises source WISE to 67.6. Latency was not measured.

How a recommendation or search team could try it

1. Count the most common conditions users add in your logs and check whether they map onto a few document attributes, such as source, language or content format. If many conditions do not fit, the gains will be limited.
2. Profile the whole corpus once offline and store the probability distributions. This step does not depend on the query, can be reused, and is the easiest cost to estimate.
3. Start with attribute rules only. The paper multiplies the attribute factors onto its own relevance score, which is a probability computed on the topic sentence with requirements removed. When attaching them to an existing ranker, I would first turn its base score into the same probability form; the paper did not test this. Use a labeled set of instructed queries to check whether the ranking moves in the requested direction, and check that queries without conditions do not lose quality.
4. Hold off on pairwise date rules; in the paper they were expensive and brought no gain.

Verdict

In one sentence: when the conditions users add can be named ahead of time as document attributes, splitting them into small questions and multiplying is a training-free, explainable and very cheap way to rerank.