System One models Are Not Better Rerankers

Any team thinking of using the new "System One" models, also called decision models, as rerankers should read this first. The conclusion is blunt: if you need a reranker, fine-tune a reranker. On the same Qwen backbone from 0.6B to 9B, plain reranker fine-tuning beats decision fine-tuning by 5 to 29 points of nDCG@10 (a top-10 ranking score out of 100). At 27B the two tie (59.6 vs. 60.1). The advertised calibration is fragile too: a slightly different question wording can make the calibration error two or three times larger.

What problem does it solve?

A decision model does not write text. It returns a probability. Ask "Does this document answer the query?" and it returns, say, 0.91 for "yes". The name borrows Kahneman's "System 1", the fast and intuitive mode of thinking. Weeks after the first such model, Jev, was released in September 2026, people reported it as a good reranker and plugged it into retrieval systems.

Reranking works simply. BM25, a classic keyword retriever, returns the top 100 candidates. The model is asked once per candidate whether it answers the query, and the candidates are sorted by the probability of "yes".

Earlier reports compared Jev with rerankers built on other, usually smaller models. That cannot separate the effect of decision fine-tuning from the effect of the underlying model. The authors therefore compare three variants of the same backbone.

All three return a probability of "yes", so each can be used for ranking and read as an absolute probability.

Core design: two input modes, two questions

In isolated mode, the model sees the query and one candidate, like a classic cross-encoder. In joint mode, it sees the query and all 100 candidates and is then asked about each one, so a candidate's probability can depend on the others.

The first research question is about ranking: do decision models rank better than the untuned model or a reranker fine-tune? The second is about calibration: are their probabilities more trustworthy, and does that help?

Calibration needs its own test because ranking only uses the order of scores. Scores of 0.9, 0.5 and 0.1 rank three candidates exactly like 0.99, 0.98 and 0.97, but the second set is useless as probabilities. Calibration matters only when you read the absolute value, for example to pass only passages above 0.5 to a language model.

How it is measured

Ranking uses nDCG@10 on TREC DL 2019 to 2023 and five BEIR collections, ten in total. TREC-COVID, Touché and DBPedia appear in no fine-tune's training data, so they form a separate zero-shot average.

Calibration uses expected calibration error (ECE). Predictions are split into 15 equal-width bins, and in each bin the mean predicted probability is compared with the observed share of relevant passages, weighted by bin size. If 100 predictions average 0.9 but only 70 are relevant, that bin is off by 0.2. If another 100 average 0.1 and 10 are relevant, it is off by 0, so the ECE is 0.1.

Two practical uses are tested. The fixed cutoff keeps every candidate with a probability of at least 0.5 and reports the F1 score (balancing precision and recall) of that set. The count test sums a query's probabilities over its judged passages as an estimate of how many of them are relevant. Probabilities of 0.9, 0.6, 0.4 and 0.1 sum to 2.0, so if three are relevant, the error is one passage. Always guessing the median count is off by 8.6 passages.

Result one: decision fine-tuning does not make a better reranker

From 0.6B to 9B, the reranker fine-tune wins on every backbone. Its margin over the better decision mode ranges from 5.2 points at 9B to 28.8 points at 0.6B. The gap is significant on 48 of 60 reranker–collection pairs, and the decision fine-tune never wins significantly, so this is not a fluke of a few datasets.

The margin is not just MS MARCO (a passage-ranking training set) leaking into the test, because it stays at 5.4 to 23.9 points on the zero-shot collections. It is not a secret recipe either: the authors' plain 9B reranker trained only on MS MARCO averages 59.4, matching Prism-9B at 59.0.

Up to 9B, decision fine-tuning moves the untuned model by only −2.5 to +0.7 points. That means asking the released Qwen3.5-4B the same question gives the same ranking, with no training at all.

27B is the exception. Kev-27B averages 60.1 and the authors' reranker from the same checkpoint reaches 59.6, which is a tie. Clef, another 27B decision model, scores 54.8, below its untuned model at 56.4.

Joint mode helps only the largest models. Kev-27B gains about one point, while Kev-0.6B loses 12.2 and Kev-8B loses 13.0. The likely reason is length: 100 candidates take about 13,000 tokens on TREC DL and over 40,000 on BEIR, while the small Kevs were trained on inputs of at most about 7,552 tokens.

Result two: the calibration advantage depends on one sentence

With the authors' question, decision models from 4B up are better calibrated than published rerankers, for example Kev-8B at 0.113 vs. Qwen3-Reranker-8B at 0.257. Jev (0.075) and Kev-27B (0.076) are the best overall.

With a looser earlier wording, which asks whether the document is relevant, ranking barely changes but calibration worsens, often by two or three times; Jev joint goes from 0.075 to 0.224. Decision models then at best match the best reranker of their backbone. "Calibrated out of the box" is always relative to a definition of relevance that the user must write into the question.

Calibration also does not make up for worse ranking. Below 9B, rerankers keep the better set at a 0.5 cutoff, for example an F1 of 51.0 for Prism-4B vs. 42.6 for Kev-4B. Only at 27B, where rankings are equal, does Kev-27B's calibration pay off (57.2 vs. 48.3).

Limitations as I see them

1. Released models differ in training data as well as objective, and Jev's training method is unpublished, so a better decision fine-tune could change the picture.
2. The controlled comparison rests on two reranker fine-tunes trained once without tuning, calibration is measured only on TREC DL, and only two wordings were tried.
3. All candidates come from BM25. Dense-retrieval candidates are harder to separate, and decision models are only about a month old.

How a retrieval team could try it

1. Rerank your own top 100 with the untuned backbone, a decision model and a reranker of the same size, and compare nDCG@10.
2. If you need thresholds or counts, measure ECE on fully judged queries with at least two wordings.
3. For usable probabilities, add a yes/no classification loss to reranker training; the authors' 9B reranker matches Clef-Flash's calibration that way (0.111 vs. 0.112).
4. Try joint mode only on large models, and keep the input within the trained length.

Verdict