How BERT-based Cross-Encoders Model Relevance in Information Retrieval?

BERT-based cross-encoders are among the most accurate rerankers, yet how they judge relevance inside has never been clear. This paper traces how information flows in three models of different sizes across several in-domain and out-of-domain datasets. It finds the three models go through the same stages. Early and middle layers contextualize tokens while dedicated attention heads look for literal matches between query and document; deeper layers bring heads that find semantic matches; finally the signals are gathered toward the query tokens and [CLS] for scoring. The details differ, though: the three models assemble the relevance signal into [CLS] in different ways, and MiniLM may even be looking for evidence of non-relevance.

This is a solid "dissection report", not a new method. For ranking teams, its most useful parts are two conclusions you can check on your own models. First, the attention heads responsible for matching can be fooled by hard negatives picked by BM25; in the paper only MonoBERT-large clearly fails, and the authors attribute this to its training loss rather than its size. Second, MiniLM may search for "evidence of non-relevance" rather than "evidence of relevance". The second conclusion is still the paper's hypothesis, and its key control experiment uses only one query-document pair, so discount it accordingly.

What question it answers

The paper studies three models: MiniLM-L12-v2 (33M parameters, 12 layers), MonoBERT-base (12 layers, fine-tuned by the authors on MS MARCO) and MonoBERT-large (340M parameters, 24 layers) (§3.1). MonoBERT-base is a deliberate "control": it has MiniLM's depth and MonoBERT-large's pre-training, so the effects of size and pre-training can be separated.

In-domain data are the 226 human-judged queries of TREC DL 2019–2022; out-of-domain data are the BEIR sets closest to classic retrieval, including financial QA FiQA, literature search on the coronavirus pandemic, TREC-Covid, medical retrieval NFCorpus and BioASQ, and open-domain QA Natural Questions (§3.2).

The question fits in one sentence: when a query and a document are fed in together, from the first layer to the final [CLS] score, which components do the work, where does information flow, and what signals flow. To describe "direction", the paper splits the input into five blocks: [CLS], the query tokens, the first [SEP], the document tokens and the second [SEP]. The attention matrix thus splits into 5×5=25 cells, each a transfer direction, such as "document tokens take information from query tokens" (§5, Figure 5).

The tools, and what each one shows

Any single interpretability method can mislead, so the paper uses four tools that check each other; the table lists three, and the fourth, the Information Bottleneck, is described after it:

The attribution step makes one change that matters for retrieval: instead of a single query-document pair it uses triplets (query, relevant document, non-relevant document) and attributes the fact that "the relevant document scores higher than the non-relevant one" (§4.2). The relevant pair and the non-relevant pair each start from their own baseline, with the studied part replaced by padding, and are interpolated in step. The output being differentiated is the sigmoid of the difference between their two scores (Eq. 2). This finds the components that discriminate rather than generic scoring components. The fourth tool, the Information Bottleneck, gives every attention head in every layer a set of learnable switches. Each switch controls one direction of transfer between input parts ([CLS], query, document and the two [SEP] tokens, 25 directions in all) and multiplies the matching block of the attention matrix. The switches are learned once on the MS MARCO training set and shared by all inputs, with the original model frozen. The loss keeps the masked score close to the original score while penalizing open switches, and the switches are binarized by a threshold at the end (Appendix E). MiniLM keeps only about 9% of directions, MonoBERT-base about 15% and MonoBERT-large about 21%. With the masks applied, none of the three models drops significantly on any dataset, for example MiniLM goes from 0.74 to 0.71 on TREC DL 2019 (Table 2), so the learned masks cause no statistically significant ranking drop on the evaluated datasets.

Finding 1: a three-stage pipeline, same stages but different details

Stacking the results of the tools, the three models go through the same stages (§5.2, §10):

1. Early and middle layers: query and document each absorb context, while the two sides also exchange information. This refutes an earlier hypothesis that query and document ignore each other at first; in fact cross-side transfers and within-side transfers happen in the same layers.
2. Later layers: information flows mainly to the query tokens, which act like a "collection box".
3. Finally, the signal is assembled into [CLS], and here the three models differ (§7.2, §10). MiniLM first refines the signal on the query tokens and only then hands it to [CLS]; masking "all tokens send information to [CLS]" does not hurt it until the last few layers (Figure 9). MonoBERT-large collects raw signals into [CLS] and refines them there over several layers. MonoBERT-base sits in between, with [CLS] starting to accumulate relevance information from a fairly early layer. The paper conjectures that 12 layers are too few for [CLS] to do all the refinement on its own (§8.4).

A concrete case: on TREC DL, masking all transfers between and within query and document from the first layer to the last drops all three models to random-ranking level (§5.3). Out of domain, though, both MonoBERT models keep part of their performance, which means they learned paths that bypass explicit interaction; MiniLM did not. Attribution shows the same difference: on out-of-domain datasets, both MonoBERT models rely on important neurons and heads that differ widely from their in-domain ones, while MiniLM's barely change (§4.4); this is the paper's basis for calling MiniLM more stable.

Finding 2: dedicated matching heads, and how hard negatives fool them

The attention contrast finds a small set of "matching heads": when the document is relevant, their attention lands much more sharply on one token on the other side, and in both directions, with query-to-document and document-to-query attention rising together (§6.2, Figure 11). Checking whether the attended token is the same token splits matching heads into two kinds. In MiniLM, for example, two heads in the middle layers mostly look for "the same token", which is literal matching; a later head mostly attends to tokens different from the source token. The paper calls a match between different source and target tokens a semantic match, a definition based only on whether the tokens are identical (§6.2). In all three models literal-matching heads come before semantic-matching heads (Figure 13), because semantic matching can only happen once tokens have absorbed context and carry meaning.

These heads really matter. Among MiniLM's top 10 attention heads by attribution, 8 coincide with the "matching heads" found earlier by causal tracing, and 6 are singled out in this paper's contrast plots (Table 3). So an attribution (NIG) ranking plus the attention contrast can locate these heads more cheaply than causal tracing.

Hard negatives are the part retrieval people should watch most. Swapping random documents for BM25 hard negatives shrinks the differences of all heads; the most prominent matching heads of MiniLM and MonoBERT-base still stand out, but the two most prominent heads of MonoBERT-large (head 13 of layers 11 and 15) no longer do (Figure 12). In other words, hard negatives look literally like the query, and MonoBERT-large's matching heads cannot tell "the words line up" from "truly relevant". The paper attributes this to the training loss: MonoBERT-large was fine-tuned with a simpler binary cross-entropy loss, MiniLM with a listwise loss and MonoBERT-base with InfoNCE (§8.4), so this is not simply "bigger is worse". Separately, §7.2 takes the hidden states of [CLS], the query tokens and the document tokens at every layer and uses linear discriminant analysis (LDA) to test whether relevant and non-relevant documents can be separated linearly, and finds all three models separate them worse out of domain, and only in deeper layers when facing hard negatives. From this the paper conjectures that cross-encoders rely more on literal matching when they have not grasped a new domain's semantics, which may be why they fail out of domain.

Finding 3: MiniLM may look for "evidence of non-relevance"

The most surprising result is a control experiment (§5.3, Figure 10b). In every layer and every attention head, the authors set all attention weights to zero except [CLS] attending to itself, then set that one weight by hand from 0 to 1 and watch the score the model gives an unrelated query-document pair. The authors treat this weight as a proxy for how much [CLS] reads: in a normally running model the attention weights sum to 1, so the higher the [CLS] self-weight, the less is left for the query and document. In the experiment all other weights stay at zero, so [CLS] does not actually read more or less content; the experiment measures how the score changes when only this proxy value changes, and its conclusion should be discounted accordingly.

MiniLM's score rises almost monotonically with this weight, correlation 0.92; MonoBERT-large shows only 0.14, and MonoBERT-base 0.73 with an erratic curve. So raising only the weight [CLS] puts on itself makes MiniLM more likely to judge the pair relevant. From this the authors hypothesize that MiniLM's default answer is "relevant", and that it reads the query and document to find reasons for "not relevant". Dataset-level results echo this: with transfers within and between query and document masked, MiniLM falls below random both in and out of domain (§5.3), and a few MiniLM heads become more focused when the document is not relevant, again both in and out of domain (§6.2). A MiniLM trained with harder negatives (picked by ColBERT-v2) is more extreme and calls the pair relevant whatever the weight.

On this basis the paper's practical advice is: MiniLM-L12-v2 is slightly better than the two MonoBERT models (Table 5) and its internal mechanism is more stable out of domain, so it is the better deal (§10); MonoBERT, especially the large version, may be over-parameterized, learning several parallel routes for the same job (§5.2).

Limitations as I see them

1. Only BERT-style models are studied. The paper explicitly excludes decoder-based LLM rerankers (§3.1), while production increasingly uses LLM reranking; whether these findings transfer is unknown.
2. The key "non-relevance evidence" experiment uses a single pair. Figure 10b comes from tuning the weight on one query and one unrelated document, so the 0.92 correlation does not represent a whole dataset. The authors also admit an information leak in the masking setup: masking sets weights to zero after the softmax, so the hidden tokens still affect the size of the one remaining weight, and information leaks in that way. That is why, when [CLS] can only see itself, MiniLM still keeps about 80% of its performance (§5.3, §8.3).
3. The hard-negative and out-of-domain conclusions are qualitative. That matching heads are fooled by hard negatives and that models rely more on literal matching out of domain come from attention-contrast plots and the layer-wise separability analysis; the paper does not quantify the effect on ranking metrics. In a separate follow-up (Vast et al., 2026) the authors used these findings to remove superfluous attention directions and report much lower latency and better out-of-domain quality (§8.5), but those results are not in this paper and cannot be verified here.

How a search team could try it

1. If your production reranker is a BERT-style cross-encoder, reproduce the attention contrast first: take a batch of queries, pair each with a relevant document, a random document and a BM25 hard negative, find the heads with the largest attention differences, and see whether they "go blind" on hard negatives.
2. Build a separate evaluation slice of "literally similar but not relevant" documents (high-BM25 non-relevant ones), and compare nDCG@10 on this slice with the overall figure to judge how much the model relies on literal matching.
3. When choosing a model, compare MiniLM-L12-v2 and a larger BERT reranker on the same out-of-domain data; the paper's conclusion is that the small model need not lose.
4. If you use MiniLM, repeat the [CLS] self-attention control experiment on a few samples to see whether your model also leans toward "relevant by default"; if so, empty or noisy documents may get high scores. This is an inference from that experiment, not something the paper tested, and is worth monitoring separately.

Verdict