Contents · 10 sections
1What problem is it solving?2The big picture first3Mechanism one: treat the listwise prompt as a query that "understands intent better"4Mechanism two: two-stage training keeps retrieval and reranking ability together5Why it is fast: no generation, one forward pass is enough6Results: ahead on average, but look at where it comes from7The quality of the top 20 sets the ceiling8Limitations as I see them9How a retrieval team could try it10VerdictSHARE
Teams running search or RAG retrieval pipelines who find LLM reranking too slow should read this. The problem it addresses: LLM listwise reranking (putting the query and dozens of candidates into one prompt and having the model output the sorted IDs) gives the best quality, but it generates the ranking token by token, so a single query can take several to more than a dozen seconds; embedding retrieval is fast, but its ranking precision falls short. The core idea in one sentence: encode the whole listwise prompt, "query + top 20 candidates", into a new query vector, then rerank by cosine similarity against precomputed document vectors, with no generation at all. Key numbers: on 8 BEIR datasets E2Rank-8B averages 54.35 NDCG@10, on par with or slightly above the same-backbone generative reranker RankQwen3-8B (53.39); per-query reranking latency on TREC-COVID drops from 16.93 s to 3.40 s, about 5×; and the same model used for retrieval does not lose embedding quality on MTEB.
What problem is it solving?
Start with an illustrative scenario. A user searches "xiaomi price", and first-stage retrieval returns 100 documents, some about phones and some about millet, the grain ("xiaomi" means both in Chinese). Embedding retrieval looks only at the query, and the word fits both meanings, so the two kinds of documents are mixed in the ranking. If you first glance at the top documents and see that most are about phones, you can infer the user most likely means the phone brand, and rerank with that understanding.
This is an old method in information retrieval: pseudo relevance feedback (PRF), which treats the top documents from the initial retrieval as "probably relevant" and uses them to rewrite or expand the query. The authors argue that LLM listwise reranking is strong for the same reason: the model sees the query and all candidates at once and can use the relations among candidates. Earlier studies had already found that having the model generate ranking IDs one by one is not the key step; what matters is the interaction between query and candidates in the prompt.
The trade-offs of the existing approaches:
| Approach | How it ranks | Strength | Problem |
|---|---|---|---|
| Embedding retrieval | Cosine between query vector and document vectors | Fast; document vectors can be precomputed | No fine-grained interaction between query and documents or among documents |
| LLM listwise reranking | The prompt holds the query and candidates; the model generates "[2] > [1] > [3]…" | Best quality | Long-prompt prefill plus token-by-token generation, high latency |
| E2Rank | Encode the query with the top 20 candidates into a new query vector, then cosine against document vectors | Keeps the interaction among candidates without generation | Depends on the quality of the top 20 |
The big picture first

The flow of one reranking pass:
| Step | What happens | Cost |
|---|---|---|
| 1. Retrieve | Use the same model's query vector to fetch the top 100 from the corpus | One encoding plus vector search |
| 2. Build the prompt | Concatenate the query and the top 20 candidates, in retrieval order, into a listwise prompt | None |
| 3. Encode the prompt | Run the whole prompt through the model once and take the hidden state at the final [EOS] position as the new query vector | One forward pass |
| 4. Rerank | Compute cosine between the new query vector and each of the 100 document vectors and sort by score | 100 vector dot products |
The key point: document vectors do not depend on the query, so those computed at retrieval time are reused for reranking, and only the query vector absorbs the candidate documents' information.
Mechanism one: treat the listwise prompt as a query that "understands intent better"
When an ordinary embedding model encodes a query, its input is "instruction + query"; for reranking E2Rank swaps the input for "instruction + top k candidates + query", with an instruction along the lines of "given a query and some relevant documents, rerank the documents", and then, as with ordinary embeddings, takes the vector at the last position.
Why does this help ranking? In the "xiaomi price" example, if 15 of the top 20 are about phones, the model sees this context while encoding, so the resulting query vector leans towards the phone meaning and documents about grain get lower cosine scores. Unlike classical PRF, how these documents are used is not a hand-written rule (such as picking expansion terms or adding document vectors to the query in fixed proportions) but something learned in training.
The authors explicitly compared two classical PRF approaches on a model without listwise training: putting the candidates straight into the prompt, or fusing the query vector with document vectors Rocchio-style (the classic PRF formula: new query = original query + a weighted average of relevant document vectors).
| Approach (0.6B model) | DL20 | BEIR average | BRIGHT |
|---|---|---|---|
| E2Rank | 70.15 | 52.09 | 30.96 |
| No listwise training, query only | 66.29 | 49.93 | 22.69 |
| Plus text PRF (candidates in the prompt) | 56.57 | 46.52 | 29.62 |
| Plus vector PRF (Rocchio fusion) | 63.96 | 49.20 | 21.85 |
This shows candidate documents do not bring gains by themselves: an untrained model actually does worse when it sees them, dropping on DL20 (the passage-retrieval set of the TREC 2020 Deep Learning track) from 66.29 to 56.57. The gain comes from training the model to read the candidates.
Mechanism two: two-stage training keeps retrieval and reranking ability together
Stage one is standard embedding training: about 1.5M public examples with the InfoNCE contrastive loss (pull queries towards the correct document, push them away from wrong ones) to train query and document vectors.
Stage two continues training with two losses at once: InfoNCE to keep retrieval ability and RankNet to learn reranking. The training data is about 87K examples, each with 1 query, 1 positive and 15 negatives, and Qwen3-32B ranks the 16 documents into a full order as the label. RankNet is a pairwise loss: for each pair of documents, if the label says A is more relevant than B, the model is penalised when it does not score A above B.
One detail is worth computing by hand. Differences between cosine similarities are usually small, so the authors divide the score gap by a temperature τ=0.1 before feeding it into the loss. Suppose A should rank above B, with cosines 0.70 and 0.50 (illustrative): with τ=1 the loss is log(1+e^(−0.2))≈0.60, close to the "cannot tell them apart" value log2≈0.69; with τ=0.1 the gap is amplified to 2 and the loss drops to log(1+e^(−2))≈0.13. In other words, the temperature makes small cosine gaps visible in the loss, which gives the model a reason to widen them. Also, the sign of the exponent in the paper's equation (3) is reversed: under the paper's own definition that a smaller rank number means more relevant, the formula as written would reward wrong orderings, so it should be read as RankNet intends.
The total loss is InfoNCE plus λ times RankNet, with λ set to 2; the authors swept λ from 0.5 to 3 and NDCG@10 changed by less than 1 point.
Label quality is a weak spot. By the authors' own count, on MS MARCO Qwen3-32B puts the dataset's annotated positive first only slightly more than 50% of the time, so they treat these labels as noisy supervision. Interestingly, the teacher scores 53.37 zero-shot on BEIR, while the student E2Rank-8B reaches 54.35.
The ablation (0.6B) shows which part matters most:
| Configuration | DL20 | BEIR average | BRIGHT | MTEB v2 |
|---|---|---|---|---|
| Full E2Rank | 70.15 | 52.09 | 30.96 | 63.41 |
| Stage one only | 63.55 | 46.31 | 15.30 | 62.40 |
| Stage two without RankNet | 66.50 | 49.24 | 22.40 | 63.31 |
| Stage two without listwise prompts | 66.29 | 49.93 | 22.69 | 63.66 |
| Without stage one | 69.32 | 51.33 | 30.66 | 60.61 |
Removing RankNet or removing listwise prompts both drop BRIGHT from about 31 to about 22, showing reranking ability comes mainly from the combination of "ranking loss + candidate context"; removing stage one barely affects reranking but costs nearly 3 points on MTEB, so stage one mainly protects retrieval.
Why it is fast: no generation, one forward pass is enough
The generative baseline RankQwen3 reranks the top 100 with a sliding window: 20 documents per window, moving up 10 at a time, so it runs 9 windows, each of which first reads in 20 documents and then generates ranking IDs one by one. E2Rank puts only the top 20 into one forward pass, and the rest is just vector dot products, which is where the speed difference comes from.
| Per-query reranking latency (seconds, one A100) | DL20 | TREC-COVID |
|---|---|---|
| RankQwen3-0.6B | 1.97 | 4.58 |
| RankQwen3-8B | 6.44 | 16.93 |
| FirstZephyr-7B (generates one token only) | 3.81 | 5.26 |
| Qwen3-0.6B, scoring one document at a time | 0.28 | 0.40 |
| E2Rank-0.6B | 0.45 | 0.63 |
| E2Rank-8B | 2.55 | 3.40 |
E2Rank's latency here includes encoding the 100 documents. In a real system document vectors are computed when the index is built, and overview panel (c) shows this part is most of E2Rank's latency, so the actual online reranking cost is lower. A same-size model scoring one document at a time is even faster than E2Rank, but trained on the same data with the same RankNet loss it reaches only 45.97 on BEIR, versus 52.09 for E2Rank-0.6B.
Results: ahead on average, but look at where it comes from
| BEIR average NDCG@10 over 8 datasets | 0.6B | 4B | 8B |
|---|---|---|---|
| BM25 retrieval (no reranking) | 43.43 | 43.43 | 43.43 |
| RankQwen3 (generative reranking) | 48.03 | 52.38 | 53.39 |
| E2Rank (mixed training data) | 52.09 | 54.14 | 54.35 |
At 0.6B, E2Rank's BEIR average leads by 4.07 points. The authors ran per-query paired significance tests on each dataset, covering these 8 BEIR sets plus TREC DL 2019 and 2020, 10 in total: at 0.6B, 9 are significantly better and none is significantly worse, the cleanest result in the paper. At 4B and 8B, however, the average lead comes mainly from one dataset, Touché2020: Touché alone leads by 10.50 at 4B and 10.33 at 8B. Dropping Touché and recomputing the average gap over the other 7 datasets gives about +0.51 at 4B and about −0.38 at 8B (my estimate), meaning that at 8B, without Touché, E2Rank is slightly worse than generative reranking. Moreover, BM25 alone scores 44.22 on Touché, RankQwen3-8B falls to 31.73 after reranking and E2Rank-8B gets 42.06, so the "lead" there is more about losing fewer points than generative reranking does than about ranking better.
On the reasoning-heavy BRIGHT, E2Rank-8B averages 33.4, above the same-size RankQwen3's 32.0 and most reasoning rerankers trained with reinforcement learning, behind only ReasonRank (7B) at 35.7. As a retrieval model, E2Rank-8B averages 65.03 across MTEB's 56 datasets, slightly above the 64.26 from stage one alone.
The quality of the top 20 sets the ceiling


These two figures show that E2Rank's gain rests on the premise that "most of the top candidates are relevant", not on longer prompts being better. The authors also ran a gradual contamination test: replacing a growing share of the relevant documents in the top 20 with irrelevant ones takes DL20 from 70.15 all the way down to 51.55, and with everything replaced it is only slightly above BM25's 47.96, far below the 65.25 from the query alone.
Limitations as I see them
First, it amplifies errors when first-stage retrieval fails. For queries where retrieval is already poor (few relevant documents in the top 20), the feedback signal is misleading and the result can be worse than not using listwise at all. The authors concede in their limitations that it can fall below the first stage itself when relevant evidence is extremely sparse. Online, consider falling back to query-only scoring for such queries.
Second, the advantage at larger sizes is not stable. A large part of the average lead at 4B and 8B comes from Touché alone, several BEIR datasets get significantly worse, and most TREC DL differences are not significant. If your use case resembles MS MARCO-style web QA, at 8B it may not beat generative reranking; the main benefit is speed.
Third, both labels and comparisons are noisy. Training labels come from an LLM that agrees with the dataset's positives only about half the time, with no human-label check; the generative baseline RankQwen3 uses 40K GPT-4-labelled examples, while E2Rank's mixed data has about 87K, so the two differ in volume and source. Only the group trained on MS MARCO is a same-data comparison, and in that group the 8B average lead is just 0.47.
How a retrieval team could try it
- Check retrieval quality first. On your own evaluation set, count how many of the top 20 are relevant on average; if it is low, E2Rank's premise does not hold, so fix retrieval first.
- Run stage-two training on your existing embedding model. The paper also validated it on GTE-Qwen2-1.5B, which uses bidirectional attention, where the BEIR average rose from 51.56 to 53.57 with MTEB essentially unchanged, so you do not necessarily need a new backbone.
- Set the feedback window to about 20 documents in retrieval order, and keep query-only scoring as a fallback, switching back for queries with low retrieval confidence.
- When comparing latency, leave out document encoding and count only the single forward pass over the prompt, against your current reranker on the same GPU.
Verdict
In one sentence: the quality of listwise reranking comes mainly from "letting the model see the candidates together", not from making it generate a ranking, and building that into an embedding model replaces generation with one forward pass and cosine similarity.
| Dimension | Assessment |
|---|---|
| Importance of the problem | High. Reranking latency is the main obstacle to LLMs in search and RAG pipelines |
| Novelty of the method | Medium to high. PRF and RankNet are old; what is new is learning the listwise context into a query vector while keeping document vectors reusable |
| Experimental evidence | Medium. Solid at 0.6B; at larger sizes the advantage concentrates on individual datasets, and label noise is high |
| Engineering cost | Low. One model handles retrieval and reranking, needs no generation, and supports batch inference |
| Worth trying? | Yes. Especially for latency-sensitive settings where retrieval quality is already good |
How did this read for you?
Hoot made a note. Thanks for the feedback.
One tap, no sign-in; tap again to undo. Only anonymous counts are kept, never who you are.
Discussion
Add experiment details, reproduction notes or a different take. Requires a GitHub sign-in.
Loading the discussion…