4.45
LEAD STORY ·
Elastic
ElasticIs it always better to give a reranker more candidates? Jina's team at Elastic tested its own listwise reranker on more than two dozen retrieval benchmarks, and the answer is no. With BM25 as the first stage, reranking the top 100 candidates works best, cutting about a fifth of the error on average, and adding more makes it worse. With semantic search as the first stage, the best point moves down to 20 candidates and the gain is much smaller, and sending 200 is on average worse than not reranking at all. The reason is that a listwise model reads the whole candidate list at once, so with too many candidates its attention is spread thin and noise accumulates. A model that scores candidates one by one does not have this problem, since an extra candidate never changes the order of the others. The worse the first stage, the more candidates reranking needs and the more room it has to fix things. Size the candidate pool by the quality of the first stage, because sending too many hurts
4.05
RAG usually stuffs whole retrieved passages into the LLM, though often only a few sentences in them can answer the question. Picking those sentences is not easy: a sentence can look irrelevant on its own, and whether to keep it may depend on other retrieved text. Judging each sentence with the full context is too expensive. Decision models can score many candidate sentences in one pass, with each judgment seeing the surrounding sentences.
4.05
Multimodal retrieval has two kinds of embeddings. Discriminative embeddings encode the raw input directly, which is fast but misses fine-grained meaning. Generative embeddings first have the model reason about the query intent and distinguishing evidence and then encode, which is more accurate but adds a reasoning pass to every query. AntEmbed, from Soochow University and Ant Group, lets one model do both while sharing a single candidate index.
3.85
Beihang UniversityWhen multimodal LLMs are used for embeddings, a recent trend is to have the model write a chain of thought before encoding, injecting intermediate semantic evidence into the representation. The problem is that these chains are written for the query and the candidate separately, never seeing the hard negative that looks like the positive, so they cannot say why the positive is right or where the hard negative falls short. UMER switches to pairwise reasoning: it puts the query and candidate side by side and writes the evidence that matches the instruction and the evidence that does not. One MLLM learns both an embedding that can be indexed alone for full-corpus retrieval and a pairwise relevance judgment, and the two then distill reliable preferences into each other. In serving one can retrieve with the embedding alone and turn on pairwise reranking when the budget allows. On MMEB-V2, UMER with a Qwen2-VL-7B backbone scores 70.6, 2.0 points above the strongest baseline on the same backbone. To tell positives from hard negatives, reasoning has to compare them side by side
3.05
An LLM's accuracy on medical exam questions cannot say which individual answer can be trusted. The author takes two 7B open models and checks whether their option probabilities are reliable on English MedMCQA and Arabic MedAraBench multiple-choice questions. In all four combinations the raw probabilities are overconfident, and a single temperature parameter removes more than 80% of the calibration error. Conformal prediction reaches its promised coverage in every setting, but the answer sets are large when accuracy is low: to guarantee 90% coverage, Jais's sets on Arabic questions contain 89% of the options on average. Answering only the most confident fifth of questions cuts Qwen's English error from 0.439 to 0.098. The two benchmarks differ in subject and difficulty, so the author does not attribute the differences to language itself. When a model is not accurate enough, a coverage guarantee alone cannot yield useful answer sets