AI SEARCH · 2026-11-03
AI Search — Paper Daily, 2026-11-03
Today's AI search track is about whether the evaluation itself can be trusted. One paper finds that the stronger the retriever, the less reliable the shortcut of judging answerability from relevance labels; another shows that attack success rate hides the new errors RAG defenses introduce; a third maps what visual document retrieval benchmarks actually test. On methods, DecisionRank teaches a single-pass reranker to rank like a generative teacher at a fraction of the latency, and another paper takes apart how BERT cross-encoders decide relevance step by step.
4.35
Conversational RAG benchmarks often take a shortcut when deciding whether retrieved context is enough to answer: if none of the judged relevant documents were retrieved, the turn counts as unanswerable, and refusing is scored as correct. Re-running RECOR with four retrievers, the authors find the shortcut gets worse as the retriever gets better: among contexts with no judged document, the share that actually contains the answer rises from 28% to 72%. Relevance judgments are incomplete, and strong retrievers keep finding useful documents nobody labelled. Scoring with this shortcut rewards readers that refuse when they should answer. Refusals should be scored against what the retrieved text says, not against whether a judged document was hit.
4.35
LLM listwise reranking works well but is slow, because the model generates document IDs one by one. Scoring every document ID at the first token and sorting once is far faster; the hard part is teaching such a student the ranking skill of a large teacher. DecisionRank lets the student sort documents by its own scores first. At each position of that list, with the earlier positions fixed, the teacher gives a probability distribution over which remaining document should come next, and the student's scores over the remaining documents are turned into a distribution and pulled toward it. A 1.7B student beats its 27B teacher on TREC DL, and on TREC DL an 8B student beats generative listwise reranking of the same size while running 24.5 times faster.
4.10
BERT-based cross-encoders are among the most accurate rerankers, yet how they judge relevance inside has never been clear. The paper traces how information flows in three models of different sizes across several in-domain and out-of-domain datasets, and finds the three go through the same stages. Early and middle layers contextualize tokens while dedicated attention heads look for literal matches between query and document; deeper layers bring heads that find semantic matches; finally the signals are gathered toward the query tokens and [CLS] for scoring. The details differ: the three models assemble the relevance signal into [CLS] in different ways, and MiniLM may even be looking for evidence of non-relevance.
4.05
Researchers looking for experimental procedures often carry several conditions at once, such as culturing a cell line at a set temperature without DMEM medium. One methods paragraph often describes several setups side by side, and a match only counts if all conditions hold within the same setup. ProSearch turns this need into a retrieval benchmark across six experimental science domains. The best of 19 retrievers reaches an nDCG@10 of only 0.290, and reranking lifts it only to 0.474. A manual error analysis finds that nine in ten top-ranked wrong results fail at least one condition.
4.05
Off-the-shelf retrieval embeddings carry taxonomy information that can be read out. The authors find that embeddings trained with Matryoshka Representation Learning (MRL) put taxonomy information mainly in the leading dimensions. Cutting a short slice in the original order groups same-category concepts better than cutting after a shuffle or keeping the last dimensions. Building on this, PEEL trains only a 98K-parameter projection on top of frozen embeddings that learns to tell, from two concept vectors, which one is the parent, and assembles these judgments into a taxonomy graph. Pairing the nodes of its graph one-to-one with the gold taxonomy and scoring how similar their names and neighbors are (Graph F1, maximum 1), it reaches 0.885 on Wikipedia and 0.931 on arXiv, given the full concept vocabulary and a set of gold parent-child edges.
3.95
RAG poisoning defenses are usually judged by one number: how much the attack success rate drops once the defense is on. Attack success rate is the share of answers in which the model repeats the attacker's answer. The authors store each query's answer under four conditions, clean or poisoned retrieval, each with and without the defense, and compare them one by one. Defenses do block some attacks, but they also make other queries that were not fooled start giving the attacker's answer, and the success rate nets this out. Retrieval-score detectors have a blind spot too: with one poisoned passage in context they stop 59 of 60 poisoned answers, and with five poisoned passages they stop none. A defense report should show both what it prevented and what it introduced.
3.75
Screenshot-based RAG keeps images and layout, but written facts inside screenshots, such as names, dates and numbers, are hard to retrieve. Pix2Evidence encodes each screenshot tile together with its transcription into one vector, so layout and text count together at retrieval time. During training, a vision-language model writes, for each training question, a description of the screenshot content that supports the answer. Besides standard contrastive learning, the retriever is trained so that the tile ranking produced by the question matches the ranking produced by that description. Without reranking, its answer accuracy beats the text-extraction and screenshot-only baselines on four of five Wikipedia QA benchmarks, and on MMSearch it reaches 26.0%, 4.4 points above the screenshot-only PixelRAG.
3.75
At every step an agentic searcher must choose: continue with what it already knows, or search first. Rewarding only final answers cannot tell a bad search choice from bad retrieval or reasoning. This paper scores each step on its own. The model rewrites the step's intermediate conclusion several times, the mean negative log-likelihood of the generated tokens is taken as uncertainty, and a curve fitted before training converts it into a probability of being right, the reliability. A no-search step scores its reliability minus the average reliability on a calibration set. A search step keeps the same query, rewrites the conclusion with and without the returned passages, and scores the gain in reliability. Trained with GRPO, its average CEM (whether the gold answer appears in the response) across seven QA benchmarks beats HiPRAG by 2.1 points with 6.8% fewer searches.
3.70
With only BM25 and keyword lookup as retrieval tools, plus a search loop that rewrites queries and decides when to stop, an LLM answers 85% of BrowseComp-Plus. The authors argue accuracy is no longer the main problem; context is. The loop repeats the same searches, rereads the same documents and fills the context window with passages it never uses. RAGFlow's answer is to have an LLM split documents at ingestion into single claims, each carrying its verbatim evidence sentence. At query time the loop checks these claims first, and when a gate judges the evidence sufficient it reads only those sentences instead of whole passages. Compared with the same loop reading raw passages, total tokens per query drop by 91% when that evidence suffices and rise only 1.9% when it does not, for a 38% average saving on FRAMES. Graphs, wiki pages and similar structures show no measurable accuracy gain.
3.70
Word files, PDFs and slide decks are organised by page, yet most visual document retrieval benchmarks test one thing: given a question, find a single page. This paper first separates the task by granularity, either finding pages or finding whole documents. It then separates it by mode, which can mean ranking items one by one, finding a set of pages that only together form the evidence, or searching over several rounds. Finally it checks 42 existing benchmarks against eight criteria. Most cover only single-page, one-by-one retrieval. Questions from real user needs and graded relevance labels are rare, and whether labels cover the whole corpus varies widely. Before picking a benchmark, check that it tests the ability the application actually needs.
NeurIPS 2026
EMNLP 2026