ProSearch: Benchmarking Multi-Constraint Protocol Retrieval in Experimental Science

ProSearch is a benchmark for multi-constraint protocol retrieval: researchers looking for experimental procedures usually carry several conditions at once, while a single methods paragraph often describes several experimental setups side by side, so a retriever only gets it right if all conditions hold within the same setup. The paper's contribution is to turn this need into an evaluation set of 1,515 queries across six experimental science domains, and to use it to show that current retrievers are weak at this: the best of 19 retrievers, Qwen3-Embedding-8B, reaches an average nDCG@10 of only 0.290, and reranking lifts it only to 0.474 (Tables 6 and 8). I think retrieval and RAG teams should pay attention to this benchmark, because it exposes a general weakness: retrievers find paragraphs on the right topic, but nine in ten of their top-ranked errors fail at least one condition.

The problem it tests: conditions must be bound to one setup

Take the paper's example: a researcher wants a procedure in which a certain cell type is cultured at 37°C without using DMEM (a common cell culture medium) (§1). The hard part is not finding paragraphs that mention these words, but the fact that one methods paragraph often describes several experiments, for example one group using DMEM and another group with a different medium and temperature. All three conditions appear in the same paragraph yet may belong to two different experiments, and scoring by keywords or semantic similarity ranks such paragraphs high.

The paper writes this requirement into its relevance definition (§3.1): each paragraph is first split into "protocol records", each corresponding to one coherent experimental configuration; a paragraph is relevant only if at least one record satisfies all of the query's conditions at once. The paper calls this joint constraint binding, Among existing benchmarks, MultiConIR also requires binding, but over structured items; ProSearch differs by requiring binding within unstructured methods text (Table 1, §2).

Each query carries 3–5 explicit conditions, each made of a field, an operator and a value, for example "culture temperature equals 37°C" or "medium is not DMEM". Some conditions are numeric, and some are negations that exclude a reagent or procedure (§3.1, Figure 3).

How the data is built

Construction has four steps (§3.2, Figure 1):

1. Expert modeling. For each of six domains (cell culture, immunotechnology, electrochemistry, organic synthesis, inorganic materials, polymer materials) an expert picks a representative experiment type, and together they define 60 experimental fields and the dependencies between them. For example, "confluence" (the share of the culture surface covered by cells, usually a percentage) is defined by the experts as depending on the "culture mode" field, so combinations with confluence but no culture mode are filtered out.
2. Corpus collection. Open-access papers from 2020–2025 are collected from OpenAlex, and only paragraphs from methods sections are kept; each domain has between 30,371 and 93,315 paragraphs (Table 2).
3. Query generation. GPT-5.2 extracts each experimental configuration in each paragraph into a separate structured record, filling expert-defined experimental fields and keeping the supporting text span for every value. One record covers one coherent configuration and never combines evidence from different configurations in the same paragraph (§3). Then templates are filled with the real values from one record to form a set of conditions. Only condition sets satisfied by 3–8 paragraphs in the corpus are kept, removing sets that are too broad or have almost no evidence, GPT-5.4 then removes scientifically uninformative sets and writes the rest as natural-language queries, and Claude Sonnet 4.6 checks each query for faithfulness, clarity and retrieval usefulness.
4. Relevance labeling. There are two kinds of candidate paragraphs: those whose extracted records satisfy all conditions, and "near-miss" paragraphs that fail only one. Whether a candidate is relevant is decided by GPT-5.4 reading the original text and judging configuration by configuration. This produces 1,551 queries; 36 have no associated relevance judgments (qrels) and are excluded from retrieval evaluation, leaving 1,515. Paragraphs outside the candidate pool count as non-relevant, as is usual; on 300 sampled queries the authors also judged the previously unlabeled paragraphs in all 19 retrievers' top 10, and only 1.86% were relevant, with retriever rankings almost unchanged (Appendix D.1).

On label quality, experts audited 285 query-paragraph pairs and agreed with the model labels in 89.1% of cases (Table 4). One number stands out: 59.8% of all relevant paragraphs come from the near-miss group (Table 5), which means the extracted records often miss or misstate conditions, and structured matching alone would miss most of the positives.

Main results: current retrievers struggle

The numbers come from Tables 6 and 8. Some observations:

• General dense retrievers do best, while retrievers trained specifically on scientific text are generally poor; MatSciBERT scores only 0.005 (Table 6). The paper reads this as domain adaptation on scientific text not helping. However, these scientific models are mostly small BERT models never trained for retrieval, differing in size and training objective, so the low scores cannot all be blamed on domain adaptation.
• Reasoning-oriented retrievers do not beat the strongest general model either; the best of them, DIVER-Retriever, scores 0.255 against 0.290 (Table 6), so the reasoning ability these retrievers have learned helps little here.
• Domains differ widely. The best retriever scores only 0.092 on immunotechnology but 0.495 on inorganic materials (Table 6).
• Query rewriting helps little. All with Qwen3-Embedding-8B retrieval and compared with the original query's 0.290: splitting the query into sub-queries loses 0.058, HyDE loses 0.018, and the best variant, DIVER-QExpand, gains only 0.027 (Table 7). The authors explain that splitting loses the links between conditions, and HyDE's invented details dilute the specific parameters.
• Reranking clearly helps: Qwen3-Reranker on the top 100 candidates reaches 0.474 (Table 8), yet immunotechnology and electrochemistry stay at only 0.301 and 0.294.
• Expanding the search from methods paragraphs to all body paragraphs makes the corpus 5.8 times larger on average, and Qwen3-Embedding-8B falls from 0.290 to 0.206, while the relative order of the four representative retrievers compared stays the same (Table 9).

Taken together, general dense models with broader training data do best (the paper's conjecture), but the overall level remains low. Even the best combination stays below 0.5 nDCG@10, far from being usable to find protocols for a scientific agent. My reading is that reranking helps a lot because a reranker reads the query together with the whole paragraph and has a chance to check conditions one by one, while a retriever that relies only on vector similarity is better at judging whether two texts are about the same thing. This is my interpretation; the paper only says accurate ranking requires judging multiple conditions jointly.

The authors also ran a small open-web test on 30 queries, measuring the share of queries with at least one correct piece of evidence in the top 10. Plain Tavily search reaches 0.300, and Tavily Agent, which rewrites queries and searches in several rounds, reaches 0.800 (Table 10). But only 0.353 of Tavily Agent's top 10 results satisfy all conditions (Table 10): query rewriting and multi-round search find more evidence, yet many top results still fail some condition.

Diagnosis: the problem is condition checking

The authors run two diagnostics that point to the same conclusion.

The first is a "structured diagnostic" (Table 11): ignoring text similarity, it uses the extracted records to count, for each paragraph, the largest number of conditions satisfied by a single record, and ranks by that count. This ranking reaches an average nDCG@10 of 0.774, and using Qwen3-Embedding-8B similarity to order paragraphs with the same count lifts it to 0.814, both far above the 0.290 of text-only retrieval.

The second is a manual error analysis (§4.3, Figure 2): for 25 queries per domain, the authors inspect the two highest-ranked wrong results of Qwen3-Embedding-8B, 300 pairs in total. 90% of them violate at least one query condition; the remaining few are topic-only matches that lack the required evidence, or unlabeled positives.

Together these show that retrievers already find paragraphs on the right topic; they fail because they do not check whether each condition is truly met. The paper attributes this to condition binding, but the error analysis only counts which kind of condition is violated; it does not separately count errors where all conditions are present but belong to different experiments. The error types differ by domain: cell culture and organic synthesis are dominated by categorical violations, immunotechnology and polymer materials by numeric ones (§4.3). Each error gets one primary cause, and when several conditions are violated it is classified by the order negation, numeric, categorical, so the shares partly reflect that order.

Limitations as I see them

• The 0.774 of the structured diagnostic should not be read as the score of a usable method. Candidate paragraphs were selected with the same extracted records, so this score is tied to the labeling process; it is better seen as a reference for how good ranking could be if conditions were extracted accurately. The re-labeling check in Appendix D.1 supports comparisons between retrievers but does not remove this concern.
• Labels depend on GPT-5.4. Re-judging 16,700 pairs with Qwen3-235B gives only 66.5% agreement, and with Claude Sonnet 4.6 90.2% (§3.3). Pairwise judgments differ considerably, but retriever rankings barely change, with Spearman correlations of 0.970 and 0.982 (Appendix D.2). Comparisons between retrievers are therefore reliable, while any single score deserves some margin.
• Each query has on average only 4.0–9.8 relevant paragraphs (Table 2), and evaluation covers single paragraphs only, not evidence spread across paragraphs, tables or supplementary material (§6), which is some distance from how protocols are really assembled.

How a search team could try it

1. Collect queries from your own product that carry several conditions which must belong to the same object, for example an item that must be in stock in a given model, color and size, or a service in a given place, time slot and price range; sample 100 of each and label whether the top results satisfy all conditions.
2. Measure current retrieval plus reranking on these queries, and count how many top errors come from conditions spread across different objects.
3. If such errors are common, try a light approach: use an LLM to split candidate documents into per-object structured records, add a feature for "how many conditions one record satisfies" in reranking, and compare with similarity-only reranking.
4. Track latency and extraction cost as well, to decide whether this step belongs in offline indexing or online reranking.

Verdict