Contents · 8 sections
1Miss one table: recall 67%, query still wrong2Four stages: frozen retrieval, fused candidates, features, ranking3The pool merges ranks, not scores4The ranker relies on signals beyond retrieval scores5On Spider, within one point of RASL6The ranker lifts complete@5 from 83.9% to 95.6%7On BIRD without evidence, Fiben and BEAVER, FERN falls clearly behind8Use the ranker as a first table finderSHARE
This paper shows that on public benchmarks with readable table names, finding tables for Text-to-SQL does not have to call a large model: a few off-the-shelf retrievers plus one gradient-boosted ranker get within one point of a Claude-based table selector at a budget of five tables. On BIRD (a public Text-to-SQL benchmark) without its per-question evidence hints and on a financial schema with run-together table names, it falls clearly behind the Claude-based method; on an enterprise warehouse benchmark, it recovers complete table sets less often than two retrievers trained on that warehouse.
The paper studies a setting where many databases are pooled into one catalog and the model is not told which database holds the answer, so hundreds of tables sit side by side. The BIRD (a public Text-to-SQL benchmark) dev catalog used in the paper has 80 databases and 597 tables. Before writing SQL, the system must pick the few tables this question needs out of those hundreds and pass their schema to the generator.
The step is hard because errors in both directions hurt. Miss one table and the SQL is wrong; pass extra tables and they eat the generator's context. A running example (illustrative): the question is "Which member placed the largest order in Shanghai last year?", and it needs three tables, member, orders and city. The catalog may contain five e-commerce databases that each have an orders table, while "Shanghai" appears only in the cells of the city table, not in any table or column name.
Existing table finders include pure retrieval methods and methods that use an LLM to extract keywords and then select tables. With pure retrieval, BM25 sees "order" and puts the orders tables from all five databases near the top, while city shares no word with the question and drops far down. A representative of the second kind is RASL (it asks Claude 3.5 Haiku to extract keywords for retrieval, then asks Claude 3.5 Sonnet to read the candidate schemas and pick the tables it judges relevant). RASL's recall is high, but each question costs two hosted model calls; in enterprise settings, sending schema and question text to an external service may not be acceptable.
FERN (the authors' table finder, short for Feature-Engineered Ranking with No generative LLM) asks how far one can get without any generative model, using only components that run locally.
Miss one table: recall 67%, query still wrong
First, the scoring. recall@k is the share of gold tables (the tables the reference SQL uses) that land in the top k, averaged over questions. In the example, suppose the top five contain member and orders but not city: recall@5 is 2/3≈67% (illustrative). The number looks acceptable, yet the SQL for this question is certainly wrong.
So the paper also reports complete@k: the share of questions whose top k contains every gold table. The question above scores 0 on complete@5. A complete table set is necessary for a correct query, not sufficient, but an incomplete set is always wrong. The results below should be read with both metrics: a system can have high average recall and a clearly lower complete rate.
Four stages: frozen retrieval, fused candidates, features, ranking
FERN trains no retrieval model; it fits only a ranker in the last stage. The whole path:
| Stage | Input | What it does | Output | Training only? |
|---|---|---|---|---|
| 1 Frozen retrieval | the question; each table's full card and name card | BM25 and two off-the-shelf embedding models (Qwen3-Embedding-0.6B, BGE-large) score every table in the catalog | five retrieval scores per table | no |
| 2 Candidate pool | ranks from BM25, Qwen3 and BGE on full cards | fuse the ranks with RRF and keep the top 100 | 100 candidate tables | no |
| 3 Features | question, catalog, cell values, retrieval scores | compute up to 120 features per candidate | a 100×120 feature matrix | no |
| 4 Ranking | the feature matrix | a LightGBM ranker fitted on BIRD training questions scores and sorts | the final table list | the ranker is fitted once |
Each table appears to the retrievers as two cards. The full card holds the table name, the database name and every column name, with a column's natural-language name added in parentheses when it differs from the stored name; in the example, city's full card looks roughly like city (database shop a): city id, city name (name of the city), province (illustrative). The name card holds only the table-name words, with camelCase and underscores split, plus the database name, such as city (database shop a). Neither card contains types, keys or values.
The five retrieval scores come from BM25 on full cards and on name cards, Qwen3-Embedding-0.6B on full cards and on name cards, and BGE-large on full cards only.
The pool merges ranks, not scores
The next stage uses only three of them: the ranks from BM25, Qwen3 and BGE on full cards. Why not simply add these three retrievers' scores? BM25 scores have no fixed upper bound, while dot products of normalized embeddings never exceed 1, so the scales do not match. FERN uses RRF (reciprocal rank fusion), which looks only at positions: if a retriever ranks a table at position rank (counting from 0), the table's pool score is the sum of 1/(60+rank) over the three retrievers.
Here is the pool score for two tables (illustrative). The orders table is ranked 0, 2 and 1 by the three retrievers, so its pool score is about 1/60+1/62+1/61≈0.049. Another table is ranked 0 by BM25 but past position 80 by both embedding models, so its score is about 0.017+0.007+0.007≈0.031. A table that all three retrievers place near the top beats one that only a single retriever likes.
The pool keeps the top 100, and on BIRD dev its average recall of gold tables already reaches 99.4. This means that on BIRD, finding candidates is mostly solved; the hard part is ordering them.
The ranker relies on signals beyond retrieval scores
Sorted by pool score, city may still sit beyond position 20 in the example. The ranker has to pull it up with signals other than retrieval scores. The 120 features fall into about a dozen families; here is a walk through them for the city candidate (illustrative):
- Value match: the literal "Shanghai" from the question appears among city's cell values, so this family fires.
- Foreign-key graph: city connects to orders, the top pool candidate, through member, so its join distance is 2; it is also a shared neighbor of two tables in the top five.
- Database: city belongs to the same database as orders and member near the top, and that database's best pool score is high.
- Retriever agreement: whether the five retrieval signals give city similar positions, and how many of them place it in their top 15.
The ranker is LightGBM trained with the lambdarank objective: the 100 candidates of one question form a group, and the model learns which table should come before which. It is fitted once on 9,394 BIRD training questions, with hyperparameters set in advance and never tuned, and then applied unchanged to the other benchmarks.
Why not use retrieval scores alone? The authors compared feature groups. On BIRD dev with evidence, a ranker over retrieval scores alone reaches complete@5 of 86.2, and adding the other features raises it to 93.9, a gain of 7.7 points. At fifteen tables, each group alone already reaches recall above 98. This means the extra features matter mainly when only five tables can be handed to the generator.
FERN has three variants that differ in their inputs. FERN-ev appends BIRD's per-question evidence hint (a sentence naming relevant columns and values) to the question and uses all 120 features; FERN-q uses the question alone with 116; FERN-schema also drops cell values, column types and row counts, leaving 102, for catalogs that lack them.
On Spider, within one point of RASL
The comparison uses RASL's published numbers; RASL was not rerun and its per-question results are unavailable, so the differences below are point differences, not significance tests. The cleanest case is Spider (another public Text-to-SQL benchmark): FERN was never fitted on Spider, and no design decision looked at it. In the table, RASL retriever is RASL ranking by retrieval without LLM table selection; DBCopilot is a generative routing model trained on synthetic questions over the target databases.
| Method | BIRD R@5 | BIRD R@15 | Spider R@5 | Spider R@15 | Fiben R@15 |
|---|---|---|---|---|---|
| DBCopilot | 85.8 | 94.6 | 91.6 | 97.6 | 56.9 |
| RASL retriever | 70.5 | 92.6 | 72.1 | 94.1 | 64.5 |
| RASL full (Claude selects tables) | 97.5 | 97.8 | 97.0 | 98.0 | 69.2 |
| FERN pool alone | 86.8 | 96.2 | 89.6 | 98.0 | 39.3 |
| FERN | 96.6 | 98.9 | 96.7 | 99.1 | 54.2 |
| FERN, BIRD question only | 89.3 | 97.1 |

At five tables, FERN is 0.3 points below RASL on Spider and 0.9 points below on BIRD with evidence. This shows that on public benchmarks with readable names, ranking without a large model already comes close to Claude's table selection; on Spider this is FERN-q, which sees only the question.
At fifteen tables, FERN is 1.1 points higher on both, but this cell needs a discount: RASL's fifteen is a cap, and its LLM returns only the tables it judges relevant, whereas FERN always returns exactly fifteen. So this compares a full list of fifteen with a possibly shorter one.
A more surprising result: on Spider, FERN's candidate pool alone reaches 98.0 at fifteen tables, level with RASL. This means most of the gap between earlier non-LLM methods and RASL is closed simply by fusing frozen retrievers; the ranker earns its keep at small budgets.
The ranker lifts complete@5 from 83.9% to 95.6%

The figure isolates the ranker: the same 100 candidates, ordered by pool score, by BM25 score, or by FERN. On Spider, complete@5 rises from 83.9 for the pool to 95.6; on BIRD with evidence, from 75.0 to 93.9.
Reranking the same pool by BM25 is worse, with complete@5 of only 31.4 on BIRD. This shows the gain comes from the combination the ranker learns, not from just any reordering. The ranker's value lies less in pushing recall a little higher than in fitting every required table of more questions into the top five. Back to the example: the pool already contains city, just low down; the ranker uses the value match and the join distance to pull it into the top five, and only then does this question's complete@5 go from 0 to 1.
In the fitted ranker, the feature with the largest gain is the pool score's gap to the top candidate, at 49.5% of total gain. My reading is that the ranker's backbone is still the retrieval signal, and the other features move tables near the cutoff up or down.
On cost, FERN-ev with evidence takes a median of 208 ms per question on one A100, of which 122.7 ms is feature computation in plain Python, with no hosted model calls. Fitting the three rankers takes about 3.5 minutes on a CPU.
On BIRD without evidence, Fiben and BEAVER, FERN falls clearly behind
The first limit is the evidence hint. BIRD's evidence often names the exact columns a question needs. Without it, FERN-q reaches R@5 of 89.3 on BIRD, 8.2 points below RASL, and needs twenty tables to exceed RASL's recall at five. For deployments without per-question evidence hints, the question-only row is the more relevant one.
The second limit is Fiben, a financial database: its table names are run together and unreadable (for example ACCOUNTSPAYABLEANDACCRUEDLIABILITIES), it has no cell values, it is a single database, and each question needs almost four tables; the paper cannot tell which of these factors is at work. In terms of the example (illustrative): without cell values, the clue that "Shanghai" appears in city's cells does not exist, and FERN-schema does not compute that family at all. FERN's pool caps recall at 82.8 within 100 candidates; at fifteen tables its recall is 54.2, against RASL's 69.2.
Fiben offers one more comparison. FERN reaches 80.7 recall when returning fifty tables and drops to 54.2 at fifteen, a loss of 26.5 points; RASL, letting its LLM select at most fifteen from a fifty-table pool, loses only 11.9 points. On Fiben, FERN's candidate pool is weak and keeping relevant tables under a small budget is also hard; the cause cannot be separated among the factors above. The two systems also use different question sets and embeddings, so this comparison does not isolate what LLM selection contributes.
The third limit is the 121 real questions of BEAVER (a benchmark built from real enterprise data warehouses). FERN-schema has seen no BEAVER question, and its recall at fifteen, 82.1, falls between two retrievers trained on BEAVER's synthetic questions (80.8 and 84.5).
But its complete@15 is only 49.6, which is 5.0 and 12.4 points below those two retrievers, or 6 and 15 fewer complete questions. This shows that with almost four gold tables per question, average recall can look fine while queries still miss a table. Because there are only 121 questions, the interval on the complete rate is about plus or minus 9 points, so these differences are descriptive only.
One more point affects what "no training" means: FERN's rankers use 9,394 labeled BIRD questions, while RASL uses 200 calibration questions per dataset. FERN avoids training on the target domain, not supervision.
Use the ranker as a first table finder
Back to the "Shanghai member" question: on catalogs like BIRD and Spider, whose table names are English words, three off-the-shelf retrievers put member, orders and city into the top 100. The LightGBM ranker then uses value matches, join distances and database signals to pull city into the top five, at about 0.2 seconds per question, with the schema never leaving the machine.
Three degradations were actually observed: on a catalog like Fiben, which combines run-together names, no cell values and many tables per question, the recipe falls clearly behind RASL. On BIRD without evidence, it is 8.2 points below RASL at five tables, although its recall at fifteen is still 97.1. On BEAVER, its complete rate is below two retrievers trained on that warehouse. The authors see deployment in two steps: first a ranker fitted on public data, then supervision on the target catalog once labeled data exists there.
How did this read for you?
Hoot made a note. Thanks for the feedback.
One tap, no sign-in; tap again to undo. Only anonymous counts are kept, never who you are.
Related papersMore from Paper Daily
Retrieval · 2026-10-26PersonalQuery-Benchmark:Evaluating Retrieval Robustness to Personalized User Expression DifferencesTwo shoppers want the same product. One asks for strawberry, lactose-free Greek yogurt in a short phrase; the other writes a longer sentence and misspells a word along the way. Will search return the same results? This paper builds a benchmark, PQB, from real user reviews in three Amazon categories to test whether results hold up when the same need is worded differently. They often do not. With t…
Retrieval · 2026-10-27LightRetriever: A LLM-based Text Retrieval Architecture with Extremely Faster Query InferenceIn LLM-based text retrieval, documents can be encoded offline, but queries arrive in real time, so every query runs the full LLM, which keeps throughput low and serving costly. LightRetriever breaks the symmetry: documents still go through the full LLM, while the query side is reduced to a table lookup. During training, a shared prompt plus a single token is fed through the LLM to get that token'…
Retrieval · 2026-11-09Semantic Prior-Guided Incomplete Cross-Modal Retrieval with Noisy LabelsCross-modal retrieval, such as searching images with text or text with images, is often trained on data where some samples lack one modality, and the class labels contain errors. Existing methods treat labels as the semantic reference, so a wrong label makes the filled-in content wrong too. SPIRN gives each class a fixed, mutually orthogonal reference vector that is not updated during training. T…
Retrieval · 2026-11-03How BERT-based Cross-Encoders Model Relevance in Information Retrieval?BERT-based cross-encoders are among the most accurate rerankers, yet how they judge relevance inside has never been clear. The paper traces how information flows in three models of different sizes across several in-domain and out-of-domain datasets, and finds the three go through the same stages. Early and middle layers contextualize tokens while dedicated attention heads look for literal matches…
Retrieval · 2026-11-02xPRT: Cross-Passage Retrieval PretrainingWhich passages of the same web page talk about related things is, by itself, enough to train a retriever. xPRT first takes passages whose vectors are close enough to the page's mean vector as anchor passages, then, by vector similarity and word overlap, finds for each anchor the four most related passages of the same document. It then trains the embedding: the anchor must pick out its most relate…
Retrieval · 2026-11-03Better Retrievers, Worse Labels: How Incomplete Relevance Judgments Distort Abstention Evaluation in Conversational RAGConversational RAG benchmarks often take a shortcut when deciding whether retrieved context is enough to answer: if none of the judged relevant documents were retrieved, the turn counts as unanswerable, and refusing is scored as correct. Re-running RECOR with four retrievers, the authors find the shortcut gets worse as the retriever gets better: among contexts with no judged document, the share t…
Retrieval · 2026-10-29LTRR: Learning To Rank Retrievers for LLMsRAG systems usually wire in one fixed retriever, yet different questions suit different retrievers, and some are better answered with no retrieval at all. This paper treats choosing a retriever as a ranking problem. For each training question it records how much each retriever improves answer quality over no retrieval, then trains a ranking model to order the retrievers, with no retrieval as one…
Discussion
Add experiment details, reproduction notes or a different take. Requires a GitHub sign-in.
Loading the discussion…