What the Query Remembers: Query Abstraction Governs Lexical and Dense Retrieval of Classical Arabic Poetry

When searching a classical Arabic poetry archive, which works better: keyword search with BM25 or an embedding model? This paper finds that the answer depends mainly on how much of the original wording the user still remembers. When users half-remember a few lines, BM25 beats all nine embedding models, including the 8-billion-parameter Qwen3-Embedding-8B (Recall@5 of 0.746 versus 0.717). When users can only describe what the poem is about, BM25 drops to 8th of 10 systems (0.392 versus 0.748). By rewriting the same queries, the authors show that the ranking is driven mainly by whether the query contains original wording. Adding half a verse of the original to a query raises BM25 recall by 37.8 points; adding a BM25 arm to fused retrieval gains 6.1 points on the half-remembered query set but loses 9.7 points on the theme-only query set.

This is worth reading for any team building content search, not just teams working on Arabic or poetry. The conclusion transfers directly. The same retrieval system can have opposite best configurations for different kinds of queries, so before choosing a design you need to know whether your users' queries carry original wording. The experimental design is also worth copying. Instead of only comparing two datasets, the authors rewrite the same queries to test cause and effect. The limits are that the corpus is only classical Arabic poetry, the queries were written by the authors, and there are only about 500 known-item queries.

What problem is it actually solving?

Classical Arabic poetry is hostile to retrieval. Arabic words change form heavily around their roots, vowel marks are usually omitted in writing, and poems are full of wordplay and metaphor, so the literal wording is far from the meaning. That makes it a common stress test for whether embedding models retrieve by meaning rather than by surface form.

The authors started with a simple question: how well do current embedding models retrieve classical Arabic poetry? They found the question has no single answer. Users remember different things. Some remember half a line, and some remember only what the poem was about. Where a query sits on this abstraction scale decides which retriever wins, by how much, and whether common improvements help or hurt.

How the experiments are set up

The authors use two corpora. Each poem is split into overlapping three-line chunks that are encoded separately, and a poem's score is the maximum of its chunk scores.

• Fann or Flop: 4,687 poems with human-written explanations of the verses. Its 244 queries simulate a user who half-remembers the poem. The query writers had seen the poem and could freely include remembered wording, and 32.8% of queries share an exact three-word run with the target poem.
• Diwan: to keep computation manageable, 3,500 poems were sampled from a 14-million-line corpus, 70 each from the 50 most frequent poets. Its 258 queries are deliberately written as abstract descriptions of content, and only 2.7% contain original wording.

For example, one half-remembered query is "a satirical poem about an ugly lover that opens: say much or say little, we are tired of you". One abstract query is "a philosophical poem that denies reincarnation and urges trust in reason alone", which shares no words with the poem.

The contestants are nine embedding models (including Qwen3-Embedding 0.6B and 8B, BGE-M3, multilingual E5 and jina-v3) plus BM25. There are three index types: the original verses, plain-language explanations generated by Gemini in place of the verses, and the human explanations that come with Fann or Flop. The main metric is Recall@5, the share of queries whose target poem appears in the top five results.

Finding one: the winner depends on the query

On Fann or Flop, BM25 ranks first and edges out Qwen3-8B (0.746 versus 0.717, not significant). On Diwan, BM25 falls to 8th and Qwen3-8B leads by 36 points. Scaling the same model family from 0.6B to 8B gains 17 and 22 points on the two corpora, but the 8B model still does not pass BM25 on Fann or Flop. Model size alone does not close this gap.

However, the corpus and the query style change together, so this comparison alone cannot say which factor matters. The authors therefore run two checks inside the same corpus.

The first check is stratification. The 244 Fann or Flop queries are split by whether they share an exact three-word run with the target poem: 80 quote it and 164 do not. BM25 finds every target in the quoting group (Recall@5 of 1.000) and drops to 0.622 in the non-quoting group. On non-quoting queries, BM25 and Qwen3-8B are no longer significantly different (0.622 versus 0.646). So BM25's overall lead comes entirely from the quoting queries.

The second check rewrites the same queries while keeping the information need and target poem fixed:

The table shows that BM25 gains the most when wording is added and loses the most when it is removed, and its difference from Qwen3-8B is significant in both directions. The embedding model is affected too, just less. Once the quote is removed, BM25 and Qwen3-8B tie, just as in the naturally non-quoting group. A regression controlling for query length, proper nouns and poem length leaves the overlap effect essentially unchanged.

Finding two: the same fix can help or hurt

Since the verses are hard to understand, a natural idea is to index plain-language explanations instead. Whether that helps depends entirely on the query type.

On Diwan, with abstract queries, replacing verses with AI explanations improves all ten systems, and BM25 rises from 0.392 to 0.760, a gain of 37 points. On Fann or Flop, with half-remembered queries, the same replacement hurts most systems, and BM25 falls from 0.746 to 0.357, a loss of 39 points, because the remembered wording no longer appears in the explanation.

The figure shows that the effect is continuous rather than all-or-nothing. The less a query overlaps the poem's wording, the harder the verse index finds it and the more the explanation index helps. The authors also find that for BM25, the change in lexical overlap brought by an explanation best predicts the outcome. For embedding models, the change in embedding similarity is the better predictor. In other words, BM25 bridges by words and embedding models bridge by meaning.

Finding three: two evaluations give opposite rankings

Besides finding one specific poem, the authors test thematic browsing. There are 200 theme-only queries, such as "white hair and its psychological effect", with no single correct answer, and a large language model rates the top three results from 1 to 5.

On thematic browsing, BM25 is last on both corpora and significantly worse than eight of the nine embedding models. The conclusion holds under blind ratings from two annotators and under two open judges from different model families. Removing the literal-match criterion from the rubric barely changes BM25's rank. BM25 ranks first only on whether results repeat the query's wording.

Deployment: fusion depends on query type

The authors test reciprocal-rank fusion (RRF, which merges several ranked lists by summing inverse ranks). The three candidate arms are Qwen3-8B over the verse index, Qwen3-8B over the explanation index, and BM25 over the verses, and the authors compare their combinations. Fusing the explanation and verse arms helps on both corpora and can serve as a default. Whether to add the BM25 arm depends on the setting. On Fann or Flop, the best combination with BM25 scores 0.885, 6.1 points above the best without it (0.824). On Diwan, the best combination with BM25 scores 0.833, 9.7 points below the explanation index alone (0.930). The pattern holds with every other embedding model.

So an archive serving readers who half-remember lines should keep a BM25 arm, and one serving thematic browsing should not.

Limitations as I see them

First, the setting is unusual. Classical Arabic poetry has an exceptionally large gap between wording and meaning, and in news or product search the size of this effect may differ.

Second, the queries were written by the authors, and the two styles were produced from different instructions on purpose. Real user queries may be distributed differently. The scale is also small, with about 500 known-item queries and 200 thematic queries.

Third, the AI explanations are of moderate quality. In a 50-item audit only 60% were fully faithful and 8% misread the poem. Better explanations could change some conclusions.

How a team could try this

1. Sample a few hundred queries from production logs and measure their lexical overlap with clicked documents to see whether your users look more like "remembers the wording" or "remembers only the theme".
2. Evaluate BM25, dense retrieval and fusion separately for each overlap group instead of looking only at the overall metric.
3. If both kinds of queries are common, detect whether a query contains original wording first, then set the BM25 arm's weight in fusion accordingly.
4. Consider having a language model write explanations of documents and fusing them with the original text in the index, which helped on both query types.

One-sentence conclusion

Which retriever wins is largely decided by how much original wording the user's query carries; with quoted wording BM25 is hard to beat, with theme-only queries embedding models lead by a wide margin, and fusion should be configured by query type.