Contents · 5 sections
1Random paraphrasing separates wording from relevance2Walkthrough: one fish-tank query, and how a selected paraphraser is recorded3In all nine models, length raises the chance of being selected4What this means for people building LLM recommenders and shopping assistants5LimitationsSHARE
Search "recommend a name-brand new fish tank": does the longer write-up get picked? How random paraphrasing uncovers verbosity bias in LLM recommenders
A user asks a shopping assistant: "Recommend name-brand and newest fish tanks." The system first retrieves 5 candidates from the catalog. Some are fish-tank decorations or fish food; one is a short line, "suitable for live plants in aquariums, size 27 cm"; another describes a double-arc design, lighting, a 32-liter capacity, and a long list of selling points. A large language model then has to pick one recommendation from these 5 natural-language descriptions. The paper's Figure 2 shows this real retrieval set.
The question is: is the model choosing the relevant product for the query, or is it also being pulled by how long the description is? Catalog copy is often written by sellers, and there are rarely item-by-item relevance labels in production. Change the model, the prompt, or the catalog, and who gets recommended can change with it, which ordinary offline evaluation is slow to catch.
This paper, from the University of Granada, gives an audit that needs no labels: each product description is randomly handed to a different paraphrasing model, and the descriptions are then randomly shuffled in the prompt. The authors prove that if the recommender's choices are invariant to these paraphrases, every paraphrasing model must be selected with equal probability. From that starting point, they use a four-step statistical procedure to test for departures from uniformity and, after controlling for relevance proxies, to separate the contribution and direction of surface features such as length. They instantiate the method as a verbosity-bias case on 20,000 template queries and nine recommender LLMs. After controlling for the measured relevance, longer text raises the odds of being selected in all nine models, by as much as 38% per standard deviation. Relevance remains the stronger signal.
You can test whether an LLM recommender is pulled by how a description is written, without labels; the length effect is an association after controlling for relevance proxies. The experiments are single-turn, template queries, 5 candidates each, and decoding temperature 0.
Random paraphrasing separates wording from relevance
Systems like this usually retrieve first, then let a large model choose from the list. Positional bias has already been controlled by randomly shuffling the order: if the model does not favor a particular prompt position, each position should be selected equally often. This paper applies the same idea to how the description is written.
A paraphraser is another large model, prompted to rewrite the product description without changing its meaning, and to emit a fixed JSON object. The authors also add an identity paraphraser that changes nothing, to see whether original descriptions are favored or avoided relative to rewritten ones. If paraphrasers were always pinned to retrieval rank, the effect of rewriting would be tangled with retrieval relevance. So they draw two permutations independently: one decides which product goes to which paraphraser, and one decides the order of the rewritten descriptions in the prompt. The recommender returns only the prompt position it selected, which is then traced back to the paraphraser.
If the recommender decides by meaning alone, and the paraphrases truly do not change the meaning, each paraphraser should be selected with probability 1/k. Any significant departure from uniformity means at least one of those things is false: either the recommender is looking at surface wording, or some paraphrases have already pushed the meaning across a decision boundary. Uniformity itself is not a sufficient condition for "no bias"; effects in opposite directions can cancel.
Walkthrough: one fish-tank query, and how a selected paraphraser is recorded
The walkthrough follows the example in the paper's Figures 1 and 3. The query is "Recommend name-brand and newest fish tanks," and the candidate count is k=5. The paraphraser assignment comes from the paper's Figure 3.
Step 1. Retrieve. The retriever bge-small-en-v1.5, an embedding model, takes the top 5 items from the catalog by the cosine between the query and the product title. The descriptions vary widely in length: one is a very short live-plant line, one is a 32-liter tank packed with selling points, and fish food and decorations sit in between.
Step 2. Assign paraphrasers at random. This draw is 1, 5, 3, 2, 4, corresponding to DeepSeek-R1 7B (Qwen), the identity paraphraser, Gemma-3 12B, GPT-OSS 20B, and Falcon-3 7B. The fish-food item is assigned the identity paraphraser, so the original text is left unchanged; the short plant description is rewritten by Falcon-3 7B as "suitable for live-plant aquariums, size 27 cm"; the long item goes through Gemma-3 12B and remains a long selling-point paragraph.
Step 3. Shuffle the five items in the prompt, so that prompt position and paraphraser are independent, and a position preference cannot systematically favor a particular paraphraser. The recommender selects one position at temperature 0.
Step 4. Trace that position back to the original product, and then to the paraphraser it was assigned. This query contributes one count: some paraphraser was selected once.
Step 5. Repeat the same procedure on 20,000 queries. If selection is invariant to paraphrasing, the five paraphrasers should each have about 20%. The authors compare this against the uniform distribution with a chi-square goodness-of-fit test at significance level 0.05. They then move to a conditional logit: within the 5 candidates of the same query, they estimate how much length, the relevance proxies, and semantic preservation each explain, and whether the coefficients are positive or negative.

What this can show is that, without relevance labels, one can still test whether paraphrasers are selected uniformly, and then separate length from the relevance proxies. It cannot, on its own, say whether any single recommendation is right or wrong, because there are no human labels here.
In all nine models, length raises the chance of being selected
Queries are formed from 54 templates, 18 categories, and a set of attributes, and 20,000 are sampled. The catalog comes from the Amazon ML Challenge 2023: descriptions shorter than 40 characters are dropped, then 50,000 items are sampled uniformly. Retrieval again uses bge-small-en-v1.5. The four paraphrasing LLMs and the identity paraphraser all run at temperature 0. The nine recommenders belong to five families: DeepSeek-R1, Falcon-3, GPT-OSS, Gemma-3, and Llama-3.1, also at temperature 0.
Seven of the nine models depart significantly from uniformity in the chi-square test (p less than 0.05). The two GPT-OSS models do not reach significance, with p-values 0.0597 and 0.5626. The departures are not large, at most about ±1.3 percentage points, but 20,000 queries still let seven of the models be detected. The Gemma-3 12B paraphraser is selected more often by almost every recommender; rewrites by DeepSeek-R1 and Falcon-3 are more often avoided. This does not look like "recommenders prefer rewrites from their own family": the two DeepSeek-R1 recommenders actually select DeepSeek-R1's descriptions less often. The identity paraphraser also has no single direction: some models prefer the original text, and some avoid it.
Uniformity is only the first step. The authors then fit a conditional logit grouped by query, standardize every feature globally, and estimate each feature's unique contribution. Unique contribution is the McFadden R² of the full model, minus the R² of the model refit after that feature is dropped. The part jointly explained by length and relevance is not fully allocated among these unique values. Relevance uses two proxies: a dual-encoder cosine between the query and the description, and a cross-encoder score from a model that reads the query and the description together. There is also the vector similarity between the paraphrase and the original text, used to control whether the rewrite has shifted the meaning. Length is the token count after spaces and punctuation are removed.
| Recommender | Uniformity p-value | Length coefficient | Odds increase per SD |
|---|---|---|---|
| DeepSeek-R1 7B (Qwen) | 0.000029 | +0.3220 | +38.0% |
| DeepSeek-R1 8B (Llama) | 0.000025 | +0.2811 | +32.5% |
| Gemma-3 12B | 0.003874 | +0.2423 | +27.4% |
| Gemma-3 27B | 0.014408 | +0.2271 | +25.5% |
| Llama-3.1 8B | 0.000075 | +0.1944 | +21.5% |
| Falcon-3 7B | 0.000018 | +0.1826 | +20.0% |
| GPT-OSS 20B | 0.562556 | +0.1484 | +16.0% |
| Falcon-3 10B | 0.000050 | +0.1469 | +15.8% |
| GPT-OSS 120B | 0.059720 | +0.1167 | +12.4% |
(p-values from the paper's Table 1; length coefficients from the unique coefficients in the paper's Table 3. The odds ratio is e to the power of the coefficient, and features are standardized. The two GPT-OSS uniformity tests do not reach significance, but the length coefficients are still positive.)
After these relevance proxies are controlled for, the length coefficient is positive for all nine models. The smallest is 0.1167 for GPT-OSS 120B, and the largest is 0.3220 for DeepSeek-R1 7B, a gap of about three times. Turning the coefficients back into odds: for each extra standard deviation of length, the odds of being selected rise by 12.4% to 38.0%. The two GPT-OSS models that did not reject uniformity in the first step still show a positive length coefficient here. That matches the point that uniformity is not a sufficient condition for "no bias." Hoot says
The article already walks Figure 1: how a selected paraphraser is recorded for one fish-tank query. This walkthrough only pursues a point that walk did not unpack: if the uniformity test is not significant, whether that means there is no length bias. The p-values and coefficients below come from Tables 1 and 3; they are not illustrative.
the uniformity chi-square only looks at how often each paraphraser is selected → the two GPT-OSS models do not reach significance (p=0.059720 and 0.562556) → after relevance is controlled for, the length coefficients are still positive (0.1167 and 0.1484) → turned into odds, each extra standard deviation raises the odds by 12.4% and 16.0% → uniformity is not a sufficient condition for "no bias"
- the uniformity chi-square test → it only looks at how often each paraphraser is selectedIf the recommender only looks at meaning, each paraphraser should be selected equally often. The chi-square test only compares how often each paraphraser appears; it does not look at which description is longer inside a given query.what is tested = how often each paraphraser is selected, not length
- the two GPT-OSS models → the uniformity test does not reach significanceSeven of the nine models have p less than 0.05. For GPT-OSS 120B the p-value is 0.059720, and for the 20B model it is 0.562556; neither rejects uniformity.120B p=0.059720 > 0.05; 20B p=0.562556 > 0.05
- the conditional logit after controlling for relevance proxies → the length coefficients are still positiveThe authors then fit a conditional logit grouped by query, and control for relevance proxies. The length coefficient is 0.1167 for GPT-OSS 120B and 0.1484 for the 20B model, both greater than 0.120B coefficient=0.1167; 20B coefficient=0.1484
- turning the coefficients back into odds → each extra standard deviation of length raises the odds by 12.4% and 16.0%Features are standardized, and the odds ratio is e to the power of the coefficient. For the 120B model that is about 1.124, a 12.4% increase; for the 20B model it is about 1.160, a 16.0% increase.e to the power 0.1167 ≈ 1.124 (+12.4%); e to the power 0.1484 ≈ 1.160 (+16.0%)
- a non-significant uniformity test → it cannot be read as no length biasUniformity only means the paraphrasers are selected about equally often; a length preference can still remain among the candidates inside a query. So uniformity is not a sufficient condition for "no bias".uniformity is not sufficient for "no bias"
Figure 4 shows the length and similarity distributions under each paraphraser: rewritten descriptions are usually a little shorter than the original, so length still differs across candidates. The walk above only asks why a non-significant uniformity test can still leave a length coefficient; it does not read this figure as the uniformity test, and it does not retell the Figure 1 fish-tank example.

Source
The article and paper Remark 3: uniformity is a necessary but not sufficient condition for no structural bias. Table 1: chi-square p=0.059720 / 0.562556 for GPT-OSS 120B / 20B. Table 3 length coefficients 0.1167 / 0.1484. §4.2.3: e^{β} is the multiplicative change in the odds of selection per standard deviation. Figure 4 is only used to show that length still varies after paraphrasing; it is not a figure of the uniformity test.The unique contribution of the relevance proxies is usually larger. In McFadden R², the unique contribution of the cross-encoder score is about 2.2% to 4.1% across models, while length ranges from 0.28% for GPT-OSS 120B to 2.25% for DeepSeek-R1 7B. On that most sensitive DeepSeek model, the unique contribution of length is already comparable to the cross-encoder. This kind of R² is typically lower than the numbers familiar from linear regression; what is being compared is relative size.

When length and relevance point in the same direction, the selection gap is largest. When they conflict, the more relevant description is still selected more often, but the average advantage is less than half of what it is when the two agree. When relevance is similar, all nine models select the original candidate more often if it is the longer one; if the original candidate is shorter, seven models select it less often, and the gaps are also smaller. Relevance remains the stronger signal, and length is a stable secondary signal that can partly offset relevance.
What this means for people building LLM recommenders and shopping assistants
Among the models tested here, sensitivity to length can differ by about a factor of three; when choosing a model, this kind of black-box audit can sit alongside accuracy and cost.
The paraphrases can be reused. If the model, the prompt, or the catalog changes, a few thousand to ten or twenty thousand queries are enough to rerun the audit. On a platform where sellers write their own descriptions, a recommender that rewards longer text will push copy to get longer rather than clearer. Mitigations worth trying include truncating descriptions to similar lengths before they reach the recommender, or rewriting every candidate with the same paraphraser. The same test can also check whether those treatments have reduced the bias.
Limitations
The empirical results are limited to single-turn product recommendation, template-generated queries, 5 candidates each, and deterministic decoding at temperature 0. The later feature analysis estimates associations between features and selection, not causal effects.
Reading "a preference for longer descriptions after controlling for relevance" as verbosity bias depends on the relevance scores themselves being insensitive to length. The paper lists constructing a length-insensitive relevance signal as future work.
The queries are not live production logs, and the catalog is 50,000 items sampled after short descriptions were filtered out. The authors also do not use human labels to count how often this bias would change a recommendation to a less relevant product.
How did this read for you?
Hoot made a note. Thanks for the feedback.
One tap, no sign-in; tap again to undo. Only anonymous counts are kept, never who you are.
Discussion
Add experiment details, reproduction notes or a different take. Requires a GitHub sign-in.
Loading the discussion…