τ-Rec-Books: A Disclosure-Controlled Benchmark for Requirement Elicitation in Agentic Recommender Systems

Bharath Sivaram Narasimhan, Karthik R. Narasimhan
23 minread4.35scoreNo. 511-05 Generative issue
Contents · 9 sections1What problem is it solving?2What a task looks like3How the benchmark is built4Result 1: timing matters most5Result 2: "stated when asked" tasks fail because agents do not ask6Result 3: hidden-requirement tasks fail because agents do not verify7Limitations as I see them8How a recommendation team could try it9Verdict
Rel 4Nov 4Prac 5Evid 4Time 5

People building conversational recommendation agents should read this: it treats when the user reveals a need as its own variable and exposes the two most common ways agents fail. The hard part of conversational recommendation is that users rarely state all their requirements up front. τ-Rec-Books keeps each book request fixed and changes only when the user reveals requirements: all at the start, one at the start and the rest only when asked, or some never. A requirement that is never stated can only be inferred by proposing books and watching whether the user accepts or rejects them. On solvable tasks, average success across four agents falls from 0.975 when everything is stated, to 0.756 when requirements come only on request, to 0.283 with hidden requirements. Ask-only tasks fail mainly because agents never ask. When the prompt tells them to ask, success returns to above 0.93. On hidden tasks, agents often give as the answer a book they never showed the user. Evaluating a recommendation agent means checking whether it asks, and whether it uses rejections to rule out possible requirements.

My take: the value of this paper is that it turns "when the user states their requirements" from a vague difficulty into a variable that can be measured on its own. Changing only when the same request is stated moves success far more than changing the search interface or the catalog size. That sets a concrete bar for evaluating recommendation agents: testing only the case where the user says everything at once is not enough.

What problem is it solving?

Real users rarely state every requirement when they open. The authors counted 11,348 human recommendation dialogues in ReDial: for the first genre, era or origin requirement, users stated only 36% unprompted, 52% when asked, and 12% only after seeing a suggestion.

Most existing conversational recommendation benchmarks do not treat this as a variable. For the same book request, a user who says everything at once and a user who only answers when asked are two very different tasks for an agent. τ-Rec-Books measures this on its own.

What a task looks like

The paper uses task family 060 as an example. The user wants one book with three requirements: the genre includes science fiction, the genre includes young adult, and it was published no later than 2001. The three conditions state these requirements differently:

RequirementAll stated at the startStated when askedWith a hidden requirement
Genre includes science fictionStated at the startStated at the startStated at the start
Genre includes young adultStated at the startStated when askedNever stated
No later than 2001Stated at the startStated when askedStated when asked

In the "stated when asked" version, one agent asked about language, year and other genres, the user then gave both requirements, and the agent's book was accepted. In the hidden version, the user gave the year requirement but said "I'd rather not say more about the genre." The agent ran 93 searches, proposed one book that was rejected, and then directly recommended Fahrenheit 451. That book is 1953 science fiction, but it is not young adult, and it was never shown to the user first.

When the user says "I'd rather not say", they are in fact telling the agent that this field holds a requirement, just not what it is. The only way to find it is to propose some books, see whether the user accepts or rejects them, and rule out the possibilities that do not fit.

How the benchmark is built

The catalog comes from Wikidata: 20,000 books, plus 5,000- and 10,000-book subsets that sit fully inside the larger catalog. There are 450 tasks in 150 families. Each family fixes the requirements, the answer set, the books already read and the accepted formats, and varies only when the requirements are stated, so the three conditions of one family can be compared directly. Requirements use only four fields: genre, language, country of origin and publication year.

Fifteen families are built so that no book in the catalog fits. There, the right move is to give up on recommending; the paper calls these NVR (no valid recommendation) tasks.

The agent talks to a rule-based simulated user. It answers by fixed rules: asked about a field it can share, it shares; asked about a hidden field, it says "I'd rather not say". When the agent proposes a book with its catalog ID, the user replies accept only if the book meets every requirement, is unread, and comes in a format the user accepts; otherwise reject, never with a reason. Proposing only asks for the user's verdict and can happen many times. To finish, the agent calls a separate recommend tool with one book, or with none to give up, and the dialogue ends. Scoring checks that book against the catalog metadata to see whether it meets every requirement, plus a few rule checks, such as never recommending a book already read and giving up when there is no answer. No LLM is used as a judge. During the dialogue the agent sees the user's replies, the accepts and rejects, and what search and other tools return, but it is never given the answer set.

Hoot saysProposing only gets a verdict; recommending is the answerClose ✕
Proposing only gets accept or reject; calling the recommend tool ends the dialogue and is scored.
Top row: the agent asks, the user answers or says rather not say, the agent proposes a book, the user only says accept or reject. Bottom row, illustrative: book A already read and rejected, book B accepted, the agent calls the recommend tool with a book or none, and the dialogue ends and is scored on the catalogAgent asksabout some fieldUser answersor “rather not say”Propose a bookwith catalog IDUser only saysaccept or rejectIllustrative: book Aalready read, rejectedIllustrative: book Bfits all, acceptedCall recommend toolone book or noneDialogue endsscored on catalog
Top row: the agent asks, the user answers or says rather not say, the agent proposes a book, the user only says accept or reject. Bottom row, illustrative: book A already read and rejected, book B accepted, the agent calls the recommend tool with a book or none, and the dialogue ends and is scored on the catalogAgent asksabout some fieldUser answersor “rather not say”Propose a bookwith catalog IDUser only saysaccept or rejectIllustrative: book Aalready read, rejectedIllustrative: book Bfits all, acceptedCall recommend toolone book or noneDialogue endsscored on catalog

The top row is what the agent can do in the dialogue. It can ask, and it can propose books. For a proposal the user only says accept or reject, never why. The bottom row is an illustration: book A was already read and is rejected; book B is accepted. At the end the agent calls the recommend tool with one book or none. Only that step is scored.

Sourcepaper §3 interaction protocol: proposals get only accept/reject; a trial ends with the recommend tool, called with one book or none

Four agents take part: three open-weight models, DeepSeek, GLM and Qwen, and one frontier model, GPT-6.1 Sol. There are two search interfaces: keyword search returning the top 20, and structured search that filters by field. There is also one scripted reference agent, called the "popularity" agent. It asks one question covering every field, then proposes books in order of popularity until one is accepted.

Result 1: timing matters most

Left: success under the three conditions; every agent drops sharply when a requirement is hidden. Middle: with hidden requirements, as more rejections are allowed, LLM agents' success barely rises. Right: success on hidden tasks under different prompts
Left: success under the three conditions; every agent drops sharply when a requirement is hidden. Middle: with hidden requirements, as more rejections are allowed, LLM agents' success barely rises. Right: success on hidden tasks under different prompts

On the 45 solvable task families of the evaluation panel (20,000-book catalog), averaged over the four agents and two interfaces, success is 0.975 when everything is stated at the start, 0.756 when requirements are stated only when asked, and just 0.283 with hidden requirements. Hidden tasks with no answer are counted separately and average only 0.047. Every model and every interface shows the same order. By comparison, structured search beats keyword search by only 7.1 points on average, mostly because of Qwen; without Qwen the gap is 3.3. The structured interface's prompt also lists the fields, so the comparison is between tool-plus-prompt bundles. Growing the catalog from 5,000 to 20,000 books also matters little. The catalog-size study used only the three open-weight models and only the all-stated and stated-when-asked conditions: keyword search drops 4.2 points, structured search only 0.8.

Result 2: "stated when asked" tasks fail because agents do not ask

How failed dialogues end. Left: in "stated when asked" tasks, most failures are no question at all or a field never asked about. Right: with hidden requirements, most failures are recommending a book the user never verified
How failed dialogues end. Left: in "stated when asked" tasks, most failures are no question at all or a field never asked about. Right: with hidden requirements, most failures are recommending a book the user never verified

GPT-6.1 Sol is nearly perfect when everything is stated at the start, yet it is the lowest of the four when requirements must be asked for. In 40% to 49% of those dialogues, it draws out not a single askable requirement. Yet its 101 dialogues where it did draw out every askable requirement all succeeded.

So it can find books; it just does not ask. The authors tried several prompts on an 18-family reliability subset (structured search). Adding one line to the system prompt, "the user may have requirements they have not stated yet; ask about them before you recommend", raises GPT-6.1 Sol by 59.0 points on these tasks, and all four agents reach at least 0.931.

Result 3: hidden-requirement tasks fail because agents do not verify

With hidden requirements, 78% of failures are the agent recommending a book the user never got to give a verdict on; for GPT-6.1 Sol, 415 of its 416 failures are like this. Across all such unverified recommendations, 73% happen after the user has already said "I'd rather not say", meaning the agent had the signal that a hidden requirement exists and did not use it.

Adding, on top of the ask line, "propose the book first and wait for the user's verdict before recommending" cuts unverified recommendations to at most 30% of hidden-task failures. Success reaches 0.483 to 0.617, about the same as the 0.600 of the scripted agent that asks once about every field and then proposes books by popularity. Then adding a note explaining what "rather not say" and a rejection mean adds 22.5 more points on average, to 0.708 to 0.825. The authors conclude that explaining the rules closes part of the gap.

Limitations as I see them

First, the simulated user is mechanical. It speaks only in catalog labels, never explains a rejection, and has not been validated against real people. When the authors had GLM and Qwen play the user for DeepSeek (structured search), success fell by 10.2 and 7.4 points, mostly on stated-when-asked tasks, because the LLM users shared everything they could in only about half the dialogues. This hints that more human-like users may be harder, but it has not been checked with real people.

Second, requirements are hard filters over just four metadata fields. Soft preferences common in real book requests, like "similar to this book" or "an easy read", are out of scope. In "stated when asked" tasks, one question naming every field draws out every requirement, so the test measures whether an agent asks, not how well.

Third, the catalog leans toward English-language and American works: 83.2% of books are in English and 41.5% are from the US. Only one frontier model was tested.

How a recommendation team could try it

  1. Split your own conversational recommendation test set into three versions of each request: all stated at once, stated when asked, and one requirement never stated, and report success separately.
  2. Count how many failed dialogues asked nothing, and how many recommended something the user never confirmed.
  3. Try the cheapest change first: tell the agent in the prompt to ask before recommending, and to propose a candidate and wait for the user's verdict.
  4. If hidden requirements matter in your setting, explain in the prompt what "rather not say" and a rejection mean.

Verdict

In one line: when evaluating recommendation agents, test separately whether they ask on their own and whether they verify before recommending.

DimensionAssessment
Problem importanceHigh. Users not saying everything at once is the norm, and most benchmarks ignore it
NoveltyMedium-high. Holding the request fixed and changing only when it is stated is a clean design
EvidenceHigh. Nearly 33,000 dialogues, full statistical tests, all traces released
LimitationsMechanical simulated user, hard filters only, English-leaning catalog
Worth borrowing?Yes. The three-version split can be applied directly to your own evaluation

How did this read for you?

One tap, no sign-in; tap again to undo. Only anonymous counts are kept, never who you are.

Discussion

Add experiment details, reproduction notes or a different take. Requires a GitHub sign-in.

Loading the discussion…