Contents · 9 sections
1Where current recommendation agents get stuck2The simulated user only goes by "what I need this time"3How the reward works when hits are rare4Walking through it: the fishing-rod order5Inserting the target into the candidates during training6Results7Which behaviors training changed8What this means for people building recommendation agents9LimitationsSHARE
The shopper says "not telescopic." What should the recommendation agent do next? SimRec trains the whole conversation with a simulated userYou tell a shopping assistant that you want a high-quality fishing rod and reel combo with a smooth drag, solid construction and good value. The assistant runs a product search, pulls up a summary of your past purchases, and recommends a travel-friendly telescopic rod.
You reply: not telescopic, I want a one- or two-piece rod with a smoother reel and a sturdier build. Now the assistant has several options. It can search again with new words, open the details of one candidate to check it, or ask you another question. In the paper's own example, the assistant opened the details of the second candidate, confirmed it was a standard two-piece combo, recommended it instead, and this time it hit.
This paper, from Zhejiang University, Tsinghua University and Hikvision Research Institute, asks how such a "search, inspect, ask, recommend, revise after rejection" agent can learn what to do at each step. Most existing agents follow a hand-written workflow that fixes when to search, when to ask and when to recommend. Existing training touches only one step, such as rewriting the search query or ranking a given list, so whether the whole conversation succeeded never reaches the model that made the decisions. SimRec trains the whole interaction with reinforcement learning. A simulated user anchored on the current need, with no access to the target item, answers each recommendation with what is missing. The reward looks at the final hit, and also at how good the search results were and whether the tools were used validly. With Qwen3-4B as the agent, NDCG@100 rises from 32.23% to 35.42% on ESCI, a dataset with query-product relevance labels, and from 22.94% to 24.72% on Amazon C4, a set of complex shopping requests.
Let the outcome of the whole conversation decide when to search, inspect, ask and recommend, and a 4B agent beats much larger untrained models on the retrieval ranking metric; but the users it trains and is tested against are all simulated, and the paper has no real-user experiment.
Where current recommendation agents get stuck
An LLM-based recommendation agent has a few tools: product search, item details, and user preferences. It can also ask the user a question, and it can keep going after a rejection. The problem is that a human-written workflow fixes the order of these actions, so the model never learns from outcomes.
The paper is concrete about the consequences: such agents rarely check the candidates they retrieve, rarely look at the user's history, and rarely recover once a recommendation is rejected. A bigger backbone does not make these habits go away.
Why not just copy the reinforcement-learning recipe of search agents? A search agent's environment comes ready-made: the index returns the same results for the same query, search is the only tool, and the answer is either right or wrong. Recommendation has none of this. The environment is a user who reacts to every recommendation; evidence comes from several tools with different costs; and the outcome, a hit, is rare and does not say which step went wrong. So training a recommendation agent first requires three things: the user, the actions and the objective.

The simulated user only goes by "what I need this time"
Real users cannot supply the amount of interaction reinforcement learning needs, so an LLM plays the user. How it plays matters. A simulator that knows the target item tends to leak it. A simulator given a reconstructed full user profile can drift away from what the user actually wants to buy this time.
SimRec's simulated user is anchored on the current shopping need, which in the implementation is a reference query such as the fishing-rod request above. It also reads the item the agent just recommended or the question it just asked, plus the recent turns of dialogue. It never sees the target item's ID and gets no full user profile. After each recommendation, the prompt tells it to state one or two concrete unmet requirements in natural language, such as "not telescopic," and it must never reveal any product ID.
Whether a recommendation hit is decided by the environment, not by the simulated user: the environment checks the recommended item against the target set. The simulated user only talks; the environment keeps score.
Long-term preferences are not role-played by the simulator either. They are a tool the agent can call. The agent's actions, in no fixed order, are:
| Action | What the agent gets | What it is for |
|---|---|---|
| Product search | 100 results per query, of which the agent sees short snippets of the top 20 | Find candidates |
| Item details | Full information on one item | Check hard constraints such as price, size, compatibility |
| User preference | A one-sentence LLM summary of purchase history, with the target removed | Guide ranking and query wording |
| Clarifying question | The simulated user's answer | Fill in a missing key constraint |
| Recommend | The environment's verdict plus the simulated user's feedback | End on a hit, continue on a rejection |
A conversation ends in one of three ways: a hit, the recommendation attempts run out, or the turn limit is reached.
How the reward works when hits are rare
The most direct reward is 1 for a hit and 0 otherwise. That has two problems. The catalog is large and hits are rare, so most conversations get 0. And a 0 does not say what went wrong: the target may never have been retrieved, or it may have been retrieved and not chosen.
So SimRec adds up several parts and clips the total to between 0 and 2.5:
Hit reward: 1 if the final recommended item is in the target set, 0 otherwise.
Retrieval reward: among all "non-redundant searches" in the conversation, take the one where the target ranks best and compute its NDCG@100. Non-redundant means the query was not used before and its top 100 contains at least one new item. With a single target, this score depends only on the target's rank: rank 1 gives 1, and the score falls as the rank gets worse.
Tool reward: search, item details and user preference each earn a bonus once they are validly used, 0.03, 0.01 and 0.01 respectively. Using one type ten times still counts once, and invalid or repeated calls do not count.
Penalties: if the agent searches with the same query after the user's correction, it loses 0.05; past 4 turns, each extra turn costs 0.02, up to 0.20. There is also a tiny format bonus for each recommendation that contains a valid item ID and a description.
The training algorithm is GRPO. For each task the current model runs 8 conversations, each gets a total reward, and the model looks at how far each is above the group mean (divided by the group's standard deviation). That difference adjusts every token the model generated in that conversation. Within one conversation, the search queries, the detail calls and the final recommendation all share this one score. When an external LLM plays the user, only the recommender's tokens are updated.
Walking through it: the fishing-rod order
Here is the paper's Figure 2 case. The dialogue comes from the paper; the ranks, turn count and reward values are illustrative.
Step one, the user's need becomes a search. The agent writes the query "high-quality fishing rod and reel combo with smooth reel…", the retriever returns 100 items, and the agent sees the top 20, including Item1, Item2 and Item3. Illustration: the target ranks 3rd in these natural results, so the retrieval reward is 1 ÷ log2(3+1) = 0.5.
Step two, preferences shape the first recommendation. The agent calls the preference tool and gets "the user prefers adjustable, high-quality products," then recommends Item1, a travel-friendly telescopic rod. The environment finds Item1 is not in the target set, so the attempt fails and the conversation continues.
Step three, feedback changes the next action. The simulated user replies: "Not telescopic. I'd prefer a one- or two-piece rod with a smoother reel and a sturdier build." The reply names no product; it only says what is missing. The agent does not search again. It opens Item2's details and sees "a standard two-piece combo." Since there is no new search after the feedback, the repeated-query penalty does not apply.
Step four, recommend after checking. The agent recommends Item2, the environment marks it a hit, the conversation ends, and Hit@1 = 1.
Step five, this conversation's score is compared within its group. Illustration: hit 1 + retrieval 0.5 + tools 0.03 + 0.01 + 0.01 = 1.55; the conversation took 6 turns, 2 beyond the free 4, so it loses 0.04, for a total of about 1.51. Suppose another conversation in the same group did not inspect details after the rejection and pushed a different telescopic rod; it gets only 0.5 + 0.03 + 0.01 = 0.54. If the first is above the group mean, its tokens, including "inspect details after rejection," are reinforced; if the second is below the mean, its tokens are weakened.

The example shows two things. First, the simulated user's "not telescopic" is itself part of the training signal: it determines whether the agent has anything to check next. Second, the reward arrives only once, at the end of the conversation, so whether the agent's choice after the rejection was good can only be told apart by comparing conversations within the group.
Inserting the target into the candidates during training
Beyond the reward, one problem remains. The catalog is large, and after several searches the target may still not be among the 20 items the agent can see. Then the agent never gets to practice picking the right one among similar items.
SimRec's fix is a training-only shortcut: once the user has spoken at least 2 times and the target is still not in the visible candidates, the target is inserted at a random position in the candidate list. The position is random so the model cannot guess from it. The retrieval reward is always computed on the natural results without the insertion, so the inserted target earns no free retrieval credit. Insertion is turned off at evaluation.
During training, the share of conversations that needed insertion fell. In the earliest window, 38.18% of eligible conversations needed it; by the last window, 31.48%. Meanwhile the share of targets ranked in the natural top 10 rose from 61.81% to 68.52%, and the share with no target in the top 100 fell from 19.70% to 14.50%. This shows the agent's own queries kept getting better and it relied less on insertion.
Results
The paper evaluates on 5 ESCI categories and 4 Amazon C4 categories. NDCG@100 measures how high the target ranks in the search results, taking the best of all non-redundant searches; Recall@100 measures whether the target made the top 100; Hit@1 measures whether the final recommended item is the target.
| Method | ESCI NDCG@100 | ESCI Hit@1 | C4 NDCG@100 | C4 Hit@1 |
|---|---|---|---|---|
| Qwen3 Dense retriever alone | 32.62 | 19.98 | 19.80 | 6.45 |
| Qwen3-4B untrained | 32.23 | 21.20 | 22.94 | 8.30 |
| GLM-5.1 | 34.99 | 36.96 | 24.61 | 16.09 |
| Kimi-K2.6 | 34.50 | 40.40 | 22.69 | 16.26 |
| SimRec-4B | 35.42 | 24.08 | 24.72 | 9.76 |
(Numbers from the paper's Table 1, in %. GLM-5.1 and Kimi-K2.6 run untrained with the same tools and simulated user.)
This shows training did improve how well the 4B model finds candidates: its NDCG@100 is the highest in the table on both datasets, and the ESCI Electronics category gains 7.66 points. But for actually hitting the target, large models are still much stronger: Kimi-K2.6 reaches 40.40% Hit@1 on ESCI versus SimRec's 24.08%. On C4, BLAIR-LARGE, a retriever trained on Amazon data, also has higher Recall@100, 63.01% versus 59.18%.
When the retriever is swapped for the weaker BM25, SimRec's gain over BM25 alone is larger: ESCI NDCG@100 goes from 19.68% to 24.02%, and C4 from 7.68% to 15.88%. Rewriting queries over several turns makes up for part of a weak retriever.
Which behaviors training changed
Tool use changed the most. On the C4 test set, tool calls per conversation rose from 2.43 to 5.19. Most of the increase is item details (0.58 to 1.83 calls) and user preferences (0.02 to 0.86). This shows the trained agent is more willing to check items and consult the user, whereas the untrained model almost never looks at preferences.
Recovery after a rejection also changed. The paper calls the share of first recommendations that missed but ended in a hit the recovery rate. On ESCI, the untrained model's Hit@1 gains only 1.56 points from the first recommendation to the last, while SimRec gains 3.44. The recovery rate rises from 1.93% to 4.34%, improving in four of five categories, with Sports slightly down. This shows multi-turn feedback is actually being used, but only a minority of misses are recovered.

Item comparison was also tested on its own. The paper gives every model the same 16 candidates in shuffled order, hides the retrieval scores, and asks for a reranking. NDCG@10 rises from 24.75% untrained to 28.06%, against 19.73% for the random order. With identical candidates, this difference reflects the ability to compare items.
The ablations remove parts and measure the loss. On C4, removing the user-preference tool cuts NDCG@100 the most, by 3.55 points; removing item details cuts Hit@1 the most, by 3.72 points. Interestingly, without item details Recall@100 actually rises from 59.18% to 63.65%, yet fewer final recommendations hit. Retrieving more candidates does not mean picking the right one. Removing multi-turn feedback lowers both metrics.
The reward ablation is on ESCI, compared at training step 500. The full reward gives 32.38% NDCG@100 and 17.22% Hit@1. Without the hit reward they fall to 25.45% and 14.28%; without the retrieval reward Hit@1 falls to 14.58%; without the tool reward Hit@1 falls to 14.34%. The last point shows that even with the hit reward kept, rewarding "gather the evidence" still helped the final recommendation.
What this means for people building recommendation agents
If your recommendation agent calls tools in a human-written order, try letting the outcome of the whole conversation decide the order.
Start by looking at what your agent does after a rejection: does it check item details, does it consult the user's preferences, or does it reword and push a similar item? SimRec's untrained backbone almost never looks at preferences, and a bigger model alone may not fix that kind of habit.
When building the simulated user, give it the current need, the agent's latest recommendation or question and the recent dialogue, have it say what is missing, and keep the answer hidden from it; let the environment decide hits. Besides the final hit, give retrieval quality its own share of the reward; grant tool rewards once per tool type rather than per call. The paper designs it this way to encourage using tools validly rather than often.
Limitations
Both training and testing use LLM-played users. Whether real users talk and give feedback like this is not verified in the paper. Changing the simulated user changes the results: with GLM-5.1 playing the user at evaluation, NDCG@100 rises to 36.93%; with Kimi-K2.6, NDCG@100 and Hit@1 both drop.
Inserting the target during training but not at evaluation creates a mismatch between training and inference. The paper lists this as a limitation itself.
The self-play version, where the agent also plays the user, comes close to the external simulator on ESCI (34.72% versus 35.42%) but falls well short on C4 (19.77% versus 24.72%).
The final hit rate is still clearly below large models, and the experiments cover only two scales, 4B and 1.7B, and two datasets, ESCI and Amazon C4.
How did this read for you?
Hoot made a note. Thanks for the feedback.
One tap, no sign-in; tap again to undo. Only anonymous counts are kept, never who you are.
Discussion
Add experiment details, reproduction notes or a different take. Requires a GitHub sign-in.
Loading the discussion…