AutoSim: A Self-Improving User Simulator for Conversational Search Evaluation

Ask a strong LLM to play a search user and it writes 37.5 words per message on average, while real users write only 12.8. A discriminator that never saw the training data recognizes its utterances with AUC 0.88, and in the same turn, the top 5 passages its query retrieves barely overlap with the top 5 retrieved by the real user's query (overlap@5 is only 0.01). The conversational search user simulator AutoSim, studied in this SIGIR-AP 2026 paper, is in exactly this state: its rankings of retrieval systems agree with NIST in three TREC iKAT years (Kendall τ of 0.66, 0.91 and 0.91), yet the way it talks is far from human. The authors constrain the simulator's self-improvement with held-out real-user data, and the conclusion is clear: changing only the prompt cannot push discriminator AUC below 0.85; changing the model weights brings AUC down to 0.45, to the point that people reading whole conversations cannot tell. More importantly, sounding human changes evaluation conclusions. Where real users retrieve poorly, a prompt-driven simulator overestimates nDCG by 0.136, enough to change a launch decision.

1. The problem: can a simulator be trusted, and what should change to make it more human?

With a user simulator, a conversational search system does not need a human study for every design change, and tuning a ranker or a clarifying-question policy can be compared reproducibly. But the simulator's persona, prompts and rules are fixed at design time, and when the system under evaluation changes, the simulator does not change with it. Prior work also found that it is unreliable in itself: swapping only the underlying user model swings the evaluated agent's task success rate by about 9 percentage points, and general LLM simulators systematically overestimate success. And a more human-like simulator is not necessarily a more accurate evaluator.

The authors therefore ask three questions. First, can AutoSim rank retrieval systems the way NIST does? Second, can self-improvement narrow its behavioral gap with real people, and should it change the prompt or the weights? Third, once the gap narrows, is it a better evaluator? The authors first run a diagnosis and find that the default simulator's relevance judgments already agree with NIST; the gap lies in "what the user says and asks", and changing the generation prompt cannot close it. So the overall design is: leave the relevance judge untouched and rewrite only the utterance generator, at the weight level.

2. Method: AutoSim and its self-improvement loop

At its core AutoSim is still an ordinary LLM generator, wrapped in a layer of user state. The state holds a persona, the current information need, the conversation history, and latent emotions such as satisfaction, frustration and patience. The persona comes from iKAT's personal text knowledge base (PTKB), that is, the set of statements describing the user's background attached to each topic. In each turn an orchestration policy decides who acts: routine actions like rewriting the query, clicking and stopping go to rule-based deterministic simulators; answering clarifying questions, giving feedback, expressing satisfaction and writing the next utterance go to the LLM.

Two parts of the simulator can be changed. One is the relevance-judgment instruction, which decides whether the simulator is satisfied with results and therefore decides how it ranks systems. The other is the utterance generator, the model that writes what the simulated user says; it can be changed through its prompt or its parameters, and this is the core comparison of the paper.

To measure how far the simulator is from real people, the authors combine "does the system ranking agree with NIST" and "does the behavior distribution match real people" and call it the Conversational-search Sim2Real Gap (CSG). The communication-style part is measured in two ways. One looks at statistics such as average word count, the share of polite phrasing and the share of very short messages; these are easy to read and also easy to write directly into a prompt. The other trains a discriminator on held-out data to tell real utterances from simulated ones and reports ROC AUC, with 0.5 meaning indistinguishable. The latter is an adversarial standard that cannot be passed by matching a few numbers. As we will see, the two measures reach different conclusions, and that is itself a result.

Each round of the self-improvement loop first records the simulator's full trajectory, including what action it chose each turn, what it retrieved, what it clicked and what it said. It then clusters the differences between simulated and real behavior into a few groups (say, "messages too long" or "too polite"), and has a model from a different family than the generator write an evaluator for each group that scores simulated trajectories on that difference. Improvement uses reflective population search: looking at the gaps the evaluators point out, this model writes N candidate edits at once (for example, rewrites of the generation prompt). Each is scored by these evaluators on a calibration set (the paper does not say how the per-cluster scores are combined), and the best is kept only if its score is no lower than before the edit; otherwise nothing changes that round. A rolling held-out set is watched to see the effect but plays no part in accepting edits.

The weight-level change is more direct. The authors first LoRA-fine-tune Qwen2.5-3B on real iKAT utterances as a starting point, then run 4 rounds of adversarial rejection sampling. The held-out turns here are the 82 iKAT turns not in the fine-tuning data; the discriminator's AUC is also computed on them, with five-fold cross-validation grouped by turn, and they differ from the rolling held-out set the prompt loop only watches. Each round generates several candidate utterances for each held-out turn, keeps the ones the frozen discriminator considers most human, fine-tunes further on them, and retrains the discriminator.

The evaluation uses three years of data, iKAT 2023, 2024 and the multi-session 2025, with 176, 116 and 45 human-judged turns respectively, all based on the ClueWeb22-B corpus and NIST passage-level relevance labels. Each year the authors build 12 retrieval systems from these parts: three first-stage retrievers (BM25, SPLADE, RRF), three query forms (the raw query, a context-resolved query, and a persona-expanded query), and with or without a cross-encoder reranker. The 12 combinations are picked so that effectiveness spans from very weak to very strong. 2023 also includes two real official runs, for 14 in total. The reference ranking uses nDCG@5 on NIST labels, computed only over judged passages (the paper's condensed nDCG@5) so that systems outside the judging pool are not penalized for finding unjudged relevant passages. AutoSim's ranking comes from having it judge, in persona, whether it is satisfied with each of a system's top 5 passages (satisfaction is set by its relevance-judging prompt), averaging per system and sorting by the average. To avoid depending on closed APIs, all validation uses two open instruction models from different families (an 80B-class Qwen and a 31B-class Gemma) at temperature 0.

3. First question: it is a valid evaluator

On the 12 iKAT 2023 systems, the τ between AutoSim's and NIST's rankings is 0.66, statistically significant. The result is also stable: dropping each of the 12 systems in turn gives τ between 0.59 and 0.73, and having AutoSim re-run and re-score the whole pool five times moves τ by only ±0.01. So the agreement is not propped up by one system or by one lucky run. With the two official runs added, τ is 0.65. Like NIST, AutoSim puts the weak automatic-rewrite run last; the biggest disagreement is the strong run that uses manually rewritten queries, which NIST ranks 3rd and AutoSim ranks 10th.

On iKAT 2024 and 2025, τ reaches 0.91. On TREC CAsT 2020, which has no personas and a different corpus, τ drops to about 0.56, so it transfers, but at a discount. Where does this ranking ability come from? The authors run two controls. Without personas, the relevance judge's τ is 0.79, about the same as with personas; with a judge from another family (Gemma instead of Qwen), the 2024 ranking does not change at all. So valid ranking comes from the strong LLM already knowing how to judge relevance, not from the persona. Its weakness is systems of similar quality: looking only at the 6 closest systems, τ for 2023 drops to 0.33, which is no longer significant.

4. Second question: prompts have a ceiling, weights do not

First, changing only the prompt. Across 12 random seeds, the full loop with calibration checks raises communication-style fidelity on held-out data (how close three statistics are to real users: mean words per message, share of messages with pleasantries, and share of very short messages; higher is more human-like, and the paper does not give the formula combining them) from 0.56 to 0.76, a significant gain. Without calibration checks it reaches only 0.63, not significantly different from no improvement, which shows the calibration check is necessary. But a closer look shows the loop learned the wrong thing: average word count did fall from 37.5 to 11.0, but the share of polite phrasing shot up to 39%, far above the 12% of real people, and it never learned the 5% of very short messages real people write.

An even more telling comparison follows. A one-line hand-written prompt, "write a short message of about 12 words and skip the pleasantries", scores higher on the same turns (0.80) than the prompt the loop found (0.68); these turns are not the setup behind the 12-seed mean of 0.76 above, which the paper does not detail, so the two figures are not directly comparable. The hand-written prompt wins because writing three statistics directly into a prompt is easier than reaching them through search. A few-shot prompt that gives real short utterances as examples already matches the length (12.3 words), yet discriminator AUC is still 0.94. In other words, the statistics match, and the discriminator still spots it at a glance.

Changing the weights works completely differently; discriminator AUC changes as follows:

The authors rule out two alternative explanations. One is that the discriminator is too weak. A separately trained XLM-RoBERTa discriminator recognizes the default simulator almost perfectly (AUC 0.94) but gets only 0.51 on the weight-level simulator; and it recognizes real utterances with just a little polite phrasing added at 0.99, so the discriminator is not weak. The other is that the result holds for only one model. With Qwen2.5-7B or Mistral-7B as the generator, the default version is almost fully exposed and the weight-level version sits around 0.5.

Human judgment agrees. Three annotators read whole conversations and judge whether a simulator wrote them, 30 trials each, where random guessing averages 15 correct. "Recognize" means correctly judging that a conversation came from the simulator; the main text does not say what each trial shows (one conversation, or a human and a simulated one side by side). They recognize the default simulator 28 to 30 times out of 30, but the weight-level simulator only 4 to 11 times, all below 15, that is, worse than guessing. Whether queries target the information need also shows the difference. iKAT provides for each turn a rewrite that fills in the context and stands on its own, representing the turn's full need. The authors measure word overlap between the simulator's next-turn query and that next turn's rewrite (the paper does not say whose word count is the denominator): real people score 0.56, the weight-level simulator 0.51, and the default simulator only 0.23, because pleasantries dilute the need. The shortest, terse prompt (9.6 words) is actually the worst at 0.19, which shows a short query is not the same as a focused one.

The relevance judge is different again. Under calibration checks, no edit to the judge prompt was accepted; without checks, edits moved τ by at most about 0.1, within sampling noise. So self-improvement can change how the simulated user talks, but not how it judges relevance.

5. Third question: sounding human changes evaluation conclusions

The system-ranking τ did not change, because the ranking was already right. The difference shows up in something training never optimized: how good retrieval actually is when using the queries the simulator writes itself. On 157 held-out turns, the authors run BM25 with each simulator's queries and compare nDCG@5 with that of real users' queries. In the table, "length-matched truncation" cuts the default simulator's verbose query down to about real-user length (11 words on average); the paper does not say whether each is cut to its matching real query or all to one length.

The 2024 data is harder and was entirely unseen in training. Real users' average nDCG here is only 0.016; the default simulator overestimates it by 0.136, the terse prompt and truncation to the same length still overestimate by seven to nine points, and only the weight-level simulator's bias is not significantly different from zero. This shows the problem is not length: length explains only 2% of the variance in retrieval effectiveness. On the 2023 held-out set, where real users retrieve reasonably well, no simulator has a detectable bias. So the overestimate appears exactly where real users struggle most and where evaluation most needs to be trusted.

This bias also cannot be removed with a single overall correction. In these 2024 turns, real users found nothing relevant in 86% of cases. On the turns where people failed, the default simulator's queries still retrieved relevant passages 57% of the time, while the weight-level simulator did so only 14% of the time, meaning it fails in the same places people do. In terms of an actual decision: with a launch threshold of nDCG 0.05, evaluating 2024 with the default simulator reads 0.15, well over the line; real users get only 0.02, and the weight-level simulator reports 0.03, so both say this system should not launch.

6. Limitations and practical value

The authors list several limitations. The system pool is built by the authors plus two official runs, not a full set of participating teams. The behavioral reference is iKAT utterances written by organizers, which only stand in for users in real logs. Human discrimination involves only three annotators. Ranking validity holds only for systems with clear quality gaps, and near-ties are hard to separate; 2025 has only 45 turns, and barely significant results there should be read as descriptive only. Behavior was measured only for communication style, query reformulation and clarification; clicks, satisfaction and when to stop are not yet measured. Note also that the weight-level "self-improvement" actually relies on real utterances as the LoRA starting point and is not fully self-bootstrapped.

For teams doing offline evaluation of search or conversational products, the paper offers two directly usable reminders. First, judging simulator quality cannot rest on statistics like word count and politeness rate, nor on system-ranking correlation alone: a simulator with high τ can still overestimate absolute effectiveness by a wide margin. It is best to add a learned discriminator and also check how far the retrieval effectiveness of simulated queries is from that of real people. Second, if you have a batch of real user utterances, LoRA-fine-tuning a small model makes simulated users more human than repeatedly polishing prompts; for the relevance judge, a strong model works as is and is not worth tuning. The framework, protocol, LoRA adapters and the human discrimination study tools are all open source.