Contents · 9 sections
1What problem is it actually solving?2Overall framework: what each of the three parts does3Satisfaction reward: combining reformulation, gap and retention into one score4DRPO: shifting each query's advantages by one offset5TRAF: letting objective weights reference each other6Experiments: which conclusions hold up7Limitations as I see them8How a recommendation or search team could try this9One-line conclusion and assessmentSHARE
This is worth reading if you work on multi-objective fusion in search or recommendation ranking, especially if you want to wire long-term goals such as retention into fusion weights. Read it with one question in mind: much of the offline satisfaction gain is the model being scored by its own reward model. The paper addresses the fact that fusion weights in short-video search have been tuned on immediate feedback such as clicks and long plays, which does not match whether users are actually satisfied. The core idea in one sentence: whether the user reformulates right after seeing results, how long until their next search, and whether they search again the next day are combined into one satisfaction reward, and a modified GRPO learns which set of fusion weights each request should use. Over two months on 10% of Kuaishou search traffic, the query reformulation rate fell 0.319%, watch time rose 0.611%, and the retention gap over the baseline grew steadily from 0 to about 0.06%. I think the satisfaction reward is worth borrowing, but the gain from the algorithm's "dual-relative advantage" should be discounted.
What problem is it actually solving?
First, multi-task fusion (MTF). A ranking model usually does multi-task learning (MTL) first, predicting several scores at once such as click rate, long-play rate, watch time and relevance; MTF then combines them into one final score that decides the order. The paper uses a weighted sum:
is video n's predicted score on objective j, and the log flattens scale differences between objectives; are the fusion weights, which must sum to 1 and each have bounds so no objective is fully suppressed. All candidates in one request share the same weights, but different users and queries can get different ones. So what is learned is a policy: given the user and query, output a set of weights.
An illustrative scenario shows the difference between two weightings. A user searches "how to make braised pork", and the candidates include a 15-second viral clip and an 8-minute full tutorial. With a high click weight, the viral clip ranks first; the user clicks and watches it, learns nothing, and reformulates to "braised pork detailed steps". With a high watch-time weight, the tutorial ranks first; the user watches it and leaves, then comes back the next day to search for another dish. Judged by immediate clicks the first is better; judged by satisfaction the second is.
Why not reuse recommendation's reinforcement learning (RL) approach to fusion weights? The authors give two reasons. Search has explicit intent and relevance constraints, so content cannot be swapped freely for exploration. And data is sparse: at Kuaishou, positive feedback density in search is about one tenth of the recommendation feed, and search behavior sequences are two orders of magnitude shorter. This means long-term retention is hard to attribute to any single ranking in search, so a satisfaction signal closer to each request is needed.
Overall framework: what each of the three parts does

| Module | Input | Output | Solves |
|---|---|---|---|
| Satisfaction reward model | query, user features and the ranked list | a satisfaction score in 0 to 1 | turns sparse long-term signals into a reward every request has |
| DRPO (Dual-Relative Policy Optimization) | several queries in a batch, several weight sets sampled per query | policy gradient | stable policy updates under sparse feedback |
| TRAF (Task-Relation-Aware Fusion) | policy hidden state and per-objective logits | refined weights | lets objective weights reference each other instead of being output independently |
The policy network is small, 0.91M parameters, with 4.5 ms online inference.
Satisfaction reward: combining reformulation, gap and retention into one score
The authors take two query-level signals from search logs. One is reformulation: after seeing the current results, the user immediately issues a semantically related new query, which usually means they did not find what they wanted. The other is the session gap: the time from the current query to the user's next query. Shorter gaps usually mean more engagement. The paper's Figure 3 shows retained users reformulate less and have shorter gaps.
The gap cannot be used raw, because it strongly tracks user activity: heavy users naturally have short gaps. The authors first normalize by each user's own 60th-percentile historical gap to get , then apply exponential decay. The final satisfaction reward is:
marks reformulation, marks searching again the next day, is a temperature controlling decay speed, and . The leading factor means any reformulated request scores 0.
A hand calculation (illustrative, ). If this gap equals the user's own 60th percentile, and ; if they search again the next day, satisfaction is . If the same user's gap is 3 times their usual, , and they do not return the next day, satisfaction is only about 0.02. If they reformulate right away, it is 0 regardless of what follows.
These signals exist only after the fact in logs and are unavailable at ranking time, so the authors train a list-level reward model : given the query and ranked list, predict the satisfaction the list will get. It is trained as a regression onto this satisfaction label.
How is chosen deserves a note. The authors sweep from 0 to 1 and compute the correlation of with the normalized gap and with retention; their sum peaks at , about 1.45. This criterion naturally favors an even split, because is itself a weighted sum of those two terms. It shows 0.5 is not arbitrary, but not that 0.5 is best for online retention.
DRPO: shifting each query's advantages by one offset
First, what an action is. The authors split each weight's bounded interval into bins and keep only bin combinations whose sum is within a tolerance of 1; the policy is a categorical distribution over this feasible set. This does not quite match the task-relation section's description of weights as a softmax over per-objective logits; one reasonable reading is that the network's outputs set the probability of each feasible combination, which is then sampled. That is my interpretation. GRPO samples G weight sets for a query from that distribution, ranks a list with each, computes rewards, and subtracts the group mean and divides by the group standard deviation to get advantages. The authors see a problem with group-only comparison: if a query is inherently hard and every weight set ranks badly, the best in the group still gets a positive advantage and is reinforced as a good action.
DRPO (Dual-Relative Policy Optimization) adds a "batch-relative" advantage:
and are the mean and standard deviation of rewards within query i's group, and and are the mean and standard deviation of the group means across the batch. The second term is the same for all actions of one query.
An illustrative hand calculation: two queries in a batch, 3 weight sets each. A is an easy query with rewards 0.9, 0.8, 0.7; B is a hard one with 0.4, 0.3, 0.2. Both groups have within-group advantages of about +1.22, 0, −1.22. Group means are 0.8 and 0.3, the batch mean is 0.55 and the standard deviation 0.25, so the second term is +1 for A and −1 for B. Combined, A's three sets are +2.22, +1, −0.22 and B's are +0.22, −1, −2.22. B's best set drops from +1.22 to only +0.22.
The authors prove that this term leaves the advantage gap between two weight sets of the same query unchanged and only scales the whole query's gradient. I would add one angle: in policy gradients, adding the same constant to every action of a state is a change of baseline and leaves the expected gradient unchanged. This means the second term carries no new direction in expectation; what it actually does is a finite-sample effect. On easy queries, all sampled actions are pushed up and the policy concentrates; on hard queries, all sampled actions are pushed down and probability flows to unsampled actions, which amounts to more exploration. So DRPO behaves more like adjusting exploration by query difficulty, and difficulty depends largely on the query itself. An ambiguous query rarely gets a high reward whatever the weights, so the policy keeps being pushed to explore on such queries.
The composite reward is a plain sum of three parts: an immediate engagement reward, the satisfaction reward, and format constraints. The engagement reward is an NDCG-style ranking score: the higher the videos the user clicked or long-played sit in the list, the higher this term. There are two constraints: weights summing to more than a tolerance away from 1 cost 1 point, and any video below a relevance threshold in the top k costs 2. The latter writes search's relevance requirement as a hard penalty, a real difference between search and recommendation.
TRAF: letting objective weights reference each other
Previous approaches have the policy network output one logit per objective independently. TRAF (Task-Relation-Aware Fusion) projects the hidden state into one vector per objective, takes pairwise inner products as task-relation scores, and refines each objective's logit as
where is objective 's original logit, is the relation score between objectives and , and is objective 's own gate coefficient. The first term keeps the objective's own logit; the second is a relation-weighted sum of all objectives' logits. The refined then goes through a softmax to give the weights. Intuitively, when the long-play weight rises, the watch-time weight should often move with it, and TRAF lets the network learn that coupling explicitly. It adds only 0.04M parameters and 0.2 ms of inference.
Experiments: which conclusions hold up
Offline data comes from Kuaishou search: 400M users, 806M videos and 2.15B sessions, split 9:1. Metrics are NDCG@10 on each of four objectives, plus a satisfaction score and retention. Main results, excerpt (paper Table 1):
| Method | Type | NDCG average | Satisfaction score | Retention |
|---|---|---|---|---|
| LambdaRank | learning to rank | 0.6743 | 0.7421 | 0.7112 |
| CEM | black-box weight search | 0.6454 | 0.7206 | 0.6824 |
| DDPG | general RL | 0.6884 | 0.8132 | 0.7356 |
| SAC | general RL | 0.7004 | 0.8343 | 0.7865 |
| AURO | retention-oriented RL | 0.7040 | 0.8467 | 0.8026 |
| SaFRO | this paper | 0.7102 | 0.8656 | 0.8173 |
First the trustworthy part. Average NDCG is 0.0062 above the strongest baseline, AURO (an RL method optimized for retention), about 0.9% relative, and SaFRO is best on all four objectives. These come from real logs and are fairly reliable.
Then the part to discount. The satisfaction score comes from the same satisfaction model SaFRO uses as its training reward, and retention is predicted by a simulator trained on the same data. This means offline satisfaction and retention measure how close each method is to this reward model's preferences, and SaFRO, which optimizes that model directly, is expected to win. The ablation makes this very clear:

| Variant | Click NDCG | Duration NDCG | Relevance NDCG | Satisfaction score | Retention |
|---|---|---|---|---|---|
| Full SaFRO | 0.6431 | 0.6871 | 0.7085 | 0.8656 | 0.8173 |
| No satisfaction reward | 0.6312 | 0.6728 | 0.7032 | 0.7221 | 0.7384 |
| No batch-relative advantage | 0.6287 | 0.6621 | 0.6991 | 0.8478 | 0.7952 |
| No TRAF | 0.6294 | 0.6698 | 0.7114 | 0.8515 | 0.8089 |
Without the satisfaction reward, the satisfaction score falls 0.14 while NDCG falls only about 0.01, because the satisfaction score is produced by that very reward model. Without the batch-relative advantage everything drops, but per the previous section it does not change the expected gradient direction, and only single runs are reported, so how much of the gap is random variation is hard to tell. Removing TRAF actually raises relevance NDCG from 0.7085 to 0.7114, which means TRAF's coupling trades a little relevance for the other objectives.
In the hyperparameter study, raising group size from 2 to 32 lifts the satisfaction score from 0.7912 to 0.8656, so more candidates per group make comparison more effective; but NDCG peaks at and dips slightly at 32. The entropy coefficient has an inverted-U effect: NDCG, satisfaction score and retention all peak at , and retention falls to 0.7425 at .
The online test is the most convincing. On 10% of search traffic for two months, the control is a DDPG (an actor-critic RL algorithm) fusion policy optimized for immediate engagement:
| Metric | Reformulation rate (lower is better) | CTR | Long-play rate | Watch time |
|---|---|---|---|---|
| SaFRO vs baseline | −0.319% | +0.136% | +0.495% | +0.611% |

The reformulation rate is directly penalized in the reward, so its drop means fewer "didn't find it" cases online. The retention gap widens over time rather than jumping at launch, which is consistent with satisfaction accumulating into retention.
Limitations as I see them
First, offline evaluation is circular. The satisfaction score comes from the training reward model and retention from a simulator, both heavily overlapping SaFRO's objective. The only independent evidence is NDCG and the online A/B, and the NDCG lead is only about 0.9%.
Second, the online control is weak. The baseline is DDPG optimized for immediate engagement, which in the offline table trails AURO and RLUR (another retention-oriented RL method) by a lot, 0.7356 vs 0.8026 retention. Online there is no comparison with a method that also optimizes retention, so you cannot tell whether the gain comes from adding the satisfaction reward or from DRPO and TRAF. Figure 8's axis is in percent; a gap of about 0.06% is not trivial at 400M daily users, but the absolute size really is small.
Third, the theory behind the batch-relative advantage is thin. As discussed, adding the same constant to every action of a query leaves the expected gradient unchanged. Its roughly 0.02 retention gain in the ablation comes from one run with no variance across seeds. Also, the assumption that shorter gaps are better does not always hold in search: quickly issuing an unrelated new query is not a reformulation and gets counted as satisfied.
How a recommendation or search team could try this
- Start with the satisfaction reward alone. Combine "reformulation scores 0, gap normalized by the user's own percentile, next-day return" into a query-level label, check on your own data whether it correlates with long-term retention, then decide whether to train a reward model.
- In offline evaluation, score satisfaction metrics with separate data or a different model, not the one used as the training reward; otherwise every comparison is biased toward whichever method optimizes it directly.
- To try DRPO's batch term, run it and plain GRPO with 3 to 5 random seeds each, compare means and variances, then see whether the gain is significant.
- Use your current strongest fusion policy as the online control, run at least a full month, and look at the daily retention gap curve rather than only the last day.
One-line conclusion and assessment
The most borrowable point: treat "did the user reformulate immediately" as a veto on satisfaction, then add the gap normalized by the user's own rhythm and next-day return, and search too can have a long-term reward available on every request.
| Dimension | Assessment |
|---|---|
| Problem importance | High. Fusion weights decide the ranking, and search has long lacked an optimizable long-term satisfaction signal |
| Method novelty | Medium. The satisfaction reward is novel; DRPO's batch term and TRAF are small changes |
| Experimental evidence | Medium. The two-month online A/B is credible, but offline satisfaction and retention are self-scored and the control is weak |
| Engineering cost | Low. A 0.91M-parameter policy with 4.5 ms inference; the main cost is maintaining the reward model |
| Worth trying | The satisfaction reward, yes; run multi-seed controls on DRPO's batch term before adopting it |
| Fits | multi-objective fusion in short-video and content search, and other ranking settings with explicit intent and sparse feedback |
How did this read for you?
Hoot made a note. Thanks for the feedback.
One tap, no sign-in; tap again to undo. Only anonymous counts are kept, never who you are.
Related papersMore from Paper Daily
Personalization · 2026-07-22Uncertainty as Remedy: Mitigating Satisfaction Label Bias in Short Video Multi-Objective Ensemble RankingDeep dive: Uncertainty as Remedy: Mitigating Satisfaction Label Bias in Short Video Multi-Objective Ensemble Ranking.
Generative · 2026-07-31Taiji: Pareto Optimal Policy Optimization with Semantics-IDs Trade-off for Industrial LLM-Enhanced RecommendationDeep dive: Taiji: Pareto Optimal Policy Optimization with Semantics-IDs Trade-off for Industrial LLM-Enhanced Recommendation.
Generative · 2026-06-05Bridging Short Videos and Live Streams: Reasoning-Guided Multimodal LLMs for Cross-Domain Representation LearningDeep dive: Bridging Short Videos and Live Streams: Reasoning-Guided Multimodal LLMs for Cross-Domain Representation Learning.
Retrieval · 2026-10-28ACARE: Agent-Context-Aware Reranker for Iterative SearchSearch agents often send sub-queries of just a few keywords, so a sub-query-only reranker promotes documents that match the words but do not help the real question. ACARe gives the reranker the original question and the agent's reasoning too. The key control: without changing Qwen3-Reranker-4B's weights, adding the question alone lifts a small agent's accuracy from 48.2% to 54.8%, and fine-tuning…
Retrieval · 2026-08-05From Relevance to Authority: Authority-aware Generative Retrieval in Web Search EnginesDeep dive: From Relevance to Authority: Authority-aware Generative Retrieval in Web Search Engines.
Retrieval · 2026-08-05SPEAR: Selection-aware Personalized End-to-end Adaptive Rewriting and Retrieval for Community SearchDeep dive: SPEAR: Selection-aware Personalized End-to-end Adaptive Rewriting and Retrieval for Community Search.
Discussion
Add experiment details, reproduction notes or a different take. Requires a GitHub sign-in.
Loading the discussion…