KL-MOPD: Discrepancy-Aware Multi-Objective On-Policy Distillation for Generative Recommendation

When a generative recommender has to serve both clicks and value, a common recipe is to fine-tune two single-objective teachers, one for clicks and one for conversion or GMV, and then distill both into one student. At every generated token the student is pulled toward both teachers with equal weight. This paper from Zhejiang University and Alibaba shows that the equal split is unfair. On TencentGR, at the first token the student's KL divergence to the value teacher is 3.13 times its divergence to the click teacher, and after equal-weight training it is still 3.38 times. The harder teacher also changes along the sequence: the value teacher is harder on tokens 1 to 3, the click teacher on tokens 4 and 5. Their method, KL-MOPD, changes one thing. At each token it weights each teacher by how far the initial student is from it, so the farther teacher gets more weight. On an industrial ad dataset with about 1.49 billion interactions, the joint metric is 3.19% higher than equal weighting, and in a 10% traffic online A/B test ad revenue rose 1.63% and GMV 2.20%.

Teams doing multi-objective generative recommendation, or distilling from several teachers, should read it because the change costs almost nothing. The weights are computed once before training, there is no extra forward pass, and no new module, so an existing distillation pipeline only needs a different loss weight. The more useful contribution is the observation behind it: on all three public datasets, the equal-weight student is worse on the joint metric than the best single-objective teacher. After equal-weight distillation, TIGER has a higher joint metric than the initial student but remains below the best single-objective teacher. Keep in mind that the public experiments use a tiny model with two encoder and two decoder layers, absolute numbers are very low, and on industrial data the method is compared only with equal weighting.

What problem does it solve?

A generative recommender encodes every item as a short sequence of tokens, a Semantic ID (SID), and generates them one by one. This creates a hard constraint: if any single token is wrong, the item cannot be retrieved. Multi-objective optimization therefore cannot just align item-level probabilities; every token has to be learned well.

Consider an illustrative ad recommender in which each ad has 5 tokens. Early tokens pick a broad category, later tokens narrow down to a specific ad. The click teacher favors eye-catching, cheap ads; the value teacher favors high-conversion, high-basket ads.

With equal weights at every step, the student drifts toward whichever teacher is closer, because that loss is easier to reduce. The result is that early steps under-learn the value teacher and late steps under-learn the click teacher. Each objective is missed at some step, and since any wrong step loses the item, both objectives lose recall.

The paper measured the real per-token KL on TencentGR before training (Table 1):

The value teacher is farther on the first three tokens and the click teacher on the last two, matching the illustrative shape.

Background: multi-objective on-policy distillation

The starting point is multi-objective on-policy distillation (MOPD), a method used for large language models to merge several expert models into one. It has three steps.

Why learn on the student's own paths? Because at serving time the student follows its own paths. Learning only on teacher-generated or logged sequences creates a train-serve mismatch. The paper also proves that, if the teachers are optimal KL-regularized policies, this loss is equivalent to directly maximizing click reward plus value reward minus a penalty for drifting from the initial model.

Why align per token? Classical multi-objective methods align the probability of the whole item, which is the product of token probabilities. The whole product can match while one step still has low probability, and decoding goes wrong at exactly that step.

KL-MOPD: weight each teacher by distance

KL-MOPD only changes the weights. First run the initial student, the same pretrained model from which both teachers were fine-tuned, over the data and compute the average KL $d_{j,t}$ from the student to teacher j at token t. The weight of teacher j at token t is
$${\alpha }_{j,t}=\frac{d_{j,t}}{d_{c,t}+d_{v,t}}$$
where c is the click teacher and v the value teacher. The farther teacher gets the larger weight. When both are equally far, the weights return to one half each and the method reduces to plain MOPD.

A hand calculation with Table 1: at token 1 the value teacher's weight is 0.01995 / (0.00637 + 0.01995), about 0.76, so the click teacher gets about 0.24. At token 5 it flips: the click teacher's weight is 0.05058 / (0.05058 + 0.01467), about 0.78. The same student mostly catches up with the value teacher early and with the click teacher late.

Why use distances measured before training instead of updating them during training? The authors' reason is cost. The weights depend only on the initial student and two fixed teachers, so they are computed once; training needs no extra forward pass and no extra memory. The dynamic-weighting baseline TrustMOPD updates weights at each step, trains more slowly per step in the paper's speed comparison, and scores lower.

The theory gives a lower bound: the improvement of KL-MOPD over equal weighting on the joint objective is at least a positive term minus a constant, and the positive term grows with the squared gap between the two teachers' distances at each token. The intuition is simple: the more the two teachers differ in difficulty, the more weighting by difficulty helps. The authors plug in estimates from TencentGR and verify that the condition for a positive bound holds.

Does it actually work?

The metric is objective-weighted recall: for the click objective, retrieved items are weighted by click reward; for the value objective, by conversion, GMV or eCPM. The joint metric averages the two. The table shows the joint metric at top 10 with a TIGER backbone.

The third row is the most striking. The equal-weight student is below the best single-objective teacher on all three datasets, which means that with two teachers the student under-learns a different teacher at different tokens and ends up good at neither. KL-MOPD achieves the best joint metric in every setting in Table 3, by 4.12% on average over the strongest baseline or teacher. With a second backbone, REG4Rec, KL-MOPD still achieves the best joint metric. However, equal-weight distillation already beats both teachers on the joint metric on Taobao at top 10.

Are the distances really evened out? Table 1 also reports post-training KL. Equal-weight training cuts item-level KL to each teacher by more than half; KL-MOPD reduces it by a further 13.70% on average, and on the first three tokens the distance to the value teacher is clearly smaller than under equal weighting. So the gain comes from learning the harder teacher better.

Why per token, and why KL? The ablation on TencentGR and YooChoose swaps the weighting rule:

On TencentGR a single weight per item gives almost nothing, because the harder teacher switches midway and one weight can only lean one way. Restricting KL to the top 16 tokens also fails, because teacher disagreement often sits on tokens the student does not rank highly, and truncation hides it.

The industrial validation is more convincing because model and data are real-scale: about 1.49 billion interactions and 8 tokens per ad. The joint metric at top 100 goes from 0.3139 with equal weighting to 0.3236. Online, over 5 days with 10% traffic per arm, ad revenue rose 1.63% and GMV 2.20%, so both reported online business metrics improved.

Limitations

First, the public models are tiny and absolute numbers are low. The public experiments use a two-layer model with hidden size 64, and the joint metric on TencentGR is around 0.02, so relative gains there should not be extrapolated to large models. The industrial and online results carry more weight.

Second, on industrial data the method is compared only with equal weighting. The authors say resource limits kept them from running the other seven baselines there, so "better than every method" is established only on the small public setups.

Third, only two objectives are handled. The weight formula extends directly to three teachers, but the theory and all experiments cover two, and the authors list more objectives as future work. Real ad systems usually also care about watch time, diversity and more.

How a recommendation or search team could try it

1. If you already distill a generative model from several teachers, run the initial student over a validation set and compute per-token KL to each teacher. If the distances are similar at every token, the gain will be small and you can skip this.
2. If they differ clearly, replace the equal weights in the loss with KL-proportional weights and change nothing else. The only cost is one extra inference pass before training.
3. When evaluating, compare not only against the joint baseline but also against the best single teacher. If the distilled student is worse than one teacher, the weighting is the problem.
4. Online, first watch metrics that depend on both objectives, such as revenue and GMV, then split clicks and conversions to check that neither objective was sacrificed.

Bottom line

When several teachers teach together, which one to listen to at each step depends on which one the student is farther from at that step. The weights are computed once before training. In the reported offline experiments, the distilled student outperforms both single-objective teachers on the joint metric.