Unifying Behavior Modeling and Semantic Generation for Generative Recommendation

Generative recommendation gives every item a string of Semantic IDs and has the model generate the next item's code token by token. This KDD 2026 paper from South China University of Technology finds a systematic bias in that approach: the fewer items in a user's history that belong to the same category as the target, the worse it does compared with old-style sequential recommenders. On test samples where zero or less than a tenth of the history shares the target's category, TIGER (the representative model that first used Semantic IDs for generative recommendation) trails SASRec (the classic self-attention sequential recommender that looks only at behavior) by 6% to 30%. The authors propose OMG (a generative recommender that unifies behavior modeling and code generation), putting behavior modeling and code generation in one model. What really opens the gap is the few steps of "latent reasoning" placed between the encoder and decoder, which move the representation step by step from the behavior space to the semantic space: adding this one component lifts every metric across three datasets by another 14.5% on average, more than all the other components combined. In the end OMG beats the strongest baseline on all four public datasets.

The value of this paper is that it quantifies the intuition that "generative recommendation favors same-category items" and offers a very cheap fix, worth a look for anyone doing Semantic ID generative recommendation. The reason: the reasoning component takes only 0.46% of the model's non-embedding parameters and four reasoning steps add only 0.16% compute, so it plugs into an existing TIGER-style model with almost no structural change. But all the evidence comes from public datasets with tens of thousands of users, and the last two components bring small, unstable gains, so it fits better as a cheap offline experiment than as a launch plan to copy directly.

A "beer and diapers" style example

Start with an example (illustrative). A user has recently viewed these toys in order; the first token of the Semantic ID roughly corresponds to a coarse product category:

Looking only at semantics, the model will think the next item is most likely early-learning again, so it generates a string of similar blocks and boards. Behaviorally, though, parents who buy early-learning toys often go on to buy bath toys; the two categories look nothing alike yet often appear one after the other. This is what the authors call "semantic preference bias": generative recommendation is good at finding "things that look alike" but tends to miss "things bought together". Traditional sequential recommenders give each item an independent ID vector and learn only who follows whom, so they catch exactly this kind of cross-category pattern.

The authors checked this with data. On Amazon Sports they compute a semantic correlation for every test sample: the share of history items whose first code token matches the target item's. After grouping by correlation, they compare the two models:

In the zero-correlation group, TIGER's NDCG@10 (a ranking metric for how high the right answer is placed) trails by 5.7%; in the group with correlation between 0 and 0.1 it trails by 29.9%; while in the group where more than 40% of the history is same-category, TIGER leads by 34.5%. So the strengths of the two models are complementary, and the question becomes how to combine them in one model.

Pipeline overview

The backbone is the same as TIGER's: a T5-style model with a 4-layer encoder and a 6-layer decoder and an embedding size of 128. Each item's Semantic ID comes from RQ-VAE (residual-quantization autoencoder) applying three levels of residual quantization to Sentence-T5 text vectors; items whose first three tokens collide get a fourth token so that every code is unique.

Hybrid encoding: giving the codes "who follows whom"

Why change the encoding? Standard generative recommendation feeds only the codes into the model, splitting one item into three or four code tokens. Similarity between items is then decided entirely by content: two same-category products that never appear together get nearly identical codes, while cross-category products often bought together get unrelated codes.

OMG gives each item an extra learnable ID vector and simply averages it with the vectors of its code tokens as the item's input to the encoder. An item now occupies one position, so the encoder can learn transitions at item granularity. After the encoder reads the whole history, the output at the last position serves as the user's current intent, and an auxiliary task uses it to predict the next item over the full catalog, the same training objective as SASRec. This task forces the encoder to learn "people who bought A then buy B" even when A and B have nothing in common in their codes.

The authors tried heavier fusion. Concatenating the ID vector with the code vectors and passing them through a small network did worse than plain averaging; removing the ID vector entirely made all three datasets clearly worse. Their explanation is that the model ultimately predicts codes, and too much transformation during fusion distorts the code vectors.

Latent reasoning: moving step by step from behavior space to semantic space

Hybrid encoding alone is not enough; in the ablation it even lowers Toys Recall@10 from 0.0557 to 0.0540. The reason is that the two tasks do not line up: the encoder is trained by the behavior objective, so its representation leans toward "which item comes next"; the decoder must generate "the code of the next item" and needs semantic information. Handing a behavior-leaning representation straight to the decoder leaves the decoder hard-pressed to use it.

OMG adds a reasoning unit between the two. It is a two-layer small network that starts from the encoder's last hidden state; at each step it adds the previous result to this initial state and passes the sum through the small network again to get a new state, for K steps in total. The idea is similar to ReaRec and related methods, but those methods send the state back through the whole backbone at every step, with cost growing with the square of the sequence length; OMG iterates on a single vector, so it adds almost no compute.

Iteration without constraints lets intermediate states drift. The authors supervise every step and let the target change gradually: at each step two losses are computed, one predicting the next item from the current state (the behavior objective) and one predicting the first code token of the next item from the current state (the semantic objective), summed with weights. The semantic weight is 1 at the last step and is multiplied by a decay coefficient for each step back; the behavior weight is 1 minus the semantic weight. So the first few steps learn almost only behavior, the last step aims fully at semantics, and the transition in between is smooth. Hoot says▾

Hoot saysWorked example K=4, decay 0.3: how much each kind of supervision gets per stepClose ✕

The semantic weight is multiplied by 0.3 per step back from the last: 1 → 0.3 → 0.09 → 0.027; behavior gets the rest, so step 1 is almost all behavior.

‹1/4›

Swipe or use the arrows to see the four steps. K=4 and 0.3 are only a worked example: the paper reports each value as best on Beauty separately, not that the pair is jointly optimal. Source: paper Eq. 11: β_k = β·γ^(K−k), α_k = 1 − β_k, β=1; on Beauty γ=0.3 and 4 steps are best; γ swept from 0.1 to 0.9

Take the example above: the state at the first step mainly answers "which specific product will this parent buy next", and may already rank the bath toy quite high; further on, the state increasingly answers "which category does the next product belong to". By then it carries traces of the earlier behavioral judgment and will not fixate only on early-learning toys. Finally, all intermediate states are appended to the encoder output and handed together to the decoder to generate the code.

An analogy: it is like translating by first thinking the meaning through in your native language and then rendering it sentence by sentence in the foreign language, rather than translating word by word from the start. The early steps "work out what the user wants", the later steps "say it in the language of codes".

The authors used t-SNE (a visualization method that projects high-dimensional vectors onto a plane) to plot reasoning trajectories for several toy categories: the starting point is far from the target code vector, and the reasoning states move along a smooth path from the start toward the target. The decay-coefficient experiment also supports the "transition slowly" design: values of 0.1 to 0.3 work best, meaning the semantic weight decays quickly going backward and the first few steps are almost entirely behavior-driven. The best number of reasoning steps varies with the data: 4 steps is best on Beauty, with more steps hurting; on Sports it keeps improving all the way to 10.

RL post-training: rewarding "right on behavior" and "spread across categories"

After supervised training, the authors observed that in a group of candidates generated by the decoder, the first code token is often identical for all of them, meaning a dozen or so candidates all crowd into one coarse category. The encoder's "predict the next item" distribution is much more spread out. OMG uses GRPO (a reinforcement-learning algorithm that updates the policy from scores relative to the group) for post-training, having the model generate 20 candidates per sequence and scoring them with two rewards:

The accuracy reward uses continuous similarity rather than "1 for a hit, 0 for a miss", because hitting the exact item among tens of thousands is hard, and a continuous score gives useful feedback to every candidate. The diversity reward uses the encoder's more spread-out distribution as the reference, keeping the decoder's output from clustering too much.

Post-training uses only the samples with the lowest semantic correlation: the bottom 10% for Sports and Beauty and the bottom 40% for Toys. This saves compute and avoids damaging the model on the high-correlation samples it already handles well. Compared with randomly picking the same share of samples, picking by correlation gives a larger gain. On Toys, after 5 epochs of post-training NDCG@10 rises from 0.0362 to 0.0366 and item coverage from 0.695 to 0.727, and the internal diversity of recommendation lists also rises steadily, which shows that it mainly spreads recommendations out while accuracy rises only slightly.

Results

The four public datasets are Sports, Toys and Beauty from Amazon reviews, plus Yelp; sequences are truncated to at most 20, the last item is held out for testing, and ranking is over the full catalog. The table below compares NDCG@10; "strongest baseline" means the opponent with the highest NDCG@10 on that dataset. The three opponents in the table are USI (a method that represents items by concatenating item ID vectors with semantic vectors), RPG (a method that generates a whole longer Semantic ID in parallel) and HSTU (Meta's generative sequential recommender that models item ID sequences directly):

The table shows two things. First, the strongest baseline differs on every dataset, and on Beauty it is the behavior-only SASRec, which shows that semantic preference bias really exists on real data. Second, OMG ranks first on all four datasets, with gains over TIGER of 10% to 28%, so combining behavior and semantics really is better than either one alone.

The step-by-step ablation shows better where the credit lies. Each row below is the gain over the previous row, averaged over the relative gains of 12 numbers: Recall@5, NDCG@5, Recall@10 and NDCG@10 on Sports, Toys and Beauty:

Latent reasoning contributes most of the gain, which means the key is not "whether behavior information is present" but whether the decoder can use it. The auxiliary task alone gains little and even dips slightly on Toys; only with the reasoning unit bridging the middle does behavior information turn into generation quality.

On cost, the reasoning unit takes only 0.46% of non-embedding parameters, and 4 reasoning steps add only 0.16% compute. Compared with TIGER, at 10 reasoning steps each training epoch takes about 12% longer and generating one recommendation list about 11% longer. This shows the cost comes mainly from the decoder reading a few more positions, not from the reasoning unit itself.

Limitations

First, all evidence comes from small offline experiments. The four datasets have only tens of thousands of users and between ten-odd thousand and over thirty thousand items, sequences are capped at 20, and absolute metric values are low, with NDCG@10 between 0.02 and 0.04. Industrial settings often have hundreds of millions of items and thousands of history events, so both the severity of semantic preference bias and the effect of the reasoning unit may differ, and no online experiment verifies it yet.

Second, the last two components bring small and unstable gains. RL post-training adds only 1.8% on average, and on Beauty Recall@10 is 0.0690 both before and after, which shows that this step does essentially nothing on some data.

The share of samples it selects for training is 10% on Sports and 40% on Toys, and the best number of reasoning steps also ranges from 4 to 10, which means these hyperparameters were tuned per dataset. Moving to new data requires searching them again, so the cost of reproducing it is much higher than the compute of the components themselves.

Third, moderately correlated samples remain a weak spot. The authors acknowledge in the appendix that in the band where the history has only a few same-category items, OMG beats TIGER but still trails SASRec. The reason is that the semantic signal for such samples is half right and half misleading, and the model cannot yet tell when to trust semantics and when to trust behavior.

If you want to try it

This suits teams already using Semantic IDs for generative retrieval who see heavily homogeneous candidates. A low-cost trial:

1. First reproduce the analysis in Figure 1: group your own data by "share of history in the same category as the target" and compare your current generative model with an ID sequence model. Only continue if the low-correlation groups show a clear gap.
2. Add only hybrid encoding and the reasoning unit to the existing model, starting from 4 reasoning steps and decay coefficients of 0.1 and 0.3, and watch the gain in low-correlation groups and overall.
3. Consider post-training only if the reasoning unit helps, and track both accuracy and category coverage, because its main effect is to spread recommendations out.
4. Measure latency before launch: the reasoning unit itself is cheap, but the cost of the extra positions the decoder reads must be measured under long sequences and large beam search.

Verdict