Beyond Semantic IDs: Atomic Creator Tokens for Large-Scale Generative Recommendation

Teams building generative recommenders on Semantic IDs should read this. This Meta paper gives every creator its own token in the language model's vocabulary and uses it in place of the SID. The reason is simple: an SID is a many-to-one code, so several creators share one SID, and even when the model generates the right SID it still has to guess which of those creators the user wants. The paper calls the method ACT (Atomic Creator Token, one dedicated token per creator). The model generates one creator token in a single step and takes the top K by probability directly, with no need to expand SIDs into creators and sort them by popularity.

On Meta's creator recommendation data, the authors swap only the token layer and keep everything else fixed. The share of cases where the user's next engaged creator appears in the top 15 rises from 23.58% to 43.41%. In the largest candidate tier, even with the SID codebook enlarged 64 times, SID still trails ACT by 8.72 points. The SID pipeline's per-user inference cost is also about 2.7 times ACT's. The bottleneck is the SID layer, not the language model. The price is that every token needs enough training examples, so the authors use ACT only for head creators.

What problem is it solving?

The task is recommending creators a user does not yet follow. Candidates are a few hundred thousand head creators above a follower threshold, and a user's history is only a short trail of engagements with previously suggested creators, such as opens, likes, shares, follows and long dwells. From this history and the user profile, the model predicts the next creator the user will engage with.

The mainstream generative recipe follows OneRec and TIGER. A pretrained GNN first computes an embedding for every creator, and residual quantization compresses it into 3 codes, which together form that creator's SID. The quantizer is fixed once trained. A language model is then fine-tuned to generate the next SID from the history.

The trouble is that there are fewer SIDs than creators. Many creators land on the same SID, which the paper calls a collision. At inference the model uses beam search to generate several SIDs, each SID is expanded into all of its creators, and the final list is ordered first by the SIDs' beam order and then by follower count within each SID. An illustrative example:

The model generated the right SID (12, 7, 3), but if only two slots are left in the list, new creator C, the one the user actually wants, is pushed out. Popularity ordering favors big accounts, the opposite of what recommending fresh creators needs.

The second problem is in training. Once the quantizer is fixed, the language model's training signal reaches only the SID code tokens and never flows back to the creator embedding. The quantizer groups creators by GNN similarity while the language model learns next-creator prediction, so the two optimize different things. Every new creator also forces a codebook rebuild.

How does ACT work?

The pipeline has three steps:

The core idea in one line: with one token per creator, collisions and the expand-and-sort step disappear, and creator representations become model parameters that can be trained directly.

Why is one token enough? At the last position the model outputs a hidden-state vector $\mathbf{h}$, and every creator $a$ has its own embedding row ${\mathbf{w}}_a$. Creator $a$'s score is the dot product ${\mathbf{w}}_a^{\top }\mathbf{h}$, and a softmax over all creators' scores gives the next-creator distribution. The training objective is the cross-entropy of the true next creator.

How does the training signal reach creator representations? The same creator token is an input when it appears in the history and a scored output at the prediction position. Looking only at the output-scoring path: the gradient of the cross-entropy with respect to a creator embedding is $(p_j-1[j=a^{\star }])\mathbf{h}$. Here $p_j$ is the model's probability for creator $j$ and $a^{\star }$ is the true next creator. Intuitively, the true creator's embedding is pulled toward the current hidden state, and the others are pushed away in proportion to their probabilities. In the SID pipeline the training signal can update only the SID code-token embeddings; the gradient reaching the frozen GNN and codebooks is zero.

A hand-checkable example (illustrative): if the true creator has probability 0.2 in one prediction, its update coefficient along $\mathbf{h}$ is 0.2 − 1 = −0.8, so gradient descent pulls it toward $\mathbf{h}$; another creator with probability 0.1 has coefficient 0.1 and is pushed slightly the other way. Hoot says▾

Hoot saysOne prediction, and how creator representations change in each scheme (illustrative)Close ✕

In ACT the creator embedding is itself a trained parameter: it is pulled in or pushed away by how far off the prediction was; in the SID scheme the creator representation is frozen in the GNN and never moves.

ACT: one trainable token per creator

True creator, probability 0.2 (illustrative)

coefficient 0.2 − 1 = −0.8, pulled toward h

Another creator, probability 0.1 (illustrative)

coefficient 0.1, pushed slightly the other way

What shapes the creator representation

the recommendation objective itself

The closer the probability is to 1, the less the true creator moves; the more confident the model, the smaller the update.

SID: creators are quantized into codes first

What the training signal can update

the SID code-token embeddings

GNN embedding and codebooks

frozen, gradient 0

What shapes the creator representation

the pretrained GNN's similarity

Creators sharing one SID get the same codes, and no amount of training separates them.

The probabilities 0.2 and 0.1 are illustrative. The coefficient is the number multiplying the hidden state h in the gradient of the cross-entropy along the output-scoring path; gradient descent moves against it, so a negative coefficient means being pulled toward h. Source: paper §2.2 Eq. (6): the gradient is (p_j − 1[j = a*]) h; §2.1: the SID pipeline's GNN encoder and codebooks are frozen, with zero gradient

Main training fine-tunes the backbone with LoRA while training the creator embeddings, on "profile + history → next creator" examples. The production model also has a prior warm-up step that freezes the backbone and updates only the new tokens; the public-data experiments skip the warm-up, and ACT still leads. New tokens are randomly initialized by default; they can also be initialized from the GNN embedding through a linear projection, but stay trainable afterwards. The added parameters are modest: 10Y creators times 3072 dimensions in bf16 is about 0.6 GB (Y is the relative unit the paper uses to mask the real scale).

The authors also tried post-training after SFT. Because a creator is a single token, DPO, GRPO and one more post-training method can all run directly over the real set of creators. None of the three beat SFT, so the served model is the SFT model.

Experiments: what if only the token layer changes?

To keep the comparison fair, the SID baseline and ACT share the same backbone, data and decoding, with SID implemented as in OneRec; the only difference is the token layer. Production data is split by day, with about one million test examples. There are three candidate tiers by decreasing follower threshold, Y, 10Y and 30Y creators, each larger tier containing the smaller one.

Two evaluation views must be kept apart. Creator-level recall checks whether the true next creator is in the final top K. SID predictions are first expanded into creators before this is computed. Cluster-level recall only checks whether the generated SID equals the target's SID; it overstates SID because one SID stands for several creators. All results below use creator-level recall.

From the top 15 to the top 200, ACT leads clearly. This shows its advantage is not limited to very short lists.

NDCG@200 is weighted by engagement value: a brief open gets the lowest weight and a follow the highest. It also rises by 83.4%, which shows the recall gain carries over to a metric that cares more about ranking and engagement value.

How much does SID lose to collisions?

The authors split SID's creator-level recall into two parts: first hit the SID (cluster-level recall), then that hit must survive expansion, popularity ordering and the cut to the top 15. The surviving fraction is written $\rho$, equal to creator-level recall divided by cluster-level recall. It is a ratio of two recalls computed at evaluation time; serving never computes it. $\rho =1$ means hitting the SID equals hitting the creator; the smaller it is, the more is lost. The paper calls the lost part the collision tax. ACT has no collisions, so its $\rho$ is always 1.

(X is the paper's masked base codebook size per level; X³ means 3 levels with X codes each.)

At the 10Y tier with the base codebook, $\rho$ is only 0.47. In other words, more than half of the clusters SID hits are lost during expansion and the popularity cut. As candidates grow from Y to 10Y, the mean collision count rises from 2.35 to 3.79 and $\rho$ falls with it. SID pays twice as scale grows: the code is harder to predict, and the collision tax is heavier. ACT only has to get the creator right and pays no collision tax, so its relative lead widens from 46.5% to 84.1%.

A larger codebook recovers part of this. Doubling the codes per level at the 10Y tier lifts SID's R@15 from 23.58% to 29.21%, which shows collisions are an important source of the gap. But at the 30Y tier, with a codebook 64 times larger, 43.3% of creators still share an SID with someone. Larger codebooks mean fewer collisions, but with shrinking returns and never down to zero, because the frozen GNN embeddings cluster tightly on their own. In this comparison SID is evaluated only on the roughly 76% of creators that have a GNN embedding while ACT is evaluated on all of them, which favors SID, and ACT still leads by 8.72 points.

The authors also compared the creator embeddings ACT learns with the GNN embeddings and found them largely decoupled, keeping only a little local neighborhood structure. ACT learns a space that serves the next-creator task, rather than relearning the GNN's collaborative relations.

How big should the vocabulary be, and when does ACT fail?

A larger vocabulary makes each prediction harder but covers more engagement. From Y to 30Y, recall when the target is in the vocabulary falls, while the share of engagement the vocabulary covers rises faster, so the product of the two, effective recall, rises at every tier. Within the head, then, a bigger vocabulary is better until each token runs out of data.

On the public KuaiRand-27K data, the authors keep only the 50,000 creators with the most training clicks. Here ACT's Recall@15 is 2.763%, SID's is 1.859%, and a global-popularity baseline gets 2.351%. This shows ACT learns something beyond popularity, while SID falls below simply recommending popular creators from K=10 on.

SID is slightly ahead of ACT only at K=1 (0.332% versus 0.294%). This fits its habit of returning the most popular creator within each SID first: the top slot benefits from popularity, and the edge vanishes as the list grows.

On the smaller KuaiRand-1K, the conclusion flips. This is again Recall@15, against the better of the two heuristic baselines, global popularity and recent history. With all creators, ACT gets only 0.75% against 1.76% for the better heuristic. Keeping only head creators raises ACT to 1.86%, still below the heuristic's 3.78%. In these two settings each creator token has only about 5 and 34 training examples on average, versus about 331 in the KuaiRand-27K setting. KuaiRand-27K also has far more total data, so this comparison gives no universal sample threshold. What it does show is that ACT's gain depends on how much supervision is available, not on one-token-per-creator by itself. That is why the authors position ACT as a head component: ACT for the head, conventional retrieval for the long tail.

Serving cost and A/B

On the same 32 A100s with beam width 15, per-user inference at the 10Y tier drops from 97.5 GPU-ms to 35.0, peak memory per card drops from 61.5 GB to 41.4 GB, and concurrent throughput is 160% of SID's. The reason is that SID outputs 5 tokens for every candidate SID it generates (a start token, 3 codes and an end token) before that SID is expanded into creators, so every beam takes more decode steps and holds more KV cache. ACT generates 1 token, and the only extra cost is one wider softmax.

In the online A/B test, the ACT arm is compared with the SID-enhanced production arm and with the classic production ranker. The online metrics are masked and fall into two overall consumption metrics, one product consumption metric and one product engagement metric. The product engagement metric moved most: +0.29% against the SID arm and +0.66% against the classic ranker (p≈0.05, approaching significance). The other three consumption metrics moved only 0.01% to 0.11%. This means the offline recall gain points the same way online, but the size is small, and the authors treat it only as directional evidence.

In the ablation, at beam width 200, GNN initialization beats random initialization by 0.75 points of R@15 (43.91% to 44.66%), and adding a contrastive loss lowers every metric. The served version uses the simpler random initialization.

Limitations as I see them

First, the online evidence is weak. All four metrics point the right way, but only product engagement against the classic ranker (+0.66%) approaches significance (p≈0.05), and against the SID production arm it is just +0.29%.

Second, ACT depends on ample supervision. On KuaiRand-1K, with about 5 to 34 examples per token on average, ACT cannot even beat a simple heuristic baseline (1.86% versus 3.78%). So long-tail items with sparse supervision must go to conventional retrieval, and ACT cannot cover the full catalog.

Third, a larger vocabulary means a wider output softmax. The paper's head is on the order of a few hundred thousand creators; with millions of items, inference cost and per-token sample counts become new constraints.

How a recommendation team could try it

1. Pick head items first: sort items by training-sample count, keep only those with at least a few hundred examples each, and check what share of engagement they cover. This decides how much ACT can handle.
2. Run a controlled comparison on an existing SID generative model: keep backbone, data and decoding, swap SIDs for one token per item, report both creator-level and cluster-level recall, and compute $\rho$. A low $\rho$ means collisions are holding you back.
3. Start with random initialization and plain SFT, without the contrastive loss or post-training. None of these extras helped in the paper, so measure the gap with the simplest version first.
4. Put ACT on the head route of existing retrieval, leave the long tail on the old retrieval, and watch high-value engagement metrics online alongside inference latency and memory.

Conclusion

For head items, rather than keep enlarging the codebook to ease collisions, give each item its own token and let the recommendation objective's gradient train item representations directly.