Contents · 5 sections
1Recommendation: measure what it uniquely adds to the candidate pool2The three codebooks are residual; MTP training and inference are not autoregressive3Search quality: MTP alone is not enough, and a lightweight ranker is what makes it stand up4They also tried a unified student in training5What to take awaySHARE
Shopify has to retrieve over a large, multi-merchant catalog that keeps changing, under tight latency. SID-LLM's approach is to encode each product with Shopify's in-house product-search embedding model (768 dimensions), then turn that vector into a short Semantic ID (SID) with residual-quantized K-means (RQ-KMeans). The three codebooks are residual: the first quantizes the 768-dimensional vector, the second quantizes the residual, and the third quantizes what remains, so the three codes approximate one vector layer by layer rather than independently quantizing the same vector three times. Search and recommendation both become "generate relevant SIDs from the current context": search uses the query (optionally plus a buyer profile), recommendation uses interaction history; training pairs that context with the target product's SID for next-token prediction, and inference supplies a context of the same form. Generated SIDs are then resolved into purchasable products and handed to downstream ranking. What is shared is the SID representation and the joint pretraining data; online, search and recommendation are still two separately pretrained models and two serving paths.

Recommendation: measure what it uniquely adds to the candidate pool
Generative retrieval is easy to report as its own Recall@50. The authors take the top 200 from each generator in the production pool and, with leave-one-out, measure how much pool recall drops when that generator is removed — incremental pool recall, counting purchases that only it found. On 70,747 users with a purchase in the held-out week, SID-LLM with an LFM2-2.6B backbone (the 2.6B-parameter SID generation model chosen in the paper) has Recall@50 of 24.61% and incremental pool recall of 2.65 percentage points, ahead of recent engagement (1.64, recalling from recent interactions) and atomic-ID HSTU (0.78, a sequential generator over unshared item IDs rather than content SIDs). Because the incremental number asks "which purchases did it uniquely contribute," it sits closer to this generator's place in a multi-retriever pool than a standalone Recall@50 does.
The home-feed A/B is more direct: the control is the home feed without SID-LLM, and the treatment arm adds SID-LLM candidates to the pool (with joint-probability resolution). On the portion of home-feed traffic that actually received those candidates, attributed orders are +7.95%, converting users +8.60%, and conversion rate +9.85%, all significant after Dunnett correction. That is not a site-wide number. The backbone is LFM2-2.6B because it had the lowest p95 among the models they compared; the Qwen3 family has higher average quality but much higher latency. This LFM2-2.6B is the downstream SID generation model, not the 768-dimensional product encoder above.
Of the SIDs the recommendation model actually predicts, 78.9% are shared by more than one product (across the whole SID inventory, about 70% map to a single product). Joint-probability resolution adds the beam's log p(SID | user) to a popularity-allocated log p(product | SID) inside the cluster, lets each SID contribute at most 10 products, and keeps the 200 highest-scoring items. Relative to resolving one product per SID, this step raises Recall@50 by 15.8%. Online recommendation still uses autoregressive decoding, plus this resolution step.
The three codebooks are residual; MTP training and inference are not autoregressive
The metric needs a definition first. OneDCG@10 starts with an LLM judge assigning each returned product a relevance grade from 0 to 3, and a query-independent product-quality score is added (ratings, popularity, brand, shop sales, and so on). The quality score s is a weighted sum of four signals in [0, 1] — ratings, popularity, brand prevalence, and shop sales — with weights that sum to one. The gain is 100(2^{r+s}-1)/15, so s is worth at most one extra grade and the gain runs from 0 to 100. A discounted cumulative gain is then taken, divided by the constant DCG of k results that all receive the maximum gain of 100. So the measure is absolute relevance: the denominator does not change with the list itself. Purchase Recall@10 is the fraction of purchased items in converting sessions that land in the top 10. Evaluation protocols (judge, query set, index) differ across tables, so absolute scores are comparable only inside the same table.
Production uses 8192-8192-8192, with 8192 centers in each codebook: a larger 32768-8192-8192 scores higher OneDCG, but a larger first codebook means the first generation step chooses among more centers, and p95 reaches 50 ms at 100 requests per second, so they kept latency headroom and did not adopt the larger first codebook.
In training, multi-token prediction (MTP) predicts all three codeword distributions from the request prefix alone, with no codeword attending to another. That is a different computation graph from "autoregressive code-by-code, later codes seeing earlier ones." At inference the three heads emit in parallel, then a beam walks a prefix index built from catalog SIDs and constrained by the request's market eligibility, assembling only SIDs that resolve to purchasable products. Because training already does not rely on code-to-code autoregression, inference can run the three heads in parallel and let the catalog index prune illegal prefixes. Hoot says
Code-by-code autoregression (contrast, illustrative)
- Code 1
- Sees only the request prefix, picks 1
- Code 2
- Sees the already chosen 1, then picks 2
- Code 3
- Sees 1-2, then picks 0 or 3
- How the catalog is used
- Legal-prefix filtering is mixed into each step
Later codes depend on earlier ones, so the three heads cannot all run first and filter afterwards.
The paper's MTP
- What the three heads read
- Each sees only the same request prefix, not the other codes
- Training
- All three distributions are predicted at once, with no codeword attending to another
- Inference
- Three heads emit in parallel, then a beam walks the catalog prefix index
- Illustrative outcome
- 1-2-0 and 1-2-3 remain; prefix 1-0 is pruned by the catalog
Because training does not rely on code-to-code autoregression, inference can predict in parallel and let the catalog index drop illegal prefixes.
Both columns use the same illustrative request and a tiny catalog: the only legal SIDs are 1-2-0 and 1-2-3, and 1-0 is not a legal prefix. The codes are illustrative, to contrast whether later codes see earlier ones; the paper does not use this set.
Source
Paper: MTP training predicts all three codeword distributions from the request prefix alone, with no codeword attending to another; at inference the three heads emit in parallel, then a beam walks a prefix index of catalog SIDs (with the request's market eligibility) and assembles only SIDs that resolve to purchasable products. The toy catalog and code values are added here.Search quality: MTP alone is not enough, and a lightweight ranker is what makes it stand up
Under protocol E5, autoregressive code-by-code generation scores 0.3083 OneDCG@10 and 0.3246 purchase Recall@10. MTP alone is about 0.280 OneDCG and 0.284 purchase recall, and widening the beam barely moves either; what it buys is roughly a halved median server-side latency. On the search path, each generated SID is first resolved to the single eligible product with the highest static quality score. Those 250 products then go to a learned lightweight ranker, which reads the query / profile and the candidate, for a relevance score. The serving ranker then combines the ranker score, an auxiliary quality score that uses the same algorithm as the query-independent quality term in OneDCG, and the generator's log-probability of the candidate (the MTP score), with weights 0.5 / 0.3 / 0.2. All three scores are min-max normalized over the request's candidates so the scales line up. After that step, the same-protocol numbers are 0.3365 OneDCG@10 and 0.3450 purchase recall. Both are above autoregression, and the combination is faster at every percentile and load. The margin is small at low load and opens up on high-load tails. Because the auxiliary score and the quality term in OneDCG use the same algorithm, purchase recall is the more independent of the two. Production search uses MTP plus the ranker; recommendation stays autoregressive.
| Protocol E5 | OneDCG@10 | Purchase Recall@10 |
|---|---|---|
| Autoregressive | 0.3083 | 0.3246 |
| MTP only (beam 100 / 250) | 0.2804 / 0.2811 | 0.2839 / 0.2840 |
| MTP + lightweight ranker (250 candidates) | 0.3365 | 0.3450 |
These numbers show that the latency won by emitting codes in parallel needs the lightweight ranker afterwards to pull quality back above autoregression; reading MTP's OneDCG alone would misread the combination as a quality-for-speed concession.
A search-stack ablation under a different protocol, E6, must not be compared in absolute terms with E5 above. The full stack with SID-LLM scores 0.327 OneDCG@10 when queries without a buyer profile and personalized queries with a profile are scored together, and 0.357 on personalized queries with a profile only. Removing it — both its candidates and the features it feeds the rankers — drops the scores to 0.306 / 0.330. SID-LLM alone (its own ranker, bypassing other retrievers and the two learned ranking stages L2 and L3 that follow candidate merging in the full stack) is 0.285 in the combined setting, below the full stack without SID-LLM, and 0.336 in the personalized-only setting, slightly above that stack. The authors summarize the latter as "close"; the full stack is still best. Online search went through a series of A/Bs, and the authors write that the results supported a production deploy; the rest of this article quotes only the offline numbers given in the text and the recommendation home-feed A/B, and does not invent a single headline search lift.
They also tried a unified student in training
Taking joint supervised fine-tuning as the baseline, search and recommendation experts are then trained separately from the same autoregressive continued-pretraining checkpoint: the recommendation expert raises Recall@50 by about another 1.7% but drops search quality; the search expert has no detectable advantage on search and clearly hurts recommendation. A routed multi-teacher distillation student (each example is routed to the matching expert, with reverse KL plus cross-entropy on the gold SID): versus joint SFT, Recall@50 rises 1.76%, discovery Recall@50 (counting only target products the buyer has never interacted with) rises 10.03%, and the two search metrics show no detected change — the confidence intervals still allow a small drop. This is an experimental student initialized from the same autoregressive continued-pretraining checkpoint; distillation, extra budget, and teacher choice are not written as fully independent ablations, and production serving remains two paths, MTP for search and autoregression for recommendation. "Unified" first means a shared SID and joint pretraining, and only secondarily this distillation line.

Offline, still under the same continued-pretraining plus MTP initialization and without a ranker, they also compare search post-training with cross-entropy, DPO, and GRPO. GRPO trained on LLM-judge feedback has the highest OneDCG, and purchase recall can fall. Under exposure bias, a relevant product that was never shown cannot become a purchase positive, and the authors themselves say this drop does not mean substitutes would bring fewer purchases.
This is first of all an industrial system report: retrieve over a changing multi-merchant catalog, and write search and recommendation as the same SID generation. Absolute offline scores are bound to their evaluation protocols; E5 and E6 must not be compared across tables. The distillation student is experimental, and production serving stays task-specific. The text does not quote a single headline search-online lift that can be cited; the recommendation home-feed A/B does give order and conversion numbers on impacted traffic.
What to take away
For SID generation to stand up at Shopify's scale, the point is not only "it can generate IDs." Recommendation is judged by incremental pool recall and home-feed orders; search trades latency with MTP and then uses a lightweight ranker to pull quality back above autoregression. The three codebooks are residual approximations, not three independent quantizations; MTP already trains the code positions not to attend to one another, so inference can emit them in parallel and constrain them with a catalog index. Catalog encoding, joint pretraining, and task distillation are parts along the same representation path; at serving time, search and recommendation can still be — and in fact are — served separately.
How did this read for you?
Hoot made a note. Thanks for the feedback.
One tap, no sign-in; tap again to undo. Only anonymous counts are kept, never who you are.
Discussion
Add experiment details, reproduction notes or a different take. Requires a GitHub sign-in.
Loading the discussion…