CARE: Contrastive-Anchored Residual Encoding with Listwise Reconstruction for Retrieval-Oriented Semantic IDs

Yuan Xu, Dawei Chen, Xiaoguang Lin, Xin Chen, Yulong Ou, Haijun Wu
33 minread4.10scoreNo. 211-18 Generative issue
Contents · 6 sections1Common SIDs only compress in document space2CARE-SID: pull the space toward retrieval, then quantize3Walkthrough: the same error, but A and B can swap4How much the offline numbers move on industrial video-search data5What this means for people building generative retrieval6Limitations
Rel 5Nov 4Prac 4Evid 4Time 5

Reconstruction error won't tell you if A and B swapped: how video-search SIDs keep the order under a query

A user types "latte art tutorial" into a video app. Generative retrieval does not look up nearest neighbors in a vector index. It first gives each video a short Semantic ID, called an SID, then has a language model write the target SID token by token from the query and the videos the user has already watched, and returns the videos attached to that ID. The query and numbers in the paper's Figure 1 are illustrative.

The SID is the only address the generative model can see. Videos that share an SID can only be recalled together. Information lost in quantization cannot be recovered later. Common encoding methods optimize document-side distortion, reconstruction, lattice fit, or bucket capacity; they do not directly constrain whether relevant videos still rank first once a query arrives. Two reconstructions can have the same per-document error, yet one keeps the order and the other swaps A and B under that query.

This paper, from Tencent, is about the SID tokenizer used in industrial video search. Existing methods such as RQ-KMeans and RQ-VAE optimize distortion or reconstruction error on document embeddings and never look at queries. R3-VAE, a 2026 quantization method, encodes SIDs with a shared reference vector and per-layer rating residual quantization; here it is the quantization kernel. The authors propose two methods. CARE-SID places a residual alignment network before the quantizer and uses contrastive learning on queries and documents to pull the space toward what retrieval needs. CARE-LR then distills the similarity distribution of the original embeddings over a candidate list onto the SID reconstructions. Encoding still uses only the document's own embedding, so new videos can be encoded incrementally with the same weights. In the offline evaluation, reconstruction-view Recall@10 is 62.75% for RQ-KMeans, 65.23% for CARE-SID, and 66.09% for CARE-LR. CARE-SID is already used in the generative retrieval stage of the industrial video search system.

Encoding and evaluating SIDs by query ranking raises reconstruction-view Recall@10 by 3.34 points over RQ-KMeans; these figures are offline SID-level metrics, CARE-LR is still a small-scale trial, and end-to-end results from the generative model have not been reported.

Common SIDs only compress in document space

RQ-KMeans shrinks the residual layer by layer, RQ-VAE optimizes reconstruction and codebook losses, and other work uses a fixed lattice or adjusts bucket capacity after the fact. All of these objectives are defined on document embeddings. Retrieval asks something else: when the same query arrives, can relevant documents rank above irrelevant ones? Quantization always discards information; the question is where it is discarded. If the loss falls on the direction a query uses to separate relevant from irrelevant items, distortion can look small and recall still drops.

The authors therefore ask two questions. In which space should one quantize, so that the directions a query needs are less damaged? After quantization, how can each query's relative order over candidates be preserved?

The evaluation also skips the generative model. The paper uses row-budget recall and looks at the SIDs directly. The main metric was fixed before the CARE-LR experiments: bucket by the full three-digit SID, use the mean of the reconstructions in each bucket as the key, rank buckets by query cosine, and compute recall under a budget of 10 documents. This is reconstruction-view Recall@10. It measures how much retrieval ability remains in the SID plus the reconstruction path, without first training a generative model.

CARE-SID: pull the space toward retrieval, then quantize

Quantizing the upstream embedding directly assumes that this space is already the best one to partition. CARE-SID does not assume that. It adds a residual alignment network in front of the quantizer: the document embedding d goes through a two-layer network to produce a correction, which is added back and normalized to get z. The output layer starts at zero, so z equals d at the beginning of training, and the codebooks already warmed up with RQ-KMeans and R3-VAE remain valid.

During training, the same network is also applied to queries. A contrastive loss pulls positive pairs together in the aligned vectors, and an anchor loss keeps z from leaving the original d. The reconstruction target is also fixed as the original d, not the aligned z, so the alignment network cannot lower the reconstruction loss by collapsing the space. Queries appear only in the training objectives. When a video is actually encoded, only its own embedding is used: it is aligned, the shared reference direction is removed, and three levels of rating residual quantization produce a three-digit SID.

A simpler approach is to add the contrastive loss directly on the codes after quantization. The authors tried that in an earlier development round; it can only push the codebook, and tends to pack many documents into the same buckets. CARE-SID places the contrast before quantization and holds the space with the anchor. The reconstruction head is used only in training and in the offline evaluation; serving-time encoding does not need it. The reference coefficient is the inner product of the aligned vector z and a shared reference vector r; the reference component is v0=(z⊤r)r, and removing it leaves the residual e0=z-v0. At each level the residual is first normalized, the codeword with the largest cosine is selected, the rating is that cosine, the level's component is the rating times the selected codeword, and that component is subtracted from the residual before the next level. The reconstruction head adds the reference component and the three level components, passes the sum through an MLP, which is a small multilayer network, and normalizes the result to obtain the reconstructed vector. The SID records only the three selected codeword indices; it does not contain the ratings or the reference coefficient, so the same SID can still correspond to different reconstructed vectors.

Hoot saysIf contrast is added after quantization, what does it actually push?Close ✕

The article already walks Figure 4: with the same per-document error, A and B can swap, and the listwise KL can tell the two reconstructions apart. This walkthrough only pursues a point that walk did not unpack: if the contrastive loss is added on the codes after quantization, what it actually pushes. The counts 16,305 and 9,069 below come from an extreme attempt in §6.1; they are not illustrative.

contrast on the codes after quantization can only push the codebook → in an extreme attempt, first-level codewords dropped from 16,305 to 9,069 and retrieval got worse → CARE-SID places contrast on the aligned vector z before quantization; the output layer starts at 0, so z=d at the start → the anchor holds z, and the reconstruction target stays the original d, so the alignment network cannot collapse the space to drive reconstruction loss down

  1. contrastive loss added on the codes after quantization → it can only push the codebookThe codes are already chosen discrete indices, so the contrastive signal cannot reach the alignment network and can only change the codewords in the codebook. The authors tried this in an earlier development round.what is pushed = the codebook, not the alignment network
  2. constraining the discrete codes directly → first-level codewords drop from 16,305 to 9,069, and retrieval gets worseIn one extreme attempt, the number of first-level codewords in use dropped from 16,305 to 9,069, a loss of 7,236. Many documents were packed into the same buckets, and retrieval got worse.16,305 − 9,069 = 7,236 first-level codewords dropped
  3. CARE-SID places contrast before quantization → it pushes the alignment network, and z still equals d at the startThe contrastive loss is instead applied on the aligned vector z. The alignment network's output layer starts at 0, so at the beginning of training z equals the original d, and the already warmed-up codebooks remain valid.output layer = 0 → z = d at the start
  4. the anchor and a reconstruction target fixed at the original d → the alignment network cannot collapse the space to drive reconstruction loss downThe anchor loss keeps z from leaving the original d, and the reconstruction target stays the original d rather than the aligned z. So the alignment network cannot collapse the space to drive the reconstruction loss down.reconstruction target = original d, not z

Figure 8 compares the offline gap between CARE-SID (alignment before quantization) and R3-VAE (without this alignment). It is not a figure of the 16,305-to-9,069 trial. On the search-tuned embedding, that gap shrinks to a very small number.

Paper Figure 8: in the same pool and at the same codebook size, changing the embedding moves Recall@10 more than changing the quantizer; the right panel is the gain of CARE-SID over R3-VAE.
Paper Figure 8: the offline gap of CARE-SID over R3-VAE. The walk above only asks what contrast pushes when it is placed before or after quantization; it does not read this figure as the 16,305→9,069 trial, and it does not retell the main result about changing the embedding.
SourceThe article and paper §6.1: an earlier attempt to constrain the discrete codes directly dropped first-level codewords from 16,305 to 9,069 in an extreme case, and retrieval got worse. §5.1: the residual alignment network's output layer is initialized at zero, so z=d at the start. §5.2: the contrastive loss is applied on the aligned vectors, the anchor keeps z close to d, and the reconstruction target is the original d, not z. Figure 8 is only used to mark the offline gap after adding CARE-SID's alignment; it is not a figure of the 16,305→9,069 trial.

Walkthrough: the same error, but A and B can swap

The walkthrough uses the numbers in the paper's Figure 4. The temperature is 0.05, and the three candidates are A, B, and C. These cosines are the worked example listed in the figure.

Step 1. Look at the teacher. The original embeddings' cosines with the query are A 0.70, B 0.66, and C 0.55. A gap of 0.04, divided by the temperature 0.05, is 0.8; e to the power 0.8 is about 2.2, so A's probability is a little more than twice B's. After softmax, the probabilities are 0.67, 0.30, and 0.03, and the order is A ahead of B.

Step 2. Look at a reconstruction that keeps the order. The cosines become 0.63, 0.59, and 0.49, and the probabilities become 0.66, 0.30, and 0.04. The order is unchanged, so the listwise KL, the divergence between the two probability lists, is only 0.001.

Step 3. Look at a reconstruction that swaps the order. The cosines become 0.59, 0.63, and 0.49, A and B trade places, and the probabilities become 0.30, 0.66, and 0.04. The listwise KL rises to 0.29. That means the per-document reconstruction errors can look similar, while first place under the query has already changed from A to B. Comparing each document only with its own reconstruction does not reveal the swap; comparing the probability distribution over the whole list does.

Step 4. What CARE-LR supervises is this list. Each training query comes with a candidate set: at most 32 items from the real search funnel, plus at most 16 neighbors and 16 random documents. The teacher applies softmax to the original-embedding cosines, the student applies softmax to the reconstruction cosines, and the two distributions are trained with KL at the same temperature 0.05. The funnel source is used only for stratified sampling; clicks and ranking scores are not the target. There is also a reconstruction loss on background documents, so reconstruction does not worsen over the document distribution as a whole.

Step 5. So that distillation starts already attached to CARE-SID, the new layer-resolved reconstruction head is initialized at zero, the remaining parameters are copied as they are, and training starts as the same function as CARE-SID. The alignment network, the reference vector, and the first-level codebook are frozen; only the last two codebooks and the reconstruction head are updated. The encoding interface is unchanged, and each video's first SID digit stays the same.

Figure 4. Softmax probabilities of three candidates under the same query, at temperature 0.05. Left: the teacher distribution from the original embeddings; middle: a reconstruction that keeps A ahead of B, with KL 0.001; right: a reconstruction that swaps A and B, with KL 0.29. Source: paper Figure 4
Figure 4. Softmax probabilities of three candidates under the same query, at temperature 0.05. Left: the teacher distribution from the original embeddings; middle: a reconstruction that keeps A ahead of B, with KL 0.001; right: a reconstruction that swaps A and B, with KL 0.29. Source: paper Figure 4

The authors also tried constraining the discrete codes directly, for example by making related documents for the same query share a prefix. That easily merges buckets too aggressively. In one extreme setting, the number of first-level codewords fell from 16,305 to 9,069, and retrieval got worse. So CARE-LR supervises the reconstruction path and leaves the encoding interface unchanged.

How much the offline numbers move on industrial video-search data

Training uses production-scale document embeddings and query-document positive pairs from this video search service. The evaluation pool has 1,001,700 documents. Queries are split into four slices: ideal documents for the general and long-tail slices come from human annotation, and those for the high-click and high-impression slices come from seven days of user behavior, for 2,990 queries in total. Queries always use the original embedding; they do not go through the alignment network and are not quantized. RQ-KMeans and R3-VAE are the first two stages of the CARE-SID warm start, so the gap from CARE-SID to R3-VAE measures the contribution of the third stage as a whole.

The CARE-LR numbers come from a small-scale trial: the training queries and candidates come from about one million search-funnel lists, each step uses 8 queries and 512 background documents, and training runs for 1,000 steps. The paper calls this the method's first validation.

MethodRecall@10vs RQ-KMeans
RQ-KMeans62.75%—
R3-VAE65.15%+2.40
CARE-SID65.23%+2.48
CARE-LR66.09%+3.34

(Numbers from the paper's Table 3; the metric is reconstruction-view recon_P3@10. The paired 95% interval for CARE-LR versus RQ-KMeans is +2.30 to +4.36.)

Figure 6. Left: gains over RQ-KMeans, with a band for the paired 95% interval of CARE-LR; right: the paired difference of CARE-LR over CARE-SID. Source: paper Figure 6
Figure 6. Left: gains over RQ-KMeans, with a band for the paired 95% interval of CARE-LR; right: the paired difference of CARE-LR over CARE-SID. Source: paper Figure 6

CARE-LR is another 0.86 points above CARE-SID at @10, with an interval of +0.55 to +1.19, and the gain is concentrated at the head of the ranking; by @50, the interval already contains 0. The ablations show that this gain comes from the listwise KL: adding it to the original sum head is +0.70, and adding it to the layer-resolved head is +0.85, and neither interval contains 0. Changing only the head structure, and the interaction of the head with the KL, both have intervals that contain 0. In the experimental group that trains only the layer-resolved head and freezes the codebooks, Recall@10 is 66.12%; full CARE-LR minus this group is -0.03 percentage points, with an interval of -0.12 to +0.05. That means that in this trial the gain travels through the reconstruction path, and updating the last two codebooks has no extra measurable benefit.

Figure 7. Paired differences of the listwise KL and the layer-resolved head on reconstruction-view Recall@10. Solid markers mean the 95% interval does not contain 0. Source: paper Figure 7
Figure 7. Paired differences of the listwise KL and the layer-resolved head on reconstruction-view Recall@10. Solid markers mean the 95% interval does not contain 0. Source: paper Figure 7

What the generative model actually consumes is the buckets cut by the integer SIDs. CARE-LR freezes the first level, so the one-digit buckets are identical to CARE-SID; the bucketing recall of two-digit and three-digit prefixes differs by at most 0.02 points. Bucket quality is almost unchanged, even though some videos change in the last two digits.

How much contrastive learning can add depends heavily on the original embedding. On a general multimodal embedding that was not trained on search data, CARE-SID's Recall@10 is 2.10 points above R3-VAE. After switching to the embedding contrastively trained on search data in this system, that gap shrinks to 0.07, and the two methods are essentially tied at @10. Moving from the general embedding to the search-tuned one raises CARE-SID's Recall@10 from 27.55% to 65.22%, and the embedding itself from 55.65% to 88.14%. On the same embedding, the quantizers compared in this paper usually differ by only a few points. The authors' judgment is that, in their system, the largest return comes from first raising the retrieval quality of the embedding, and only then encoding SIDs.

Figure 8. In the same pool and at the same codebook size, changing the embedding moves Recall@10 more than changing the quantizer. The right panel is the gain of CARE-SID over R3-VAE, which narrows clearly on the search-tuned embedding. Source: paper Figure 8
Figure 8. In the same pool and at the same codebook size, changing the embedding moves Recall@10 more than changing the quantizer. The right panel is the gain of CARE-SID over R3-VAE, which narrows clearly on the search-tuned embedding. Source: paper Figure 8

CARE-SID is already used in the generative retrieval stage of the industrial video search system. Offline, every video is encoded into a three-digit SID, and new videos are encoded incrementally with the same weights. Serving does not use the reconstruction head. To deploy CARE-LR, because the last two codebooks will move, the SID map must be refreshed at the same time, along with the generative model's training labels and user histories.

What this means for people building generative retrieval

First get the retrieval embedding right, then encode SIDs in that space by query ranking; in this paper's experiments, the quantizers compared on the same embedding differ in recall by only a few points.

If the embedding has already been contrastively trained on search data, the pre-quantization contrastive alignment used by CARE-SID shrinks the Recall@10 gap to 0.07 points. One can then ask whether listwise distillation after quantization can still keep the order under a query. It is also worth checking whether these gains have entered the integer SIDs, or remain only in the reconstruction head that does not go to production. The generative model sees the SID; it does not see the reconstructed vector.

Limitations

All results are offline, SID-level metrics. CARE-LR has been trained only in the small-scale trial above. End-to-end results from the generative model, and a larger-scale CARE-LR, are stated in the paper as forthcoming in a later version.

The reconstruction views of R3-VAE and the CARE models use ratings and reference coefficients that are not part of the integer SID, so the view is not determined by the SID the generative model writes alone. In this trial, CARE-LR's gains travel through the reconstruction path; whether that helps a generative model that consumes only integer SIDs still needs end-to-end confirmation.

The confidence intervals reflect only query sampling, not variation across training seeds, and there is no multiple-comparison correction. The set of ideal documents used in the evaluation had already been used in earlier development rounds, so this is not a fresh blind test.

How did this read for you?

One tap, no sign-in; tap again to undo. Only anonymous counts are kept, never who you are.

Related papersMore from Paper Daily

  1. Generative · 2026-10-28PinGER: Token-Efficient Generative Retrieval with Semantic IDsGenerative retrieval encodes each item as a multi-level SID and generates it level by level, but on the input side every history item costs several tokens. During training, Pinterest's PinGER places one dense-vector token after each item's SID and requires that any token reading another item may see only that item's dense token, never its SID. At serving time all SIDs in the history can be droppe…
  2. Generative · 2026-11-04Beyond Semantic IDs: Atomic Creator Tokens for Large-Scale Generative RecommendationThis Meta paper gives every creator its own token in the language model's vocabulary and uses it in place of the SID. The reason is simple: an SID is a many-to-one code, so several creators share one SID, and even when the model generates the right SID it still has to guess which of those creators the user wants. On Meta's creator recommendation data, swapping only the token layer lifts the share…
  3. Generative · 2026-10-29REGER: Modeling inter-item Relations with Semantic IDs for Generative RecommendationCross-entropy tells a generative recommender only which token is correct, with no signal about how other candidates compare. REGER, from the University of Electronic Science and Technology of China, splits a conventional sequential model's scores over all items into branch preferences along the SID prefix tree and distills them along the true path. Added alone to TIGER, it lifts Beauty Recall@10…
  4. Generative · 2026-10-28MuSID-KD: Multigranularity Knowledge Distillation for Generative Recommendation with Semantic IDsWhen a SID-based generative recommender is distilled into a small model, the student is usually asked only to match the teacher's distribution over which item to recommend in the end. ECNU's MuSID-KD (a multi-granularity distillation method that works level by level over the SID) finds this is not enough: a student aligned only on items disagrees with the teacher on the order of candidate groups…
  5. Generative · 2026-07-31CapsID: Soft-Routed Variable-Length Semantic IDs for Generative RecommendationDeep dive: CapsID: Soft-Routed Variable-Length Semantic IDs for Generative Recommendation.
  6. Generative · 2026-04-30K-CARE: Knowledge-driven Symmetrical Contextual Anchoring and Analogical Prototype Reasoning for E-commerce RelevanceDeep dive: K-CARE: Knowledge-driven Symmetrical Contextual Anchoring and Analogical Prototype Reasoning for E-commerce Relevance.
  7. Generative · 2026-10-29Latent Residual Reasoning for Ranking in Generative RecommendationGenerative ranking models usually score every candidate in one forward pass. Refiner, from Meituan and the University of Science and Technology of China, adds a few more steps. Each candidate gets a reasoning token, and each step predicts only a correction added to the original score. The base model stays frozen; only LoRA and a new output layer are trained. On Steam it lifts MTGR's AUC from 0.76…
  8. Generative · 2026-08-28E2Rank: Your Text Embedding can Also be an Effective and Efficient Listwise RerankerDeep dive: E2Rank: Your Text Embedding can Also be an Effective and Efficient Listwise Reranker.

Discussion

Add experiment details, reproduction notes or a different take. Requires a GitHub sign-in.

Loading the discussion…