Contents · 7 sections
1What problem is it actually solving?2The big picture3Up close: how pairs are mined and how the loss is computed4Is this approach really better?5Limitations as I see them6How a recommendation/search team could try this7Bottom lineSHARE
Teams working on I2I retrieval or item embeddings should read this, mainly because it answers a question people argue about all the time: when you swap the item encoder for a multimodal large language model (MLLM), does the gain come from the model itself or from the supervision signal used to train it? CAMIE (Co-Engagement-Aware Multimodal Item Embeddings) comes from Snap's Dynamic Product Ads (DPA) team and has already replaced two production encoders. The core idea in one sentence: encode an item's image and text together into one vector with Qwen3-VL-Embedding 2B, then fine-tune it contrastively on "co-engaged item pairs" mined from user journeys, so that a single model serves both the multimodal and the text-only retrieval lines. Key numbers: on the same backbone, fine-tuning raises R@10 by 37.7%; under an identical training recipe, swapping backbones (MLLM, SigLIP2, CLIP) changes multimodal R@10 by at most 5.8%; and replacing the old multimodal I2I encoder online lifts that line's CVR by +10.832%.
What problem is it actually solving?
I2I retrieval takes a seed item and returns "related" items. In production the whole catalog is usually encoded offline into an ANN index, and each request does one lookup, which keeps latency low and also covers cold-start items.
Problem one: similar is not the same as bought together. An illustrative example:
| Illustrative | Seed: white linen shirt | Content similarity retrieves | What users actually view and buy together |
|---|---|---|---|
| Item A | – | Another white shirt (looks alike) | Linen trousers from the same shop |
| Item B | – | White T-shirt (same color and category) | The same line in another color |
| Item C | – | White bed sheet (same material words) | Matching straw sandals |
Models like CLIP/SigLIP learn "does this image match this text". Used directly for I2I, they retrieve items that look or read alike, while ads care about the items users will click or buy next.
Problem two: fragmented modalities. Production systems often maintain separate text-only, image-only and multimodal encoders, each with its own fine-tuning, refresh pipeline and ANN index. The paper wants one model and one training run that can serve all of these lines.
What the paper really wants to answer is an attribution question: how much of the gain comes from a stronger MLLM backbone, and how much from co-engagement supervision. It separates the two with a controlled experiment, which is the step it takes beyond similar work such as Xiaohongshu's NoteLLM.
The big picture

| Step | What it does | Why it is needed |
|---|---|---|
| 1. Mine pairs | Find two items the same user engaged with within a short time, from 30 days of user journeys | Direct evidence of "viewed and bought together" |
| 2. Render input | Concatenate the image and "Title: … Brand: … Description: … Category: …" into one record | Switching modality only changes which fields are included; the model stays the same |
| 3. Encode | Frozen Qwen3-VL-Embedding 2B plus LoRA, last-token pooling, then a projection head | The backbone already has a multimodal interface, so no separate fusion module is needed |
| 4. Train | Symmetric InfoNCE, with negatives from the rest of the batch and from other GPUs | Pull co-engaged items together |
| 5. Serve | Encode offline when the catalog refreshes, write into the ANN index, one lookup per request | The online path matches the old content I2I, so the latency profile does not change |
Up close: how pairs are mined and how the loss is computed
What rules are used to mine pairs? For each user, the 30-day journey is deduplicated by item, and at most 2 pairs are sampled per user. Four behaviors are kept (view, click, add-to-cart, purchase), not just high-intent ones. A set of filters follows. Attribution must be of what the paper calls the "exact match" type: literally, the engagement can be attributed precisely to a specific ad item, and the paper gives no finer definition. Beyond that, title and image must be non-empty, the two items must belong to the same advertiser, their top-level categories must match, and the time-decay weight must be at least 0.5. Time decay uses a 7-day half-life: the weight is 0.5 raised to the power (days apart / 7).
Working through what this rule means: two engagements 3 days apart get weight 0.5^(3/7) ≈ 0.74 and are kept; 7 days apart gives exactly 0.5, right on the line; 10 days apart gives about 0.37 and is filtered out. So "weight at least 0.5" is equivalent to keeping only co-engagements within one week. Why "same advertiser"? The paper's general rule requires both items to belong to the same "serving scope", meaning the set of items that are allowed to retrieve each other online; in the DPA experiments this becomes "same advertiser". Pairs outside the scope can never be retrieved online anyway, so learning them is useless. Offline experiments use 10M training pairs and 100K test pairs; the production model uses 100M pairs mined with the same pipeline.
How is symmetric InfoNCE computed? A batch has N item pairs. For pair n, treat i_n as the query, j_n as the positive and the other N−1 j's as negatives, and compute one softmax cross-entropy; then reverse it, with j_n as the query and the i's as candidates, and average the two.
Illustration: a batch of 3 pairs, where i_1 has cosines 0.8, 0.2 and 0.1 with the three j's, and the temperature is 0.5. Dividing by the temperature gives 1.6, 0.4 and 0.2; exponentiating gives about 4.95, 1.49 and 1.22; the positive's probability is 4.95/7.66 ≈ 0.65, and this term's loss is −ln 0.65 ≈ 0.44. Why symmetric? Because co-engagement has no direction: shirt→trousers and trousers→shirt are the same relation, so both directions should be learned. The paper actually uses a learnable temperature initialized at 0.07.
Why is no retraining needed to switch modality? For the text-only line, the image is simply left out of the input; for the image-only line, only the image is included. The backbone and projection head are unchanged. That is what "one checkpoint, several lines" means.
Is this approach really better?
Per Snap's policy the paper reports only relative values. Offline, "un-fine-tuned Qwen3-VL-Embedding 2B with multimodal input" is 100%, and at test time each query item must find its paired item among all items that appear in the test set.
| Model | Type | Input | R@1 | R@10 | MRR |
|---|---|---|---|---|---|
| Qwen3-VL-Embedding 2B | CAMIE fine-tuned | Multimodal | 126.4% | 137.7% | 130.5% |
| Qwen3-VL-Embedding 2B | Not fine-tuned | Multimodal | 100.0% | 100.0% | 100.0% |
| Qwen3-VL-Embedding 2B | CAMIE fine-tuned | Text only | 117.8% | 122.5% | 116.4% |
| Gemini Embedding 2 | Commercial API | Multimodal | 129.1% | 129.9% | 129.4% |
| Gemma 4 E2B | CAMIE fine-tuned | Multimodal | 94.5% | 108.2% | 100.0% |
Three points come out of this table. First, on the same backbone, fine-tuning lifts R@10 by 37.7%, which shows behavioral supervision itself is the bulk of the gain. Second, before fine-tuning, adding the image is actually worse than text alone (text-only 111.1% vs multimodal 100%); only after fine-tuning does the image start to help (137.7% vs 122.5%), so the value of images is unlocked by behavioral supervision. Third, CAMIE beats the strongest commercial model, Gemini Embedding 2, by about 6% on R@10 with much smaller vectors (512 vs 3072 dimensions), but Gemini is still slightly ahead on R@1.
Can one model replace the text-only encoder? When the multimodally trained model is fed text alone, its R@10 is 87.9% of multimodal, versus 89.0% for a model trained specifically on text, a gap of only 1.1 percentage points. This shows the text-only line can be merged into the same model. Image-only input reaches just 67.0%, and a dedicated image-only version only 68.8%, which shows the problem is not the merging but that this backbone is simply weak at images alone.
Does the gain come from the backbone or the supervision? This is the key experiment of the paper. The authors trained two non-LLM dual encoders with the exact same recipe: SigLIP2-large and CLIP ViT-L/14. Pairs, loss, number of negatives, output dimension, LoRA settings, learning rate and steps are all identical; the only difference is the backbone.

This shows behavioral supervision is the bulk of the gain: after training the three backbones on the same pairs, multimodal R@10 differs by at most 5.8%, while fine-tuning itself brings 37.7%. The MLLM's advantage lies in two places: on multimodal it beats SigLIP2 by 4.3 percentage points, and the authors ran paired bootstrap and permutation tests with p<0.001; on text-only its lead is much larger. Image-only should be left to a dedicated vision tower. The authors also give a training curve: at 40% of one epoch the two dual encoders have essentially saturated, while the MLLM keeps improving to the end, and its gap to SigLIP2 widens from 2.3 percentage points at 20% to 4.3 at the end.
There is also a comparison of data volume versus purity. Using only add-to-cart and purchase pairs gives about one third of the full set. Taking "a random one-third sample of all-behavior pairs" as the baseline: at the same one-third scale, high-intent pairs alone are only 1.6% higher on R@10; using all-behavior pairs at full scale is 8.9% higher. This shows that for this task, keeping more data pays off more than keeping only high-intent behavior.
How did it do online? The online test is a randomized A/B with all other serving components unchanged:
| Comparison | CTR | CVR |
|---|---|---|
| Overall DPA traffic | +0.211% (not significant) | +1.911% |
| Line where the multimodal I2I encoder was replaced | +0.390% (not significant) | +10.832% |
| Line where the text-only I2I encoder was replaced | +18.958% | +13.12% |
The three CVR lifts and the text-only line's CTR lift are significant at 99% confidence. The old multimodal production encoder was itself SigLIP2-large, just trained on content labels. Combined with the matched experiment above, this shows the online gain comes mainly from switching the supervision to co-engagement, not from switching the backbone.
Limitations as I see them
First, the offline test set itself favors this kind of training. Test pairs and training pairs are mined from co-engagement with the same pipeline, so any model trained on this signal has an advantage, and un-fine-tuned backbones and commercial APIs are naturally handicapped. The authors themselves admit the offline split "cannot price that difference fairly". So the 37.7% and "beats Gemini by 6%" say more about "aligning to your own behavior distribution helps" and cannot be read as a comparison of general embedding quality.
Second, the matched comparison may be unfavorable to the dual encoders. The effective batch is only 32 to 64, because the 2B MLLM cannot fit a larger batch on 40GB GPUs, so the authors fixed the number of negatives for all backbones at the MLLM's limit. Contrastive models like CLIP/SigLIP usually rely heavily on large-batch negatives; pinned to such a small batch, part of their roughly 5% gap may come from this constraint. The paper says this makes "compute and negatives matched rather than favorable to CAMIE", but gives no large-batch control for the dual encoders.
Third, the online and offline models are not the same. The model in the offline tables is trained on 10M pairs and outputs 512 dimensions; the production model is trained on 100M pairs and, to keep the index the same size as the old one, uses only the first 128 dimensions. The paper does not say whether training accounted for the truncation, nor does it report 128-dimension offline numbers, so the offline tables do not directly describe the production model's quality. The old text-only encoder was also never evaluated offline, so its +18.958% CTR has no offline counterpart.
How a recommendation/search team could try this
- Do not switch backbones first. Take the content embedding model already in production (for example a CLIP or SigLIP variant), fine-tune it contrastively on item pairs co-engaged by the same user within a week, and denoise with the paper's filtering rules. By the paper's own conclusion, this step captures most of the gain.
- Use two test sets: one of co-engagement pairs mined the same way, and one of human relevance labels unrelated to the training signal, so the first set's bias toward behavioral supervision does not mislead you.
- If you maintain both multimodal and text-only encoders, then try an MLLM backbone that serves both lines by including or omitting the image, and compare the text-only line against the dedicated text model. Keep a dedicated vision tower for the image-only line.
- When mining pairs, compare "high-intent behavior only" against "all behavior, more data", once at equal scale and once at different scales, to see whether volume or purity matters more on your own data.
- Before launch, measure offline quality at the truncated dimension, then decide the index dimension.
Bottom line
Item embeddings should align with user behavior, not with content itself; CAMIE uses a clean controlled experiment to show that behavioral supervision delivers the vast majority of the gain, and that the MLLM's value lies mainly in letting one model support both the multimodal and the text-only retrieval lines.
| Dimension | Assessment |
|---|---|
| Importance of the problem | High. I2I retrieval and item embeddings are everywhere in e-commerce and ads, and "should we switch to an LLM backbone" is a real decision |
| Novelty of the method | Medium. Co-engagement contrastive learning and MLLM embeddings are not new; the novelty is the rigorous attribution control |
| Experimental evidence | Medium-high. Matched-recipe control with significance tests, plus a randomized A/B; but the offline set favors behavioral supervision and the online model differs from the offline one |
| Engineering cost | Medium. A 2B backbone plus LoRA costs more than a dual encoder to encode the catalog offline; the online path is unchanged |
| Worth trying? | Yes. First add behavioral supervision to your existing content model, then consider switching backbones |
| Where it fits | Item I2I with user journey logs that must serve both multimodal and text-only retrieval |
How did this read for you?
Hoot made a note. Thanks for the feedback.
One tap, no sign-in; tap again to undo. Only anonymous counts are kept, never who you are.
Related papersMore from Paper Daily
Personalization · 2026-11-11Query- and Candidate-Aware Long User Sequence Modeling for Sponsored Product Ranking in SearchWhen sponsored search ranking uses long-term behavior, the usual approach searches history for actions similar to the candidate product, which can miss history highly relevant to the current query but only loosely tied to the candidate. Beyond recent behavior, Wayfair retrieves one batch of long-term actions by candidate and another by query, then feeds them to a Transformer with two context toke…
Personalization · 2026-10-29Meta Andromeda: Powering the Next Generation of Ads Retrieval Through Model-System Co-DesignMeta's next-generation ads retrieval engine Andromeda brings near-ranking-quality models to retrieval, scoring every eligible ad. The new version replaces the single user tower with several specialized user towers as experts, and encodes the ad tower with real-time request context. An attention-based gate in the interaction layer activates only the most relevant experts per ad, cutting inference…
Personalization · 2026-09-02SetMIR: Multi-Interest Retrieval as Set PredictionDeep dive: SetMIR: Multi-Interest Retrieval as Set Prediction.
Discussion
Add experiment details, reproduction notes or a different take. Requires a GitHub sign-in.
Loading the discussion…