Taobao search's fine-ranking CTR models mainly remember historical clicks through user, query, and item ID features. The same item with a different main image or a rewritten title keeps the same item ID, so the model's score barely changes; newly listed, rarely clicked items have learned little from the ID itself. Alibaba's FineSID splits item images and text into fine-grained discrete codes and looks them up together with ordinary ID features into the same ranking network: images become 30 position-aware codes across four VAR tokenizer scales from 1×1 to 4×4; titles and OCR text are tokenized per field. Training has two stages: first train the whole network on LLM-labeled query–item relevance, then swap in a click head and fine-tune all parameters on click logs. Offline AUC is 0.25 points above the HiFormer baseline and 0.38 points higher on cold items; a seven-day Taobao main-search A/B test shows click +0.21% and GMV +0.85%.
The overall structure has three stages: offline, turn content into discrete codes; pretrain the shared network on relevance labels; fine-tune for personalization on click labels. Stages two and three share one network and only swap the output head.
1. Problem: how content enters an ID model
Industrial CTR models are mainly ID embeddings plus a feature-crossing network. Multimodal content usually enters in one of three ways. First, jointly train an image encoder end to end—too expensive. Second, concatenate continuous vectors from a pretrained model, for example the LEARN-style baseline that concatenates Qwen3.5 content vectors; continuous vectors and sparse IDs have different distributions and often only fuse in late layers. Third, semantic IDs (SID): quantize a whole item's content vector into a few discrete codes and look them up like IDs—RQ-VAE SID is representative.
The authors agree with the third direction but say the granularity is too coarse. RQ-VAE compresses a whole image into a few codes and loses spatial detail such as "is there a promo badge in the top-left" or "what color is the model wearing"; two main images of the same item can get the same codes. FineSID wants to keep those details without running an image model online.
2. Method
2.1 Content → discrete codes
Images use VAR's pretrained tokenizer. VAR generates images scale by scale; its tokenizer is trained for reconstruction and encodes an image into multi-scale code maps. FineSID takes scales 1×1, 2×2, 3×3, and 4×4—1+4+9+16 = 30 codes per image. Coarse scales capture overall style; fine scales capture local regions.
Codes are computed offline and stored; online serving only adds 30 table lookups and does not run an image model. The ranking network's embedding tables for these codes are randomly initialized and trained with the task, independent of VAR's reconstruction codebook. Flattening keeps a fixed "which scale, which cell" order, so the same position always enters the same input slot.
For text, query, title, and OCR each use their own BPE and n-gram vocabularies; each field stays an independent token sequence rather than being collapsed into one summary vector.
2.2 Shared prediction network
The three input types meet as follows. Text fields first do cross-attention: query embeddings as queries, title and OCR token embeddings as keys/values, yielding text representations weighted by the current query. Then ordered fusion concatenates, in a fixed order, the original ID-feature embeddings, the 30 visual-code embeddings, and each field's text representation.
The long vector is projected to 768-d, then through two HiFormer layers. HiFormer is Google's feature-crossing structure; here its latent-slot attention maps inputs onto a small fixed set of slots to avoid quadratic self-attention over thousands of tokens. An MLP 768→256→64 yields a 64-d shared representation for the output head.
2.3 Relevance first, then clicks
Stage 2 pretrains on relevance. Each impression gets a four-way soft relevance distribution from a Qwen3.5 model fine-tuned on millions of human labels; the network fits soft-label cross-entropy:
is the teacher's probability for tier , the network's prediction, shared-network parameters, the relevance head. The intent is for SID embeddings and crossing layers to learn "does this image and title match this query" first—clicks alone learn that slowly because clicks mix price and brand preference.
Stage 3 drops the relevance head, attaches a randomly initialized click head, and fine-tunes all parameters—including SID embeddings, ID embeddings, crossing layers, and normalization stats—with binary cross-entropy on click logs. The authors stress this is full-parameter transfer, unlike migrating only embeddings and discarding upper layers.
3. Experiments
Data are real Taobao search impressions: seven days Aug 25–31 for training (~51.2B impressions, 251M users, 418M items); one day Sep 1 for eval (453M impressions). Metrics are AUC, query-grouped GAUC, and the same two on the cold-item subset with under 1,000 views that day. Taobao numbers are absolute differences versus the HiFormer baseline, in percentage points.
Main results versus other content-injection methods. This shows fine-grained codes beat whole-image vectors and coarse SIDs. Controls all add content on the same HiFormer backbone: Visual ID quantizes visual vectors into discrete IDs; COINS aligns SID content with collaborative signals for cold start; PRECTR and Joint relevance + CTR are two ways to bring relevance supervision into CTR.
FineSID leads every column; cold-subset gains exceed full-set gains, matching "content codes fill in when ID signal is scarce." LEARN-style continuous concatenation only gains +0.05 AUC; Joint relevance + CTR is below the baseline on the cold subset.
Same-item creative discrimination. This rules out "the model only memorized item IDs." Item ID is fixed; only creatives (main image, title) differ. Metric is NSCTR: for each impression, the model picks the highest-scoring creative among that item's candidates; a match with the logged creative counts, and matching impressions' CTR is weighted by the item's original impression count. Candidates share the exact same ID, so score gaps can only come from content. FineSID's relative NSCTR gain is 0.75%; COINS is 0.27%.
Ablations: what each stage contributes.
Dropping SID nearly zeros the gain and slightly hurts cold AUC—most of the lift is the content codes themselves. Relevance pretraining adds another 0.05 AUC and matters more for NSCTR (0.51% → 0.75%). Mean pooling cuts more than half of the AUC gain, so keeping position and field identity helps.
Public MMCTR. MMCTR has no relevance labels, so only the version without stage 2 is tested. HiFormer baseline AUC 0.9767; FineSID reaches 0.9844 (+0.77 points), above COINS 0.9825 and LEARN-style 0.9823.
Online A/B. Seven days on Taobao main search versus the production model: click +0.21%, GMV +0.85%, both significant. The authors report no significant fine-rank latency increase because the 30 visual codes are extracted offline and online only adds lookups and a wider first projection.
4. Limitations
The authors list three: the VAR tokenizer is trained for reconstruction and frozen in stage 2, so codes are not optimized for search semantics; relevance pretraining is only shown transferring to CTR, not to retrieval or conversion; 30 codes widen the concatenated vector and need more work for million-candidate retrieval.
One more: MMCTR only tests the version without relevance pretraining, so the full three-stage pipeline has evidence from only one Taobao data source; online GMV +0.85% with click only +0.21% is not broken down by which item types drive the GMV lift.
5. Takeaways for fine-ranking teams
If your fine-ranker already has a content-vector feature with little gain, try FineSID's idea: replace images with multi-scale, position-aware discrete codes that share the same embedding lookup path as IDs; online cost is mainly storage and lookups.
If you already have relevance labels or a relevance model, pretrain the whole ranking network on it, then fully fine-tune on clicks. In the ablation this step helps same-item creative discrimination (NSCTR) the most—useful when you need multi-image, multi-title creative selection.