DuoLM reads item content offline and reserves online computation for deciding whether a particular user will like it. An encoder language model compresses each item, while a ranker uses those representations with user history and feedback. The system design turns that reuse into measured serving gains.
1. Problem and motivation
A full-text cross-encoder reads item descriptions together with user context. Its representations depend on the request and cannot simply be cached across users. Repeating this work for every candidate makes language-model recommendation expensive.
A compact item vector is cheaper, but a generic text embedding may discard details needed for personalized judgments. DuoLM trains content compression and personalized ranking together.
There is also overhead outside the model. Candidates in one request share a user history, yet input construction, transmission, and scheduling can repeat. Prefix caching does not necessarily remove those operations. As model execution gets faster, they become a larger share of total cost.
2. Method
The encoder reads templated item text and uses its final EOS hidden state as one item representation. A continuous interface normalizes and calibrates its scale against the ranker’s native token embeddings. Both history items and candidates use this interface; textual feedback remains beside the relevant history entries.
During training, ranking gradients flow through the interface into the encoder. Item vectors are recomputed while learning and cached only after training. A frozen full-text teacher supplies scores for the exact histories and candidate groups used by the student. The student learns both observed targets and the teacher’s relative preferences.
The paper compares feedback in the encoder with feedback in the ranker. The former caches an item–rating pair; the latter caches one representation per item. Their quality is close, and the latter provides a simpler reuse boundary.
At serving time, virtual token IDs reference a GPU-resident item-vector table. Requests carry IDs and feedback instead of transmitting complete vectors. These IDs are an access mechanism for continuous representations, rather than generated semantic item codes.
The multi-item scoring path prefills the user history once and shares its key–value cache across candidate branches. A tree-structured mask keeps branches independent. CUDA Graph replay further reduces launch and scheduling overhead. These changes save work at different points in the request path.
3. Main contribution
The central design boundary separates reusable content understanding from request-specific preference judgment. The representation is shared across users, the history computation is shared across candidates, and candidate scores remain independent.
The paper measures complete requests rather than treating lower model FLOPs as proof of faster serving. Runtime ablations help explain where the end-to-end improvement comes from.
4. Results
Recommendation quality is evaluated on Amazon Books and Video Games. NDCG@10 rewards relevant items appearing earlier in the ranking; absolute values should not be compared across datasets.
| Model | Books | Video Games |
|---|---|---|
| HLLM | 0.5508 | 0.5025 |
| DuoLM | 0.5586 | 0.5298 |
| Full-text teacher | 0.5440 | 0.5313 |
The student exceeds the teacher on Books and approaches it on Video Games. This demonstrates useful compression in these settings, not lossless compression for every domain.
On one H100, with 100 candidates per request, high-concurrency throughput reaches about 26,400 candidate scores per second. The unit is candidate scores, not user requests. Full-request p99 latency is about 537 ms at that concurrency.
The teacher comparison includes vLLM with automatic prefix caching. The gain combines a different model representation with a specialized serving path; it cannot all be attributed to switching inference engines.
Ablations also favor restraint: the simple EOS readout beats the tested learned readouts, stronger distillation does not keep improving quality, and extra hidden-state self-alignment hurts the reported setting.
5. Limitations
The evidence covers two product categories and specified hardware, without an online UGC search experiment. It does not establish behavior on rapidly edited posts, long comment threads, or breaking events.
Two engineering implications follow from the architecture. First, cached item vectors need versioning, incremental refresh, and memory budgeting. Moving encoding offline relocates its cost rather than eliminating it. Second, prefix reuse requires identical user history and context; candidate-isolation masks must remain correct.
Independent scoring also differs from list-level optimization for diversity or competition. Evaluation should preserve the paper’s candidate construction before expanding to a production recall pool. NDCG values from different candidate protocols are not interchangeable.
6. Practical implications
A content-rich ranking stage is a useful trial setting. Compare compressed and full-text scoring on long-tail items and narrow preference segments, where a single vector might hide important details.
Measure cache hits, history prefill, input preparation, and GPU execution separately. Report both candidate throughput and request throughput, plus tail latency. For UGC, include new and edited posts, recording the time until refreshed vectors become visible and the representation version used for ranking.
Sources: Sections 2–4 and Appendices A, C, and D. All figures are original assets from the paper’s arXiv HTML.