Advancing Generative Recommendation Inference: our collaboration with NVIDIA on TensorRT LLM

Spotify runs generative podcast recommendation with semantic IDs, and a single online request needs three kinds of inference: beam search to generate candidates, scoring a given batch of candidates, and inserting the user vector into the prompt as one soft token. These used to be split across two inference engines, vLLM and SGLang, and vLLM's beam search could not saturate the GPU, so Spotify fell back to lower-tier cards. Spotify and NVIDIA added the missing features to TensorRT-LLM and unified all three in one engine. On Qwen3-1.7B, beam search cost drops by up to 54% and scoring cost by up to 7%.

Teams that have shipped or plan to ship generative recommendation should read this. It is not a methods paper but a very concrete engineering record: the inference features generative recommendation needs are quite different from chat, and the mainstream open-source engines each support only part of them. The post breaks the requirements down clearly and shows throughput comparisons after closing the gaps. The most useful part is the three requirements themselves, which almost every team using semantic IDs for retrieval and LLMs for ranking will run into.

Background: what inference a recommendation request needs

In January 2026 Spotify launched LLM-based generative retrieval for podcast recommendations. The approach follows PLUM (a paper that adapts pre-trained LLMs into generative recommenders): each podcast is represented by a short string of semantic IDs, the model reads the user information and generates that code one token at a time, and the generated code is mapped back to a podcast.

To turn this into a service across many surfaces, the inference engine has to support three things at once.

The first is beam search. Generating a code cannot follow only the single most likely path; it keeps several of the highest-scoring prefixes, expands and prunes at each step, and the complete codes left at the end are the candidate podcasts. So the engine must not only support beam search but return the results of multiple beams, not just the best one.

The second is candidate scoring. Sometimes the candidates already exist and the LLM only has to judge which are better. The candidates go into the prompt, the model's log probabilities for label tokens such as "yes" or "no" are read out, and the candidates are ranked by them.

The third is personalization. Recommendations must know who is listening, and Spotify already has a mature set of dense user embeddings. How to hand that vector to the LLM is another feature the engine has to support.

Why beam search could not saturate the GPU in vLLM

The January launch used vLLM. It is one of the most widely used open-source inference engines and handles ordinary completion requests very well, but its beam search is weaker. The post names two reasons. First, vLLM implements beam search outside the core GPU engine, so much of the scheduling falls on the CPU and the CPU becomes the bottleneck. Second, it recomputes partially filled KV blocks, adding extra GPU work. My reading: because beam search lives outside the engine, each expansion step effectively resubmits every beam as a new request; vLLM's prefix cache only reuses full KV blocks, so the tokens already in the last, partially filled block of each beam are recomputed at every step.

As a result, high-end GPUs sat underused. To control cost, Spotify launched on lower-tier cards and kept the system at a better load point. This is a typical trade-off: if the engine cannot saturate high-end cards, cheaper cards give a lower unit cost.

Scoring many candidates at once

The most direct way to score candidates is to fan out: send one request per candidate, read each one's label probabilities, and aggregate. More candidates means more requests, and the same user prefix is computed over and over.

SGLang has a more efficient approach, which LinkedIn's engineering team also contributed to; call it multi-item scoring. The figure shows it clearly: the user prefix and all candidates are joined into one sequence, an attention mask lets each candidate see only the user prefix and itself, and a single forward pass scores every candidate. This avoids the overhead of one request per candidate while keeping compute linear in the number of candidates. That is why Spotify used SGLang for scoring.

Getting the user vector into the prompt: three approaches

The post compares three ways of giving user information to the LLM, which is very useful for anyone building LLM-based recommendation.

The first is to write the user's listening and interaction history as text in the prompt. Describing a user accurately takes a lot of text, and long prompts push latency beyond the request budget and raise GPU usage. So this is used sparingly, for example only for the user's most recent real-time streams.

The second is to give users semantic IDs too, as in PLUM: tokens representing users are added to the vocabulary, a handful of tokens stand for a user, prompts stay short, and inference needs no special support. The model is fine-tuned with the extended vocabulary. Early results were promising, but the post says this path needs more experimentation and is not in production yet.

The third works best: soft tokens, from Spotify Research's "Embedding to Prefix" method. The idea is to learn a projection that maps the existing user vector into the LLM's embedding space as a single token-sized vector. Fine-tuning trains two parts together: the projection layer (one linear layer) is learned end to end, and the LLM itself updates only a subset of its weights so it learns to use the vector. At inference the user vector is projected, inserted into the prompt, and then the first forward pass runs. The figure above shows the process. Podcast recommendation uses this method, and Spotify is extending it to other tasks on its internal LLM.

The benefit is reusing years of user embeddings from the recommender without lengthening the prompt. The cost is that the engine must let you replace the vector at one position before the forward pass, while most engines accept only token IDs by default.

Moving all three into TensorRT-LLM

After the launch, generative recommendation is expanding from podcasts to more content and surfaces, so an efficient, maintainable inference stack matters more. The goal of the Spotify and NVIDIA collaboration was clear: unify efficient beam search, scoring and user soft tokens in one engine.

TensorRT-LLM had better hardware utilization for beam search but initially lacked two things: a scoring API like SGLang's, and soft-token injection plus multi-beam result return like vLLM's. The two teams added the following, all now in released versions:

In TensorRT-LLM, beam search is implemented inside the sampler, which cuts CPU-side overhead, and together with overlap scheduling (the GPU keeps working while the CPU prepares the next step) it gets better GPU utilization. The old C++ sampler was also replaced by a more flexible design that relies more on PyTorch, and beam search on the new sampler is notably faster than before.

Soft tokens are built on out-of-tree model support. With NVIDIA's help, Spotify wrote a Qwen3 extension plugin: the user vector comes in through multimodal request parameters, and the plugin does the projection and replaces the placeholder token. This is a neat trick, because multimodal models already have to insert image vectors into the prompt, and reusing that path avoids touching the engine core.

Scoring got an extension to the Python API equivalent to SGLang's scoring API. For now, a custom serving layer is still needed for efficient tokenization and request batching.

Performance: how to read the two benchmark charts

Start with beam search. The test setup is close to generative retrieval: 2,000-token prompts, only 5 generated tokens (a semantic ID string is short), and beam width 30. Read the chart by where the latency curve turns sharply upward; that is where the system can no longer keep up. On the 1.7B model, vLLM loses control around 15 QPS, while TensorRT-LLM holds up to about 33 QPS.

Now scoring. With 500 candidates per call, the two engines' saturation throughput is almost the same, with TensorRT-LLM slightly ahead. SGLang was already good at scoring, so the main gain from moving is not speed but no longer maintaining a separate engine.

Cost

The post's conclusion: on Qwen3-1.7B, beam search workloads cost up to 54% less and scoring workloads up to 7% less. This roughly matches the saturation numbers above: the same card handles more than twice as many beam search requests, so the GPU cost per request falls by about half; scoring throughput is only a few percent higher, so the saving is small.

The indirect gains matter more. In the vLLM days only low-tier cards made sense; once the engine can saturate high-end cards, there is more room in choosing hardware. With all three workloads in one engine, deployment, monitoring and upgrades involve only one stack.

Limitations as I see them

First, this is a company engineering blog, and the benchmarks run under fixed conditions: 1,000 requests per rate point, a concurrency cap of 64, and unique prompts with no KV cache reuse. In real traffic a user's prefix is often reused, so results may differ; replay your own traffic to check.

Second, the versions and parameters of the compared engines are fixed. vLLM and SGLang move fast and their beam search implementations keep changing, so the gap may shrink in six months. Treat this as a record of where the bottlenecks were at the time, not a final verdict between engines.

Third, scoring currently needs a custom serving layer and soft tokens need a model plugin. For companies without a dedicated inference team, the migration cost is not small.

How a team could try it

1. List the inference features your generative recommendation needs: multi-beam return, batched candidate scoring, vector injection, and which engine handles each today.
2. Measure beam search saturation QPS with your own prompt length, beam width and decode length. Generative retrieval decodes very little with very wide beams, which is far from chat workloads, so generic benchmarks do not transfer.
3. If you want to give user vectors to the LLM, try soft tokens first: reusing the multimodal vector input path is usually easier than changing the engine core.
4. If scoring already runs well on SGLang, there is no rush to migrate; beam search is the better place to optimize first.

Bottom line

Generative recommendation needs different inference from chat: beam search, batched scoring and vector injection are all required. Filling those gaps and unifying them in one engine that saturates the GPU cut beam search cost roughly in half.