KVStream: Adaptive Parallel Storage Management for KV Caching in Large-Scale LLM Serving
Long-context inference keeps reusing the same prefixes: system prompts, whole documents, code repositories and multi-turn session histories.
Looking for a title? Use site search.
Long-context inference keeps reusing the same prefixes: system prompts, whole documents, code repositories and multi-turn session histories.
By moving generative podcast recommendation off two separate inference engines and onto TensorRT-LLM together with NVIDIA, Spotify cut beam search cost by up to 54% and scoring cost by up to 7% on Qwen3-1.7B.
Quantizing a trained LLM straight to FP4 lets Blackwell GPUs use native FP4 matrix multiplies, but accuracy drops noticeably.
Mixture-of-experts (MoE) models activate only a few experts per token, so compute is cheap, but every expert's parameters must stay in GPU memory, and memory is usually the first thing that blocks deployment.
In generative recommendation with semantic IDs, the model writes an item's code one token at a time, and beam search keeps only the highest-scoring prefixes at each step.
In Adobe Express's AI assistant, retrieval metrics that assume the user saw a ranked list and acted on it stop working.
Ask a strong LLM to play a search user, and a discriminator that never saw the training data recognizes its utterances with AUC 0.88.
Ask a question about election participation, and a complete answer should cover three kinds of people: those who voted, those who did not, and those who were not eligible.
In RAG, in-context example selection and agent memory, the same query usually goes through two models: an embedding model first encodes it into a vector for retrieval, and then a generation model reads it again to answer.
Across the rerankers the authors evaluated, nDCG cannot tell them apart on 45.6% of NanoBEIR queries.
On LinkedIn, a software engineer who starts looking for a job usually wants to open the profiles of recruiters, hiring managers and HR, not other engineers.
A recommender places a film in a row of items, and the user does not click it. Should the system record that the user doesn't like it?
Move a large model to a different GPU or framework version, and the training recipe that used to be fastest often no longer is.
When NVIDIA's GDS reads data from SSD straight into GPU memory, the authors measured that 99.6% of the time for asynchronous reads was stuck at the step of submitting requests.
The KV cache a large model keeps in GPU memory grows with the text, and one way to compress it is to make the vector stored for each word shorter, but most existing methods only save storage.
When an agent goes back and forth with tools such as search, web pages and code execution for hundreds of turns, it pays a recurring cost.
In long-context decoding, every generated token reads the whole KV cache from HBM, so the longer the context, the slower attention gets.
Ask LLMs to "name a running shoe brand" and they will most likely say Nike; ask them to "recommend a running shoe brand" and they are more likely to say Brooks.
Two headphone listings are identical in price, star rating and specs, except that one adds the line “Our best-selling model, chosen by over 10,000 customers.” Across six such pairs, GPT-4o picked the listing with that line in all 144 trials.
Hybrid retrieval fuses the results of term-matching sparse retrieval such as BM25 with those of dense vector retrieval into a single ranking, and it always wins on average.
Chunking schemes for RAG are almost always compared on one static corpus, yet the documents in a knowledge base keep changing.
People writing proofs and coding agents alike must first find suitable theorems in Mathlib, the mathematics library of the Lean proof assistant.
In the Google Discover ads team's model iteration, offline and online results did not line up.
When a recommendation team maintains ranking models, contextual bandits, uplift models and sequential recommenders at the same time, it usually ends up with one library for each, plus separate glue code for evaluation, each with its own interface.
More and more inference requests now come from multi-step agents and reinforcement-learning rollouts, which share very long prefixes and arrive in bursts.
Generative recommendation generates a product's semantic IDs layer by layer, and before the first layer it has seen no product-side information at all; this is its gap with discriminative ranking models.
When a large language model serves as a recommendation foundation model and first writes a natural-language reasoning trace before outputting the next item's semantic ID, the reasoning text is far longer than the semantic ID and holds back both decoding time and throughput.
The Netflix homepage used to be assembled by a multi-stage recommender that retrieved candidate rows and titles, ranked them separately and then built the page; GenPage instead generates the whole page with one decoder-only Transformer.
Generative recommendation generates the next item's semantic IDs one code at a time, and attention cost grows quadratically with sequence length, so these models usually take only the most recent few dozen behaviors and leave the long-term history unused.
Perplexity releases 0.6B and 9B multi-vector retrievers that encode text, images, and document screenshots: each document token stores a 128-d vector, and scoring sums MaxSim of each query vector against document vectors.