C 2 KV: Compressed and Composable KV Cache Reuse for Efficient LLM Inference

Teams running long-context inference, especially RAG (retrieval-augmented generation) services, should read this. It points out something easy to miss: when you reuse a document's KV (key-value) cache, the real bottleneck has moved from recomputation to moving large caches from host memory onto the GPU. The core idea in one sentence: attach a small trainable module to a frozen LLM that compresses each document's KV cache to a quarter of its size, in a form that can be concatenated in any order without repair computation. In the paper's Figure 1 example, time-to-first-token for 8 documents drops from 1.06 s of full prefill to 0.1 s of pure loading; on Llama3.1-8B with 4× compression, the MuSiQue multi-hop QA score is 0.3587, higher than full computation at 0.3198.

What problem is it solving?

Before generating, an LLM runs a prefill pass that computes keys and values for every input token; stored, these form the KV cache. Longer inputs mean slower prefill. A RAG service injects several retrieved documents into each request, and the same document is reused across many requests, so precomputing and storing each document's KV cache is the obvious move.

The difficulty is that documents do not appear at fixed positions. An illustrative case: request A uses documents 3, 4, 7 in that order, request B uses 7 then 3.

Repair is needed because document 3's cache was computed alone and never "saw" document 7, so it deviates from a joint computation. Recomputing fewer tokens means larger deviation.

Even with fast repair, loading the full cache alone takes 0.42 s. Caches usually live in host memory and are moved to the GPU on demand, so transfer dominates as context grows. Compressing the cache is the natural fix, but the authors show that stacking a generic compressor (SnapKV) on top of reuse methods collapses accuracy: on HotpotQA, Block-Attention drops from 0.4271 to 0.0743.

Pipeline: extract, store, concatenate

Offline, each document is passed alone through the frozen model plus the C2 Extractor, producing a compressed KV (C2KV) that is stored. Online, the needed C2KVs are retrieved, given positional encodings for their new placement, and concatenated directly before the user query.

C2 tokens as memory slots

With compression ratio k, one C2 token is appended per k document tokens. All C2 tokens share one learnable embedding and carry no text. All base model weights stay frozen, and the original query/key/value projections act only on document tokens; C2 tokens get their own trainable per-layer query/key/value projections, about 10% extra parameters. The projected C2 tokens then join the document tokens in standard attention, and everything after that, such as the output projection and the FFN, still runs on the frozen base weights: the paper adds separate weights only for the C2 tokens' query/key/value projections. Only the C2 tokens' KVs are kept. With k=4, a 4,000-token document is stored as 1,000 KV entries.

Structured Information Flow

An attention mask enforces three rules. Document tokens cannot attend to C2 tokens, so their computation is identical to the base model. Each C2 token attends only to its own block of k document tokens plus the sink block B(0); in the official code this is the document's first k tokens (the first block itself, no extra slot), since attention in LLMs tends to concentrate on the earliest tokens. For k=4, slot 2 sees tokens 1–4 and 5–8, slot 3 sees 1–4 and 9–12; all other blocks are masked. C2 tokens attend causally to earlier C2 tokens, accumulating document-level context. For k=4, the second C2 token summarizes tokens 5–8 plus what the first slot already holds, and never sees token 9 onward. Because each slot summarizes only its own span, it carries no wrong "preceding context" when moved. Hoot says▾

Hoot saysSlots see only their own block: vs a plain causal maskClose ✕

Under a plain causal mask, later slots would absorb the whole preceding text. C2KV lets a slot see only the sink block and its own block, so its cache stays valid wherever it is spliced in.

• document cells a slot sees

• document cells a slot sees
• slots: causal among themselves

• document → document: the base model's causal mask, unchanged
• sink block B(0): every slot sees it
• slot sees: its own block + earlier slots (and itself)
• masked

1. Document cells the slots see (illustrative: 3 slots, 12 tokens, k=4): plain causal 3×12 = 36; C2KV only sink + own block: slot 1 sees 4 (its block is the sink), slots 2 and 3 see 8 each, 20 in all.
2. Rule 2, row s2: blk1 (sink) + blk2 visible, blk3 masked, so it summarizes tokens 5 to 8 and never sees token 9 onward.

Rows are queries, columns are keys. Illustrative: k=4, a 3-block document of 12 tokens and 3 slots; slots are drawn after the document only for the picture. Slot 1's own block is the sink block itself. Source: the three mask rules and the k=4 example in this dive's structured-information-flow section (per the official code, sink block B(0) is the document's first block)

Positions at concatenation

KVs are stored before RoPE (rotary positional encoding) is applied. At inference, positions are reassigned by placement, and each compressed segment occupies the span of its original tokens. During extraction slot j takes the position of the last token in its block (3, 7, … for k=4). At concatenation each segment is shifted by the accumulated offset, so a 400-token first document keeps slots at 3, 7, …, 399 and the second document starts at position 400, its first slot at 403.

Training compression and concatenation together

Training uses multi-document QA from HotpotQA, 2WikiMultiHopQA, and LongMagpie. Each document is extracted separately, the caches are concatenated, and the model answers from the concatenated cache; only the answer's language-modeling loss trains the extractor. Since document count and order vary across samples, the extractor must produce caches that work under any composition.

Results

From Table 1 (Llama3.1-8B). MuSiQue and 2WikiMQA are question answering, scored by F1 (word overlap between the model's answer and the reference answer); SAMSum and MultiNews are summarization, scored by ROUGE-L (overlap between the generated and reference summaries, based on their longest common subsequence). All scores run from 0 to 1, higher is better:

At 4×, three of four tasks beat full computation, which suggests the multi-document QA training also helps the model use several documents. Ablations confirm each design: removing position re-alignment drops Qasper from 0.3755 to 0.2827, and replacing the structured flow with anchor tokens, extra tokens from another compression family that attend bidirectionally with document tokens and therefore rewrite them, drops it to 0.2750. Decode time also benefits: from 16k to 128k context, full-cache per-token latency grows almost linearly while C2KV's curve stays much flatter.

Limitations I see

First, every base model needs its own trained extractor plus multi-document QA data; a model upgrade means retraining. Second, summarization loses quality and the ratio–quality relation is unstable: MultiNews at 4× is 0.2532 versus 0.2685 for full compute, and SAMSum at 8× is 0.2572 while 16× recovers to 0.3556, so each ratio must be tested per task. Third, training focuses on QA, so the extractor learns what helps answer questions rather than faithful reconstruction; code, tables, and long dialogues are not evaluated.

How an inference team could try it

Measure first: split TTFT in your RAG service into cache loading and prefill; this only pays off when loading dominates. Then train a 4× extractor with the official code on your current base model for a frequently reused document set, compare answer quality with full compute and your current reuse scheme, evaluate QA and summarization separately, and only then try higher ratios to see whether memory and bandwidth savings buy larger batches or longer contexts.

Verdict

The key lesson: the next step for KV reuse is moving less data, not computing less, and caches must be designed for concatenation from the start to be compressible.