BellowsKV: One Latent for Many Layers, Expanded on Demand

In long-context inference, the biggest memory cost is often not the weights but the KV cache that every layer keeps for itself. BellowsKV's judgment is that the caches of adjacent layers look almost nothing alike, while the hidden states they are computed from are very similar. So what should be shared is the input, not the cache. Without any training, it folds the hidden states of every n layers into one shared vector per token, and each layer rebuilds its own keys and values from it with its own matrix. The vector keeps its full dimension; compression comes from cross-layer sharing, with further savings from fewer bits and optional eviction of unimportant tokens. On Llama-3.1-8B and two other models, it keeps 97–98% of full accuracy at about 14x compression. The strongest earlier cross-layer method keeps at most 84% at 16x. Decode speed stays essentially the same as with an uncompressed cache.

Engineers building LLM serving, long-context or agent backends should read it, because it offers a very practical rule: when compressing the cache, hold the dimension first, cut bits second, and drop tokens last. The rule holds across three models and three evaluation protocols. The method computes one projection offline and needs no retraining, so the bar to try it is low. However, decode speed was measured only with the default HuggingFace implementation and no custom kernel. One set of Qwen2.5 results is also unexplained even by the authors, so retest before shipping.

What problem is it actually solving?

Start with the arithmetic, using the paper's Llama-3.1-8B example as an illustration. One 131K-token sequence carries about 16.8 GB of KV cache at 16-bit precision, already more than the model itself. When serving, weights load once, but every concurrent sequence needs its own cache.

So real savings need an order of magnitude of compression. Existing methods all lose accuracy in that range. Quantization collapses below 2 bits. Dropped tokens are gone for good, even if a later turn needs them. Cutting channels deletes information that attention still uses. Recent work lets adjacent layers share one compressed cache, which works at 4x but also degrades sharply at 16x.

Three measurements first

Instead of jumping to a method, the authors measured three things on uncompressed models. Each one maps to a design decision.

The first measurement shows that adjacent caches differ while their inputs are alike. Cosine similarity between neighboring keys is close to zero, and the same holds for values. Hidden states, however, reach 0.82–0.87 similarity and grow more similar with depth. Two adjacent layers read almost the same input, because residual connections accumulate it layer by layer. But each layer uses different projection matrices to compute keys and values. This means lending one layer's cache to another must fail; the input is what should be shared.

The second measurement shows that the shared vector has no standout channels. A normal KV cache has a few channels with very large values. During quantization they set the scale for the whole group, so the other channels get coarse treatment. The usual fix rotates the cache to spread values out, but every read must rotate back. Each coordinate of the shared vector is already a linear mix of all channels across n layers, so it is naturally flat. Its 4-bit error is 0.10, below the raw cache's 0.156.

The third measurement shows that adjacent layers agree on which tokens matter. The rank correlation of their token importance scores is 0.76–0.80. So one eviction decision per block can replace one per layer.

One more finding is negative, and the experiments test it later. The shared vector's dimension cannot be cut. Mixing channels from many layers does not lower the rank of the information, so storing fewer coordinates deletes information.

How it works

The model's layers are split in order into non-overlapping blocks of n layers. For each block:

1. Compute one projection offline. On calibration data, collect second-moment statistics of the block's hidden states and solve in closed form. The result is one down-projection and one up-projection per layer that minimize the error between rebuilt and true keys and values. This runs once per model and is folded into the weights.
2. Store the shared vector at prefill. After reading the prompt, stack each token's hidden states from the n layers and multiply by the down-projection. This yields one key vector and one value vector that replace the original 2n caches. A few initial tokens, a short recent window and newly generated tokens stay exact and per layer.
3. Add a small correction. At prefill each sequence has both true keys and rebuilt keys, and their difference before positional rotation is the rebuild error. The key errors from all layers in the block are concatenated into one matrix. A randomized low-rank approximation produces a small basis shared by the block for that sequence, plus a few coefficients per token, which are stored in the cache. At read time, each layer multiplies the token's coefficients by the part of the basis corresponding to its own key coordinates, adds the result to its rebuilt key, and then applies positional rotation. This is the only per-sequence work, and it takes little space.
4. Rebuild on demand at decode. Each layer first takes the attention-weighted sum of the shared value latents. It then applies the output projection with that layer's value up-projection folded in, producing the attention output without rebuilding individual values. Keys must be rebuilt, since position encoding is applied to keys. The rebuild is a small matrix multiply that hides behind memory-bound cache reads.

Why keep the shared vector as wide as one layer's key? Here, keeping dimension means setting the latent width R to one layer's key width p, without reducing R further. Projecting the stacked hidden states into this latent still incurs reconstruction error.

Two compression operators

The first operator, BellowsKV-Quant, lowers precision. It applies plain uniform integer quantization to the shared vector, with one scale per group of 128 adjacent coordinates. No rotation or per-channel handling is needed, because the vector was measured to be flat already. The main experiments use two settings: 2 layers per block at 8 bits, and 4 layers per block at 4 bits.

Here is where the second setting's ratio comes from. Four layers store one shared vector of the same width, giving 4x from layers. Going from 16 bits to 4 bits gives another 4x, for 16x in theory. Because scales, the recent window and the correction all take space, the measured ratio with everything counted is 13.8x.

The second operator, BellowsKV-Evict, drops tokens. Since each block holds one shared vector per token, the eviction decision is made once per block. After prefill, the most recent queries' attention weights score every token, summed across all layers and heads of the block. Each token is then assigned the maximum score in a 13-position window centered on it, giving neighbors of high-scoring tokens a better chance of being kept. The first 4 positions and the last 64 prompt positions are always kept. Among the remaining positions, the fraction q with the highest updated scores survives; the rest are dropped once for the whole block.

Results

The evaluation covers 5 models and three protocols. They are absolute LongBench scores on eight tasks, retention relative to the uncompressed model, and an agent task that must read retrieved evidence verbatim.

This shows that in the 14–18x range it barely loses accuracy, while existing cross-layer methods fall behind. Raw-cache sharing already loses accuracy near 1.3x: on Llama, MiniCache retains 74.8% and KVSharer 88.2%, which supports the claim that caches themselves should not be shared.

Why does xKV collapse at 16x? The authors ran a control. The same xKV shared basis normally reaches 16x by cutting rank to a quarter. Keeping the rank and storing 4 bits instead gives 15.1x. Retention then recovers from 83.5% to 97.6% on Llama and from 79.5% to 99.2% on Mistral. So xKV loses accuracy because it cuts dimension, not because it shares.

The conclusion also holds on the stricter agent task. The test covers all 830 BrowseComp-Plus queries in a single-turn retrieve-then-read setting. Two new models are used: Qwen3-32B and the mixture-of-experts Qwen3-30B-A3B. At about 14.6x, BrowseComp-Plus's own judge gives BellowsQ scores of 86.0 and 85.8 on the two models, against 85.5 and 86.1 uncompressed. xKV on Qwen3-32B falls to 26.5 already at 8x. Tasks that must quote evidence verbatim are more sensitive to cutting dimension.

Eviction experiments also favor the block. At equal memory, one decision per block beats per-layer eviction by 0.9–3.0 points at about 8x and by 5.3–6.6 points at about 16x. The reason is that a block evicts shared vectors, so the same memory keeps more tokens.

On decode speed, an H100 GPU with 32K context and default HuggingFace code runs at 0.995x of uncompressed speed. The rebuild cost is therefore negligible. Under the vLLM serving framework, the freed memory grows the KV cache pool by 14.8x.

Limitations as I see them

1. It saves capacity, not speed. By the authors' own formula, decode speedup depends only on dimension and bits read, not on the number of shared layers. Dimension is kept and reads are 16-bit, so decode does not get faster. If the bottleneck is single-request latency rather than concurrency, it will not help.
2. There are unexplained failures on Qwen. On Qwen2.5, every setting that stores the xKV basis at 4 bits collapses, one to 23.4% retention, and the cause is unknown. Under vLLM the pool grows 14.8x, above the 13.8x byte ratio, also for an unknown reason. Such effects may reappear on other models.
3. The engineering is early. Decode timing uses no custom kernel. The vLLM integration reports only pool size, with no end-to-end throughput. Each sequence also fits a correction at prefill, and that extra prefill time needs measuring.

How an inference serving team could try this

1. Confirm the cache is the bottleneck. Check what limits concurrency for long-context requests. If it is memory for the cache, this method fits; if it is single-request latency, the gain is small.
2. Reproduce retention offline. Use the open-source code to compute projections for your own model. Start with 4 layers + 4 bits and compare retention on your own long-text evaluation set.
3. Test stability across models. Given the Qwen anomaly, run both the 2-layer 8-bit and 4-layer 4-bit settings on any new model family. Look for individual tasks that drop sharply.
4. Then measure serving metrics. After integrating with your serving framework, measure extra concurrency at equal memory, added prefill time, and any change in tokens generated per second.

Bottom line

BellowsKV pushes cross-layer cache compression to about 14x with almost no accuracy loss by sharing inputs, holding dimension and cutting bits first. Inference teams should retest it on their own models.