ZipFusion: Fusing Lossless KV Decompression with Attention for Efficient LLM Decoding

Lossless KV cache compression has mostly been used to save transfers: compress before moving, then decompress on the GPU so attention can read the original data. ZipFusion's judgment is that the decompression step eats most of the benefit. So it lets attention read compressed data directly and restores the original values in registers with a few integer bit operations. The data format and the attention kernel are designed together, and the freed memory goes to the scheduler to admit more requests. On an H100, the attention kernel is about 1.25x faster than FlashInfer. With 8K context and the same memory, whole-model decode throughput is 26.3% higher than uncompressed SGLang and 2.76x the best earlier lossless scheme, with bit-exact results.

Teams running LLM inference, especially those with zero tolerance for accuracy changes, should read it. It shows that lossless compression can make decoding faster, not just save bandwidth, as long as you avoid a separate decompression pass. The idea in one line: compress only exponent bits, keep the fraction untouched, and make restoring cheap enough to fit inside the attention kernel. However, its ceiling is a 25% saving, since a block shrinks at most to 75%. It was tested on one H100, two 8B models and fixed 8K prompts, so treat it as a stackable optimization.

What problem is it actually solving?

First, why "decompress then compute" does not pay off. This illustration uses the measurement setup from the paper's Figure 1. Llama 3.1 8B decodes with batch 32 and 2048 input tokens each. Three existing lossless codecs are plugged into SGLang, and at every step they decompress one layer's cache into a buffer before attention runs.

So decompression is not a small cost: even in the best case it is a third of decode time. Worse, attention still reads full-size data after decompression. The bytes reread at every step do not shrink, and the restored copy needs extra memory. A KV cache is written once and read at every later decode step. If it stays compressed while being read, one compression pays off across many steps.

The authors chose lossless over quantization because prior work found that most models are more sensitive to KV cache quantization than to weight quantization on long-context retrieval and QA. For deployments that need results identical to the original model, lossless is the only option.

The pipeline

The system has three parts, from bottom to top:

1. Compression format. The KV cache is organized in pages of 16 tokens. When a page is full, it is split into blocks of 512 values that are compressed independently. Eligible blocks shrink to 75%; the rest stay as is. Pages that are not yet full stay uncompressed so new tokens can be appended directly.
2. Attention kernel. The kernel copies blocks from GPU memory into on-chip shared memory while they stay compressed. Each thread takes its 8 values, restores them in registers and feeds them straight to the matrix units. Nothing is written back to GPU memory.
3. Memory management. Memory freed by compression can be reassigned between layers. The allocator keeps computing how many new pages still fit and tells the scheduler how many requests it can admit, while reserving room for requests still generating.

Compression format: change the exponent, keep the fraction

Why compress only the exponent? A BF16 value, a 16-bit floating-point format, has 1 sign bit, 8 exponent bits and 7 fraction bits. Values in one block tend to have similar magnitudes, so their exponents cluster in a small range and can be compressed. Fraction bits look nearly random and are hard to compress.

Here is how it works. Take the smallest exponent in the block and round it down to an even number; that is the base b. If every exponent lies between b and b+15, the block is compressed; otherwise it is not. Each value keeps its low 8 bits unchanged, and those bits hold the 7 fraction bits plus the lowest exponent bit. What remains is the sign bit and the top 3 bits of the exponent offset, called d, which is the offset divided by 2 and rounded down. Together they form a 4-bit code. Each value goes from 16 bits to 8+4=12 bits, exactly 75%.

A worked example. A block's exponents range from 120 to 133, so b=120. One value has exponent 127, giving offset 7, or binary 0111. The lowest bit, 1, is already in the retained low byte, so only the top 3 bits, 011, need storing; here d=3. To restore, multiply d by 2, add b, then add that lowest bit from the low byte: 120+6+1=127, exactly the original. The whole process uses integer additions and shifts with no floating-point math, so there is no rounding error.

Each page also stores a little metadata so the kernel can find each block. For a 16-token page with head dimension 128, each head's keys and values are each split into 4 blocks. The 4 key blocks share 8 bytes and the 4 value blocks use another 8. Half of those bytes record where the first block starts. The other half give one byte per block, either marking it uncompressed or storing half the base, b/2. So the kernel knows whether each block is full size or 75%, and summing earlier block sizes locates the next one.

Restoring in registers: one addition for four values

Restoring must be cheap, or the compute cost cancels the read savings. The authors shape the data layout around the hardware. GPU matrix instructions require each thread to hold 8 specific elements, so the cache writer stores those 8 elements together. Keys and values play different roles in the two matrix products, so each gets its own layout.

Each thread reads 8 low bytes and 4 code bytes. The key observation is that a BF16 high byte equals the sign times 128 plus the top 7 exponent bits, and those 7 bits equal b/2 + d. So if the sign sits in bit 7 and d in bits 0–2, adding b/2 yields the high byte directly. The 4-bit codes are packed in pairs to match. In one byte, value 1's sign goes in bit 7 and its offset in bits 0–2. Value 5's sign goes in bit 6 and its offset in bits 3–5.

The thread treats the 4 code bytes as one 32-bit integer. One branch masks each byte to keep bits 7 and 0–2, giving the codes of the first four values. The other branch shifts and masks the middle four bits into the same positions, giving the codes of the last four. Each branch adds a constant with b/2 in every byte, so one addition restores 4 high bytes. In the paper's example, the code byte is 0xD0 and b/2=63. Value 1 has sign 1 and offset 0, so its high byte is 128+63+0=0xBF. Value 5 has sign 1 and offset 2, so its high byte is 128+63+2=0xC1. Since b/2+d never exceeds 127, the addition cannot carry into the sign bit or the next byte.

Memory management: put the savings to work

Memory freed by compression is useless unless the scheduler knows about it. The hard part is that this space is dynamic. Full pages release space when compressed, while running requests keep asking for new pages.

The authors handle this in three ways. First, a new page is always allocated at uncompressed size, because it must accept writes before it fills up and gets compressed. Second, a compressed page may move, so a lookup table maps the unchanged page ID to its current location. Third, each layer has private space plus a shared pool. When a chunk in one layer becomes fully empty with no pending reads, it returns to the pool and can go to another layer. The allocator then computes how many new pages fit across all layers, and the scheduler admits requests based on that number.

Here is why cross-layer sharing matters. Every page of a new request needs space in every layer, so capacity is set by the tightest layer. Suppose layer 0 compresses well and frees a large chunk while layer 1 barely compresses. Without a shared pool, layer 0's savings sit idle. With one, that chunk moves to layer 1 and the total number of pages that fit actually grows. Hoot says▾

Hoot saysTwo layers, 10 blocks each: how many new pages fit with and without a shared poolClose ✕

Capacity is set by the tightest layer: layer 0 compresses well and frees a lot, which sits idle without a shared pool; with one, its freed blocks go to layer 1 and the same memory takes half again as many pages.

• 40 idle units

• compressed old pages
• this layer's new pages (4 units each)
• new pages for layer 1 via the pool
• free

1. No pool: layer 0 has 100 − 20 = 80 free, layer 1 has 100 − 60 = 40. Each new page needs 4 in every layer, so at most min(80, 40) ÷ 4 = 10 pages, and 40 units of layer 0 sit idle.
2. With a pool: 80 + 40 = 120 free in total, 8 per new page across both layers, so 120 ÷ 8 = 15 pages; layer 1 needs 15 × 4 = 60, and the 20 beyond its own 40 come from blocks layer 0 returned to the pool (a block goes back only when fully free with no reads in flight).
3. If pages were sized after compression: layer 0 would think each needs 1 and admit 80; but a page cannot be compressed before it fills, and 80 pages being written need 80 × 4 = 320 units, 240 more than the 80 free. That is the reason for rule one, allocate at uncompressed size.

All numbers are illustrative: each block is 10 units and each layer owns 100; 20 existing pages take 1 unit each in layer 0 and 3 in layer 1 after compression; new pages are always allocated at the uncompressed 4 units. Source: paper §5: new pages allocated at uncompressed size; compressed pages may move, with a lookup table from page id to location; per-layer private space plus a shared pool, fully free blocks with no reads in flight return to the pool; the allocator reports how many new pages all layers can still hold

Results

Tests run on one H100 with 80 GB of memory, using Llama 3.1 8B and Qwen3-8B. Each request has 8192 input tokens and generates 256. All methods share the same memory budget, and restored values are checked bit for bit.

The gains come from two sources that add up. At the same batch size, attention reads fewer bytes than uncompressed SGLang, so it is about 15% faster. Against other lossless schemes it also skips the separate decompression step, so the gap exceeds 2.5x. The freed memory fits about 24% more requests, and together these give 26.3% at the largest batch. Qwen3-8B shows nearly identical numbers, with batch size rising from 40 to 50 and throughput up 25.8%. The effect does not depend on one particular model.

A comparison with related work is also telling. A prior system that fused weight decompression into matrix multiplication reported that restore overhead could cancel bandwidth savings on A100 and H800. ZipFusion achieves a net gain on H100, which suggests its restore instructions are light enough.

Limitations as I see them

1. Low compression ceiling. A block shrinks at most to 75%, and blocks with scattered exponents do not compress at all. So it cannot save much memory alone; the largest batch only grows from 46 to 57. It works better as a layer stacked with other optimizations than as the main compression tool.
2. Narrow test conditions. There is one H100, two 8B models, BF16 only, and fixed 8192-token essay prompts. Prompts were compressed in advance, and timing excludes prefill and first compression. Longer contexts, multi-GPU deployments or real request mixes may change the gains.
3. Compression during generation was not tested. By design, a page can be compressed once full. In the experiments, though, newly generated cache stays uncompressed for every method, and only the precompressed 8192-token prompt is compressed. For reasoning requests with long outputs, generated tokens dominate the cache, so the cost and benefit of online compression need your own measurements.

How an inference serving team could try this

1. Measure your exponent distribution first. Sample KV caches from production models and count how many 512-value blocks have an exponent span within 16. That decides the real savings.
2. Reproduce the kernel gain on one GPU. Compare attention latency at fixed context length and batch size. Confirm that on your GPU model the restore cost stays below the read savings.
3. Wire it into scheduling and watch concurrency. Let the allocator report freed space to the scheduler. Measure the largest batch and throughput under equal memory, and check how gains decay for long outputs.
4. Evaluate alongside quantization. If your product tolerates a slight accuracy loss, test a quantization scheme too. See whether the two stack before choosing what to ship.

Bottom line

ZipFusion shows that lossless compression can make decoding faster, as long as you skip the separate decompression pass. With a 25% ceiling, it fits best as a layer on top of other optimizations.