Contents · 5 sections
1In a 128×128 state, 99% of the output comes from the top 2–16 directions2An exact SVD per step takes 4000 ms, so only the top is touched, and only after buffering3Touching only the top lets the tail pile up, so the idle CPU cleans it4State memory is roughly halved, and end-to-end decoding is 1.25× faster on average5Adaptive ranks and the frozen tail hold up after 32K generated tokensSHARE
Most layers of a hybrid model are linear attention, and each head keeps one fixed-size matrix per request, the recurrent state, which must be read and written in full for every generated token. At large batch sizes this traffic dominates the time of each step. Existing low-rank compression is either done once before serving or picks in advance which directions to keep, but the state is rewritten at every step, and a choice fixed in advance may not hold for unseen input. The paper measures that these states carry their information in only a few directions: out of at most 128 components, the first 2–16 recover 99% of the output. SinkState therefore keeps SVD factors at each head's actual rank plus a small buffer throughout decoding, and the few heads whose factored form would not save memory fall back to dense. On four hybrid models on a single RTX PRO 6000, peak recurrent-state memory drops to 0.52 of the baseline on average and end-to-end decoding is 1.25× faster on average. Accuracy changes range from −1.6 to +2.6 points with no consistent direction. So the linear-attention state need not be stored and accessed in dense form: maintaining low-rank factors at the actual rank during decoding saves both this memory and per-step traffic without systematically hurting accuracy. The saving is in recurrent-state memory, not total GPU memory.
Most layers of a hybrid model (the Qwen3.5 and Nemotron 3 families) are linear attention, with only a few full-attention layers mixed in. Each linear-attention head keeps one fixed-size matrix per request, called the recurrent state. Taking Mamba2 as the example, for every new token the matrix is first scaled by a decay factor and then gets the new token's added. Variants such as GDN (Gated DeltaNet) have more complex updates; the illustrations below all follow Mamba2. The output is the query multiplied by this matrix, a step the paper calls readout. The state does not grow with the context, which is where linear attention saves memory over full attention.
The trouble is that the whole matrix is read once and written once at every decoding step. Illustration: a head's state is 128×128, i.e., 16,384 FP32 numbers; to generate one token, readout reads all 16,384 numbers and the update writes them all back. The real scale is larger: Nemotron-3-Nano-4B (Nemo-4B below) holds about 258k states of size 80×128 at batch size 128. Each time the batch doubles, this traffic doubles too, and it soon dominates the time of each step.
The natural idea is low-rank compression, but the existing routes the paper lists do not fully exploit the redundancy inside each state. SVD compression of weights and KV caches happens once before serving, because those tensors are fixed or only appended to; the state is rewritten at every step. ReplaySSM-style methods buffer recent inputs and write the state back periodically, removing most writes, but for the head above, readout still reads all 16,384 numbers each step. SketchSSM approximates only the reads, while writes stay full. State pruning and offline-calibrated projections decide in advance which directions to keep and how many, and that choice may not hold for an unseen input.
In a 128×128 state, 99% of the output comes from the top 2–16 directions
The authors first measured q, k, and v in three models: Qwen3.5-9B (GDN layers), Nemotron-3-Nano-4B (Mamba2 layers), and Kimi-Linear-48B (KDA layers). On PG-19 book text, the top 8 directions capture 73–88% of the energy of q, k, and v in most cases. On random tokens the concentration is even stronger, with the top 8 directions capturing 87–96%. This shows that the energy still concentrates in a few directions on input with no semantics. The authors argue that the concentration does not depend on semantics and call it the linear attention sink.
Since the written into the state at each step mostly lies in these few directions, the state itself is dominated by them too. Figure 2 is the direct evidence: out of up to 128 components, the top 2–15 already hold 99% of the state's energy, and the top 2–16 recover 99% of the output . This means most numbers in a dense state are redundant, yet every step reads and writes all of them.

Turning the redundancy into storage, go back to the illustrative 128×128 head: storing SVD factors of rank r means storing and , each 128×r, for 256r numbers. If this head has rank 8, it needs only 2,048 numbers, 1/8 of the dense form, and readout reads proportionally less. At rank 64 the two break even, and beyond 64 the factors are larger than the dense form, which matters later.
An exact SVD per step takes 4000 ms, so only the top is touched, and only after buffering
The factored form is attractive; the difficulty is that the state changes every step. The most direct approach is to recompute an exact SVD at every step. For Nemo-4B at batch size 128, an exact SVD of all 258k states takes about 4000 ms, while a decoding step takes only 20–30 ms, over two orders of magnitude apart. The authors consider an offline-calibrated fixed low-rank basis neither general nor scalable.
Their way out comes from another observation: incoming and mostly fall on the top few components of the existing state. On Qwen3.5 and Nemotron, the top component alone captures 18–65% of a new vector's energy; components beyond the 16th capture even less than random directions. So an update only needs to touch the top.
SinkState splits the components of each state into two parts. The active part is the 8 components with the largest singular values, and the frozen part is the rest. New tokens first go into a buffer of length K=8 and are merged only when it is full; this merge is called compaction. Compaction concatenates only the active part with the buffer and runs a small SVD on them, puts the frozen part back unchanged, and finally drops the merged components below ( is a threshold and is the largest singular value). For the illustrative head, what gets concatenated is 8 active components plus 8 buffered items, and with Brand's incremental SVD only a 16×16 core matrix is decomposed, never the full 128×128 matrix. Hoot says
Stored after local merge (illustrative)
- Component 1
- direction u·vᵀ, weight 2
- Component 2
- same direction u·vᵀ, weight 1
- Drop line
- 0.1×2=0.2, both kept
- Stored components
- 2
Same direction; the weights are just coefficients, not singular values of the whole block.
Exact SVD of the whole block
- Actual matrix
- 2·u·vᵀ + 1·u·vᵀ = 3·u·vᵀ
- Singular values
- 3 and 0
- Drop line
- 0.1×3=0.3, the 0 is dropped
- True rank
- 1
Between compactions, the CPU does exactly this on the frozen part.
Contrast: two orthogonal directions (illustrative)
- Components
- u₁·v₁ᵀ weight 2, u₂·v₂ᵀ weight 1
- Exact SVD singular values
- still 2 and 1
- Stored components = true rank
- 2
Only when directions are mutually orthogonal do the stored weights equal the singular values.
Illustrative: the two components point in exactly the same direction, with weights 2 and 1; the drop line is 0.1 × the largest weight, as in the paper’s toy example. Compaction’s small SVD only covers the active part and the buffer and puts the frozen part back as is, so duplicate directions can survive, and the stored count ends up above the true rank.
Source
paper §3.2 Observation 2 (with local SVD alone, components with similar directions accumulate in the frozen part, so the stored rank exceeds that of an exact SVD); §4.3 CPU consolidation and Fig. 7(a) (an exact SVD on the frozen part merges redundant components, rank 3 to 2)
| When | Input | Operation | Output | Where |
|---|---|---|---|---|
| End of prefill | Dense state | Blocked randomized SVD, stopping once a block yields a singular value below | Factors at each head's own rank | GPU |
| Every decode step (Mamba2 as the example) | New token's k, v, q, decay | Decay applies to all singular values (including buffered items), k and v are appended to the buffer, output computed directly from factors and buffer | Readout result | GPU |
| Buffer holds 8 items | Top 8 components + 8 buffered items | Small SVD, merged with frozen, components below dropped | New factors | GPU, staggered across layers |
| Between two compactions | Frozen part | Exact SVD, merging components with similar directions | Fewer frozen components | CPU, asynchronous |
The paper works through these steps on a 4×4 toy example (, K=2, prompt "After 25 years, Wikipedia is still"). After prefill, the first block of randomized SVD gives singular values 8.0 and 1.95, both above the threshold 0.1×8.0=0.8, so it computes another block and gets 1.10 and 0.06. Since 0.06 is below 0.8, it is truncated, and this state's rank is set to 3.
When decoding the first token, the decay factor is 0.9; SinkState multiplies the three singular values by 0.9, giving 7.2, 1.76, and 0.99, while and stay unchanged. The new token's v and k are attached to the buffer as a 4th component with weight 1. This shows that during buffering, each step writes the changes to the singular values and the newly added vectors, without rewriting the whole state.
When the second token arrives, the buffer is full and compaction is triggered. The active part is just the largest item, 5.76; together with the two buffered items it forms a 3×3 core matrix whose SVD gives 8.99, 2.08, and 0.04. After merging with the frozen 1.41 and 0.79, the 0.79 and 0.04 fall below 0.1×8.99=0.899 and are dropped, leaving 3 components. This shows that compaction only needs to decompose a small 3×3 matrix.
Touching only the top lets the tail pile up, so the idle CPU cleans it
The frozen part never enters the small SVD, and the cost is that later updates may store an existing direction again as a new component. In the same example, after compaction the frozen components 2.08 and 1.41 are nearly parallel, with cosines of 0.89 and 0.99 between their left and right singular vectors. This also shows that after compaction the stored factors are only an approximate SVD whose components need not be orthogonal. When two nearly aligned components are combined, their singular values roughly add up: 2.08+1.41≈3.5. Redoing an exact SVD on the CPU gives 3.41 and 0.02, and 0.02 is dropped, so the rank goes from 3 to 2.
This duplication is common in real models. After 128 decoding steps on GSM8K, the small SVD alone stores 1.19–1.32 times as many components as an exact SVD (Nemotron) and 1.67–1.75 times as many (Qwen3.5). Letting the small SVD cover more columns would help, but it slows down every step.
The authors note that host CPU utilization during decoding is only about 2%. The frozen part does not enter the local SVD, so SinkState downloads it to the CPU, runs a multithreaded exact SVD, and uploads the result before the next compaction, where the GPU only swaps a pointer; the paper calls this step CPU consolidation. The CPU cannot keep pace with the GPU, so layers are processed in turn; if a layer is compacted again before its SVD finishes, the result is discarded and that layer is retried first in the next round.
Compaction also has a timing problem. If all layers compact at the same step, a latency spike appears every 8 steps. Staggered compaction lets only 1/8 of the layers compact at each step; for example, a 24-layer model compacts 3 layers per step.
What remains is at the kernel level. Ranks differ across heads, so SinkState stores the uneven factors compactly with a CSR-like offset array (CSR is the row-offset storage format common for sparse matrices), and heads whose factored form would not save memory (for a 128×128 head, rank above 64) fall back to dense after initialization. Readout is done by one fused kernel that computes directly from the factors and the buffer, without rebuilding the dense matrix.
State memory is roughly halved, and end-to-end decoding is 1.25× faster on average
The experiments run four hybrid models on a single RTX PRO 6000 (96 GB): Nemo-4B and Nemo-120B (Mamba2 layers), and Qwen-9B and Qwen-80B (GDN layers). The baselines are the dense state kernels of Mamba-SSM and FLA (the flash-linear-attention kernel library), adopted by vLLM and SGLang, with both sides using the same batch size.

Across six benchmarks, end-to-end decoding is 1.25× faster on average and up to 1.58× (Nemo-4B on MATH-500 and IFEval). Nemo-4B gains the most (1.30–1.58×) because most of its step time is spent on the state; Nemo-120B gains the least (1.04–1.16×), since its MoE layers take more of the time and its larger weights keep the batch small. This shows that the speedup depends on how large a share of each step the state reads and writes take.
Peak recurrent-state memory drops to 0.38–0.63 of the baseline, 0.52 on average. Nemo-4B saves the most and Qwen-80B the least, which the authors attribute to the higher ranks of GDN states. This refers only to recurrent-state memory, not total GPU memory. Accuracy changes by −1.6 to +2.6 points across 24 model-benchmark pairs, rising in 12 and by +0.3 on average, with no consistent direction, which shows that the compression does not systematically hurt the output.
Looking only at the state kernels makes the picture clearer. The dense kernel time grows linearly with the batch, while SinkState grows much more slowly, running 1.64–2.41× faster on Nemo-4B and 1.03–1.95× faster on Qwen-9B.

The speedup is lowest at batch size 32 and levels off beyond 128. The reason is that at small batches the kernels do not yet saturate memory bandwidth, so moving fewer numbers saves little time. Because the factors are smaller than dense states, the same GPU fits larger batches: the baselines run at most 640 on Nemo-4B and 768 on Qwen-9B, while SinkState runs 1280 on both.
What is saved is the bytes that must be moved at every step, so the larger the batch and the larger the state's share, the bigger the gain.
The two ablations correspond to staggered compaction and CPU consolidation. Staggered compaction cuts P99 step time by 15.9–40.9 ms, at the cost of a 1.4–6.1 ms increase in P50, largest on Nemo-4B. With CPU consolidation on, state memory drops further, and more on the two Qwen models than on the two Nemotron models. This shows that GDN-layer states accumulate more duplicate components under the small SVD alone, so the CPU step brings them a larger memory saving.
Adaptive ranks and the frozen tail hold up after 32K generated tokens
Nemo-4B's rank distribution shows why a fixed rank does not work. On RULER, after prefilling 8K tokens and then generating 8K more one token at a time, 22% of heads keep at most 8 components, 13% keep more than 32, and a few are dense. So a uniform cut at 8 would lose information in the latter, and a uniform 32 would waste memory on the former.
Since the frozen tail is not updated for long stretches, the main worry is error accumulation over long generations. On RULER's 16K retrieval tasks, SinkState differs from the baseline by at most 0.1%; on MATH-500 and GPQA Diamond with thinking enabled and up to 32K generated tokens, it loses at most 0.5%. Nemo-4B gains 4.1 points on GPQA, which the authors attribute to the baseline hitting the generation length limit on more questions, not to improved capability.
The method's scope also has concrete limits: all measurements are on a single GPU; Kimi-Linear appears only in the earlier measurements, and end to end only Mamba2 and GDN layers are validated. CPU consolidation relies on idle CPU cores, and the test machine has 32. Gains are limited at small batches: at batch size 32, the Qwen-9B kernel is only 1.03× faster.
Back to the 128×128 head from the opening: SinkState maintains its SVD factors at each head's actual rank plus a buffer, and the few heads whose rank is too high for the factored form to save anything still fall back to dense.
How did this read for you?
Hoot made a note. Thanks for the feedback.
One tap, no sign-in; tap again to undo. Only anonymous counts are kept, never who you are.
Discussion
Add experiment details, reproduction notes or a different take. Requires a GitHub sign-in.
Loading the discussion…