SinkState: On-the-Fly SVD of Recurrent States for Efficient Hybrid LLM Serving

Haolin Jiang, Weihang Li, Zhongzhu Zhou, Jinwen Pan, Santosh Pandey, Hang Liu
32 minread4.10scoreNo. 311-13 Infra issue
Contents · 5 sections1In a 128×128 state, 99% of the output comes from the top 2–16 directions2An exact SVD per step takes 4000 ms, so only the top is touched, and only after buffering3Touching only the top lets the tail pile up, so the idle CPU cleans it4State memory is roughly halved, and end-to-end decoding is 1.25× faster on average5Adaptive ranks and the frozen tail hold up after 32K generated tokens
Rel 4Nov 4Prac 4Evid 4Time 5

Most layers of a hybrid model are linear attention, and each head keeps one fixed-size matrix per request, the recurrent state, which must be read and written in full for every generated token. At large batch sizes this traffic dominates the time of each step. Existing low-rank compression is either done once before serving or picks in advance which directions to keep, but the state is rewritten at every step, and a choice fixed in advance may not hold for unseen input. The paper measures that these states carry their information in only a few directions: out of at most 128 components, the first 2–16 recover 99% of the output. SinkState therefore keeps SVD factors at each head's actual rank plus a small buffer throughout decoding, and the few heads whose factored form would not save memory fall back to dense. On four hybrid models on a single RTX PRO 6000, peak recurrent-state memory drops to 0.52 of the baseline on average and end-to-end decoding is 1.25× faster on average. Accuracy changes range from −1.6 to +2.6 points with no consistent direction. So the linear-attention state need not be stored and accessed in dense form: maintaining low-rank factors at the actual rank during decoding saves both this memory and per-step traffic without systematically hurting accuracy. The saving is in recurrent-state memory, not total GPU memory.

Most layers of a hybrid model (the Qwen3.5 and Nemotron 3 families) are linear attention, with only a few full-attention layers mixed in. Each linear-attention head keeps one fixed-size matrix per request, called the recurrent state. Taking Mamba2 as the example, for every new token the matrix is first scaled by a decay factor and then gets the new token's added. Variants such as GDN (Gated DeltaNet) have more complex updates; the illustrations below all follow Mamba2. The output is the query multiplied by this matrix, a step the paper calls readout. The state does not grow with the context, which is where linear attention saves memory over full attention.

The trouble is that the whole matrix is read once and written once at every decoding step. Illustration: a head's state is 128×128, i.e., 16,384 FP32 numbers; to generate one token, readout reads all 16,384 numbers and the update writes them all back. The real scale is larger: Nemotron-3-Nano-4B (Nemo-4B below) holds about 258k states of size 80×128 at batch size 128. Each time the batch doubles, this traffic doubles too, and it soon dominates the time of each step.

The natural idea is low-rank compression, but the existing routes the paper lists do not fully exploit the redundancy inside each state. SVD compression of weights and KV caches happens once before serving, because those tensors are fixed or only appended to; the state is rewritten at every step. ReplaySSM-style methods buffer recent inputs and write the state back periodically, removing most writes, but for the head above, readout still reads all 16,384 numbers each step. SketchSSM approximates only the reads, while writes stay full. State pruning and offline-calibrated projections decide in advance which directions to keep and how many, and that choice may not hold for an unseen input.

In a 128×128 state, 99% of the output comes from the top 2–16 directions

The authors first measured q, k, and v in three models: Qwen3.5-9B (GDN layers), Nemotron-3-Nano-4B (Mamba2 layers), and Kimi-Linear-48B (KDA layers). On PG-19 book text, the top 8 directions capture 73–88% of the energy of q, k, and v in most cases. On random tokens the concentration is even stronger, with the top 8 directions capturing 87–96%. This shows that the energy still concentrates in a few directions on input with no semantics. The authors argue that the concentration does not depend on semantics and call it the linear attention sink.

Since the written into the state at each step mostly lies in these few directions, the state itself is dominated by them too. Figure 2 is the direct evidence: out of up to 128 components, the top 2–15 already hold 99% of the state's energy, and the top 2–16 recover 99% of the output . This means most numbers in a dense state are redundant, yet every step reads and writes all of them.

Fraction of state energy (black) and output H_t q_t (red) retained when only the top r components of the state are kept, for three models; dotted lines mark where 99% is reached, and the gray dashed line is the isotropic reference
Fraction of state energy (black) and output H_t q_t (red) retained when only the top r components of the state are kept, for three models; dotted lines mark where 99% is reached, and the gray dashed line is the isotropic reference

Turning the redundancy into storage, go back to the illustrative 128×128 head: storing SVD factors of rank r means storing and , each 128×r, for 256r numbers. If this head has rank 8, it needs only 2,048 numbers, 1/8 of the dense form, and readout reads proportionally less. At rank 64 the two break even, and beyond 64 the factors are larger than the dense form, which matters later.

An exact SVD per step takes 4000 ms, so only the top is touched, and only after buffering

The factored form is attractive; the difficulty is that the state changes every step. The most direct approach is to recompute an exact SVD at every step. For Nemo-4B at batch size 128, an exact SVD of all 258k states takes about 4000 ms, while a decoding step takes only 20–30 ms, over two orders of magnitude apart. The authors consider an offline-calibrated fixed low-rank basis neither general nor scalable.

Their way out comes from another observation: incoming and mostly fall on the top few components of the existing state. On Qwen3.5 and Nemotron, the top component alone captures 18–65% of a new vector's energy; components beyond the 16th capture even less than random directions. So an update only needs to touch the top.

SinkState splits the components of each state into two parts. The active part is the 8 components with the largest singular values, and the frozen part is the rest. New tokens first go into a buffer of length K=8 and are merged only when it is full; this merge is called compaction. Compaction concatenates only the active part with the buffer and runs a small SVD on them, puts the frozen part back unchanged, and finally drops the merged components below ( is a threshold and is the largest singular value). For the illustrative head, what gets concatenated is 8 active components plus 8 buffered items, and with Brand's incremental SVD only a 16×16 core matrix is decomposed, never the full 128×128 matrix.

Hoot saysTwo stored components, one actual directionClose ✕
Two components with the same direction should become one. The small local SVD cannot see this; an exact SVD of the whole block merges them.

Stored after local merge (illustrative)

Component 1
direction u·vᵀ, weight 2
Component 2
same direction u·vᵀ, weight 1
Drop line
0.1×2=0.2, both kept
Stored components
2

Same direction; the weights are just coefficients, not singular values of the whole block.

Exact SVD of the whole block

Actual matrix
2·u·vᵀ + 1·u·vᵀ = 3·u·vᵀ
Singular values
3 and 0
Drop line
0.1×3=0.3, the 0 is dropped
True rank
1

Between compactions, the CPU does exactly this on the frozen part.

Contrast: two orthogonal directions (illustrative)

Components
u₁·v₁ᵀ weight 2, u₂·v₂ᵀ weight 1
Exact SVD singular values
still 2 and 1
Stored components = true rank
2

Only when directions are mutually orthogonal do the stored weights equal the singular values.

Illustrative: the two components point in exactly the same direction, with weights 2 and 1; the drop line is 0.1 × the largest weight, as in the paper’s toy example. Compaction’s small SVD only covers the active part and the buffer and puts the frozen part back as is, so duplicate directions can survive, and the stored count ends up above the true rank.

Sourcepaper §3.2 Observation 2 (with local SVD alone, components with similar directions accumulate in the frozen part, so the stored rank exceeds that of an exact SVD); §4.3 CPU consolidation and Fig. 7(a) (an exact SVD on the frozen part merges redundant components, rank 3 to 2)
Overview of SinkState: SVD initialization of the state after prefill; during decode, buffering and an SVD covering only the active part run on the GPU and are merged with the frozen part; the frozen part is sent asynchronously to the CPU for SVD
Overview of SinkState: SVD initialization of the state after prefill; during decode, buffering and an SVD covering only the active part run on the GPU and are merged with the frozen part; the frozen part is sent asynchronously to the CPU for SVD
WhenInputOperationOutputWhere
End of prefillDense stateBlocked randomized SVD, stopping once a block yields a singular value below Factors at each head's own rankGPU
Every decode step (Mamba2 as the example)New token's k, v, q, decayDecay applies to all singular values (including buffered items), k and v are appended to the buffer, output computed directly from factors and bufferReadout result GPU
Buffer holds 8 itemsTop 8 components + 8 buffered itemsSmall SVD, merged with frozen, components below droppedNew factorsGPU, staggered across layers
Between two compactionsFrozen partExact SVD, merging components with similar directionsFewer frozen componentsCPU, asynchronous

The paper works through these steps on a 4×4 toy example (, K=2, prompt "After 25 years, Wikipedia is still"). After prefill, the first block of randomized SVD gives singular values 8.0 and 1.95, both above the threshold 0.1×8.0=0.8, so it computes another block and gets 1.10 and 0.06. Since 0.06 is below 0.8, it is truncated, and this state's rank is set to 3.

When decoding the first token, the decay factor is 0.9; SinkState multiplies the three singular values by 0.9, giving 7.2, 1.76, and 0.99, while and stay unchanged. The new token's v and k are attached to the buffer as a 4th component with weight 1. This shows that during buffering, each step writes the changes to the singular values and the newly added vectors, without rewriting the whole state.

When the second token arrives, the buffer is full and compaction is triggered. The active part is just the largest item, 5.76; together with the two buffered items it forms a 3×3 core matrix whose SVD gives 8.99, 2.08, and 0.04. After merging with the frozen 1.41 and 0.79, the 0.79 and 0.04 fall below 0.1×8.99=0.899 and are dropped, leaving 3 components. This shows that compaction only needs to decompose a small 3×3 matrix.

Touching only the top lets the tail pile up, so the idle CPU cleans it

The frozen part never enters the small SVD, and the cost is that later updates may store an existing direction again as a new component. In the same example, after compaction the frozen components 2.08 and 1.41 are nearly parallel, with cosines of 0.89 and 0.99 between their left and right singular vectors. This also shows that after compaction the stored factors are only an approximate SVD whose components need not be orthogonal. When two nearly aligned components are combined, their singular values roughly add up: 2.08+1.41≈3.5. Redoing an exact SVD on the CPU gives 3.41 and 0.02, and 0.02 is dropped, so the rank goes from 3 to 2.

This duplication is common in real models. After 128 decoding steps on GSM8K, the small SVD alone stores 1.19–1.32 times as many components as an exact SVD (Nemotron) and 1.67–1.75 times as many (Qwen3.5). Letting the small SVD cover more columns would help, but it slows down every step.

The authors note that host CPU utilization during decoding is only about 2%. The frozen part does not enter the local SVD, so SinkState downloads it to the CPU, runs a multithreaded exact SVD, and uploads the result before the next compaction, where the GPU only swaps a pointer; the paper calls this step CPU consolidation. The CPU cannot keep pace with the GPU, so layers are processed in turn; if a layer is compacted again before its SVD finishes, the result is discarded and that layer is retried first in the next round.

Compaction also has a timing problem. If all layers compact at the same step, a latency spike appears every 8 steps. Staggered compaction lets only 1/8 of the layers compact at each step; for example, a 24-layer model compacts 3 layers per step.

What remains is at the kernel level. Ranks differ across heads, so SinkState stores the uneven factors compactly with a CSR-like offset array (CSR is the row-offset storage format common for sparse matrices), and heads whose factored form would not save memory (for a 128×128 head, rank above 64) fall back to dense after initialization. Readout is done by one fused kernel that computes directly from the factors and the buffer, without rebuilding the dense matrix.

State memory is roughly halved, and end-to-end decoding is 1.25× faster on average

The experiments run four hybrid models on a single RTX PRO 6000 (96 GB): Nemo-4B and Nemo-120B (Mamba2 layers), and Qwen-9B and Qwen-80B (GDN layers). The baselines are the dense state kernels of Mamba-SSM and FLA (the flash-linear-attention kernel library), adopted by vLLM and SGLang, with both sides using the same batch size.

End-to-end speedup (top), accuracy change (numbers at the top), and normalized state memory (bottom, solid for average and hatched for peak) of four models on six benchmarks
End-to-end speedup (top), accuracy change (numbers at the top), and normalized state memory (bottom, solid for average and hatched for peak) of four models on six benchmarks

Across six benchmarks, end-to-end decoding is 1.25× faster on average and up to 1.58× (Nemo-4B on MATH-500 and IFEval). Nemo-4B gains the most (1.30–1.58×) because most of its step time is spent on the state; Nemo-120B gains the least (1.04–1.16×), since its MoE layers take more of the time and its larger weights keep the batch small. This shows that the speedup depends on how large a share of each step the state reads and writes take.

Peak recurrent-state memory drops to 0.38–0.63 of the baseline, 0.52 on average. Nemo-4B saves the most and Qwen-80B the least, which the authors attribute to the higher ranks of GDN states. This refers only to recurrent-state memory, not total GPU memory. Accuracy changes by −1.6 to +2.6 points across 24 model-benchmark pairs, rising in 12 and by +0.3 on average, with no consistent direction, which shows that the compression does not systematically hurt the output.

Looking only at the state kernels makes the picture clearer. The dense kernel time grows linearly with the batch, while SinkState grows much more slowly, running 1.64–2.41× faster on Nemo-4B and 1.03–1.95× faster on Qwen-9B.

State-kernel time per step versus batch size on Nemo-4B and Qwen-9B (top, SinkState split into readout and compaction) and speedup (bottom); gray regions mark where Mamba-SSM/FLA run out of memory
State-kernel time per step versus batch size on Nemo-4B and Qwen-9B (top, SinkState split into readout and compaction) and speedup (bottom); gray regions mark where Mamba-SSM/FLA run out of memory

The speedup is lowest at batch size 32 and levels off beyond 128. The reason is that at small batches the kernels do not yet saturate memory bandwidth, so moving fewer numbers saves little time. Because the factors are smaller than dense states, the same GPU fits larger batches: the baselines run at most 640 on Nemo-4B and 768 on Qwen-9B, while SinkState runs 1280 on both.

What is saved is the bytes that must be moved at every step, so the larger the batch and the larger the state's share, the bigger the gain.

The two ablations correspond to staggered compaction and CPU consolidation. Staggered compaction cuts P99 step time by 15.9–40.9 ms, at the cost of a 1.4–6.1 ms increase in P50, largest on Nemo-4B. With CPU consolidation on, state memory drops further, and more on the two Qwen models than on the two Nemotron models. This shows that GDN-layer states accumulate more duplicate components under the small SVD alone, so the CPU step brings them a larger memory saving.

Adaptive ranks and the frozen tail hold up after 32K generated tokens

Nemo-4B's rank distribution shows why a fixed rank does not work. On RULER, after prefilling 8K tokens and then generating 8K more one token at a time, 22% of heads keep at most 8 components, 13% keep more than 32, and a few are dense. So a uniform cut at 8 would lose information in the latter, and a uniform 32 would waste memory on the former.

Since the frozen tail is not updated for long stretches, the main worry is error accumulation over long generations. On RULER's 16K retrieval tasks, SinkState differs from the baseline by at most 0.1%; on MATH-500 and GPQA Diamond with thinking enabled and up to 32K generated tokens, it loses at most 0.5%. Nemo-4B gains 4.1 points on GPQA, which the authors attribute to the baseline hitting the generation length limit on more questions, not to improved capability.

The method's scope also has concrete limits: all measurements are on a single GPU; Kimi-Linear appears only in the earlier measurements, and end to end only Mamba2 and GDN layers are validated. CPU consolidation relies on idle CPU cores, and the test machine has 32. Gains are limited at small batches: at batch size 32, the Qwen-9B kernel is only 1.03× faster.

Back to the 128×128 head from the opening: SinkState maintains its SVD factors at each head's actual rank plus a buffer, and the few heads whose rank is too high for the factored form to save anything still fall back to dense.

How did this read for you?

One tap, no sign-in; tap again to undo. Only anonymous counts are kept, never who you are.

Discussion

Add experiment details, reproduction notes or a different take. Requires a GitHub sign-in.

Loading the discussion…