Sparse attention (for example the index-selected history attention used by DeepSeek-V4.1-Flash and the GLM-5 series) lets each decode step read only part of the KV cache, but unselected history may be selected later, so the full history must still be stored. With long contexts and many requests, how that history is stored decides how many requests can decode at once. Scitix.AI's UlaCache maps every GPU's KV pool and optional CPU memory into one logical address space: the inference engine only allocates and selects KV by logical address, and UlaCache moves data and updates the mapping to whichever GPU or CPU holds it. GPU and CPU therefore each store different history, instead of CPU holding a full copy and GPU a replica. In SGLang with GLM-5.3 on Multi-LexSum requests of 64K–128K input tokens, with speculative decoding on and matched GPU/CPU memory budgets, peak decode throughput versus HiSparse rises 1.16× (DP) and 1.70× (TP) on 8×H200, and 1.73× (DP) and 2.51× (TP) on 8×B300.
1. Most storage is wasted on replicas
On GLM-5.3 with 8×B300 the authors measure KV allocation for two existing designs:
Two sources of duplication:
- Cross-GPU duplication. With MLA (multi-head latent attention), head-sharded TP makes every GPU store the same KV. Under 8-way TP only 12.5% of space holds unique history.
- GPU–CPU duplication. HiSparse keeps full history on CPU and copies the selected part to GPU. Its 53.3M physical slots hold only 30M unique history positions; the GPU portion adds no capacity.
HiSparse also gives each request a fixed-size GPU cache quota. When fewer requests are active, the remaining ones cannot use free HBM. Example: on 8×B300, DP, speculative decoding, batch size 1, the GPU could hold all history, yet HiSparse TPOT is still 23.5% higher than pure-GPU SGLang.
2. Method
Unified logical addresses. Each KV entry has a fixed logical address; a map records which GPU or CPU slot currently holds it. Moving KV between GPU and CPU only updates the map.
- DP: each GPU manages its own GPU and CPU pools; capacity is their sum.
- TP: all GPUs' pools form one distributed address space; each GPU stores a different part of each request's history (logical address t on GPU t mod P). The computing GPU also has a local GPU buffer caching recently selected KV for reuse, fetching only misses from local or remote GPUs. Layers that share the same sparse selection can prefetch the next layer's KV while the current layer runs.
PB-LRU (parallel bucketed LRU). With CPU storage on, the GPU pool is shared across requests and must decide what to evict to CPU. A full LRU scan of the pool is expensive; PB-LRU buckets GPU slots by recency, updates hits in parallel, finds replaceable slots via a hierarchical bitmap, prefers the coldest bucket, and protects entries the current attention step needs. Data is moved first and the map updated after, so later reads see ready data.
3. Experiments
Setup: GLM-5.3, FP8 weights and KV, 8×H200 and 8×B300. Baselines are full-KV-on-GPU SGLang and CPU-history HiSparse. Speculative decoding uses MTP with 2 draft steps. All three systems get the same GPU HBM allocation; CPU budgets are 128 GB per H200 and 192 GB per B300. Workload is Multi-LexSum, 64K–128K input, greedy 2,048 output tokens per request.
3.1 Capacity and concurrent decodeable requests
This checks whether unified addressing holds more history and admits more requests under the same memory.
Admitted requests:
The TP gap is largest because UlaCache removes MLA's cross-GPU duplication. Under DP, HiSparse already keeps history on CPU, so the gap is smaller.
3.2 Throughput and latency
This checks whether more capacity becomes throughput, and whether extra lookup and scheduling hurt single-request latency at low load.
- Throughput. With speculative decoding, peak throughput versus HiSparse is 1.16× (H200 DP), 1.70× (H200 TP), 1.73× (B300 DP), 2.51× (B300 TP). On B300 TP without speculation, UlaCache peak throughput is 3.51× SGLang. Peaks are each system's best over all batch sizes; the lift mainly comes from decoding more requests at once.
- Low-load latency. Across eight configs at batch sizes 1–8, UlaCache TPOT is below HiSparse but above SGLang. The authors attribute the gap to extra lookup, scheduling, and TP communication.
3.3 PB-LRU ablation
This checks whether bucketing hurts hit rate and how much scheduling time it saves. Setup: 8×H200 DP; PB-LRU and timestamp LRU replay the same access log at equal cache capacity.
Result: GPU hit rates differ by under 0.10 points; at larger batch sizes PB-LRU schedules about 1.5–2.3× faster. At small batches with almost no eviction, plain LRU is faster because PB-LRU still maintains bucket metadata.
4. Limitations
Only one model (GLM-5.3) and one workload (Multi-LexSum long-document summary) are tested. Throughput compares peaks that occur at different batch sizes, so the numbers mainly reflect how many requests fit, not single-step speed at a fixed batch. At low concurrency it is slower than pure-GPU SGLang. No accuracy is reported because UlaCache does not change which tokens attention selects.
5. Takeaways
If you serve MLA + sparse-attention models (GLM-5, DeepSeek-V4.1 class) on SGLang with long contexts and the bottleneck is HBM not fitting more concurrent requests—especially under TP—UlaCache's idea is worth attention: remove cross-GPU and GPU/CPU duplication first, then talk about swapping. If the main scenario is low concurrency and per-request latency, pure-GPU SGLang is still faster.