UlaCache: Unified Logical Addressing for Sparse LLM Serving

Models such as DeepSeek-V4.1-Flash and the GLM-5 family use index-based sparse attention: at each decoding step a scoring module picks a small subset of past positions, and attention reads only the KV cache at those positions. Each step reads less data, but storage does not shrink, because a position skipped at this step may be picked at the next one, so the whole history has to stay. With long contexts and many requests, how much history fits in memory decides how many requests can decode at once. Scitix.AI's UlaCache puts every GPU's KV pool and optional CPU memory into one "logical address space": the inference engine allocates and selects KV by logical address only, and UlaCache decides which GPU or CPU holds the data, moves it, and updates the mapping. The core idea is not "use more CPU" but "make every byte of memory hold different history, not copies". On SGLang with GLM-5.3 and 64K–128K-token long-document summarization requests, with speculative decoding on and equal memory budgets, peak throughput is 1.16x to 2.51x that of HiSparse. The paper's own curves show that almost all of this gain comes from fitting more requests at once; per-step speed at the same batch size does not improve.

1. The problem: most memory holds duplicate copies

The authors first measured how GLM-5.3 allocates KV on 8×B300 and found heavy duplication in both existing approaches, from two sources.

The first source is duplication across GPUs. GLM-5.3 uses MLA, which compresses the KV of all attention heads into one shared representation. Standard tensor parallelism (TP) splits computation across GPUs by attention head, but that shared representation cannot be split by head, so every GPU stores a full copy. Under 8-way TP only 12.5% of the space holds distinct history; the other 87.5% is copies. In other words, eight GPUs' worth of memory gives the storage capacity of one.

The second source is duplication between GPU and CPU. HiSparse is the current reference design for extending sparse-attention KV into CPU memory: it keeps full histories on the CPU and, at each step, moves the selected entries to the GPU and caches them. The catch is that every entry cached on the GPU is still on the CPU too. In the figure it uses 53.3 million physical slots but stores only 30.0 million distinct history positions. Its 23.3 million GPU slots make access faster without adding any history capacity.

HiSparse also has an allocation problem: it gives each request a fixed-size GPU cache. When there are few requests, the freed memory cannot be used by the others. The authors' example is 8×B300, DP mode (data parallel: each GPU runs the whole model and serves its own requests, detailed below), speculative decoding on, a single request: the GPU could hold the entire history, but one request's quota cannot, so it keeps fetching from the CPU. As a result its time per output token (TPOT) is 23.5% higher than GPU-only SGLang.

2. Unified logical addresses: the engine tracks "which number", not "where"

UlaCache gives each KV entry a fixed logical address and keeps a mapping table recording which GPU slot or CPU slot currently holds it. When data moves between GPU and CPU, only the mapping changes; the logical address stays the same, so the engine side never needs to know the data moved. This is the same idea as PagedAttention mapping logical blocks to physical GPU blocks, extended from one GPU to several GPUs plus CPU.

In DP mode each GPU serves its own requests and manages its own GPU pool and CPU pool; a request's capacity is the sum of the two. What this removes is GPU–CPU duplication: an entry lives either on the GPU or on the CPU, never both.

TP mode changes more. All GPUs' pools form one distributed address space, and each GPU stores only part of each request's history: logical address t goes to GPU t mod P, where P is the number of GPUs. Eight GPUs now hold eight different slices of history instead of eight identical copies. The cost is that computation must fetch data from other GPUs, so each computing GPU keeps a local buffer of recently selected KV. Sparse attention picks heavily overlapping positions across consecutive steps, so the buffer hit rate is high and most of the time only a few missing entries need fetching.

Two more details are worth noting. Some layers in GLM-5.3 share one sparse selection result; UlaCache does the address lookup and deduplication once for such a group and, while the current layer computes, uses a separate CUDA stream to prefetch data for the later layers. The index keys used by the scoring module are also spread across GPUs by the same rule: each GPU scores only its local keys, and one all-gather combines the scores. This removes duplicate scoring but adds communication wait at low concurrency.

3. PB-LRU: what to evict from a shared GPU pool

With CPU storage enabled, the GPU pool is no longer carved up per request; all requests share it, recently used entries stay on the GPU and long-unused ones move to the CPU. That needs a replacement policy. Standard LRU keeps a timestamp per slot and scans the whole pool to choose victims. Doing that at every decoding step and every layer puts the cost on the critical path of decoding.

PB-LRU (the authors' Parallel Bucketed LRU) avoids the scan with an approximation. Instead of ordering slots exactly, it puts every slot accessed in the same scheduling round into the same "bucket". Each layer group keeps a ring of recent buckets plus one cold bucket. Each round the current bucket advances one position, and whatever remains in the old bucket about to be reused moves to the cold bucket; only the labels change, no data moves. When space is needed it uses free slots first, then the cold bucket, then recent buckets from oldest to newest. The current bucket is never chosen, because its data is about to be used by this step's attention.

Each bucket is a bitmap with one bit per GPU slot. On top sit several layers of summary bitmaps, where each bit says whether the segment below has any occupied slots, so selection can skip whole empty segments. Hit detection, recency updates and deduplication all run in parallel on GPU threads. Data movement also has an ordering rule: move the data first, then update the mapping and signal completion; attention waits for the signal before reading new locations, so it never reads half-moved data.

4. Experiments: capacity grows several-fold, throughput follows

Experiments use GLM-5.3 (FP8 weights and KV cache) on 8×H200 and 8×B300. Baselines are SGLang with all KV on GPU and HiSparse with history on CPU; speculative decoding uses the model's built-in multi-token prediction head (MTP) to draft two steps, which the main model then verifies. For fairness, all three systems get the same GPU memory and the same CPU budget (128 GB per H200, 192 GB per B300). The workload is Multi-LexSum long-document summarization with 64K to 128K input tokens; contexts are long enough that KV storage is the main pressure.

First, how many requests fit at once. The table gives admitted requests from paper Figure 8 (speculative decoding on):

The gap is largest in TP mode, because that is where the 87.5% of cross-GPU copies is removed. In DP mode HiSparse already keeps history on the CPU, so on H200 the gain is only about a tenth. B300 DP nearly doubles because B300 has large GPU memory, and once GPU and CPU stop duplicating each other the freed space is substantial.

Once capacity grows, the question is whether it turns into throughput. With speculative decoding on, peak throughput over HiSparse is 1.16x on H200 DP, 1.70x on H200 TP, 1.73x on B300 DP and 2.51x on B300 TP. With it off, B300 TP peak is 3.51x SGLang's.

This figure says more than the peak numbers. At batch sizes where both curves exist, UlaCache and HiSparse almost overlap, so per-step speed has not improved. The gap appears after HiSparse's curve ends: it cannot fit more requests, while UlaCache keeps climbing. On B300 TP, admitted requests rise almost 10x but throughput only 2.51x, because past a certain batch size the throughput curve flattens and more requests no longer buy proportional speed. The one exception is H200 DP with speculative decoding: only 8% more requests but 16% more throughput. The curve shows HiSparse dipping at large batches, which fits the explanation that its fixed quotas lower the hit rate.

At low concurrency (batch size 1), across eight configurations (two GPU types × DP or TP × speculative decoding on or off), UlaCache's TPOT is lower than HiSparse's but higher than GPU-only SGLang's. The reason is the extra address lookup, scheduling and, under TP, cross-GPU communication. A separate PB-LRU test replays the same access traces on 8×H200 DP: its GPU hit rate is within 0.1 percentage points of exact LRU, so the bucket approximation loses almost nothing in cache quality. At larger batches its scheduling is 1.5x to 2.3x faster. But when little replacement is needed, exact LRU is actually faster, because PB-LRU still maintains bucket labels.

5. Limits and where it applies

• One model and one workload: GLM-5.3 on long-document summarization. Different sparse selection patterns (tasks with less step-to-step overlap) would change buffer and GPU-pool hit rates, which the paper does not cover.
• Throughput comparisons take each system's best value over all batch sizes, and the best points fall at different batch sizes. So "2.51x" reflects how many requests fit, not being faster under the same load. If your production concurrency never reaches the baseline's capacity limit, you will get little of this gain.
• At low concurrency it is slower than GPU-only SGLang. For latency-sensitive, low-concurrency serving, the GPU-only setup is still the better fit.
• No accuracy test. UlaCache does not change which positions attention selects, so in principle the output is unchanged, but correctness of cross-GPU fetches and swaps rests on the implementation; no consistency check is reported.
• No report of CPU–GPU bandwidth use, and no comparison with other concurrent sparse-attention KV offloading systems.

6. Practical takeaways

If you serve MLA plus sparse attention models such as GLM-5 or DeepSeek-V4.1 on SGLang, with long contexts and the bottleneck being "memory cannot fit more concurrent requests", the most useful lesson here is to check for duplicate storage first. When deploying an MLA model with TP, work out how much KV on each GPU also exists on the others. By the paper's count it is 87.5% under 8-way TP, and removing it gains more capacity than adding CPU memory.

Second, when extending into CPU memory, make GPU and CPU hold different data and let all requests share the GPU pool instead of giving each a fixed quota. Finally, when evaluating such systems, look at the curves at equal batch sizes, not just peak multiples. The former tells you whether each step got slower; the latter only tells you how many more requests fit.