KVStream: Adaptive Parallel Storage Management for KV Caching in Large-Scale LLM Serving

Long-context inference keeps reusing the same prefixes: system prompts, whole documents, code repositories and multi-turn session histories. Their KV cache no longer fits in GPU memory or host memory, so it has to live on disk and be read back on the next hit, which puts disk reads on the critical path of every inference step. Existing systems read only from the node-local NVMe, while the parallel file system that every cluster already mounts sits almost idle. This paper builds KVStream (an engine that reads KV cache back from several storage tiers in parallel): every KV chunk is written to both the local disk and the shared storage, and at read time both tiers read at once, with whichever finishes first claiming the next batch. Plugged into LMCache, it reaches up to 1.49x LMCache's offline throughput and up to 6.1x lower mean TTFT under bursty traffic.

Teams running long-context inference services, especially multi-turn agent and coding-assistant workloads, should read this. Its value is not a new algorithm but a very practical observation: GPU nodes already have a second storage path next to them, and nobody uses it during inference. The method is simple, too. It needs no prediction of which prefixes will be reused and no tuning of split ratios. The cost is that every KV chunk is written twice, so it only pays off when reads dominate writes, and the paper measures the cases where it gets slower.

What problem is it solving?

First, the setting. Inference engines (vLLM, SGLang) cache the KV of prefixes they have computed, so a later request that shares the prefix does not recompute it. LMCache is an open-source KV cache connector. It cuts the KV that does not fit into chunks of 256 tokens, stores them in host memory and on local disk, and reads them back to the GPU on a hit.

The trouble is that the total volume of prefixes is huge. Each test node has 4 GPUs with about 140 GB left for KV after model weights, plus 512 GB of host memory. For a 14B model, the unique prefixes to cache across several document datasets are 0.5 to 1 TB, 3.3 to 7.2 times what the GPUs can hold. For a 110B model the volume approaches 2 TB. So most prefixes can only live on disk.

How slow are the reads? The figure above is a concrete case: one step of a 70B model holds 7 requests, LMCache reads them one request at a time, each takes 600 to 1300 ms, and the step spends 4.7 seconds just reading. Worse, the forward pass cannot start until the KV of every request in the step has arrived, so finishing one request early gains nothing. On LooGLE, disk reads account for 80% to 88% of LMCache's total time at all four model sizes. Even so, reading from disk still beats recomputing, by up to 2.2x on large models, because recompute cost grows with model depth while read cost grows only with bytes.

Why the local disk alone is not enough

The authors first run a set of measurements with three findings.

First, the second path sits idle. GPU nodes in supercomputing clusters and in the cloud usually mount a shared parallel file system (PFS below, such as Lustre or a managed cloud file system). Measured alone, the local NVMe reads only 2.2x (testbed 1) and 3.2x (testbed 2) faster than the PFS. Yet during inference the PFS carries almost no traffic beyond loading model weights, so its bandwidth is effectively free.

Second, reads and writes on the same disk interfere. The new KV from each step has to be written while the old KV needed by the next step is read from the same disk. Doing 512 GB of reads and 512 GB of writes concurrently takes 6% (testbed 1) and 27% (testbed 2) longer than doing them separately. So any split of reads based on idle-time bandwidth is wrong once inference is actually running.

Third, per-chunk latency has a long tail. On testbed 2, isolated NVMe reads have a median of 123 ms and a P99 of 216 ms; with reads and writes mixed, the P99 rises to 898 ms. The PFS has a 102 ms median but a 795 ms P99, because every job in the cluster shares it. The bandwidth each tier can deliver keeps changing from one inference step to the next.

KVStream's four changes

KVStream makes four changes to address these problems.

The first change is reading the whole step at once. Instead of reading one request at a time, it puts every chunk that the step's requests need into one read queue and submits them together. NVMe only reaches full speed with many reads in flight, and a one-at-a-time queue wastes that parallelism. Each chunk is also shipped to the GPU the moment it arrives.

The second change is deferring writes. Newly computed KV is held in host memory and written to disk only after the next forward pass starts. The disks are idle for reads during the forward pass, so writes rarely block reads.

The third change is writing every chunk twice. Each KV chunk is stored on both the local NVMe and the PFS. Any chunk can then be read from any tier, with no need to guess in advance which chunks will be needed or where to put them. Storage capacity is usually measured in terabytes, so a second copy is rarely a problem; the cost is twice the bytes written.

The fourth change is reading from both ends toward the middle, with the faster tier reading more. The step's chunks form a list; the NVMe reads from the front and the PFS from the back, m chunks at a time. Whichever tier finishes its current batch immediately claims the next one, and the two meet somewhere in the middle. The faster tier naturally reads more and the slower one less, with no bandwidth measurement and no ratio to tune. A few chunks near the meeting point may be read by both; since the content is identical, the first copy is used and the other is dropped.

In implementation terms, KVStream is a C++ library that does asynchronous reads and writes with io_uring, using separate queues for reads and writes, and plugs into LMCache as a storage backend. io_uring is Linux's asynchronous I/O interface: a program puts a batch of read and write requests in a queue and submits them at once instead of waiting for each one to finish. Engines like vLLM that already use LMCache need no code changes.

Meeting in the middle: a worked example

Why not just use a fixed split, say 75% from NVMe and 25% from the PFS? The paper gives an example, worked through here.

Suppose a step must read 12 GB, the NVMe reads at 3 GB/s and the PFS at 1 GB/s. Together that is 4 GB/s, so the fastest possible time is 3 seconds; this is what the paper calls the aggregate-bandwidth bound. With a 75:25 split, the NVMe reads 9 GB in 3 seconds and the PFS reads 3 GB in 3 seconds, finishing together.

Now the NVMe slows to 2 GB/s because it is also absorbing writes. Under the fixed split it needs 4.5 seconds for its 9 GB while the PFS sits idle. The combined bandwidth is 3 GB/s, so the bound is 4 seconds, and the fixed split is 12.5% slower. If instead the PFS is squeezed by another job down to 0.5 GB/s, its 3 GB take 6 seconds, slower than reading all 12 GB from the NVMe alone (4 seconds).

Meeting in the middle avoids this. Whichever side slows down, the meeting point shifts toward it, and both sides finish at about the same time. A simple model in the paper shows that this routing stays at the aggregate-bandwidth bound in every step, apart from the last batch of m chunks. A fixed split is optimal only when the bandwidth ratio happens to equal the split ratio, and loses as soon as bandwidth shifts, which it does constantly during inference.

How the experiments are set up

There are two supercomputing testbeds: testbed 1 has 4 H100s per node, testbed 2 has 4 A100s per node, each mounts a shared file system, and each node has local NVMe. Four models from 14B to 110B are used, all with TP=4, covering both the case where recompute is cheap and the case where reading is cheap.

Offline tests use three long-document datasets: LooGLE (long-document QA), NarrativeQA (whole-book reading comprehension, 46K tokens per request on average) and CUAD (legal clause extraction). Online tests replay three real traces: about 7,000 requests from a Qwen code-generation cluster on Alibaba Cloud Bailian, agent sessions from the Chutes platform, and coding-agent sessions on SWE-bench. In these workloads 80% to 96% of prompt tokens hit a cached prefix.

Baselines are: GPU plus host memory only, recomputing on a miss ("memory only" below); LMCache with local NVMe; Mooncake's KV store; and SGLang's built-in hierarchical cache HiCache. LMCache is the strongest baseline and most comparisons use it.

Result 1: offline throughput

On LooGLE, the larger the model, the larger KVStream's advantage. There are two reasons: a larger model has more KV per token, so fewer prefixes fit in memory and more reads go to disk; and recompute cost grows with depth faster than KV size does, since the 70B model has 5 times the parameters of the 14B one but only twice the KV per token.

Read bandwidth explains the gain. On one node LMCache reads 6.0 to 6.7 GB/s from the NVMe; KVStream reads about the same from the NVMe and another 2.8 to 4.9 GB/s from the PFS, 8.1 to 11.8 GB/s in total. Both read about the same total volume from disk, so the speedup comes entirely from parallel reads. Across datasets, the 32B model gains only 1.05x on CUAD, because CUAD requests are the shortest, 11K tokens on average, so disk reads are a smaller share of each step.

Result 2: how much a fixed split loses

The authors test the previous section's argument on testbed 2 with the 32B model. On a file system with no other jobs, the fixed split's distance from the bound is V-shaped: best when the split matches the average bandwidth ratio, yet still 8.0% slower, and up to 67.7% slower when the split is off. KVStream has no ratio to tune and stays within 4.3% of the bound. Even the best fixed ratio only matches on average, because bandwidth changes every step.

Then another job pushes periodic, bursty or steady traffic onto the PFS, simulating other cluster users. Now even the best fixed split is 11.8% to 23.6% slower, while KVStream stays within 4.3%. As long as someone else is using the shared storage, a fixed ratio loses, however well it was tuned.

Result 3: in online serving, queueing amplifies small gains

Online, the metric is TTFT, which is a request's own processing time plus the time it waits behind earlier requests. Faster reads not only finish the current batch sooner but also let queued requests start sooner, so the gain grows with load.

On the SWE-bench coding-agent trace, with a 1-second TTFT target, 84% of requests meet it with KVStream and 70% with LMCache; every other baseline is below 55%. The speedup from reading a second tier is something users feel directly.

On the code-generation trace, as the arrival rate goes from low to medium to high, KVStream's mean-TTFT speedup over LMCache grows from 1.07x to 1.61x to 3.12x. Per request, though, it spends only 15% to 22% less time reading than LMCache. The difference is the queue: at high load LMCache has 128 requests waiting on average, KVStream 54.3. So the big TTFT improvement comes mostly from draining the queue faster, not from any single read.

The burstier the traffic, the stronger the queueing effect. Burstiness is measured by the coefficient of variation of inter-arrival times: the standard deviation of the gaps between consecutive requests divided by the mean gap. Evenly spaced arrivals give 0, fully random (Poisson) arrivals give 1, and larger values mean requests come in bursts. Real production traffic is about 4.3. At a coefficient of 4, KVStream's mean TTFT is 6.07x lower than LMCache's; at an extreme 8, beyond real traffic, the advantage shrinks to 1.17x.

It also holds at scale. On testbed 2, from 4 to 32 nodes (128 GPUs), each replica has its own NVMe but all share one PFS. KVStream keeps a 1.23x to 1.36x throughput advantage over LMCache at every scale, because each node routes its own reads by completion and reads more from local disk when the PFS is congested, with no coordination between nodes.

When it does not help, or gets slower

This is the part I find most useful, because the authors measured the cases where their own method loses.

First, it does not help when host memory is big enough. On the Chutes trace, raising host memory from 40 GB to 80 GB per GPU lets almost every reuse hit in memory, and disk reads drop by 98.6%. KVStream is then 0.98x of LMCache, slightly slower, because it still writes a second copy.

Second, request order decides how much must be read. On the same code-generation trace, if a session's turns arrive close together, most prefixes are still in memory and KVStream is only 1.05x faster. If many sessions interleave in turn, each turn is separated by lots of other KV, and it is 3.12x faster. Online services usually must process requests in arrival order rather than reorder them for cache hits, so the second case is the common one.

Third, it gets slower when reuse is low. The authors artificially lower prefix reuse on the SWE-bench trace: at 88% reuse KVStream is 6.0x faster, at 77% the two are even (0.97x), and at 4% it is only 0.81x. Less reuse means more new KV to write and less to read back, so the cost of writing twice outweighs the gain from parallel reads.

What each change contributes

Offline throughput comes almost entirely from the second tier: with a single NVMe, its bandwidth is the cap however reads and writes are scheduled, and the first two changes move throughput by less than 1%. Adding the PFS with a fixed 70:30 split brings throughput to 1.24x LMCache, and routing by completion brings it to 1.48x.

The online column has a counterintuitive result: the fixed split pushes TTFT to 122.1 seconds, far worse than LMCache. In this online run the fixed split keeps no second copy: the first 80% of each prefix's chunks live only on the NVMe and the last 20% only on the PFS. A fixed split decides each chunk's tier when it is written, by its position in the whole prefix at that moment, and never moves it; when the prefix grows and the 80:20 boundary shifts, old chunks stay where they are. The chunks a new turn appends are always at the very end of the prefix at that moment, inside the last 20% when written, so most newly written chunks live only on the PFS. The PFS ends up serving 47% of reads with only 15% of the bandwidth and becomes the bottleneck. In multi-tier storage, assigning data in advance by position or ratio goes wrong easily; assigning at read time is what stays robust.

Limitations as I see them

First, doubling the write volume is a real cost. The authors tried two compromises in their simulator. Adding a replica only after a tier slows down cuts writes to 1.19x, but reads end up 12% to 74% above the bound, because the replica arrives after the step has already stalled. Replicating only part of each prefix's chunks (a fixed 40%) costs 1.43x writes and reads at most 5% above the bound. So if your storage is short on write bandwidth, partial replication is the direction to try first.

Second, the gain depends heavily on workload shape. It suits long prompts, short outputs, many interleaved sessions and working sets that do not fit in memory. With long outputs, decoding dilutes the share of disk reads and Amdahl's law caps the speedup: from 16 to 256 output tokens, the offline advantage stays between 1.19x and 1.37x.

Third, all experiments are on supercomputing clusters whose shared storage is a dedicated high-performance parallel file system. A managed cloud file system or object store may behave very differently in bandwidth and latency, so measure your own. Nodes also do not coordinate, so the same prefix may be stored on the PFS once per node, wasting capacity and write bandwidth; the authors leave cross-node sharing to future work.

How a team could try it

1. First confirm that disk reads are your bottleneck: measure the share of each step spent reading KV from disk. If host memory already holds most reused prefixes, this will not help.
2. Check whether nodes have a second independent storage path, such as a shared file system, a second NVMe or remote NVMe, and measure each one's read and write bandwidth under inference load.
3. If you already use LMCache, integration is cheapest. Otherwise the scheduling rule itself, reading from both ends and letting whoever finishes first claim the next batch, is easy to port into your own KV storage layer.
4. Before launch, replay your own real traffic and watch TTFT at peak and under bursts, and whether low-reuse workloads get slower.

Bottom line

The disk-read bottleneck of long-context KV caching does not necessarily need a faster disk: use the idle shared storage in the cluster as well, let each tier claim work at its actual speed, and reads stay close to the aggregate-bandwidth bound.