uGDS: A User-Space Development Kit for High-Performance GPU Direct Storage

Large-model inference services store the KV cache they have already computed on SSDs, so that the next request with the same opening can read them back instead of recomputing. NVIDIA provides GPU Direct Storage (GDS for short below) for this, letting data go from SSD straight into GPU memory without a stop in host memory. Yet when the authors measured it inside the open-source cache system LMCache, asynchronous reads through GDS on a single task queue reached only 2.36 GiB/s, less than half of synchronous read bandwidth; 99.6% of the time was stuck at the "submit request" step. This paper from Sun Yat-sen University, HKUST (Guangzhou) and others open-sources uGDS, which moves the job of "telling the SSD what to do" out of the operating-system kernel and into an ordinary program. Data still goes straight from SSD to GPU; what changes is who gives the orders. On one A100 GPU plus one Samsung 990 PRO SSD, uGDS's batched reads are up to 12 times as fast as GDS and batched writes up to 30 times; plugged into LMCache, the median time-to-first-token for long prompts (from sending a request to seeing the first output word) fell from 1.9 seconds to 0.7 seconds.

1. The problem: data no longer takes a detour, but commands still do

First, the difference between the "data path" and the "control path". The data path is where the data itself flows from and to; the control path is who tells the hardware, and through which steps, "move this piece of data to that place".

GDS fixed the data path: an SSD can DMA data over PCIe straight into GPU memory, without the CPU copying it. But GDS's control path enters the operating-system kernel every time: a read request passes in turn through the virtual file system, the actual file system, the block-device layer and the kernel's SSD driver, and NVIDIA's kernel module also looks up a GPU address for every 4 KiB page of data. When moving one large block, these fixed costs are spread thin and hardly noticed; but in KV cache reads and writes and vector search, requests tend to be small, frequent and latency-sensitive, and these costs become the main expense.

The authors point to two further problems with GDS.

First, the queueing rules are too rigid. GDS's asynchronous interface inherits the rule of GPU task queues (CUDA streams): within one queue, every request waits for the previous one to finish. But when LMCache reads back a stretch of cache, the small pieces need no particular order; they only have to all arrive before the model's computation begins. To use the full bandwidth, applications have to spread requests over many queues, effectively doing GDS's scheduling for it.

Second, problems are invisible. GDS's kernel part is open source, but the user-space library that handles batched submission, background threads and queue integration is released only as a binary. When things stall you can neither find the cause nor change the policy.

Others had already driven SSDs directly from ordinary programs, showing that this path works, but those prototypes supported only NVIDIA GPUs, assumed a single submitting thread, and did not track "requests still in flight", so a program could free a piece of GPU memory while the SSD was still writing into it.

2. Method: the CPU gives commands, the SSD moves data, the GPU only computes

uGDS separates three things.

At start-up, privileged setup is done once, by a small kernel module: it takes the SSD over from the kernel driver, maps the SSD's control registers and command queues into memory the program can read and write directly, locks the GPU memory to be used, and records its addresses on the bus.

During normal reads and writes the kernel is not involved at all. Following the SSD's protocol (NVMe, the standard command format for solid-state drives), the program itself writes a 64-byte command saying where on the disk to read, how much, and which pages of GPU memory to put it in. It then writes a number into a "doorbell" register on the SSD to signal a new command. After that it checks the completion queue for the SSD's report that it is done. The whole process is ordinary memory reads and writes, with no system calls.

The data itself still moves directly between the SSD and GPU memory, not through host memory.

The cost is that the SSD must be detached from the kernel driver and given to uGDS exclusively, and the program sees "byte number such-and-such on the disk" rather than files. The authors argue this fits cache engines like LMCache and Mooncake, which manage their own storage blocks; if files are needed, a user-space file system can be layered on top.

To make this path both fast and safe, uGDS has four designs.

First, it supports both NVIDIA and AMD GPUs. When GPU memory is registered, each vendor's memory addresses are translated into a uniform list of page addresses the SSD understands, so code above does not need to know whose card it is.

Second, command queues are split into two groups. SSDs support several sets of command queues. uGDS sets aside one deep queue just for batched requests, and the rest serve single synchronous requests. That way, when a large batch floods in, single requests are not left waiting behind it. Batched requests also gather many commands and ring the "doorbell" only once, saving repeated notification costs.

Third, it prevents "freeing while writing". This is where driving hardware yourself most easily goes wrong. For each request sent, uGDS notes "transfer in flight" against the GPU memory involved; trying to free that memory while a transfer is in flight is refused outright. Each registration also carries an increasing number, so that an old request cannot hit newly allocated memory that reuses the same address. If a command times out, the disk is marked faulty and isolated, and resources can only be reclaimed after a hardware reset confirms the SSD has stopped.

Fourth, completion notification is selectable. By default the CPU keeps polling the completion queue, which responds fastest but occupies a whole CPU core. Hardware interrupts can be used instead, with the kernel only waking the program and everything else still done in user space.

None of these ideas is new on its own: SPDK, which drives SSDs directly from ordinary programs, has long existed. uGDS's contribution is turning them into a GPU storage library that maps one-to-one onto GDS's interface and can be swapped in for production use: it has synchronous, batched and queue-ordered asynchronous interfaces, both vendors' GPUs share one abstraction, the safety mechanisms are in place, and everything from the kernel module to user space is open source. The most practical point is that a single queue can use the full bandwidth, so applications no longer need to split requests over a dozen queues.

3. Experiments: small requests, batched requests and the first batch

Tests used a server with two Intel Xeon CPUs, one A100 GPU and one Samsung 990 PRO 1TB SSD, compared against NVIDIA's current version of GDS.

uGDS starts faster: median initialisation is 42 milliseconds, against 115 for GDS, which spends almost all of it opening NVIDIA's driver session, loading configuration and probing the bus topology. For elastic services that frequently start and retire worker processes, this saves a fair amount of time.

Single small requests gain the most. With 2 threads writing 4 KiB blocks, uGDS's bandwidth is 6.2 times GDS's and average latency falls from 48 microseconds to 7.7, because each request skips a long chain of kernel steps. With more threads and bigger requests, the bottleneck moves to the SSD and the bus themselves, and the gap narrows to 1.2 to 1.4 times. On an AMD platform, against AMD's equivalent library, it is also about 1.2 times as fast.

Batched requests lead across the board. The authors tested 36 combinations of "request size × requests per batch", and uGDS won all of them: reads 2.1 to 12.1 times GDS, writes 2.3 to 30.7 times. The smaller the requests and the more per batch, the bigger the gap.

Following an anomaly, the authors tracked down a problem in GDS. More requests per batch should mean faster, yet for 128 KiB writes GDS got slower as batches grew. The cause is the first batch: with 128 requests per batch, GDS's first batch takes 91 milliseconds to read and 118 to write, nearly ninety percent of the whole 64 MiB transfer time. This is because GDS defers mapping GPU memory addresses until the first read or write, and the authors suspect it also allocates the internal state needed for batching only at the first submission. uGDS does all this during initialisation, so its first batch is almost as fast as later ones. Even leaving out the first batch and looking only at the steady state, with 4 KiB requests and 128 per batch, uGDS still reads 8.8 times and writes 23.8 times as fast as GDS.

Polling and interrupts each have their uses. For small 4 KiB requests, constant CPU polling cuts latency from 21 microseconds to 9, at the cost of a fully busy core. For large 1 MiB requests, switching to interrupts lowers the CPU use of the requesting thread from 100% to 4%, with only 5.5% more latency. So polling suits latency-sensitive cases, and interrupts are better value for large transfers.

4. Real-world effect: inside LMCache

The authors plugged uGDS into LMCache and ran end-to-end tests with the small Qwen3-0.6B model and the vLLM inference framework. To test the SSD path specifically, they turned off vLLM's own in-memory cache, so every cache hit had to be read back from the SSD.

For prompts from 3,840 to 40,704 tokens, uGDS's time-to-first-token is consistently about four tenths of GDS's. The time for the GPU to place the read-back data into vLLM's memory is the same on both sides, which shows that all of the saving comes from the storage path. Another key difference is scaling: GDS tops out at around 32 concurrent requests, while uGDS keeps scaling to 256, with 2.6 times GDS's throughput.

5. Limitations

uGDS must have the whole SSD to itself: once detached from the kernel driver, the disk has no file system and cannot be used as an ordinary disk by other programs. Currently one uGDS instance drives only one local SSD; using several disks together and reaching remote SSDs over a network are still planned. Synchronous requests are assigned to command queues in turn and hold the queue until done, so a large request slows small ones on the same queue; the authors plan a load-aware scheduler. The experiments used only one consumer SSD and one A100, with the IOMMU (the hardware that checks addresses when devices access memory) turned off, and the end-to-end test used only a small 0.6B model. How it performs with data-centre SSDs, and with several GPUs sharing a bus switch chip, remains to be verified.

6. Practical meaning

If you work on tiered KV cache storage (GPU memory, host memory and SSD) or GPU vector search, and the SSD is a dedicated local disk, uGDS is worth trying as a replacement for GDS. Small, frequent reads on a single queue are exactly where GDS suffers most and uGDS gains most.

Before going live, confirm three things: whether the SSD can be dedicated and detached from the kernel driver; whether the layer above already manages its own storage-block allocation, as LMCache and Mooncake do; and whether the server's IOMMU settings allow the SSD and GPU to transfer directly. When deploying, pick the completion method by workload: CPU polling for latency-sensitive inference nodes, interrupts for throughput-oriented or CPU-constrained jobs.

Even without switching libraries, two of the paper's findings about GDS apply directly. First, GDS spends tens of extra milliseconds preparing the first time it reads or writes a piece of GPU memory, so after a service starts it should do one warm-up read or write for each "GPU memory and disk" pair. Second, GDS's single queue gets blocked on submission, so critical reads should be split across several queues in parallel.