NVIDIA GDS moves NVMe↔GPU over PCIe P2P but still routes every I/O control path through the kernel, imposes total order on streams, and ships a closed engine. uGDS is a user-space kit (libugds + minimal kernel helper). On a Samsung 990 PRO it reaches up to 12.12× GDS batch-read and 30.69× write bandwidth; 4-KiB sync write latency −84.1% with two threads. On one stream GDS keeps only 40.8% of sync 16-MiB read bandwidth; submissions can eat 99.6% of wall time. In LMCache, median TTFT falls up to 62.7% and serving goodput rises 2.63×. GDS bottlenecks are often control-path and ordering semantics, not DMA bandwidth.
1. Problem
Small I/Os (KV offload) hate fixed per-I/O kernel cost; stream total-order is too strong when only batch order is needed; closed engines block custom scheduling.
2. Method
Setup path privileges queue creation then bypasses the OS in steady state; control path issues NVMe in user space; data path is autonomous P2P DMA. Hardware queues split latency-sensitive sync vs deep batch. Diagnostic scheduling hooks are exposed.
3. Core claim
Move the control plane off the kernel hot path and relax ordering to batch needs. Save submission cost and mis-ordering, not just chase PCIe peak.
4. Experiments
Batch read/write up to 12.12×/30.69×; 4-KiB sync write −84.1%; LMCache TTFT −62.7%, goodput 2.63×; AMD vs hipFile sync write up to 1.19×.
5. Limits
Topology-specific; user-space polling trades SM time; multi-tenant isolation is extra work.
6. Practice
For KV offload / ZeRO-offload, measure small-I/O latency and submission share before celebrating bandwidth.