At low concurrency (for example an agent app with only one or two sessions per user), each request computes little data per step, and latency is spent on kernel launches, waits between kernels, and multi-GPU communication. SGLang, NVIDIA, and AMD describe SGLang's low-latency mode in this LMSYS blog: a plugin framework that plugs vendor-written, highly fused kernels for specific models and hardware into SGLang without merging that code into the SGLang main repo. The test model is DeepSeek-V4.1-Flash with DSpark speculative decoding. On AgentX agent-trajectory replay, with 4 NVIDIA GB300 GPUs and 1 concurrent session, per-user P90 generation speed rises from 671.2 to 928.4 token/s (1.38×); with 4 AMD MI355X GPUs and 4 concurrent sessions, from 331.6 to 516.0 token/s (1.56×). Controls are standard SGLang on the same hardware.
1. Where time goes at small batch sizes
With one request, DSpark verifies only 6 token rows per step, so each kernel does little compute and dependencies plus cross-GPU communication dominate. Deep fusion for a specific model, hardware, and concurrency can cut latency a lot, but every combination needs its own implementation; putting them all in the SGLang main repo would keep growing code and review burden.
2. Method
2.1 NVIDIA: overlap dependent kernels and communication (sgl-nvidia-mega)
After warmup under a CUDA graph, the authors profile one decode layer (1 request, median per segment):
Kernel count only falls from 1,205 to 1,189; latency drops mainly by cutting waits and overlapping work, not by cutting kernel count. They do four things:
- Start the next kernel earlier. Using PDL (programmatic dependent launch, which lets a later kernel begin while an earlier one still runs), the "ready" signal is sent sooner so the next kernel can prepare barriers and load weights. Attention starts gathering KV before the query is ready; RoPE is done while loading the query, removing a separate kernel.
- Send results during the epilogue of the matmul. After the attention output projection each GPU holds a partial sum that must be reduced across GPUs. The baseline writes the matmul back to HBM and has the next kernel read it for the send; the plugin lets each thread block multicast over NVLink as soon as its tile finishes, without waiting for the whole matmul.
- Start expert matmuls before routing finishes. With at most 8 rows, tokens selected per expert fit in one 8-row tile; the plugin locates them with a bitmap and counts—no scan or sort—and starts the first expert matmul early. MoE median time falls from 44.4 µs to 36.5 µs.
- Pack speculative draft sampling into one kernel. DSpark's draft head proposes 5 tokens in order; the plugin finishes that on each GPU with one kernel launch, cutting draft time from 85.1 µs to 34.2 µs with the acceptance logic unchanged.
2.2 AMD: merge each layer into two resident kernels (sgl-amd-mono)
On AMD the authors see three large costs: even with HIP graphs each small kernel still costs 2–4 µs on the GPU; idle gaps between kernels; and no compute during two cross-GPU all-reduces per layer. The plugin uses FlyDSL to write two resident kernels covering most of a layer: K1 does mHC update, Q/KV projection, and KV-cache writes; K2 does attention, output projection, MoE, and both all-reduces. Stages pass "values tagged with the decode-step id"; matching ids start work without a separate flag. All-reduce is inside the kernel: each finished output tile is pushed to all GPUs while other workgroups keep computing. Each cross-GPU wait has a shorter timeout than the SGLang watchdog; on timeout all GPUs abort that launch instead of hanging the service.
2.3 Plugin framework
An external kernel library plugs in in four steps: register (declare supported model structures), check (before loading weights, verify hardware, precision, and parallelism, failing early on mismatch), build (prepare weights and resources at init, before KV-cache size is fixed so memory use is counted), and runtime dispatch (the plugin covers only some step types and token counts; uncovered steps fall back to SGLang's existing forward).
3. Experiments
3.1 AgentX trajectory replay
This measures how much the plugin speeds up a near-production agent load. Setup: DeepSeek-V4.1-Flash + DSpark, TP4, per-user P90 generation speed, only the plugin switched.
On NVIDIA the speedup shrinks as concurrency rises; on AMD all three tiers stay above 1.45×.
3.2 NVIDIA fixed-length load
With 8K input and 1K output and a simulated mean acceptance length of 3.51, the NVIDIA plugin raises per-user decode speed by 12%–38% at 1–8 concurrent requests.
4. Limitations
Only DeepSeek-V4.1-Flash + DSpark is supported so far; GLM-5.3, K3, MiniMax M3 and others are still in development. The blog notes that profiling itself affects timing; per-segment numbers explain mechanisms, not serve as service SLOs. External kernels skip SGLang's usual code review; CI and performance regression for the plugin API are still planned; the API itself is early and will change. The blog gives no accuracy comparison and only states that speculative acceptance logic is unchanged.
5. Takeaways
If your load is low-concurrency and sensitive to per-user latency (for example coding or agent sessions) and you deploy DeepSeek-V4.1-Flash on GB300 or MI355X, try this mode. For other models, the transferable pieces are the method: at small batches, profile waits and communication first, then decide which kernels to fuse; and isolate specialized kernels outside the main framework with plugins.