SGLang Low-Latency Mode: Kernels and Runtime

At low concurrency, such as a coding or agent app where one user has only one or two sessions open, each request has very little to compute per step, and latency goes mostly to kernel launches, idle gaps between kernels and multi-GPU communication. In this LMSYS blog post, the SGLang, NVIDIA and AMD teams introduce SGLang Low-Latency Mode: a plugin framework that brings vendor kernels, heavily fused for specific models and hardware, into SGLang without merging that code into the SGLang main repository. The test model is DeepSeek-V4.1-Flash with DSpark speculative decoding (DSpark is the draft scheme used here: a small draft head proposes 5 draft tokens one after another each step, and the main model verifies them in one pass; the blog calls it a Markov head without describing its structure). In AgentX agent-trace replay, on four NVIDIA GB300 GPUs with one concurrent session, P90 per-user generation speed rises from 671.2 to 928.4 tokens/s (1.38x); on four AMD MI355X GPUs with four sessions, from 331.6 to 516.0 tokens/s (1.56x). Both baselines are standard SGLang on the same hardware. This is an engineering blog, not a paper; the numbers are measured by the authors' own teams with no third-party reproduction.

1. The problem: at small batch sizes, time is not spent computing

With one request, DSpark verifies only six token rows per step, meaning one forward pass of the main model handles just six positions. The blog does not break this down; given the 5 draft tokens, it is presumably the 1 token settled in the previous step plus the 5 drafts. The matrix multiplications themselves are fast; what takes time is a chain of small kernels queuing one after another, the gap as each kernel starts and drains, and the waits during tensor parallelism's two cross-GPU reductions (adding up each GPU's partial results) per layer, when the GPU computes nothing. At large batch sizes these costs are spread over lots of computation; at small ones they dominate.

Deep fusion tuned to a specific model, hardware and concurrency can remove these gaps, but each combination needs its own implementation. There are several generations of NVIDIA and AMD hardware, model architectures differ, and so do parallel layouts; putting all of it in the SGLang main repository would keep growing the code and review burden. So the post really covers two things: how each vendor built low-latency kernels, and how SGLang plugs them in without taking on the maintenance load.

2. NVIDIA: almost the same number of kernels, much less waiting

NVIDIA's implementation is called sgl-nvidia-mega. The authors first profiled one decoder layer under a warmed CUDA graph, one request, taking medians over about 800 measurements per arm:

The whole layer is 29% faster, and more than half of that comes from the attention region. Most notably, the kernel count for one verify step only drops from 1,205 to 1,189. So the speedup does not come from merging kernels into fewer ones, but from overlapping consecutive kernels and waiting less. The post also warns that profiling affects timings and region medians do not add up to the whole layer; the table explains the mechanisms and does not represent serving performance.

Four specific changes:

1. Start the next kernel early. CUDA's PDL (programmatic dependent launch) lets a kernel start while its predecessor is still running, preparing barriers and loading weights before waiting for input. SGLang already used PDL; the plugin moves the "you may start" signal earlier. Attention begins gathering KV before waiting for the query, and RoPE (the position encoding that rotates query and key vectors by token position) is element-wise, so it is computed while the query is being loaded, removing a separate kernel.
2. Send results out at the end of the matrix multiply. After the attention output projection, each GPU has only a partial sum that must be reduced across GPUs. Previously the matrix multiply wrote its result to memory, and the next kernel read it back and sent it to the other GPUs. The plugin lets each thread block, as soon as its tile is done, write it straight from registers to the other GPUs via NVLink multicast, without waiting for the whole multiply to finish and without the extra read-back.
3. Start expert computation before routing finishes. With at most eight rows, each expert's selected tokens fit in one eight-row tile; the plugin records in a bitmap which tokens picked each expert, counts the set bits to get each expert's row count, and places the rows directly, with no scanning or sorting. It also launches the first expert matrix multiply while routing is finishing, so that step's setup overlaps routing. This is where the MoE region's 7.9 µs saving comes from.
4. Put draft sampling in one kernel. DSpark's draft head proposes five tokens in sequence, previously one launch each. The plugin does all five in one launch per GPU, cutting the draft stage from 85.1 µs to 34.2 µs with acceptance logic unchanged.

The workload is fixed at 8K input and 1K output. To isolate the plugin, both runs use the same setup with the speculative-decoding mean acceptance length simulated at 3.51 (3.51 draft tokens accepted per verification on average), and only the plugin flag differs. At one to eight concurrent requests, per-user decode speed rises 12% to 38%. The lower the concurrency, the bigger the gain, which matches the analysis above: what is saved is a fixed waiting cost, and the fewer the requests, the larger its share.

This figure carries one more conclusion the post does not state. With the plugin on and two requests, per-user speed is about 840 tokens/s, faster than the roughly 745 tokens/s with the plugin off and one request, while total throughput per GPU rises from about 165 to about 325 tokens/s. In other words, a service with a hard latency target can double its concurrency with the plugin and still be no slower per user than before.

3. AMD: only two persistent kernels per layer

AMD's implementation, sgl-amd-mono, supports MI355X and MI350X and takes a different route. The authors see three main costs at low concurrency on AMD. First, even with HIP graphs removing CPU-side launch work, each small kernel still costs 2 to 4 µs on the GPU. Second, in the gap while one kernel drains and the next ramps up, compute units sit idle and weight loading pauses. Third, during the two cross-GPU all-reduces per layer no computation runs.

The plugin uses FlyDSL (a tool for writing GPU kernels on the AMD side) to merge most of a layer into two persistent kernels (a persistent kernel is launched once and stays on the GPU, working through several stages itself instead of launching one kernel per small step). K1 does the mHC update (mHC is DeepSeek's manifold-constrained hyper-connections, which widen the residual stream into several parallel streams that each layer mixes with a small constrained matrix; the update is this mixing step), Q/KV projections and KV cache writes; K2 does attention, output projection, MoE and both all-reduces. One launch boundary is kept between them so that K1's KV writes are visible across the GPU before K2 reads them.

Inside a kernel, stages hand off through "values tagged with the decode step": each value a stage produces is stored together with the current decode step number. The next stage, on the same GPU or another, keeps reading that location, and once the number equals the current step it knows this step's value has been written and starts. This needs no separate flag and no per-step reset. The blog does not say exactly how value and number are written together. Weights that do not depend on routing load early; expert weights load once the router has picked the experts. The all-reduce is also brought into the kernel: each finished output tile is pushed to every GPU, each GPU sums in a fixed order, and other workgroups keep computing meanwhile.

This carries a risk: if one GPU stalls, all GPUs wait inside the kernel. So every cross-GPU wait has a deadline shorter than SGLang's watchdog; on timeout all GPUs abort that launch together instead of hanging the whole server.

4. The plugin framework: specialized kernels live outside the main repository

External kernel libraries connect in four steps. First, registration: for each supported model architecture, the library declares its requirements and build entry point. Second, checks: before loading weights, SGLang verifies that hardware, precision, parallelism and so on are compatible, failing early rather than mid-run. Third, build: weights and resources are prepared at initialization, finished before the KV cache is sized, so the plugin's memory is accounted for and does not compete with the KV cache. Fourth, runtime dispatch: a plugin may cover only some step types and token counts, and uncovered steps fall back to SGLang's original forward. All GPUs must agree on whether a step uses the plugin, or they will enter mismatched collective communication. The path chosen during CUDA graph capture is recorded, so replay needs no Python-side decision.

The benefit is that vendor teams can iterate independently while SGLang maintains only the interface. The cost is that external kernels bypass SGLang's normal code review, so correctness must be backstopped by tests at the interface, and the post says those tests are still planned.

5. Results on an agent workload

AgentX trace replay simulates real agent sessions; the metric is P90 per-user generation speed (each user session's generation speed in tokens/s, taken at the 90th percentile; the blog gives no finer definition), both sides using 4-GPU tensor parallelism with DSpark, toggling only the plugin.

On NVIDIA the gain shrinks as concurrency rises, consistent with the fixed-workload results. On AMD all three levels are above 1.45x, highest at four sessions. One possible reason is that AMD's standard path starts lower: at one session its baseline is about 405 tokens/s, only about 60% of the GB300 baseline, so there are more gaps to squeeze. The two vendors differ in both hardware and kernel approach, so these numbers are not suited to comparing which hardware is faster.

6. Limits and practical takeaways

• Only the DeepSeek-V4.1-Flash plus DSpark combination is supported for now; GLM-5.3, K3, MiniMax M3 and others are in development.
• No accuracy comparison is given, only that speculative decoding's acceptance logic is unchanged. Fused kernels may change the order of floating-point operations (for example, the order of cross-GPU sums), and the post does not say whether results match the standard path exactly.
• CI and performance regression tests for the plugin interface are still planned, and the interface itself is early and will change.
• The gains are concentrated at one to eight concurrent requests. At large batch sizes these costs are already spread thin, and the post gives no data there.

If your service is low-concurrency and sensitive to per-user latency, and you happen to deploy DeepSeek-V4.1-Flash on GB300 or MI355X, you can try this mode directly. For other models, the method is the more useful part: at small batch sizes, profile each layer's timeline first to see whether time goes to computing or waiting; if it is waiting, prioritize kernel overlap and earlier communication rather than simply cutting the kernel count. Isolating specialized kernels outside the main framework via plugins is also a maintenance approach worth borrowing.