Contents · 6 sections
1Why communication occupies compute cores2How a no-SM transfer is put together3Walkthrough: how a 144 MiB AllGather contends with GEMMs for SMs4When a training step actually gets faster5What this means for people who build LLM training systems6LimitationsSHARE
An AllGather starts, and the GEMMs beside it slow by a third: how Zero-SM NCCL moves communication off the SMsWhen you train on a B200, a matrix multiplication (GEMM) launches several hundred thread blocks and fills all 148 compute cores on the GPU (SMs, the general-purpose cores that actually do the math). The communication library NCCL, when it runs an AllGather—each GPU contributes its own slice and then receives every peer’s slice—also occupies a batch of those cores by default, to move data and wait on flags. On this node, the paper measures that a 144 MiB-per-rank AllGather slows the Qwen3-30B-A3B GEMMs it delays by more than 1% by a median of about 33%. Zero-SM NCCL, from ETH Zurich (the Swiss Federal Institute of Technology) and NVIDIA, is an attempt to move collectives and one-sided puts completely off those cores, so the transfer launches no SM kernel at all.
The problem grows with interconnect bandwidth. To saturate NVLink, NCCL needs 8 SMs on Pascal and 32 on Blackwell, about 22% of the GPU. Using fewer cores with today’s SM kernels means giving up bandwidth.
Among existing designs, NCCL’s symmetric kernels replace multi-stage pipeline protocols with a single-shot load/store algorithm and, by default, occupy only a few CTAs (cooperative thread arrays), but they still move data and wait on flags on the SMs. NCCLX (a GPU collective library that can offload bulk transfers to dedicated hardware) hands the bulk of the data to specialized units, yet still launches a small kernel for synchronization. Zero-SM NCCL builds a fully SM-free path from three pieces: CUDA stream memory operations for synchronization, copy engines for data movement inside a scale-up domain, and a CPU proxy that drives cross-node RDMA while staying ordered with the CUDA stream. On a GB200 NVL72 (a 72-GPU rack that shares one NVLink domain) and a B200 NVL8 (8 GPUs in a single-node NVLink domain), this path reaches up to 1.36× the AllGather bandwidth of NCCL’s legacy SM kernels. Under the same statistic, the median slowdown among GEMMs delayed by more than 1% falls from 33% to 4%. Plugged into Megatron-LM, it speeds up FSDP by up to 3.5%, and one-sided puts speed up pipeline parallelism by up to 14.9%.
Moving communication off the SMs can make training faster when messages are large and the GPU is already full; the same path is slower on small messages, or when the host can no longer stay ahead of launch.
Why communication occupies compute cores
NCCL collectives are host-initiated: the CPU enqueues operations on a CUDA stream, and the GPU then executes them. A CUDA stream is an ordered work queue, so a later operation on the same stream starts only after earlier work has finished. Inside a scale-up domain—a set of GPUs that can load and store one another’s memory, usually over NVLink—the older pipeline protocols launch tens of CTAs whose threads copy data, perform reductions, and spin on flags. Across the scale-out network (nodes connected by RDMA NICs (network interface cards) such as InfiniBand or RoCE), an SM kernel also packs data and posts descriptors for a CPU proxy to issue RDMA. Synchronization on both paths, including flags, barriers, and completion tracking, runs on the SMs. A DMA engine can move bytes, but it cannot express control flow such as “wait until the flag arrives, then proceed.”
GEMMs launch far more thread blocks than a GPU has SMs, so every core held by communication is a core denied to compute. Figure 1 lays this out. Panel A is the number of SMs NCCL needs to saturate NVLink in each GPU generation. Panel B is how much a 144 MiB AllGather on one B200 slows neighboring GEMMs as the CTA count changes. Panel C converts end-to-end step-time gains into estimated savings over a 15T-token pretraining run, using CoreWeave on-demand prices.

How a no-SM transfer is put together
Zero-SM NCCL sets itself two rules. A transfer must launch no GPU kernel and occupy no SM. It must also keep CUDA stream semantics: the communication may start only after earlier work on the same stream has finished writing the send buffer, and later work may read the receive buffer only after the data has arrived.
Data movement uses two hardware paths. Inside a scale-up domain, a copy engine takes over—the hardware unit that runs copies such as cudaMemcpyAsync. Once the copy is issued, it no longer occupies SMs. Across nodes, a CPU proxy thread posts RDMA writes to the NIC, and the NIC reads and writes peer GPU memory itself. Synchronization uses stream memory operations in the CUDA driver, such as writing a 64-bit value on a stream or waiting until a memory location reaches a given value. Both are enqueued on the stream and launch no kernel. Send buffers, receive buffers, and signal locations are symmetrically registered: every peer learns the others’ addresses at registration time, so a copy engine or NIC can write directly into a peer’s already-registered user buffer, with no extra staging copy inside the library and no runtime address exchange. If the application keeps the data in unregistered ordinary memory, it still has to copy into that registered buffer before sending.
The authors also add two one-sided APIs. ncclPutSignal writes data to a peer and then updates a remote signal; ncclWaitSignal blocks the local CUDA stream until that signal reaches a specified sequence number. Inside a scale-up domain, publishing the signal takes two steps: a stream memory operation writes the sequence number into a staging word on the sender’s GPU, then a copy engine copies that word to the peer. The reason is concrete: a stream memory operation cannot write memory on another GPU, and a copy engine only moves bytes—it cannot atomically increment the destination—so the value must already sit in local device memory, as an immediate operand, to serve as the copy source.
A cross-node put is split between the main thread and the proxy. The main thread appends a descriptor to a queue, writes a ready flag on the stream, and enqueues a wait on a done flag. The proxy posts RDMA to the NIC only after it sees ready; when the transfer completes, it writes the done flag, and the stream proceeds. That way the CPU cannot start a network transfer before the GPU has finished producing the send buffer, and the GPU cannot consume the receive buffer before the transfer has arrived. The peer’s signal is not this sender-side done flag. The network plugin exposes an ordered put-with-signal (iput_signal) that makes the remote signal visible only after the payload has been transferred; the receiver’s ncclWaitSignal waits on that signal.
These primitives compose into pure data-movement collectives: AllGather, AlltoAll, Gather, and Scatter. Reduction collectives that do arithmetic on the data, such as AllReduce and ReduceScatter, cannot yet take the zero-SM path. A cross-node AllGather treats GPUs with the same local rank as a rail. The proxy exchanges data along the rail with RDMA, then copy engines scatter each newly arrived chunk inside the node, so local distribution can start before the entire network exchange has finished. Figure 2 shows this division of labor.

Walkthrough: how a 144 MiB AllGather contends with GEMMs for SMs
The walk-through uses the paper’s overlap benchmark, to show how SM-backed communication slows GEMMs that already fill the GPU. The setting is one B200 NVL8 node, 8 GPUs, 148 SMs each. The compute is 100 MXFP8 (microscaling 8-bit floating-point) GEMMs from Qwen3-30B-A3B, each launching 640 thread blocks, so the GPU is already saturated. On a second CUDA stream runs a 144 MiB-per-rank AllGather: each of the 8 GPUs contributes 144 MiB and receives the other 7 slices, so by the definition of AllGather the output on each GPU is 8 × 144 = 1152 MiB.
Step 1. Take NCCL’s legacy ring, which by default launches 32 CTAs. Figure 1A draws the SMs needed to saturate NVLink on Blackwell as 32, about 22% of this B200’s 148 SMs. The paper maps the cores occupied by communication onto those CTAs, and those cores cannot run GEMMs while the collective is in flight. Under the paper’s statistic—only the GEMMs this AllGather slows by more than 1%—the median slowdown is 32.9%, and 47 of the 100 GEMMs fall in that set. This means the default communication core count is already high enough to slow neighboring compute substantially.
Step 2. Can you just give the collective fewer cores? Dropping the CTA count to 4–8 lets the legacy ring’s slowdown in Figure 1B fall below Zero-SM. But then this AllGather itself becomes 3.2–5.6× slower than Zero-SM. Fewer cores are paid for with bandwidth.
Step 3. Switch to the symmetric kernel, the single-shot load/store algorithm described above. At its default of 4 CTAs, the median slowdown among GEMMs delayed by more than 1% falls to 6.2%. Forcing 32 CTAs finishes the AllGather sooner, and only 10 GEMMs slow by more than 1%, but those that are hit suffer more, with a median slowdown of 24.0%. This means more cores move the data faster and hit neighboring compute harder.
Step 4. How Zero-SM moves this 1152 MiB. First comes a readiness barrier: each GPU uses the copy engine to publish a “buffer is ready” flag to the other 7 GPUs, then uses stream memory operations to wait until all 7 flags have arrived, and only then may peers write into its output buffer. Next, each GPU copy-engines its own 144 MiB directly into the matching slot of the 7 peers’ symmetrically registered buffers. The local 144 MiB that belongs in this GPU’s own output also has to be copied into the local output buffer. An ordinary same-device copy would default to the SMs; Zero-SM NCCL uses cudaMemcpyFlagPreferOverlapWithCompute to force it onto the copy engine, so the zero-SM guarantee holds. That self-copy is only 1/8 of the output and runs concurrently with the copies to peers. Finally a completion barrier runs, and only then may later compute read the AllGather result. The entire path uses 0 CTAs.
Step 5. The overlap result. Again counting only GEMMs slowed by more than 1%, the copy-engine path’s median slowdown is 3.9%, which the authors attribute to memory contention: the data still travels over the same HBM and NVLink, but the SMs have been left free. 34 of the 100 GEMMs still fall in that set, at much smaller magnitude. In the same benchmark, a smaller 26 MiB AllGather from Llama-3-8B slows no GEMM by more than 0.2% on the copy-engine path; the legacy ring’s median among GEMMs slowed by more than 1% is still 9.9%. This means that once SMs are free, a large message still leaves some memory contention, while the slowdown on a small message almost disappears. Figure 7 plots the slowdown of every GEMM along the compute chain.

These numbers show that SMs held by communication clearly slow neighboring GEMMs that already fill the GPU, and that more occupied cores usually mean a harder hit on the GEMMs that overlap. Moving both data movement and synchronization onto copy engines and stream memory operations can cut that slowdown. They cannot tell you how much faster a training step will be. The paper itself notes that in real training, which phase of a collective lines up with which GEMM type shifts from step to step, and several collectives often overlap at once. So end-to-end timing has to be measured separately.
When a training step actually gets faster
The authors hook Zero-SM NCCL into two paths in Megatron-LM. One is the parameter AllGather of FSDP (parameters are sharded across GPUs and gathered back when needed). The other is the activation and gradient exchange between pipeline stages, replacing two-sided Send/Recv with one-sided put_signal / wait_signal. The evaluation covers 12 FSDP recipes and 7 pipeline recipes, dense, MoE, and hybrid models from 15B to 405B parameters, on 32 or 64 GPUs. The two systems are a GB200 NVL72, whose entire rack is one NVLink domain, and a B200 NVL8, where each node is its own domain and every multi-node collective takes the hierarchical AllGather. Groups are compared in paired blocks on the same nodes, to avoid the 1–5% noise that comes from different allocations.
Against the legacy ring, the copy-engine path speeds up the GPU-bound dense FSDP recipes on both systems, by up to 3.5% on GB200 and 1.9% on B200. GPU-bound here means the host stays far enough ahead that the GPU never waits for a new kernel to be launched. Against the symmetric kernel, which already uses only a few CTAs, the picture is mixed: the copy-engine path wins clearly on Llama-70B-G on GB200, mostly ties or wins narrowly on the other GPU-bound dense recipes, and loses on host-bound recipes and on Qwen3-235B-G on B200. From profiles of each recipe, the authors distill three conditions that all have to hold.
For the gain to show up, the step must first be GPU-bound. In end-to-end profiles, communication kernels slow the compute they overlap by 3–4% by occupying SMs. The copy-engine path removes that cost, but each collective does more work on the host. Issuing one copy-engine AllGather takes 26–41 microseconds on GB200, about 2–3× the 11–17 microseconds of launching the symmetric kernel, and 77–86 microseconds versus 5–7 microseconds on B200. So this extra host work stays off the critical path only when the host is already running ahead of the GPU.
The higher fixed cost also means messages have to be large enough to amortize it. On one node, a 1 KiB AllGather takes 25–29 microseconds on the copy-engine path and 12–18 microseconds with SM kernels; copy engines overtake the symmetric kernel in bandwidth only from 128–512 MiB per rank. Training shows the same threshold: on GB200, FSDP gathers of about 240 MiB or less tie or lose against the symmetric kernel, and 820 MiB wins. This means small messages expose the latency penalty of the zero-SM path, and only large messages put its bandwidth to use.
The route also has to match the other traffic on the links at the same time. A multicast injects a single copy and advances at the pace of its most congested destination, which fits more even flows such as the expert all-to-all of MoE recipes. Unicast uses independent copies, which, when stacked with tensor-parallel traffic on GB200, are less likely to stall the whole AllGather on one congested link. The copy-engine path is slower on Qwen3-235B-G on B200, and the authors’ reason is that plan fusion—packing consecutive AllGathers into one synchronization envelope—does not yet cover the faster rail schedules.
On the pipeline path, the gain is not mainly from freed SMs—releasing them speeds compute by at most 1–2%—but from removing the rendezvous of Send/Recv. Pipeline stages rarely finish a micro-batch together, because of uneven layer splits, the embedding and loss on the end stages, and varying expert loads in MoE models. With two-sided Send/Recv, a fast stage waits for its slower neighbor at every boundary. A one-sided put writes into a preallocated slot and moves on; only the receiver waits, and only when it needs the data. The gain therefore grows with the bubble. The bubble share is b = (PP - 1) / (GA × VPP (virtual pipeline stages) + PP - 1), where PP is the number of pipeline stages, GA is the number of gradient-accumulation steps, and VPP is how many interleaved pieces each stage is split into. Hoot says
The body already walked how a 144 MiB AllGather contends with GEMMs for SMs on one B200. This walk is a different path: activation transfer between pipeline stages. The recipe numbers and speedups below come from the paper; they are not illustrative.
stages drift apart → two-sided Send/Recv makes the fast stage wait at each boundary → a one-sided put lets only the receiver wait, and only when it needs the data → the gain grows with bubble share, but too many transfers eat it with fixed cost
- stage finish-time gap → Send/Recv rendezvousPipeline stages rarely finish a micro-batch together, because of uneven layer splits and the embedding and loss on the end stages. With two-sided Send/Recv, a fast stage waits for its slower neighbor at every boundary.Freed SMs speed compute by at most 1–2%, so the gain here is not mainly from freeing cores
- one-sided put → the sender moves onThe paper replaces stage-to-stage two-sided Send/Recv with one-sided puts. After the sender writes the data to the peer and updates a signal, it can return; the receiver, when it needs that data, uses wait_signal to wait for the matching signal.Send/Recv can proceed only when both sides have reached the boundary; a put can return once the sender has written the data and posted the signal
- bubble share b → step-time speedupQwen3-Next has 8 pipeline stages, 2 virtual pipeline stages, and 8 gradient-accumulation steps. The bubble share is 30.4%, and step time is 14.9% faster on GB200 and 9.4% faster on B200.b = (8 − 1) / (8 × 2 + 8 − 1) = 7 / 23 ≈ 30.4%
- two-stage pipeline → little rendezvous to removeTwo-stage Qwen3-32B, with 2 virtual pipeline stages and 8 gradient-accumulation steps, has a bubble share of only 5.9%, and is 3.9% slower on B200.b = (2 − 1) / (8 × 2 + 2 − 1) = 1 / 17 ≈ 5.9%
- 322 transfers per step → fixed cost eats the waiting-time gainLlama-70B has 4 stages but is deeply interleaved (5 virtual pipeline stages, 32 gradient-accumulation steps), so the bubble share is only about 2%, with about 322 transfers per step. The fixed host cost of each put, plus the copy into a registered staging buffer, eats the waiting time that was removed.b = (4 − 1) / (32 × 5 + 4 − 1) = 3 / 163 ≈ 1.8% (about 2% in the body)
So one-sided puts pay off when the pipeline is deep, the bubble is around 10% or more, and transfers are few and large.

Source
Paper §IV-D2, Table II, and Figure 9. Bubble share b = (PP − 1) / (GA × VPP + PP − 1).The table below keeps only a few comparisons; the full results are in the paper’s Figures 8 and 9. The dollar figures convert step-time speedups to a 15T-token pretraining run at CoreWeave on-demand prices ($10.50 per GB200 GPU-hour, $8.60 per B200 GPU-hour). The authors note that this is not a full deployment cost model.
| Setting | Baseline | What Zero-SM changes | Source |
|---|---|---|---|
| Qwen3-30B, 144 MiB AllGather, GEMMs slowed by more than 1% | Legacy ring median slowdown 32.9% | Copy-engine median slowdown 3.9% | Fig. 7, §IV-C |
| FSDP, Llama-70B-G, 64 GB200 GPUs | Versus the legacy ring | Step time 3.5% faster, estimated $396k saved | Fig. 8, Fig. 1C |
| FSDP, Nemotron-15B-H, 32 B200 GPUs, host-bound | Versus the legacy ring | Step time 4.9% slower | Fig. 8 |
| Pipeline, Qwen3-Next, 64 GB200 GPUs, b = 30.4% | Versus two-sided Send/Recv | Step time 14.9% faster, estimated $2.6M saved | Fig. 9, Fig. 1C |
| Pipeline, two-stage Qwen3-32B, 64 B200 GPUs, b = 5.9% | Versus two-sided Send/Recv | Step time 3.9% slower | Fig. 9 |
(Numbers from paper Figures 7, 8, and 9 and §§IV-C, IV-D)
Figure 9 lines up the 7 pipeline recipes by bubble share. Llama-70B at b ≈ 2%, and the two-stage Qwen3-32B and Nemotron-15B, gain almost nothing and sometimes slow down. At b = 30.4% on Qwen3-Next, the speedup is 14.9% on GB200 and 9.4% on B200. The authors’ conditions for one-sided puts are that the pipeline is deep, stages drift apart, and a substantial share of the step is spent in the bubble (PP of at least 4, b around 10% or more); transfers must also be few and large. Otherwise the fixed host cost of each put, plus the sender’s copy of activations or gradients from ordinary PyTorch memory in Megatron-LM into a registered staging buffer, eats the waiting time that was removed. FSDP usually does not pay this staging cost: when NCCL user buffers are enabled, Megatron-FSDP already allocates parameter and gradient buffers from an ncclMemAlloc pool, and registering that pool as a symmetric window lets the parameter AllGather take the zero-SM path directly. A deeply interleaved recipe such as Llama-70B, with about 322 transfers per iteration and b of only about 2%, falls outside this boundary.

What this means for people who build LLM training systems
Check whether the step is already GPU-bound, whether messages have reached a few hundred MiB, and whether the pipeline bubble is deep enough, before turning on zero-SM communication.
If FSDP parameter gathers are already large, and host launch stays ahead of the GPU, switch NCCL’s CTA policy to the zero-SM path and compare step time against both the legacy ring and the symmetric kernel; do not trust microbenchmark bandwidth alone. If expert all-to-all or tensor-parallel traffic is also on the links, try both multicast and unicast: multicast is often a better fit for more even flows, and unicast is less likely to stall a whole AllGather when a few links are congested. If a pipeline has four or more stages and a bubble share around 10% or higher, it is worth replacing stage-to-stage Send/Recv with one-sided puts. A two-stage pipeline, or a recipe like Llama-70B with almost no bubble and hundreds of transfers per step, should not be expected to get faster. Reductions such as AllReduce and ReduceScatter cannot take this path yet, so do not assume the entire communication stack can go zero-SM.
Limitations
Inside a scale-up domain, small-message latency is still higher than with SM kernels, because every operation is issued from the host and synchronized through stream memory operations. On one node, a 1 KiB AllGather takes 25–29 microseconds versus 12–18 microseconds. Capturing operations into a CUDA graph and replaying them requires a replay-safe flag protocol, with extra sets and acknowledgments on every replay, so the cost is even higher. Scale-out synchronization depends on a CPU proxy thread and therefore consumes host resources. The number of copy engines on the GPU may also limit highly fan-out concurrent copies. Reduction collectives are not yet supported. The authors attribute the regression on Qwen3-235B-G on B200 to plan fusion for the hierarchical AllGather not yet covering all of the faster rail schedules.
How did this read for you?
Hoot made a note. Thanks for the feedback.
One tap, no sign-in; tap again to undo. Only anonymous counts are kept, never who you are.
Discussion
Add experiment details, reproduction notes or a different take. Requires a GitHub sign-in.
Loading the discussion…