Contents · 7 sections
1A two-GPU top-2 example: where each token goes2The sender picks its own slot, and a token is written once per GPU3The grouped GEMM reads in place by slot index4The remaining fixed cost is two synchronizations5Sentinel: the data signal their own arrival6At one token per rank, the round trip falls from 47.9 to 17.4 µs7It requires the whole EP group to sit in one NVLink domainSHARE
NVIDIA's NVLinkOneSided makes one point: at small batches, MoE all-to-all in inference is slow because of fixed costs, not data movement. Within one NVLink domain, let each GPU use its own routing result to write tokens straight into per-source slots in the peer's memory, skipping metadata exchange and per-expert regrouping; at small batches, also let the data itself signal arrival and skip the fence. That removes most of the fixed cost.
Start with the setting. A common way to serve large MoE models runs attention data-parallel (DP) and the MoE layers expert-parallel (EP). Each GPU (a rank, in the paper) serves only its own requests in attention and holds only a subset of the experts in the MoE layers. So every MoE layer needs two all-to-all exchanges. The first, dispatch, sends each token to the GPUs that hold its selected experts. The second, combine, sends the expert outputs back to the token's own GPU, where they are weighted and summed. Who sends how much to whom depends entirely on the router, so this is an all-to-all with variable sizes.
The difficulty is that each GPU holds few tokens during inference. Decoding must meet a latency target per output token, which caps the batch at a few hundred tokens per rank. The paper's example is DeepInfer decoding DeepSeek-V4.1-Flash: 64 requests per rank, each with 3 draft tokens, so a step processes 256 tokens. With only a few tokens per rank, the bytes to send would take less than a microsecond at the theoretical NVLink bandwidth, yet the measured dispatch-plus-combine round trip takes tens of microseconds. The extra time goes to fixed costs that do not grow with the number of tokens, chiefly the synchronization between producers and consumers on different GPUs.
Existing designs take a detour here. In NCCL and DeepEP (DeepSeek's open-source EP communication library), the sender first writes data into an intermediate buffer, and the receiver then copies or regroups it into the layout the next kernel needs. DeepEP V2.5 with DeepGEMM goes further: DeepGEMM wants each expert's tokens stored contiguously (an expert-major layout, in the paper's term), so each received token is copied again once per local expert it selected. At small batches this extra pass adds latency; at large batches it consumes memory bandwidth. A two-GPU example shows what it does in concrete terms.
A two-GPU top-2 example: where each token goes
The paper uses one example throughout, and so do I (illustrative: only 2 GPUs and 4 experts). R0 holds experts E0 and E1, R1 holds E2 and E3, and each token selects 2 experts. R0 has four tokens, T0 to T3; R1 has three, T4 to T6.

| Token (home GPU) | Selected experts | Target GPUs |
|---|---|---|
| T0 (R0) | E0, E2 | R0, R1 |
| T1 (R0) | E0, E1 | R0 only |
| T2 (R0) | E1, E3 | R0, R1 |
| T3 (R0) | E2, E3 | R1 only |
| T4 (R1) | E1, E2 | R0, R1 |
| T5 (R1) | E2, E3 | R1 only |
| T6 (R1) | E0, E3 | R0, R1 |
T1 is the row that matters most. Both of its experts are on R0. With an expert-major layout, R0 stores one copy of T1 for E0 and another for E1, so the same hidden state appears twice in memory. In addition, R0 and R1 both write into the same GPU. If all source GPUs shared one pool of slots, they would need either atomics on a remote counter or an exchange of per-GPU token counts to compute offsets. The costs the paper lists for common designs are two others: producer-consumer synchronization, and the extra copy out of the intermediate buffer.
The sender picks its own slot, and a token is written once per GPU
NVLinkOneSided starts from symmetric memory. Within one NVLink domain, each GPU uses CUDA's virtual memory management to map every peer's memory into its own address space, and a kernel reads and writes peer memory with ordinary loads and stores. The 8 GPUs of an NVIDIA HGX (GPU server board) B300 and the 72 GPUs of a GB200 NVL72 (NVIDIA 72-GPU rack) each form such a domain.
With direct writes, each GPU knows from its own routing which GPUs each token must reach, so it can push the token straight into the target GPU's memory and record locally which GPU and which slot it wrote. Metadata such as token counts or expert ids does not need to go out first. With the Fence protocol, the token goes directly into the dispatch workspace that the grouped GEMM reads; with the Sentinel protocol, it first lands in a set of landing slots, described later.
The problem is that several GPUs write into the same target GPU at once, and their slots must not collide. As noted above, a shared slot pool needs either remote atomics or a count exchange, which brings back the metadata exchange. The paper's answer is the rank-major layout: the target's dispatch workspace is split into one region per source GPU, and only that source GPU writes its region, taking the next slot by atomically incrementing a local counter for that target. Each region has T̂ slots, where T̂ is the largest token count of any GPU in this step; inference engines with DP attention already share the batch size every step, so every GPU knows T̂.
A token takes one slot per target GPU, and its k expert ids and router weights are stored in that slot. Back to the example (illustrative, filled as with the Fence protocol), R0's dispatch workspace ends up like this:
R0's dispatch workspace (T-hat = 4, 8 slots across two regions)
Region from R0: slot0 T0 [E0,E2] slot1 T1 [E0,E1] slot2 T2 [E1,E3] slot3 empty
Region from R1: slot4 T4 [E1,E2] slot5 T6 [E0,E3] slot6 empty slot7 empty
T1 takes only slot 1. The saving grows with the number of experts: a token is sent at most min(k, P) times instead of k times. With k=16 on 8 GPUs, a token is sent at most 8 times whatever the routing, at most half the bytes of a per-expert transfer.
The workspaces also shrink. For DeepSeek-V4.1-Flash on 8 GPUs with up to 4096 tokens per rank, this layout's dispatch workspace takes 160 MiB (MXFP8) and its combine workspace 320 MiB (BF16). DeepEP V2.5 with DeepGEMM needs a 242 MiB communication buffer plus an expert-major GEMM input whose hidden states alone take 1020 MiB. This shows that dropping the per-expert copy of the input changes the memory bill several-fold; neither protocol copies per expert, and all these buffers grow linearly with the EP size. Sentinel additionally needs a set of landing slots and one copy from a landing slot into the same slot of the dispatch workspace; the 160 MiB above counts only the dispatch workspace.
The grouped GEMM reads in place by slot index
If tokens stay in their slots, how does each expert find its tokens? The paper adds a small indexing kernel before MoE compute that turns the expert ids in the slots into a list of slots per expert. Continuing the example, the index on R0 is (illustrative):
E0 -> slots [0, 1, 5] (T0, T1, T6)
E1 -> slots [1, 2, 4] (T1, T2, T4)
Empty slots 3, 6, 7 carry invalid expert ids and are skipped
Slot 1 appears in both lists, but T1's data are stored once. FC1 (the expert's up and gate projections) uses gather instructions to load its input by these indices straight from the dispatch workspace and never loads empty slots; FC2 (the second grouped GEMM) applies the down projection. Finally, finalize adds up each token's outputs from the local experts, weighted by the router, and writes the sum into the same slot number of the combine workspace. T1's two outputs from E0 and E1 are summed locally on R0 into slot 1.

The results are back in the same rank-major positions as in dispatch, so combine needs no metadata either: each source GPU uses the target GPU and slot recorded during dispatch to pull the partial results from the targets' combine workspaces and add them.
The remaining fixed cost is two synchronizations
With a simpler data path, one thing still cannot be avoided. A remote store does not wait for the peer and may arrive out of order, and GPU memory is weakly ordered. So two points need synchronization: at the end of dispatch, a target GPU must know that every source GPU has finished writing before it computes; at the start of combine, a source GPU must know that every target GPU has finished computing and writing before it reads.
The first protocol is Fence. Each thread block handles one token, and the last block to finish wraps up for the whole GPU: it writes the slot counts to each target GPU, first executes one system-scope release fence, then publishes a completion flag to every peer, and waits until the flags of all peers have arrived. A small kernel then marks the slots past each count invalid (expert id −1). This chain is serial: atomic counting, the fence, writing flags and waiting for flags, none of which can be skipped.
Sentinel: the data signal their own arrival
Sentinel removes exactly this serial chain. It relies on an NVLink guarantee: each aligned 32-bit word stored to peer memory is seen whole or not at all, never half. So the source GPU does not write the dispatch workspace directly; it pushes the token into a landing slot on the target GPU. Every 32-bit word of a landing slot is preset to a sentinel value, a bit pattern the data never take. BF16 data use 0xFFFFFFFF (two BF16 NaNs), and a producer that meets this value stores two 0x7FFF NaNs instead. The target GPU polls a landing slot; once no word equals the sentinel, the slot is complete, with no fence. The target then copies it into the same slot of its dispatch workspace, which MoE compute reads as with Fence, and resets the landing slot to the sentinel. NCCL's LL protocol interleaves flag words with the data; Sentinel adds none, because the data themselves are the flags.
A simpler scheme does not work here: polling only the slots that have a token. The target does not know in advance which slots will receive data, so it cannot tell a slot still in flight from one that will never be written. Look at R1 in the example (illustrative):
Slots received by R1, Sentinel protocol (token i always lands in slot i; slots without a token keep the sentinel)
Region from R0: slot0 T0 [E0,E2] slot1 sentinel [-1,-1] slot2 T2 [E1,E3] slot3 T3 [E2,E3]
Region from R1: slot0 T4 [E1,E2] slot1 T5 [E2,E3] slot2 T6 [E0,E3] slot3 sentinel [-1,-1]
T1 does not go to R1, so slot 1 of the region from R0 never receives data. To keep R1 from waiting on it forever, Sentinel sets two rules: token i of a source GPU always lands in slot i of its region, and every source GPU writes expert ids for all T̂ slots on every target GPU, with −1 where no token goes. When R1 sees expert ids of −1 in slot 1, it knows no data will come. Slot 3 of the region from R1 works the same way, because it lies past R1's 3 tokens in this step. Combine also pushes: as soon as a target GPU finishes computing, it writes the partial results to the source GPUs without waiting for its peers.

The cost of this protocol grows with the batch, because each GPU writes, polls and resets P·T̂ expert-id slots every step, and every received word must be checked and copied into the dispatch workspace before the landing slot is reset to the sentinel. At large batches this per-slot work cancels the serial chain it saves.
At one token per rank, the round trip falls from 47.9 to 17.4 µs
The experiments run on 8 B300 GPUs and test the routed-expert layers of DeepSeek-V4.1-Flash (6 of 384 experts) and Kimi-K3 (16 of 896 experts) at 1 to 4096 tokens per rank, with perfectly even routing. The most telling case is one token per rank, because there almost everything is fixed cost: the DeepSeek-V4.1-Flash dispatch-plus-combine round trip takes 17.4 µs with Sentinel, 27.6 µs with Fence and 47.9 µs with DeepEP. NVLinkOneSided beats DeepEP at every batch: by 1.75 to 3.00 times up to 128 tokens and by 1.17 to 1.63 times from 256.
This shows that the gain comes mainly from fixed costs: the fewer the tokens, the larger the speedup. The two protocols cross over at 256 tokens per rank. The paper traced the critical path with in-kernel event timestamps: at one token, both protocols wait about equally long for their peers, and Sentinel saves the work around the wait; its dispatch skips Fence's serial chain and extra kernels and takes half as long. Sentinel shortens the work around the wait, not the wait itself.
The layer-level comparison needs two more baselines. DeepEP+DeepGEMM is the unfused version, with DeepEP for communication and DeepGEMM for compute; MegaMoE is DeepGEMM's fused version, which puts NVLink transfers and compute in one persistent kernel so they overlap. Here is the DeepSeek-V4.1-Flash layer latency (µs, from the paper's Table 4):
| Tokens per rank | Fence | Sentinel | DeepEP+DeepGEMM | MegaMoE |
|---|---|---|---|---|
| 1 | 46.0 | 36.7 | 74.7 | 49.8 |
| 16 | 79.6 | 70.5 | 122.0 | 90.3 |
| 128 | 188.5 | 183.3 | 235.9 | 188.7 |
| 256 | 208.1 | 206.5 | 251.2 | 194.1 |
| 4096 | 1185.2 | 1198.6 | 1290.2 | 766.1 |
Against DeepEP+DeepGEMM, NVLinkOneSided is faster at every batch, by up to 2.4 times, mostly in dispatch and combine. Against MegaMoE, it is faster by up to 30% up to 128 tokens; from 256, MegaMoE leads, by up to 35% at 4096. This shows that separate kernels remain competitive for decoding, but at large batches it pays more to hide communication behind the grouped GEMMs. Note that the three ratios have different baselines: 1.2 to 3.0 times is the round trip against DeepEP, 2.4 times is the layer against DeepEP+DeepGEMM, and 30% is the layer against MegaMoE.
Two further experiments rest on the same mechanism. With skewed (Zipf) routing at 1024 tokens per rank, the busiest GPU computes up to 1.9 times as many token-expert pairs as under even routing. But a token is sent to a GPU only once, so the bytes that GPU receives follow its tokens, not its token-expert pairs. So at Zipf exponent 1.25, dispatch and combine on the busiest GPU take only 160 and 135 µs, against 267 and 400 µs for DeepEP+DeepGEMM, because the latter copies tokens once per expert.
Scaled to 64 GPUs in a GB200 NVL72, the one-token round trip stays at 16.6 to 17.8 µs with Sentinel, while Fence rises from 43.0 to 47.6 µs. This shows that Sentinel has no per-peer handshake, so it barely grows with more GPUs.
It requires the whole EP group to sit in one NVLink domain
The design depends on two things: direct loads and stores to peer memory, and 32-bit words arriving whole. RDMA provides neither; the order of placement within an RDMA write is undefined. So the EP group must fit in one NVLink domain, such as an 8-GPU HGX board or a 72-GPU NVL72 rack. In addition, the dispatch workspace keeps one region per source GPU, so memory grows with the EP size. The evaluation uses perfectly even routing plus Zipf simulation and reports kernel and layer latency, not end-to-end serving throughput.
Back to the two-GPU example: T1 takes a single slot on R0, and R1 does not need to hear first how many tokens R0 will send; seeing expert ids of −1 in slot 1 tells it not to wait. In a decoding step with one or two tokens per rank, the microseconds saved are exactly those spent waiting for metadata, waiting on fences and regrouping per expert. It is already integrated into FlashInfer and vLLM; the authors also plan to bring this one-sided design into a fused kernel, to match MegaMoE's overlap of communication and compute at large batches.
How did this read for you?
Hoot made a note. Thanks for the feedback.
One tap, no sign-in; tap again to undo. Only anonymous counts are kept, never who you are.
Related papersMore from Paper Daily
Infra · 2026-11-02Nemotron 3.5 Super and Lightning: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Models for Agentic AIOpen models built for agents often have to choose between capability and inference cost. NVIDIA's Nemotron 3.5 Super optimizes both. It is a hybrid Mamba-Transformer mixture-of-experts model that activates only about a tenth of its parameters per token, and it scores 51.78% on SWE-bench Pro, ahead of the previous-generation Ultra with several times more active parameters. The deployment design ma…
Infra · 2026-11-13Multi-Future Drafting: Mixture Block Drafters for Tree-Verified Speculative DecodingIn speculative decoding, a drafter that drafts a whole block of tokens in one forward pass is fast, but it gives each position its own separate distribution. When the continuation has two equally plausible phrasings, a draft tree ranked by multiplying per-position probabilities splices them into sentences the target model would never say. The authors prove that such a tree can hold both phrasings…
Infra · 2026-11-02Rimuru: Unified Memory Management for Modern LLM Serving via Elastic Expert OffloadingPutting an MoE model's expert weights and its KV cache into one shared pool of GPU memory, and shifting how much each side gets as load changes, lets a memory-limited machine serve many more concurrent requests. Rimuru is a serving system built this way. When requests pile up, it moves rarely used experts out of GPU memory to free pages for the KV cache, and when traffic drops, it brings experts…
Discussion
Add experiment details, reproduction notes or a different take. Requires a GitHub sign-in.
Loading the discussion…