ConfluxServe: Reconciling Resource Utilization and Parallelism Efficiency across Prefill and Decode in LLM Serving

Teams running LLM inference, especially those that must handle long prompts and many live sessions at once, should read this, because it takes the old question of whether prefill and decode should run together or apart down to the operator level and gets the benefits of both. In LLM serving, running prefill and decode together lets their resource needs complement each other, since prefill uses compute and decode uses memory bandwidth. But the two want different parallelism. A long-prompt prefill must spread across more GPUs to deliver the first token on time, while decode needs only a few. ConfluxServe treats each operator differently. In the linear layers, tokens from both phases form one batch and are computed together under the same weight sharding. At the attention layers they split: decode stays on its own group of GPUs, while prefill spreads across more GPUs through two All-to-All exchanges. Both paths run at the same time and merge back into one batch afterwards. On production workloads, the request rate it sustains within latency targets is 1.26 to 1.62 times that of the strongest baselines. Whether to batch together or run apart can be decided per operator rather than for the whole model.

What problem is it actually solving?

Take a concrete scenario (illustrative). One machine runs Qwen3-14B on 4 H100s, each GPU already hosting a batch of sessions in decode, when a 32K-token prompt arrives. The service has to hold two lines at once: TTFT under 2 seconds and TBT under 100 ms.

The paper first measures why the two phases pull against each other. In resources they complement each other. Sixteen decode requests with 8K contexts take only about 20 ms per iteration, using about 16% of the GPU's compute. Under 50 ms and 100 ms TBT targets, roughly 1K and 2.2K prefill tokens can ride along, lifting utilization to 75% and 90%. Chunked prefill exploits exactly this.

In parallelism, though, they want opposite things. A 32K prompt needs at least 2 GPUs, and a 64K prompt at least 4, to produce its first token within 2 seconds, so prefill must spread out. How it spreads matters too: SP keeps 93% parallel efficiency at 8 GPUs, while TP keeps only 66%. Decode is the reverse. Each iteration computes just one token per request, a single GPU finishes an iteration within 25 ms, and spreading it to 8 GPUs drops efficiency to 39% under TP and 16% under SP, because there is too little work to amortize communication.

So existing systems pick one side:

The paper measures this dilemma on 4 H100s: keep 96 decodes running and see how many prefills per second each design completes without breaking TBT. Under a 50 ms target, LoongServe, Chunked Prefill and MuxWise reach 2.41, 4.17 and 4.38, while ConfluxServe's parallel scheme reaches 6.15. The two benefits really do stack, rather than being a trade-off to balance.

The big picture: what happens inside one Transformer layer

First, where the weights live. SP splits only the sequence, not the weights, so in Figure 7 every GPU holds a full copy of the weights. When a model is too large for one GPU, a fixed set of GPUs forms a TP group that acts as one logical GPU: weights are split by TP inside the group as usual, prefill runs SP across the groups, and each decode request lives in a single group. In the experiments, the 8B and 14B models use one GPU per group and the 32B model uses two. Now walk through the two-GPU example in Figure 7. Prefill request A has a chunk with tokens a3 and a4, split by sequence so each GPU holds one. Decode requests B and C each have one new token, b4 and c2, living on GPU 0 and GPU 1 respectively. One layer has three stages:

1. QKV projection: GPU 0 stacks [a3, b4] into one two-row matrix and computes it in a single matrix multiply; GPU 1 computes [a4, c2].
2. Attention: the paths split. b4 and c2 each compute attention on their own GPU against their own KV cache. a3 and a4 go through a first All-to-All that changes the layout from "each GPU holds part of the sequence, all heads" to "each GPU holds the whole sequence, half the heads". Each GPU computes its half of the heads, and a second All-to-All switches back to the sequence split. The two paths run at the same time on separate CUDA streams.
3. Output projection and FFN: the paths merge again, and each GPU once more computes [a3, b4] and [a4, c2] together.

Where the KV cache lives follows from this. Decode KV stays on its GPU, and prefill KV stays spread by attention head, so later chunks reuse it without moving anything. The only transfer happens when a request finishes prefill and moves to decode: its KV shards are sent once to the chosen GPU over the fast intra-node interconnect, overlapped with the last prefill chunk.

Why can linear layers be batched but attention cannot?

This is the key observation. A linear layer applies the same weights to every token. SP only changes which part of the sequence sits on which GPU; it does not change the weights or the feature dimension. So however many GPUs prefill is spread across, the prefill and decode tokens on each GPU fit into one matrix multiply. The payoff is that weights are read once: decode normally reads the whole weight matrix from memory just for a few tokens, and now that same read also serves hundreds of prefill tokens, using compute and bandwidth together.

Attention is the opposite case. Two batches of attention fit into one call only if QKV is split the same way, and prefill is split by head while decode is not split at all. Merging would not help much anyway: each request reads only its own KV, so adding requests grows compute and KV reads in proportion and does not raise arithmetic intensity. The paper therefore batches where batching pays and is possible, and lets the paths separate where it pays little and is impossible.

The split attention stage also overlaps naturally. The prefill path uses compute and communication, while decode attention uses memory bandwidth. To keep them from fighting too hard over SMs, the system uses NVIDIA's GreenContext to partition SMs inside this operator: decode gets just enough SMs to reach 90% of its standalone peak bandwidth, and prefill gets the rest. Linear layers stay inside CUDA graphs; only this narrow attention stage runs outside them.

Scheduling: how much prefill per iteration, and on which GPU

With the execution scheme in place, the system must decide how much prefill to add each iteration without breaking TBT. ConfluxServe builds a latency model. Each layer's time is the sum of two stages. The linear stage is set by the GPU with the most local tokens. The attention stage takes the slower of the prefill and decode paths, each scaled by a factor for mutual interference. The decode path is set by the GPU with the largest total decode context. The prefill path has two parts: communication depends on the largest prefill shard, and attention compute is spread evenly over all GPUs by the All-to-All. These functions are fitted by offline profiling, with mean error of 4.1% to 4.8% on the three models.

This gives the rule for dispatching prefill: make prefill plus decode tokens as equal as possible across GPUs. The paper calls it waterfill, like pouring water into uneven cups and filling the low ones first. A worked example (illustrative): the 4 GPUs hold 40, 10, 10 and 0 decode tokens. A binary search over the latency model finds that TBT allows at most 300 tokens per GPU, so the GPUs can take 260, 290, 290 and 300 more prefill tokens, a budget of 1140 for this iteration. A prompt with 2000 tokens left gets a 1140-token chunk this iteration, spread in those proportions.

When prefill finishes, the request needs a GPU to stay on for decode. While KV space is plentiful, it goes to the GPU with the smallest total decode context, keeping the next iterations balanced. Once every GPU's KV usage passes a threshold, the system packs requests onto the fullest GPU that still fits, so free space does not fragment into small pieces on every GPU that cannot hold the next long session.

The outer layer is queue order: first pick the requests that can meet their TTFT deadline under the current budget and order them by deadline, then fill in the rest shortest-remaining-first. One scheduling pass costs under 1.05 ms and runs asynchronously with GPU execution.

What do the results show?

Experiments run on NVLink-connected H100s with three models: Llama-3.1-8B and Qwen3-14B on 4 GPUs, Qwen3-32B on 8. The main workload is Mooncake-Toolagent from Mooncake production traces, with a median prompt of 6,346 tokens, a P95 of 26,114, and short outputs. The metric is goodput: the highest request rate at which at least 95% of requests get every token on time.

The first setting puts every system on first come, first served, isolating the execution scheme. The second gives every baseline a shortest-remaining-prefill-first scheduler. ConfluxServe leads in both, so most of the gain comes from execution, and its scheduler adds another 1.16 to 1.37 times on top.

The ablation switches off three things one at a time, with 24 decodes of 8K context per GPU and one prefill inserted. For a 32K prefill, removing attention overlap slows it to 1.11 times, also spreading decode out raises that to 1.16 times, and removing batching raises it to 1.34 times. All three contribute, batching the most, and the gap widens with prompt length because each chunk pays these costs again.

Other experiments each answer one question. With 32 decodes per GPU, attention overlap cuts that stage from 36.5 ms to 28.0 ms, with run-to-run variation under 3.5% over 20 runs, so the interference is stable and the latency model can rely on it. Scaling SP from 1 GPU to 2, 4 and 8 makes a 32K prefill 1.93, 4.17 and 7.94 times faster, close to linear, so most of the extra communication is hidden behind decode attention. With 128 decodes on one GPU and none on the other three, even dispatch makes the linear stage 1.32 times slower, and waterfill evens it out.

Where I think it falls short

First, it is validated on a single machine. Limited by the testbed, each run uses one 4- or 8-GPU instance, with no routing across instances and no cross-node setup. SP relies on the fast intra-node interconnect for its All-to-All, and across machines the All-to-All gets slower, so how much communication the overlap still hides is something you would have to measure yourself.

Second, gains shrink a lot with short prompts. On the QwenA workload, with a median prompt of only 1,046 tokens, it beats the strongest baseline by just 1.1 times, compared with 1.7 times on the longer-prompt Mooncake-Synthetic. If your traffic is mostly short chats, there is little room.

Third, the experiments cover only three dense models, the largest at 32B. The latency model and SM split are calibrated by offline profiling for a specific model and GPU, so a mixture-of-experts model or different hardware needs recalibration.

How an inference team could try it

1. Measure your traffic first: prompt length P50 and P95, and the time of one decode iteration on one GPU. If the P95 prompt needs several GPUs to meet TTFT while decode has plenty of TBT slack on one GPU, the paper's premise holds.
2. Build the execution scheme itself first: organize GPUs into fixed groups (single GPUs or fixed TP groups), run prefill across groups with Ulysses SP and its KV split by attention head, let each decode request live in one group with its KV kept there, and batch both phases' tokens in the linear layers. Time one long prefill under a fixed decode load and compare with the existing approach of uniform TP plus batching on the same number of GPUs.
3. Then add waterfill dispatch and the latency model. The bar is whether prediction error gets down to about 5%; with larger errors the TBT budget is miscomputed and the scheduler either wastes capacity or misses deadlines.
4. Finally compare goodput on a replay of real traffic, counted as 95% of requests getting every token on time, not just average throughput.

Verdict

The one thing to take away: the conflict between prefill and decode is not in the whole model but in a single operator, attention, so split only there and batch everywhere else.