INFRA · 2026-11-02
Infra — Paper Daily, 2026-11-02
Today's inference-systems thread replaces fixed settings in inference serving with ones measured and adjusted to load. TableTune turns the table of how many tokens speculative decoding guesses per step into something measured and self-correcting in production. FlatCake stores the KV cache with memory and SSD at the same level. ConfluxServe lets prefill and decode share a batch while each keeps its own parallelism. Rimuru lets expert weights and the KV cache share one pool of GPU memory. On the training side, TIDE removes the back-and-forth copies in FSDP2 communication, and an SGLang security write-up explains how, under disaggregated serving, KV cache data leaked into other users' requests.
4.60
Speculative decoding only pays off while the GPU has spare compute: the more requests share a batch, the fewer tokens the small drafter should guess for the big model each step. vLLM lets operators hand-write a table of how many tokens to guess at each batch size, but nothing checks whether that table is right, and users report speculation running slower than no speculation at all. TableTune times every batch size and every draft length on vLLM's own serving path and computes the highest-throughput table in minutes. Once in production, a plugin reads vLLM's counts of accepted guesses, combines them with the measured step times, recomputes how many useful tokens per second each draft length yields at each batch size, and rewrites the table with the best length. On a mixture-of-experts model, measured against the fastest fixed draft length at each load, the table from vLLM's documentation is up to 18% slower, while the measured table is at most 3% slower. This table should be measured, not copied from the docs.
4.45
LLM serving usually treats host memory as a KV cache layer in front of the SSD, but after analyzing eight production request traces the authors find that this layer barely filters any reads. Reused blocks either come back only after a long gap, or belong to agent sessions of several gigabytes each that memory cannot hold, so the SSD always ends up on the read path. FlatCake puts memory and SSD at the same level and decides each block's home when it is created. Shared system prompts and short-reply chats go to memory, while multi-gigabyte agent sessions are written straight to the SSD. A session's blocks are stored contiguously and never move, and reads stream large chunks from the SSD straight into the GPU. Replaying a production Claude Code trace, the share of prefixes it reads back instead of recomputing holds at 94% as load rises, while Mooncake drops to 64%. KV storage for long agent sessions should be planned per session.
4.35
In LLM serving, running prefill and decode together lets their resource needs complement each other, since prefill uses compute and decode uses memory bandwidth. But the two want different parallelism. A long-prompt prefill must spread across more GPUs to deliver the first token on time, while decode needs only a few. ConfluxServe treats each operator differently. In the linear layers, tokens from both phases form one batch and are computed together under the same weight sharding. At the attention layers they split: decode stays on its own group of GPUs, while prefill spreads across more GPUs through two All-to-All exchanges. Both paths run at the same time and merge back into one batch afterwards. On production workloads, the request rate it sustains within latency targets is 1.26 to 1.62 times that of the strongest baselines.
4.35
NVIDIAOpen models built for agents often have to choose between capability and inference cost. NVIDIA's Nemotron 3.5 Super optimizes both. It is a hybrid Mamba-Transformer mixture-of-experts model that activates only about a tenth of its parameters per token, and it scores 51.78% on SWE-bench Pro, ahead of the previous-generation Ultra with several times more active parameters. The deployment design matters more to people who serve models. The model carries its own multi-token prediction head for speculative decoding, and post-training trains that head again, raising the average number of accepted tokens per step by up to 21.8% over the untrained head. With NVFP4 quantization, throughput on an interactive agent workload is 2.43 times that of BF16. Base, post-trained and quantized weights are released together with the training data.
4.35
In FSDP2 training, parameters and gradients are stored one tensor per parameter, but communication wants one flat buffer ordered by GPU, so every step copies data back and forth: gathered results are copied into parameter storage, and gradients are packed into large fp32 buffers. In training a 70B model, these copies alone take 6.4% of step time. The default NCCL communication also competes for the GPU's compute units and can slow a concurrent matrix multiply by up to 69%. TIDE sends and receives data directly at the parameters' storage locations, handing the data movement to copy engines, RDMA and NVSwitch, which use almost no compute units. Across three models, step time drops by up to 27.1% relative to stock FSDP2. Beyond raw communication speed, data layout and contention for compute decide training speed as well.
4.35
To run a large MoE model on a machine with limited GPU memory, the usual approach keeps some expert weights in host memory and moves them onto the GPU when needed. Even so, GPU memory is still split into two fixed regions, one for experts and one for the KV cache. Under heavy load the KV cache runs short and concurrency stalls. Under light load the KV cache region sits idle but cannot hold extra experts, so expert weights keep being moved back and forth. Rimuru puts both into one shared page pool and adjusts it with load. When requests pile up, more experts leave GPU memory to make room for the KV cache, and when traffic drops, experts move back. To make this work, each kind of data manages the same pool with its own page size, and a rewritten MoE kernel reads expert weights that are not stored contiguously. On two consumer GPUs under heavy load, throughput reaches up to 19.03 times that of KTransformers.
4.35
In one of Cloudflare's production model deployments, a request received content from another user's session. The SGLang team traced it to the path where prefill and decode run on separate workers. A request that exceeded the prefill KV cache capacity should have been rejected, yet it still went through the handoff, and decode read cache pages that had never been filled for it. They also fixed a related hazard. When a request is interrupted midway, RDMA writes already sent are still in flight, but decode had already returned the destination slots to the allocator, so the late data could land in another request's KV cache without any error. The fix releases the slots only after those writes are confirmed to have landed. On two H100 GPUs, throughput before and after the fix differs by less than 2%. Teams running disaggregated serving should upgrade and check that context length matches cache pool capacity.
4.25
To get the full speedup of new GPUs with native low-bit math, both weights and activations must be quantized to 4 bits. A few activation channels carry very large values, and the usual fix collects calibration data first and then scales those channels down. DualQuant shows this works without calibration data. It looks only at the weights. To make the quantized weights reconstruct the original weights as closely as possible, it solves for a scaling factor per input channel and multiplies that weight column by it. The same activation channel is then divided by the same factor, which leaves the matrix product unchanged, so these factors end up scaling the activations as well. It supports many formats, including INT4, MX and NVFP4, and on most models and formats it matches or beats the accuracy of SmoothQuant, which needs calibration data. Running real packed 4-bit inference on an RTX 4090, throughput reaches up to 2.46 times that of BF16.
4.25
Post-training quantization has to settle three things: each layer's value range, what transform to apply to weights and activations before quantizing, and which way to round each weight. Most methods handle one of these at a time, each minimizing the error at that layer's output, but a small error at one layer does not mean the model's final output changes little. The authors find that a layer's error barely predicts the final impact. It does once the error is weighted by sensitivity, an estimate of how much the model's final output changes when that layer's output shifts slightly in each direction. They then use an approximate Hessian to optimize all three under one objective: how much quantization changes the model's final output. With weights, activations and the KV cache all at 4 bits, it gets the lowest perplexity on all four models, and on the smallest model it lowers perplexity from the best baseline's 17.23 to 16.05.
4.15
Diffusion language models denoise many tokens in parallel each round and promise much faster generation than producing one token at a time, but serving frameworks copy the scheduling of autoregressive models: first come, first served, and prefill first. The authors find both rules fit poorly. How many denoising rounds a request needs varies widely and is unknown at arrival, so first come, first served leaves short requests waiting behind long ones. Rounds that run only prompts leave most of the batch idle. These models generate in fixed-size blocks of tokens, and a prompt step and a denoising step each process one block at the same cost. DWISE therefore counts remaining work for both prefill and denoising in block rounds, runs requests with the least left first, and lets both phases share a round. Remaining rounds are first estimated by a small regression model from the prompt's word, line and token counts. After the first denoising round, the model's final-layer hidden states are averaged and passed to a two-layer network that re-estimates them once. Across five models on SGLang, stock SGLang's mean end-to-end latency is 5.4 times that of DWISE.