INFRA · 2026-11-10

Infra — Paper Daily, 2026-11-10

Today's infrastructure track keeps the already published work on prefix-cache collapse, speculative decoding, and expert caching. The new cards add fast clustering for large embedding sets, residue compensation that buys back FP4 accuracy, affine-folded decode over low-bit KV caches, and deviation-aware recomputation for reusing caches across requests.

4.30

2. e-means: outperforming k-means (in one-tenth the epochs)

Large-scale embedding clustering still often uses Lloyd's k-means, which takes many epochs and is awkward to tune. This paper turns the updates into mini-batches with an exponential moving average, and it gives scaling rules that set the decay rate and batch size from the number of clusters and the dataset size, so those knobs can be set without a hyperparameter sweep. Across embedding datasets, it reaches the loss of Lloyd k-means at 25 epochs in about two to three epochs, which translates to 10 to 45 times faster GPU wall-clock time than strong existing implementations. As one example, it partitions very large DINO embedding collections into many millions of clusters in just a few epochs, with a loss below Lloyd's longer training run.

4.20

4. ResServe: Accurate and Efficient FP4 LLM Inference with Residue Compensation

Block-scaled 4-bit floating-point inference lowers the cost of large language models but can hurt task quality. Decode matrices are often narrow, which leaves spare hardware capacity. This system keeps an FP4 activation base plus an FP4-quantized residue that represents what quantization discarded, uses the spare capacity to compute the residue on chip, and adds it back without extra result transfers. On NVIDIA B200, the extra request latency is at most 3.4% over uncompensated 4-bit weights and activations, while request latency falls 22.4% to 53.3% relative to 4-bit weights with 8-bit activations. Across five models, code-generation accuracy improves by a mean of 15.04% relative to same-format inference without compensation.

4.20

5. FOSCAR: Accelerating Decode Attention over 2- and 4-Bit KV Caches through Affine Folding

Decode attention streams the key-value cache for every newly generated token. Packing that cache into low-bit integers reduces memory traffic, but it does not automatically save latency, because converting each value and applying scale and offset before the Tensor Cores saturates instruction-issue throughput. This method leaves those per-value fixes off the cached values. It folds the scale and offset onto the other matrix operand or the product output, depending on the quantization scheme, and preserves the dequantized attention result in exact arithmetic. On H100, the INT2 and INT4 kernels reach geo-mean speedups of 3.36 times and 2.83 times over FlashInfer BF16.

4.15

6. A Controlled Experimental Evaluation of Configuration and Cache Architecture in LLM Serving Engines

Benchmarks often report that SGLang is far faster than vLLM and credit its prefix-cache design. This paper finds that comparisons of default deployments mix in another setting: whether a long prompt is prefilled in chunks. Under defaults on a multi-agent workload, vLLM's median time to first token is up to 64 times SGLang's. Matching chunked prefill alone removes 78% of the gap in sustained throughput where that gap is largest, and the gap in time to first token shrinks clearly too. What remains after matching is a real tail-latency advantage for SGLang at equal throughput, consistent with ordering waiting requests by what is cached. When comparing serving engines, match the configuration first, then judge the cache architecture.

4.10

7. Accelerating Offloading-Based MoE Inference by Interleaving Concurrent Requests

When a large MoE model runs locally, the GPU cannot hold every expert, so missing experts are fetched from host memory during inference. The usual remedy is prefetching, which fetches the experts the next layers will probably need while the link is idle. The authors find that prefetching can backfire when the GPU keeps an expert cache. The more needed experts the cache already holds, the harder the remaining ones are to guess, and wrong guesses waste bandwidth and memory. Bevy schedules instead of guessing. It runs several requests at once, and a request whose experts are ready runs ahead to later layers, revealing exactly which experts it needs next, so fetching overlaps with computing. On two local platforms its decode throughput reaches up to 1.55 times that of the fastest prefetching baseline. For local MoE serving, ordering requests well can beat predicting experts.

4.10

8. ResQAdam: Residual-Preserving Adaptive Optimization for Memory-Efficient Quantization Aware Training

Quantization-aware training is the most reliable way to get accurate low-bit LLMs, but AdamW keeps two extra FP32 optimizer states for every weight matrix, which costs a lot of memory. Low-rank optimizers such as GaLore save memory by keeping only the main directions of the momentum, and the authors find that they degrade badly in quantization-aware training. A large part of the momentum lies outside those main directions and is thrown away at every step, yet it stays useful even when only its sign is kept. ResQAdam therefore stores one BF16 momentum, applies Adam updates in the main directions, and for the rest keeps only the signs, scaled so this step is the same size as the Adam step, and adds both. It cuts optimizer memory by more than 70% relative to AdamW, stays close to AdamW in most settings, and lowers training throughput by less than 2%.

4.10

9. DARIC: A Position-Independent KV Cache Reuse System with Distribution-Aware Recomputation

As contexts grow, reusing key-value caches across requests without tying them to the original token positions can speed up the first generated token, but some tokens usually have to be recomputed to restore interactions across chunks. Existing methods often ignore how far a reused cache has drifted from a fully recomputed one, so token selection is either expensive or harmful to quality, and they may require a custom execution path in the serving engine. This system chooses what to recompute from that deviation distribution and keeps cache generation and recomputation loosely coupled. In an agentic coding setting it matches the pass rate of full recomputation while making TTFT 2.1 times faster; on some long-context retrieval-question datasets the speedup reaches 8.2 times.

3.35

10. Extending LLDB to Debug Applications That Span CPUs and GPUs

Meta

AI programs constantly move between CPU code and GPU kernels, and when a bug crosses that boundary, developers need to see the host and the device together. GPU vendors have built their own debuggers, such as ROCgdb and CUDA-GDB, so common debugging features are implemented again for each. Meta built a vendor-neutral accelerator plugin architecture in LLDB. It treats the CPU and the GPU as two separate debugged processes, each keeping its own instruction set, registers and address space, and presents them in one debugging session. Shared debugger features are written once, hardware-specific control lives in each vendor's plugin, and the same address pointing to different places on the CPU and GPU no longer causes confusion. Engineers who write GPU kernels and inference systems may be able to chase a bug across both sides with one debugger.

3.90

11. CodeQuant2: Enhanced Outlier Mitigation and Adaptive Clustering for Low-Precision LRMs

Reasoning models improve results by writing long chains of thought, which raises inference cost and makes low-bit quantization more important, yet aggressively quantized models often do poorly on reasoning tasks. The authors observe that activations fluctuate much more in the first decoding steps, and standard calibration data does not cover those early states. CodeQuant2 tames activation outliers with rotation and scaling, represents unevenly distributed weights with lookup tables learned by clustering, and picks calibration samples that better match real decoding. With a custom CUDA kernel, it clearly improves the accuracy of low-bit reasoning models on reasoning benchmarks and runs up to 2.85 times faster than BF16. When quantizing a reasoning model, the calibration data should cover the first decoding steps.

3.65

12. SpecPilot: Autonomous Algorithm Discovery and Systems Optimization for Speculative Decoding

Parallel drafters write a whole block of tokens in one forward pass, which in principle lets speculative decoding speed things up a lot, but their released systems fall far short of that ceiling. With one request, each step accepts only a few tokens yet reads all of the target model's weights. With batching, per-request verification cuts the speedup roughly in half. SpecPilot lets a team of agents find optimizations for each deployment. A cost model first points to the step furthest from its ceiling, several agents try directions in parallel, and every output token is checked against the original system so speed is never bought by changing the output. Without training or quality loss, the single-request speedup over autoregressive decoding rises from 4.46 times to 10.87 times. Speculative decoding systems still leave a lot on the table.

3.55

13. PagedMoE: Cross-Precision Expert Caching for Token-Level Dynamic Precision in SSD-Resident MoE Inference

Huge MoE models can run locally by keeping the experts on SSD and caching recently used ones in memory, but SSD reads still sit on the critical path. Choosing each expert's bit width per token could cut the data read, yet if the different precisions of one expert are stored as unrelated copies, the cache cannot reuse them and overlapping data is read again and again. PagedMoE treats precision as how deep one shared copy of an expert is read. A lower-precision version is exactly a prefix of a higher-precision one, so when the cache already holds the low-precision version, only the missing bit layers need to be fetched. At an average of 2.5 bits per expert weight, quality comes close to common 4-bit quantization, and on the same dynamic requests throughput is 1.33 times that of storing each precision separately.

3.40

14. Supermicro Launches KV Cache Solution with IBM and NVIDIA Demonstrating Significant Inference Performance Benefits

SupermicroIBMNVIDIA

For long-context inference, most of the time to first token goes into prefill, which recomputes the KV cache for a long prompt every time. Supermicro, IBM and NVIDIA have launched a solution that keeps computed KV cache on an all-flash storage cluster and reads it back instead of recomputing it. On a large model, the vendors measured up to 56 times faster time to first token for a 130,000-token prompt, and the benefit grows with input length. These are the vendors' own tests without independent replication, but they suggest that for long contexts used repeatedly, reading the cache from storage can be far cheaper than recomputing it.

3.40

15. Should You Split Decode? Disaggregation Across Future Models and Accelerators

In serving systems that already run prefill and decode on separate pools, is it worth splitting decode again into attention and feed-forward pools? This paper's answer is mostly no, and being able to readjust the hardware layout matters more. Hardware must be bought before future models and chips exist, so the authors compare both designs with a calibrated performance model under the same power budget. Splitting decode shows no robust advantage, and its largest lead actually comes from fitting one more prefill replica. So do not lock in the mapping, meaning which step runs on which kind of chip and how many units each pool gets. When conditions change, a fixed design delivers a median 36% less useful throughput than the best design, and readjusting on the same hardware cuts that gap to 9%.