INFRA · 2026-11-09
Infra — Paper Daily, 2026-11-09
Today's infrastructure track is about spending compute and memory more carefully in both training and inference. In reinforcement-learning post-training, FlexFloor releases GPUs as soon as their generation is done, EE-RL lowers GPU clocks stage by stage to save power, and MoEResQ shows that FP4 training corrections must be symmetric across the training and generation sides. On the inference side, SPARE compresses the recurrent state of linear attention, one study shows where the repair budget for KV cache reuse should go, which is the answer-bearing chunk, and SafeStart warns that cutting MoE experts for speed requires checking refusal behavior too. Vector index building, FP8 attention kernels on AMD, hardware-guided pruning, and FLUX 3 on Blackwell each speed up their own step.
4.35
Graph indexes for vector search, such as HNSW and Vamana, answer queries fast but take hours or even days to build on large datasets. PiPNN speeds this up by splitting the vectors into small blocks, finding neighbors inside each block with matrix multiplication, and merging the results into one graph. This paper pushes that route further. It quantizes the vectors to low precision to cut memory traffic, and it fuses the matrix multiplication with picking the top K neighbors, so the full distance matrix is never written back to memory. On CPUs, index building is up to 8 times faster than the original PiPNN. On GPUs it is up to 50 times faster than CAGRA, with query performance at high recall matching or beating the baselines. Teams that rebuild vector indexes often may no longer be held back by build time.
4.15
Reinforcement-learning post-training burns a lot of power, yet the GPU does not need to run at full clock all the time. Generation emits one token at a time and is often limited by memory bandwidth, and time spent waiting on other GPUs burns power too, so lowering the clock barely slows it down. This NVIDIA work builds an online controller, EE-RL, that picks separate GPU frequencies for generation and for training. It predicts how the default governor would perform on the same token workload, runs a few short trials, keeps the most energy-efficient setting that holds at least 98% of default throughput, and keeps adjusting as the workload changes. With training and generation on separate GPU pools in synchronous mode, it saves up to 16.9% of GPU energy at only 1.4% lower throughput. The power bill drops without touching the algorithm.
4.15
When an MoE model is trained with reinforcement learning, response generation is often held up by the few longest responses. A GPU whose own responses are finished still cannot be released, because the experts are spread across GPUs and unfinished responses elsewhere may call on the experts it holds. The paper measures that about one third of occupied time in the baseline comes after a GPU's own responses are done. FlexFloor first predicts from historical response lengths how long each response will take and groups prompts of similar expected length so they finish at about the same time. Finished GPUs then leave the expert-parallel group, the experts are rearranged across the remaining GPUs so that together they still hold every expert, and enough KV cache space is kept. Generation throughput rises by 52.74%, about 65.40% more tokens are generated per unit of occupied compute time, and rollout reward barely changes.
4.05
The usual way to prune an LLM is to set a parameter-count target, but parameter count does not determine latency, throughput, or memory. In the paper, two models with the same parameter count differ in latency: the narrow, deep one is 35% slower on an RTX 5070. AutoCompress turns this around and puts the deployment hardware's limits directly into the architecture search. It uses cost models validated against measurements from the serving engine, and adjusts the overall structure and each layer's size together, producing models whose layers are not all the same width. Across three settings, local long-context coding, batched datacenter serving, and low-latency in-game interaction, its average accuracy over eight benchmarks beats Minitron pruning by 6.57 points on an RTX 5070 but by only 0.57 points on an H100. When pruning for a specific GPU, searching the architecture against that hardware pays off most on smaller cards.
4.05
To make inference faster, MoE serving often runs only some of the selected experts for each token, and the change is usually checked on accuracy alone before launch. This paper finds that it can weaken the model's refusal of harmful requests. Under a jailbreak prompt that tells the model not to refuse, keeping about two thirds of the experts raises the share of harmful requests that Qwen1.5-MoE answers by 18.4 percentage points, while MMLU accuracy barely moves. Most of the increase appears while the model reads the prompt and generates its first few tokens. SafeStart therefore runs all experts for the prompt and the first few generated tokens and switches to the reduced expert set afterwards. It keeps at least 85% of the offline decoding speedup and brings the change under the jailbreak close to zero. Before enabling expert reduction, test refusal rates, not only accuracy.
4.05
Doing reinforcement-learning post-training in FP4 saves memory and compute but costs accuracy. For dense models there is already a fix, RolloutResQ. Quantizing activations to FP4 loses some precision, and the lost part is called the residual. The fix quantizes the residual separately and adds the two results to win precision back, but only on the response-generation side. This paper finds that on MoE models it not only fails to help but, on three of four math benchmarks, does worse than plain FP4 with no fix. The cause is routing. Experts are chosen discretely, so when only the generation side is corrected, the same token can be sent to different experts in training and in generation, and the two policies no longer match. MoEResQ applies the same correction on the training side too. On Qwen3-30B-A3B, the AIME24 accuracy gap to BF16 shrinks from 24.27 points with no fix to 3.75 points.
4.05
To save prefill, RAG and conversational memory often encode each retrieved chunk on its own and then concatenate the chunks' KV caches. The cost is that each chunk cannot see the chunks before it, so answers go wrong. Across three models and four datasets, reuse loses 19% to 100% of the context-dependent questions that full prefill answers correctly, and it loses more when the surrounding chunks resemble the answer-bearing chunk. When the repair budget is small, repairing those distractor chunks barely helps. The budget is better spent on the answer-bearing chunk, recomputed against the reused cache. In practice you do not know which chunk holds the answer. Finding it through question attention succeeds at most 18.8% of the time, and the retriever's top chunk is more reliable.
4.05
FP8 matrix multiplication has twice the peak throughput of BF16, but the softmax between attention's two matrix multiplications runs on vector units that FP8 does not speed up, so the overall gain shrinks. On the AMD MI355X, the vendor's AITER kernel reaches only 39% of peak FP8 matrix throughput at long context. ZYFA rewrites the prefill kernel. The bit pattern of an FP8 number approximates a scaled base-2 logarithm of its value, so one multiply-add on the attention score followed by an integer conversion gives a bit pattern that is read directly as the approximate FP8 probability, without computing an exponential. It also lets one group of threads compute softmax while another group runs matrix multiplications. On real inputs from three models, throughput is 1.16 to 1.76 times that of AITER, with lower numerical error.
4.05
Linear attention needs no KV cache that grows with length, but at every decoding step it must read and update a fixed-size recurrent state matrix, moving a lot of data. At high concurrency this becomes a memory-bandwidth bottleneck. SPARE builds on an observation: the effective rank of these state matrices is usually far below the original dimension, and it varies a lot across heads. Without retraining, it runs principal component analysis on each head's keys over a small calibration set and keeps the top directions as a fixed low-dimensional basis. The total dimension budget is handed out one dimension at a time, each going to the head whose state matrix has the largest remaining singular-value contribution. Reads and updates then happen directly on the compressed state. Projecting a key to the low dimension shrinks its squared length and weakens each write, so the write is divided by that squared length to compensate, with a cap. Full-model decoding becomes 2.55 times faster and peak GPU memory drops by 3.31 times.