INFRA · 2026-11-17
Infra — Paper Daily, 2026-11-17
Today's infrastructure track asks how to spend compute on what actually limits throughput or tail latency, without breaking quality or time limits. Ads ranking pushes quantization to 6 bits while still holding a very tight error bar. Collectives try to stop occupying compute units, and a compiler puts GPU-initiated communication and computation into the same intermediate representation. Storage movement picks the path with a better tail per phase, decode scheduling leaves headroom from the iteration tail, and requests whose cache has been invalidated speculate first with stale KV. On the training side, one system temporarily turns idle rollout GPUs into trainers while the long tail finishes, and another shrinks the trainable fine-tuning state to scalars on fixed directions. Edge quantization and an Ascend kernel-optimization benchmark turn the upkeep of formats, and whether existing implementations can still be made faster, into questions one can measure.
4.45
Ads ranking models are extremely sensitive to quantization error. A launch bar often requires that normalized entropy stay within 0.01% of the baseline, much tighter than the accuracy loss large language models often tolerate, so lower-bit formats have been hard to put into production.
4.20
In synchronous, colocated reinforcement learning, inference engines often sit idle waiting for a few very long generations, or decode inefficiently at very small batch sizes, and GPUs go underused. While the long tail is still running, the authors temporarily repurpose some rollout GPUs for training, and migrate unfinished generations among inference engines to raise decode throughput, while keeping on-policy training semantics.
4.15
GPU communication libraries often use compute units to move data and synchronize, and the cost of stealing compute grows as interconnect bandwidth rises. The authors instead complete collectives with copy engines and network transfers driven by a CPU proxy, without launching a compute kernel for communication.
4.15
Machine learning systems often specialize computation to the workload and the hardware, but communication still lives in generic libraries, whose device code and transport choices a high-level compiler cannot see. The authors put GPU-initiated communication into a kernel's intermediate representation, plan completions, synchronization, and batching together with computation, and lower the result to specialized transport code.
4.05
When language models are quantized to run on edge devices, every weight format must be paired with every activation format and instruction set, so the number of kernels a library must maintain grows quickly. In GGML, the tensor library used by llama.cpp, only 8 of 54 weight-activation pairs on the authors' Arm CPU test have an optimized tiled matrix multiplication, and the rest compute one block dot product at a time. The authors also put value approximations into the compiler: they specify an algorithm, choose how values are encoded and decoded, and schedule the resulting computations, generating hundreds of kernels from one matrix-product algorithm and one schedule library. The application uses about 3.7 times less code than GGML and KleidiAI. A format should not be hard-wired into a handwritten kernel; it can be a program the compiler can schedule. For teams that must maintain many quantized kernels on the edge, encoding, decoding, and computation can share one scheduling path, rather than handwriting a kernel for every format.
4.05
Parameter-efficient fine-tuning aims to change fewer parameters without losing quality, but existing methods still often leave a large trainable state, and optimizer states occupy memory as well. The authors learn only a set of scalar coefficients per layer: the input basis is fixed from task activations, the output dictionary is assigned at random, computation first aggregates output atoms before applying them to tokens, and compatible layers can share a dictionary. On commonsense reasoning, this reaches accuracy comparable to LoRA with about 4% of LoRA's trainable parameters. On Qwen2.5-7B, peak training memory is 20% lower than LoRA and throughput is 6% higher, and the method remains competitive on mathematical and code-generation tasks. The trainable state need not be a full low-rank matrix; learning scalars on fixed directions can be enough. For memory-tight fine-tuning, this puts adaptation quality, the size of the trainable state, and training efficiency together, rather than comparing final scores alone.
4.05
When a training or inference system moves state between NVMe and GPU memory on the critical path, a direct storage path is not always best for tail latency. Which path is faster changes with the operation, request size, concurrency, and contention, and the winner for bandwidth can disagree with the winner for tail latency. The authors pick a semantically equivalent delivery plan per application phase, decide from measured tail latency, and revisit the choice when conditions change. Integrated with PyTorch FSDP, this cuts step p95 by 18.4% and GPU-idle p95 by 25.3% versus the strongest fixed path. Across several eight-GPU ZeRO workloads, step tail latency also falls relative to always staging through the host or always using a direct path. Data movement should not be judged by average bandwidth alone; what matters is the tail latency a phase actually fears. For large-model training that pages state in and out of storage, always using a direct path or always staging through the host can worsen the tail in some phases.
4.05
Interactive language model services must bound both time to the first token and the gap between later tokens. A decode jitter delays every request in the current batch, and a long reply spans many steps, so even rare overruns cause many service-level misses. Schedulers that already watch those targets often fill each step's prefill budget until the predicted latency just meets the gap target, leaving no room when the prediction is off or the host stalls. The authors calibrate how much prefill each step can carry from the measured tail of its own iteration latency, leave headroom, and serve the shortest waiting prompt first. In aggregated serving, SLO goodput reaches 2.4 to 2.9 times that of the strongest baseline. With prefill and decode disaggregated, many more requests also meet the objective. To keep decoding steady, a step must not schedule prefill right up to the line. For inference services that must keep a streaming gap, leaving headroom from the tail is closer to the real experience than only pushing down the average gap.
4.05
Agent sessions often edit the context, which invalidates cached KV state, so a request must wait to prefill again before it can decode, and latency grows. In a disaggregated serving system, the authors use the invalidated KV to draft speculative tokens, then verify those drafts against the exact KV after prefill and commit only what passes. Speculation is scheduled when decode resources are underused, and it yields when drafts are unlikely to be accepted, so those resources can still serve more promising requests first. On a two-GPU testbed with coding-agent sessions across three model families, all-request speedup over vLLM is 1.05 to 1.19 times at moderate load. Cluster-level simulations driven by real traces also show that the gains can extend to larger deployments. Invalidated cache need not only be thrown away and recomputed; it can be used to speculate first. For agent services whose context is edited often, a cache miss need not wait out prefill with nothing else to do.
3.80
Kernel benchmarks on Ascend often ask whether an agent can write an implementation from scratch, and less often whether an existing implementation can be made faster on input shapes taken from real training and inference. The authors build an optimization benchmark from real workloads, split kernels from large-model training and inference into three difficulty batches, and this report evaluates the more basic batch. About 70% of model-harness combinations have a geometric-mean speedup below 1.2 times. The same model with a different harness can still move from 1.05 times to 1.69 times even when every task is solved correctly. They also score speedup together with the tokens spent in the search, to measure how much was gained inside a budget. Final speedup alone shows neither how much the search cost nor whether the process was sound. For teams training and serving on Ascend, optimization becomes a question of how much can still be squeezed under a budget, with results that must pass a correctness check, not only of writing a correct kernel.