Most of today's infra papers work through the fine print of serving costs. Two are about speculative decoding: HybridSpec finds that verification gets expensive fast on models mixing GDN and MoE layers, and another warns that speculative decoding can quietly change constrained outputs. FlagRot lets each small piece of a cached key decide whether to be rotated, while GC-MoE and EpiLoRA hide MoE communication and LoRA computation behind the main computation. On the training side, LEAP predicts a job's throughput on every accelerator type before it runs, to pick the hardware.
The same training job can often run on several accelerator types, and which one is cheapest depends on its throughput on each. A job placed on the wrong type still finishes without any error, so the extra cost goes unnoticed.
In vector search, switching to another embedding form or quantizer often means switching libraries. Existing ANN libraries either support one combination or write separate code for each, and no ANN library covers dense, sparse and multi-vector embeddings behind one interface.
Long-context inference is bound by the KV cache, and squeezing keys to very few bits would save a lot of memory if the error stays small. The usual fix rotates keys before quantizing, assuming rotation always helps.
Speculative decoding is said to leave the output distribution unchanged, but that guarantee has a precondition. Components that change token scores, such as grammar constraints, repetition penalties and watermarks, are called logits processors, and at every verification position they must receive the right prefix and state. The authors examined selected releases of Transformers and llama-cpp-python and confirmed two failure modes: a processor reads draft tokens that have not been verified yet, or it advances its internal state and does not roll it back when the draft is rejected. On one trained model with classifier-free guidance, the authors pick one prompt and count whether the first output token is a token fixed in advance. Between ordinary decoding and speculative decoding that drafts from the prompt, that probability differs by at least 60.26 percentage points. When speculative decoding speeds up constrained generation, the output may already have changed without anyone noticing.
A looped language model runs the same block of layers several times for every token, so every per-layer overhead in the serving engine is paid once per loop. The authors measure Ouro and Huginn in an open-source looped-model engine. With a single request, replaying one CUDA graph for all loops instead of submitting one per loop barely changes throughput. The real cost is the 5689 small kernels Ouro runs per token. Fusing those kernels while keeping outputs bitwise identical raises throughput to 1.11 to 1.59 times the baseline. The authors grade each optimization by whether it stays bitwise identical: one approximate fusion was no faster and answered a few fewer GSM8K questions correctly, so they did not adopt it.
In speculative decoding, drafts keep getting cheaper, so most of each round goes to the target model verifying them. How many draft tokens to verify at once is the key choice: verifying more can accept more, but all work after the first rejection is wasted. Once the target mixes Gated DeltaNet (GDN, a linear-attention layer with a recurrent state) and MoE layers, the trade-off changes. On one such Qwen model, raising the verification length from 4 to 16 makes verification 71% slower, while dense models slow down by only a few percent. HybridSpec picks the verification length again on the GPU every round, based on the estimated acceptance of the draft tokens and the measured latency of the target model. On three such models it is about 30% faster than DFlash, a method that drafts in parallel.
Finding the best serving setup for a large model means choosing the engine, how to split it across GPUs, precision and scheduler settings, and today that is done by trial and error, each trial holding one or more nodes for minutes. ROCm InferaSim builds on one observation: choices like batch size, prompt length and GPU count change how much work each GPU does, not which kernels run. So each distinct set of kernels is measured once on a single node, a measurement the authors call an anchor, and analytical formulas carry it to other configurations. On AMD GPUs, measuring the anchors took 9.1 GPU-hours, while measuring the configurations they cover directly took 204. The predicted time per output token has a median error of 8.1%. Screening candidates with predictions before tuning saves most of the trial cost.
Once a base model moves to FP8 or lower precision, its main matrix multiplications get cheaper, but LoRA adapters become a growing share of every decoding step. In multi-tenant serving, one base model carries many tenants' adapters, and they still run as separate small kernels beside the main matrix multiplication. The authors notice that the main matrix multiplication needs the down-projection only when it writes out its result. EpiLoRA therefore launches the down-projection first, lets the main matrix multiplication read its weights meanwhile, and, when the result is written, computes every tenant's up-projection inside the same kernel and adds it. On Hopper GPUs, single-adapter decoding takes up to 24% less time per token than vLLM's fastest configuration. The same low-precision weights also support LoRA reinforcement learning directly, so a 27B model trains on one 48GB GPU.
In MoE decoding each GPU holds only a few to a few hundred tokens, and the grouped matrix multiplications are bandwidth-bound, so they never use all the SMs anyway. Yet current fused operators still put token dispatch and the matrix multiplications on the same SMs, where they fight over registers and shared memory. GC-MoE uses CUDA Green Contexts to split the GPU at the driver level into a small communication partition and a large compute partition, each running and tuning its own kernel, and computation starts as soon as the first tokens arrive. On H20 GPUs, decoding a DeepSeek MoE model, it is 18.8% faster on average than the existing DeepGEMM fused operator. For MoE decoding, giving communication and computation separate SMs pays off better than making them share every SM.
NVIDIAIn distributed training, when one GPU hits a hardware fault, the first errors often come from NCCL on other nodes. The exception in the log shows where the job stopped, not where the failure began. NVIDIA has open-sourced NVIDIA Attribution, the failure-attribution system it has used internally for years. It gathers a job's application logs, system logs, GPU driver messages, scheduler state and metrics. It then applies fixed rules to the timing and links between events: if a GPU hardware error on one node is followed by NCCL errors on other nodes, the former is the root cause and the latter are symptoms. Finally it reports the root cause, the affected nodes and the evidence. In the post's example, three NCCL errors are traced back to one NVLink signal fault on a single node. NVIDIA says it analyzes millions of jobs each week. Alerts and repairs can then go straight to the faulty machine instead of being misled by downstream errors.