INFRA · 2026-11-18
Infra — Paper Daily, 2026-11-18
Today's infrastructure track keeps circling two costs that grow fast. One is a KV cache that lengthens. The other is sparse or multimodal training that gets larger. On the training side, one stack is rebuilt for mixture-of-experts parallelism and precision, another cuts a multimodal pipeline by frozen state and attention work, and a fully open model fleet releases its data and intermediate checkpoints as well. On the serving side, 4-bit latent attention, the effect of evidence order on prefix caches in offline retrieval, memory-aligned cache packing, and snapshots for preempted requests are treated as one problem. An interruption or a recompute should not throw away work already done. The serving numbers on a new rack also show that split memory and 4-bit tensor cores do not turn into throughput by themselves. The kernels and the rack-wide cache have to change with them. Long video generation asks the same kind of question. How should a model keep a layered history of what it just wrote, so the picture can roll past the training horizon without collapsing?
4.85
LEAD STORY · AAi2
AAi2
Mixture-of-experts models add parameters, but the useful compute per token barely changes. If the training stack still moves and stores parameters as a dense model would, throughput falls as the expert pool grows. Olmo-core replaces fully sharded data parallelism with distributed data parallelism, and adds expert parallelism, pipeline parallelism, a distributed optimizer, and activation rematerialization. On B300 eight-GPU nodes, with up to 512 GPUs, a 1.2-trillion-parameter run that uses MXFP8 (a microscaling 8-bit float format) and per-layer rematerialization reaches 858 useful-model TFLOPS per GPU per second. The authors also find that overlapping communication with computation can slow the overlapped kernels by more than it hides. Once experts multiply, the first bottleneck is moving parameters, not the math per step. For teams training very large MoE models, parallelism, precision, and rematerialization have to be designed together, or useful compute is spent on traffic first.
4.75
Multimodal models stitch vision or speech encoders together with a language model, so both the architecture and the data are uneven. Dropping them into a language-model training stack often leaves pipeline stages and context-parallel ranks badly balanced. When an encoder is frozen, a pipeline cut that assumed every module was trainable falls out of balance; tokens of different types also attend over different ranges,. Cornstarch splits the pipeline with frozen modules in mind, and assigns tokens in context parallelism by attention work. On 24 A40 GPUs, end-to-end training throughput rises to 2.26 times that of existing systems on average; after the frozen state is built into the cut, an iteration can be up to 2.46 times faster. The bottleneck in multimodal training is often idle time between unlike modules and unlike data. For teams training vision-language models, the parallel cut has to see frozen weights and attention masks, rather than reuse a text-only plan.
4.60
In long-context reasoning and agent workloads, multi-head latent attention already shrinks the KV cache with a low-rank latent, but the cache still grows with length and concurrency, and attention compute can saturate first. Pushing both storage and attention to FP4 is not a drop-in change: sensitive channels need a residual, and one latent cache must serve the two reduction directions of the attention matrix multiplies. On DeepSeek-R1, the method compresses the MLA KV cache by 3.20 times relative to BF16, and speeds up the attention kernel by up to 2.82 times over FP8. In disaggregated serving, per-GPU throughput rises by up to 2.30 times, while average accuracy across nine benchmarks still matches BF16. To get the benefit of 4-bit MLA, storage, the kernel, and the cache layout have to change together. For teams raising concurrency on long sessions, making this shared latent cache truly 4-bit in compute pays off more than compressing storage alone.
4.60
In offline batch question answering, related questions often retrieve overlapping evidence, but a prefix cache can be reused only when the token prefixes match exactly, so a different evidence order forces a recompute. Existing reordering methods lock groups too early, and the planner itself can spend the prefill it hoped to save. KaleidoKV plans evidence order across the whole batch, keeps a move only when it lowers planned prefill, and admits dependent requests only after their shared prefixes are ready. In all sixteen settings it records the highest throughput and the lowest mean time to first token, at about 1.24 to 1.75 times the throughput of the strongest baseline, including planning time. Evidence order is not a minor tweak; it decides whether the prefix cache can be shared at all. For offline RAG batch inference, planning shared prefixes first and then controlling when requests enter the engine works better than tuning the cache one request at a time.
4.55
Open models that both match strong peers of the same size on reasoning and agent tasks, and also release their data, training process, and intermediate checkpoints, are still rare. K2-Horizon is a fully open fleet that runs from small dense models to a large mixture-of-experts model, and it also releases training logs and infrastructure code. Its 36B-A4B model uses mixture-of-value attention and, with about one-seventh the active parameters, nearly matches a 32B dense model trained on the same data. Uno, a lossless diffusion adapter, speeds up the 7B model by about 2.2 times at batch size 1, without a separate draft model. Open weights are not enough; the data and the training process need to be visible as well. For anyone who wants to reproduce or continue training an open model, a dense and sparse pair trained on the same data and recipe is more useful than a final leaderboard score alone.
4.50
In long-context inference, the KV cache quickly fills memory and slows decoding. Existing vector quantization can push the cache below 4 bits, but when indices are not packed into standard word widths and codebooks grow large, the saved space and speed disappear. Some methods share a global codebook to decode faster, then lose accuracy at lower bit-widths; others keep accuracy, but decode even slower than an uncompressed cache. MeltKV assigns variable bit widths to grouped key channels, packs the indices into standard data types, and shares a codebook across attention heads, dropping the padding that alignment would otherwise add. At comparable perplexity, it uses 48.05% less memory than the MILLION method and cuts decoding latency by 52.91% relative to CQ, a vector quantization method, and it can compress the representation down to 1 bit. Going to fewer bits is not enough; the indices have to be packed to match memory alignment. For teams raising concurrency on long contexts, memory and decode latency have to be read together, not just the compression ratio.
4.50
Reasoning models emit long outputs, and their KV caches can occupy a GPU for minutes. When memory fills, the server preempts requests, and a preempted request may repeatedly lose its state and then write few new tokens after it returns: cache thrashing. In the authors' replays, existing systems leave more than a fifth of requests stalled for over a minute. Stint stores fixed-length KV snapshots in CPU memory, readmits requests by how long they have waited, and guarantees a stretch of steady decoding after they return. On olympiad-math replays with Qwen3-32B on H200 GPUs, no request pauses for more than 50 seconds; P99 time per output token falls by 50.7% relative to recomputation in the same engine, and the replay finishes 10.7% sooner. Preemption cannot be avoided, but a preempted request should not lose the work it already did. For serving long-output requests, keeping a snapshot and giving it an undisturbed decoding turn costs less than computing the same tokens again.
4.50
In post-training quantization for large language models, GPTQ is the usual layer-wise solver, but it optimizes continuous group scales separately from discrete integer assignments, even though the two are coupled. JointQ, in group-wise quantization, recomputes a group's optimal scale in closed form whenever it changes one integer, and periodically re-solves all scales jointly. The search starts from candidates that include the GPTQ solution, so the unregularized layer-wise calibration error is never worse than GPTQ, and the weights stay in a format that can replace it directly. The gain in six-task average zero-shot accuracy grows from 0.0044 in a wide-group 4-bit setting to 0.1559 in 3-bit per-channel quantization, and the method beats vanilla GPTQ in fourteen of sixteen comparisons. At low bit-widths, scales and integers cannot be chosen by a greedy split. For teams pushing models to 3-bit or to larger groups, fixing the solver itself may help more than preprocessing the weights alone.
4.40
IInferact
When DeepSeek-V4.1-Flash moves from a GB300 rack to a new Vera Rubin NVL72, kernels written for the older memory and math layout will not fill Rubin's split memory domains or its FP4 tensor cores. vLLM rebuilds the expert, attention, and communication paths for those two memory domains and wider matrix instructions, adds a specialized tensor-parallel decode path for the lowest latency, and uses a rack-wide KV cache plus routing to hold much higher concurrency. On the SemiAnalysis AgentX benchmark, one Rubin rack reaches 1.7 million tokens per second per GPU while each user still gets 387 tokens per second, which is 58 times the throughput of a GB300 NVL72. These figures come from rack-scale AgentX measurements, not a single-GPU microbenchmark. A new chip's throughput depends on whether the software was rewritten for its memory and instructions. For teams serving models on a new rack, moving old kernels over is not enough; the shared cache and the router have to change with them.
3.93
TencentStreaming long videos means quality has to hold past the training horizon, while the model also has to live with its own generated history and with a cache that keeps growing. MovieRoll first trains a multi-step base with window-wise teacher forcing, then compresses the historical KV cache by layer: shallow layers keep the full history, deeper layers keep only a sink and the recent tokens. Cache memory falls by up to 78%, and generation quality improves. After long-rollout distillation, the base model, trained on videos of up to 15 seconds, stays competitive at 30 seconds, and the distilled model records the highest total scores among the compared methods on longer single-event and multi-event evaluations. Long video generation has to manage the history the model just wrote, not merely train on longer clips. For minute-scale video generation, layered caches and long self-rollout during training decide whether the picture can keep rolling without collapsing.