INFRA · 2026-11-03
Infra — Paper Daily, 2026-11-03
Today's systems track is about avoiding repeated computation. LEAP reads how long a request will generate straight from the serving model's prefill hidden states and uses it to order requests. HiState and Flide deal with the recomputation caused by recurrent-layer states that cannot be cut back in hybrid models, and ACCT lets one model reuse the KV cache another model of the same family has already computed. ARK keeps KV cache transfers in disaggregated serving from colliding on the same link. On hardware, a B200 versus B300 benchmark shows the newer card is not faster everywhere, and MemX searches parallel configurations for training on mixed-vendor GPU clusters.
4.35
Running short requests first clearly lowers average latency, but only if you know roughly how long each request will generate, and existing length predictors rank worse when the domain changes while costing GPU time of their own. LEAP finds that the hidden states the serving model computes during prefill already carry an output-length signal, and a single linear layer on one layer's hidden state can read it out, with no separate encoder. The linear layer runs on CPU, in parallel with the rest of prefill on the GPU, so it takes no GPU time. After each request finishes, LEAP uses recursive least squares to correct the layer's weights from that request's true length with a closed-form update, with no gradients and no replay of old data. In single-GPU experiments on ShareGPT traffic, average per-token latency is 15.9% to 40.8% lower than with several existing predictors. A length predictor should be judged by latency after scheduling and by its own cost.
4.30
The recurrent layers of a hybrid model compress the whole prefix into a fixed-size snapshot, and it cannot be cut back to an earlier position. So even when the KV is still cached, if the snapshot at the match point was never saved or has already been evicted, a long stretch must be recomputed from an earlier snapshot, or from scratch. HiState exploits an asymmetry: a cache hit reads every matching KV entry but only one snapshot, and the snapshot's size does not depend on context length. So it keeps many historical snapshots on SSD and saves fast memory for KV, and it puts snapshots and KV into one eviction queue. Each item's score is its last access time plus the extra tokens to recompute if it is removed, divided by the space it frees; the lowest score goes first. On two public coding-agent traces, it reaches p99 TTFT speedups of up to 5.73 times over Marconi on vLLM. Extra snapshots cost no fast memory, and each hit reads only one from slow storage.
4.05
When a large model is trained on a cluster that mixes GPUs from several vendors, MemX makes memory balance across GPUs an explicit search objective, and it lets small-memory GPUs recompute more while large-memory GPUs recompute nothing. That way it finds parallel configurations that neither run out of memory nor leave memory badly uneven across GPUs. For every candidate it estimates iteration time, peak memory, and the variance of memory use across GPUs, combines the three with weights into one score, and the lowest score wins. On a three-vendor cluster, under both a time-based ranking and a memory-balance ranking, it is the only method that puts the real best configuration first; in online search it covers 31,240 configurations in 26.5 seconds. What it buys is memory headroom, not speed: in the real deployment, its pick is 9.3% slower than the fastest configuration. On a mixed cluster memory and compute are out of proportion, so the search objective must include memory.
4.05
The B300 is the upgrade to the B200, with more memory, but it is not faster on every task. The University of Chicago's Research Computing Center measured both on two eight-GPU clusters, and on convolutional network inference the B200 was up to 5.6 times faster than the B300. Per-GPU power shows why: on those tasks the B300 drew only 36% to 53% of its power limit, and the time went to the cost of each individual kernel; on ViT-Large, where the B300 wins, both cards ran close to full power. Separately, on two B300 nodes with identical chips, time to first token was 2.1 times higher on one, only because they ran different vLLM builds. Before buying a new card, measure it on your own workloads and your own software versions.
4.05
Inference engines built for generation handle the decision models used in agent systems badly. These models do not generate long text: they attach many short questions to one shared context, and each question outputs a single category. Existing engines either recompute the shared context for every question, or pack the questions together so they interfere and the output probabilities drift. Flide packs the shared context and all questions into one token sequence computed in a single pass. Indices kept on the GPU record which shared context each question belongs to. Attention layers use them so a question sees only the shared context and itself, and recurrent layers such as Gated DeltaNet restart every question from the state at the end of the shared context. The result matches a separate computation exactly, and queries are answered up to 9.5 times faster than with vLLM and SGLang.
4.05
Vector quantization can compress an LLM to about 2 bits per weight, but the harder it compresses, the more quality it loses. It splits weights into small groups and replaces each group with the closest vector in a codebook, a shared table of representative vectors. Picking the vector is a discrete choice that gradients cannot fix directly, so existing methods only make each layer's quantized output match the original, one layer at a time. TVQ keeps trainable continuous weights behind the quantized ones: in the forward pass each group of continuous weights looks up its closest codebook vector and computes with it, and the backward pass sends the gradient straight to the continuous weights. The lookup is redone after every update, so once the continuous weights move far enough, a different codebook vector is chosen. That lets the quantized model be optimized for any training objective; the paper uses block-wise knowledge distillation. LLaMA2-70B at 2 bits loses only 1% accuracy on common question-answering benchmarks.
4.05
When models of different sizes from the same family are mixed, switching models means prefilling the same context again. ACCT lets the target model use a KV cache another model has already computed: a small model reads a long context, and a large model answers without rereading the text. A small scoring module reads the source model's hidden states and scores each context token for each attention head, trained on how much deleting that token would hurt the answer. How many tokens each head keeps is a learned parameter, the counts across heads stay within one total budget, and each head keeps its highest-scoring tokens. It then uses a trained conversion module to turn the selected tokens' representations into keys and values for the matching target layers and heads. With KV cut by 75%, the target model matches or beats its original quality; on contexts of about 19K tokens, the conversion is 6.5 to 21.2 times faster than the target's own prefill.
4.05
When an LLM is pruned so that 90% of its weights are zero, one-shot pruning degrades sharply, and the strongest method, ELSA, needs optimizer state as large as the model. ETAPS tightens sparsity over several stages, pruning a little more each time, while training weight increments to make up the loss. The increments sit on a fixed, small number of positions; when one is pruned, its slot goes to a surviving weight with the largest accumulated gradient. After each stage it scores weights as the patched weight's absolute value times that position's accumulated gradient, and the next stage prunes the lowest. Across six models it beats ELSA's perplexity at every sparsity tested. Pruning LLaMA2-7B to 90% sparsity takes one H200 instead of ELSA's four.
3.95
When prefill and decode run on separate GPU pools, the KV cache must move from one pool to the other, and each transfer lasts hundreds of milliseconds to seconds, making it a huge flow on the network. Switches pick paths with ECMP by hashing packet headers, so two such flows that hash onto the same uplink share it for seconds while neighboring links sit idle. ARK uses the fact that these flows last long: senders tell each other in advance and choose source ports so that concurrent large flows hash onto different spine switches, with no switch changes and no reordering at the receiver. In packet-level RDMA simulation under moderate load, mean flow completion time drops by up to 27.3% and mean TTFT by up to 12.9%.
SC 2026 Workshop
NVIDIA