INFRA · 2026-10-29
Infra — Paper Daily, 2026-10-29
The infra track has 10 papers today, 4 with deep dives. The common thread is making LLM inference fit and run when GPU memory is tight. GateRQ lets the gate decide how many precision levels each lookup reads, shrinking Engram memory tables to about an eighth. WAKMan switches between two prefix-cache eviction policies based on recent traffic. Folding attention biases into keys lets compressed KV caches run directly on vLLM. eBay renumbers graph-index nodes before compressing edges, saving about half of edge storage without slowing queries. Also included are a bf16 rounding problem when exporting ternary models, and GPU memory isolation in CUDA 13.4. Inference serving teams can start with the bias fold and WAKMan, and vector search teams with eBay's graph-index compression.
4.45
When you quantize a language model's n-gram memory tables to low bits, keeping the most frequently accessed rows precise does worse than picking rows at random. The reason is the gate. In conditional memory layers such as DeepSeek's Engram, each fetched vector is scaled by a gate; common function words get small gates, rare entity names get large ones, and quantization error is scaled by the gate. GateRQ stores each row as a few levels of 1-bit residual codes and reads level 1 to compute the gate. It then ranks lookups within the batch by gate-based importance: the top few read to level 3, the next ones to level 2, and the rest stop at level 1. The share at each level is fixed by a preset budget of average bits read per lookup. Without retraining, it cuts DeepSeek-V4.1-Flash's table memory from 23.6 GiB to 3.0 GiB per GPU, and the freed memory clearly raises long-context throughput over FP8 tables.
4.45
Which eviction policy vLLM's prefix cache should use depends on the traffic arriving right now. Agent requests come back within seconds with the same history, so the default LRU fits. Popular system prompts and long chat histories return much later, and only S3FIFO, which protects blocks used repeatedly, keeps them. Across 20 request mixes, each policy wins some. WAKMan replays the traffic of the last 15 minutes on two simulated caches and uses whichever had the higher hit rate for the next window, gaining 5 points of hit rate over LRU alone on average.
4.45
KV cache compression can already cut a long document's cache to a tenth, but the strongest method, Attention Matching, adds a bias to every kept key, and vLLM has no slot for it. Using the queries produced while reading the document, this paper computes one direction per attention head on which those queries project to 1 on average, then moves each key along it by the bias times a fixed factor. Queries whose projection is close to 1 then see a score change about equal to the bias, leaving an ordinary KV cache. Unmodified vLLM on one T4 fits 31 concurrent requests instead of 4.
4.30
In HNSW graph indexes for vector search, the vectors are often already quantized, so the edges in the adjacency lists become the main memory cost. Edges are decoded on the fly while each query walks the graph, so compression cannot slow queries down. This work from eBay and the University of Pisa renumbers nodes offline so that neighbors get nearby IDs, then decodes with the SIMD-friendly integer compressor Stream VByte. It saves up to 52.7% of edge storage with almost no latency cost, and in eBay's production deployment it cut edge storage by up to 44.3% with no loss in query latency.
4.15
When ternary language models are released, the export first casts latent weights to bf16, and this step quietly erases most of what low-learning-rate fine-tuning changed. Ternary models are fine-tuned through higher-precision latent weights, and a ternary code changes only when its latent crosses a threshold at half the mean magnitude of its tensor. At low learning rates, most changed latents end up just past that threshold. Near the threshold, bf16's representable values are spaced further apart than these small changes, so the cast pulls a just-crossed latent back. With the documented export, Falcon-E-1B-Base's GSM8K accuracy falls from 58.79% to 0.78%.
4.15
When both weights and activations are quantized to low bits, activations in the same quantization bin collapse to one value and accuracy drops. ResCom takes back the discarded rounding residual to compensate the layer-output error. On calibration data it measures the residual's covariance and the cross-covariance between output error and residual. It uses the first to whiten the residual's scales and correlations, then takes a singular value decomposition of the whitened cross-covariance. Keeping the few directions that best explain the output error directly gives a low-rank correction matrix, with no gradient training. On Llama-3-8B with 4-bit weights and activations, five-shot GSM8K accuracy rises from 23.96% to 36.16%.
4.15
In agent serving, model inference runs on GPU clusters while tools run on distributed CPUs, and existing schedulers ignore when tool results will be ready and use heuristics to decide which sessions get a GPU and where. For each session and node, TAO computes three values. The first is how much cache it can reuse there, decaying over time. The second is whether its tool will return within the next scheduling window, estimated from tool progress reports or past durations of the same tool. The third is how many tokens it will occupy. Dynamic programming then chooses and places sessions together under capacity limits, maximizing total cache reuse first, then the number of ready sessions, then tokens. Across 12 model, workload and concurrency settings, it cuts mean time to first token by up to 36.3%.
4.00
When training hybrid-attention models on variable-length data, the longest samples need wide sequence-parallel groups spanning dozens of GPUs in full-attention layers, while sparse-attention layers balance within single-node groups. Reusing the wide groups pushes sparse-layer communication across nodes, and switching groups separately would add another data transfer. Full-attention layers already run an all-to-all exchange to swap attention heads; Layerwise Repacking uses that same exchange to also move samples into single-node groups, where hidden states stay until the next full-attention layer. Mean end-to-end training time drops by 7.61%.
4.00
When one base model serves many LoRA adapters, compressing them under a shared rank budget means deciding how many ranks each layer of an adapter keeps and how the budget is split across adapters. FRA needs no labels: on a small set of unlabeled inputs it compares each adapter's output distribution before and after compression using JS divergence. It measures this for truncating each module, builds each adapter's compression curve with dynamic programming, then allocates across the fleet to minimize the worst adapter's loss. On 78 adapters from 3 public fleets, the worst-served adapters clearly do better when budgets are tight.
CIKM 2026
NVIDIA