INFRA · 2026-10-27
Infra — Paper Daily, 2026-10-27
The Infra track has 10 papers today, 4 with deep dives. A ByteDance Seed paper finds that RL training collapse can be avoided by aligning rounding locations in three places, without changing precision. C2KV compresses retrieved documents' KV caches into small blocks that can be stitched freely. NgramRL speeds up generation in RL by drafting from a prompt's own recent answers. Qdrant 1.20 computes BM25 directly on full-text indexes. Teams running LLM training and inference can start with the first two.
4.35
LLM reinforcement learning usually generates answers with one inference engine and trains with another framework. Even with identical weights and BF16 on both sides, the numbers differ slightly, so an update meant to be on-policy quietly becomes off-policy and training can suddenly collapse. This paper from ByteDance Seed and Peking University finds that the two systems need not be aligned wholesale. It is enough to round at the same place on both sides in three spots, RoPE, SwiGLU and residual RMSNorm, keeping those intermediates in FP32 until their BF16 outputs. With all three aligned, four Qwen-family models no longer collapse, and aligning RoPE alone also suffices. The inference engine, matmul and attention kernels, and the training objective stay unchanged.
4.35
Retrieval-augmented generation stitches several retrieved documents into the context, and recomputing their KV caches from scratch every time is expensive, so some methods store each document's KV cache and concatenate them on use. But long documents have huge KV caches, so storing and loading them becomes the bottleneck, and naively compressing them before concatenation hurts accuracy badly. This paper from Shanghai Jiao Tong University and Alibaba proposes C2KV: a light extractor beside the frozen model reads each document followed by a few learnable compression tokens and keeps only those tokens' KV as the document's cache. Caches are stored without positions, which are re-applied at concatenation time, so they combine like building blocks. On long contexts, inference is up to 17 times faster with generation quality largely preserved.
4.00
In LLM reinforcement learning post-training, generating answers often takes over 70% of training time. This paper notices that the answers a prompt received in recent training rounds closely resemble what it will produce now, so they make good drafts. NgramRL builds an n-gram draft table for each prompt from only its own recent answers. During generation, the last few tokens of the current answer are looked up in the table, what followed them becomes the draft, and the large model verifies several tokens at once. It needs no extra draft model and no GPU memory, and the output distribution stays the same. On reasoning and structured-generation tasks with a warm cache, each forward pass of the large model confirms 3.6 to 7.8 tokens, up to twice as many as SuffixDecoding, a similar method that drafts from a suffix tree over global past answers.
4.00
When LLM reinforcement learning runs generation and training on separate machines, finished trajectories often wait until a whole batch is complete before training, so the trained model drifts further from the version that generated them. Trajectory data is large, and mismatches between how it is moved and how training reads it cause repeated transfers and idle waiting. KnitTrans admits finished trajectories into early mini-batches. A full batch must mix data sources at set ratios, so each admission is checked to leave enough room to fill every source's share. It moves data on demand in the order training consumes it, Several trajectories from the same prompt share its images and other multimodal inputs, so each input is encoded once per mini-batch and reused across them. The version gap drops by up to 93%, and on multimodal workloads transfer plus training is up to 5.8 times faster.
3.75
Training large models on huge GPU clusters is increasingly limited by communication. Frameworks like ZeRO++ compress traffic with a fixed quantization bit width: low bits give high throughput but leave a constant noise floor that hampers late-stage convergence, while high bits keep accuracy but slow training down. This paper hypothesizes that weight-quantization noise acts much like stochastic gradient noise. It proves that, just as the learning rate must decay for convergence, quantization precision must rise over training to keep optimization stable while maximizing throughput. The resulting Aquavit quantizes communication with few bits early in training and switches to more bits later.
3.70
Serving one large model under different inference budgets is costly if each budget needs its own trained and maintained model. Accordion is a mixture-of-experts model that serves five inference budgets with one set of weights. A run of middle layers forms a loop window; each budget runs the first few layers of that window twice in a row, higher budgets repeat more layers, and the lowest budget is the unlooped model. Pretraining samples one budget per micro-batch, with no extra consistency objective. The middle budget gets most of the top budget's gain at half the extra compute. Deeper inference raises the average across 37 tasks by 3.01 points, and the middle budget captures 89% of that.
3.70
Memory demand from data analytics and LLM applications is growing fast, so data centers are adopting tiered memory that puts cold data on cheaper, slower memory. But deciding which data goes to which tier by access hotness alone is often imprecise. This paper uses the Statistical Profiling Extension (SPE) on ARM processors for high-resolution memory profiling of eight memory-intensive cloud and LLM workloads. It splits memory into contiguous regions and builds each region's access-latency distribution: regions with long latencies deserve the fast tier. It also measures each region's longest stretch without accesses; the longer it sits idle, the better it suits the slow tier. It also evaluates how well cloud-oriented cache hierarchies work.
3.35
GPUs run analytical queries fast only when the data fits in GPU memory, which is shared among jobs and whose capacity per dollar has barely grown across generations. Compression saves memory and bandwidth, but existing approaches either tie decompression to one execution model or write decompressed columns back to GPU memory before the query runs. This paper makes decompression a standalone operator in the compiled GPU query engine Proteus: tiles are decompressed in on-chip shared memory and fused with the query in one GPU kernel, so decompressed data never goes back to GPU memory. Against HeavyDB and Sirius, query times are comparable while resident memory drops by up to 4 times and bandwidth use by up to 6 times.
KDD 2026
Qdrant
NVIDIA