Paper Daily · Infra (English)

Folding Attention Biases into Keys Makes Compacted KV Caches Run on Standard Kernels

Thu, 29 Oct 2026 08:00:00 +0800

KV cache compression can already cut a long document's cache to a tenth, but the strongest method, Attention Matching, adds a bias to every kept key, and vLLM has no slot for it. Using the queries produced while reading the document, this paper computes one direction per attention head on which those queries project to 1 on average, then moves each key along it by the bias times a fixed factor. Q…

Infra · Score 4.40 · Deep dive · 29 min · Utkarsh Kanwat · Source

WAKMan: Workload Aware KVCache Manager

Thu, 29 Oct 2026 08:00:00 +0800

Which eviction policy vLLM's prefix cache should use depends on the traffic arriving right now. Agent requests come back within seconds with the same history, so the default LRU fits. Popular system prompts and long chat histories return much later, and only S3FIFO, which protects blocks used repeatedly, keeps them. Across 20 request mixes, each policy wins some. WAKMan replays the traffic of the…

Infra · Score 4.40 · Deep dive · 26 min · Suhas Kumar Morisetty, Vaibhavi Lokegaonkar, Sachin Mathew Varghese, Wesley Harris · Source

Let the Gate Decide: Per-Lookup Precision for Quantized Conditional Memory in LLMs

Thu, 29 Oct 2026 08:00:00 +0800

When you quantize a language model's n-gram memory tables to low bits, keeping the most frequently accessed rows precise does worse than picking rows at random. The reason is the gate. In conditional memory layers such as DeepSeek's Engram, each fetched vector is scaled by a gate; common function words get small gates, rare entity names get large ones, and quantization error is scaled by the gate…

Infra · Score 4.40 · Deep dive · 35 min · Tianqi Chen, Xunchao Li, Linchuan Xie, Guanghua Yu, Jianchen Zhu, Peisong Wang, Jian Cheng · Source

Lossless Compression of Proximity Graph Indexes for Efficient Approximate Nearest Neighbors Search

Thu, 29 Oct 2026 08:00:00 +0800

In HNSW graph indexes for vector search, the vectors are often already quantized, so the edges in the adjacency lists become the main memory cost. Edges are decoded on the fly while each query walks the graph, so compression cannot slow queries down. This work from eBay and the University of Pisa renumbers nodes offline so that neighbors get nearby IDs, then decodes with the SIMD-friendly integer…

Infra · Score 4.30 · Deep dive · 22 min · Francesco Benocci, Leonard Dahlmann, Francesco Gargiulo, Tobias Heuer, Silvio Martinico, Franco Mar… · Source

TAO: Tool-Aware Joint Scheduling for LLM-Based Agentic Serving

Thu, 29 Oct 2026 08:00:00 +0800

In agent serving, model inference runs on GPU clusters while tools run on distributed CPUs, and existing schedulers ignore when tool results will be ready and use heuristics to decide which sessions get a GPU and where. For each session and node, TAO computes three values. The first is how much cache it can reuse there, decaying over time. The second is whether its tool will return within the nex…

Infra · Score 4.10 · Shuhuai Li, Shengdong Ye, Dongdong Ge, Yinyu Ye, Jianghao Lin · Source

ResCom: Activation Residual Compensation for Large Language Model Quantization

Thu, 29 Oct 2026 08:00:00 +0800

When both weights and activations are quantized to low bits, activations in the same quantization bin collapse to one value and accuracy drops. ResCom takes back the discarded rounding residual to compensate the layer-output error. On calibration data it measures the residual's covariance and the cross-covariance between output error and residual. It uses the first to whiten the residual's scales…

Infra · Score 4.10 · Ning Yang, Bohan Cui, Zhuowen Liu, Yuting Wang · Source

Lost in the bf16 Cast: Exporting Ternary Language Models Can Revert Most Low-Learning-Rate Code Changes

Thu, 29 Oct 2026 08:00:00 +0800

When ternary language models are released, the export first casts latent weights to bf16, and this step quietly erases most of what low-learning-rate fine-tuning changed. Ternary models are fine-tuned through higher-precision latent weights, and a ternary code changes only when its latent crosses a threshold at half the mean magnitude of its tensor. At low learning rates, most changed latents end…

Infra · Score 4.10 · Avichal Sahai, Nishant Raj, Animesh Srivastava · Source

Memory-Enforced GPU Sharing in Kubernetes with NVIDIA CUDA 13.4

Thu, 29 Oct 2026 08:00:00 +0800

Many AI workloads need only a fraction of a GPU, yet existing sharing modes fall short: time-slicing does not limit memory, and MPS, which runs processes concurrently, has no application-transparent memory cap. This NVIDIA blog post introduces the memory boundary added in CUDA 13.4: a workload gets exactly the memory it asks for and allocations beyond it fail, or it gets a guaranteed minimum with…

Infra · Score 4.00 · Source

LoRA on a Budget: Function-Aware Post-Hoc Compression for Adapter Fleets

Thu, 29 Oct 2026 08:00:00 +0800

When one base model serves many LoRA adapters, compressing them under a shared rank budget means deciding how many ranks each layer of an adapter keeps and how the budget is split across adapters. FRA needs no labels: on a small set of unlabeled inputs it compares each adapter's output distribution before and after compression using JS divergence. It measures this for truncating each module, buil…

Infra · Score 4.00 · Shenghui Li, Thiemo voigt · Source

Layerwise Repacking: Variable-Length Hybrid-Attention Training without an Extra Redistribution Pass

Thu, 29 Oct 2026 08:00:00 +0800

When training hybrid-attention models on variable-length data, the longest samples need wide sequence-parallel groups spanning dozens of GPUs in full-attention layers, while sparse-attention layers balance within single-node groups. Reusing the wide groups pushes sparse-layer communication across nodes, and switching groups separately would add another data transfer. Full-attention layers already…

Infra · Score 4.00 · Jiahao Zhang, Junwei Feng, Dongliang Huang, Xiaoling Zhou, Yiqiao Liao, Chengru Song · Source

HiLO: Towards Product-Level Graceful Degradation with a Quality-Aware, Load-Aware Control Framework

Wed, 28 Oct 2026 08:00:00 +0800

During traffic spikes, each service degrades on its own, so every service survives while the product's quality can collapse. Microsoft's HiLO moves degradation from the service level to the product level. It predefines a few validated product states. Each state sets a level for every service, such as dropping some features, reducing fan-out or using a cheaper model. At runtime services report loa…

Infra · Score 4.75 · Deep dive · 35 min · Knut Magne Risvik, Zhuowei Li, Nirupam Purushothama, Hunter Su, Greg Dunham, Yan Reznikov · Source

Failure-Resilient and Verifiable Distributed Approximate Nearest-Neighbor Retrieval

Wed, 28 Oct 2026 08:00:00 +0800

Replicas in a distributed vector database keep results available but do not make them trustworthy. A compromised replica can return validly signed but tampered nearest neighbors. TrustANN adds middleware above the router. For each shard it greedily picks a set of replica replies that meets a set minimum size, scores high on pairwise similarity (candidate overlap plus rank agreement), and spans a…

Infra · Score 4.40 · Deep dive · 30 min · Abdullah Al-Mamun · Source

Cross-Modal Data Co-Locality for Heterogeneous Storage Systems

Wed, 28 Oct 2026 08:00:00 +0800

A multimodal query first searches text vectors and then fetches the matching image vectors. Both nearest-neighbor searches are fast. The slow part is moving data when the two vectors sit in different storage tiers. CoLoc changes neither the vectors nor the index. It only decides which tier holds what, keeping vectors that are queried together in the same tier. It optimizes where data lives, not w…

Infra · Score 4.40 · Deep dive · 31 min · Abdullah Al-Mamun · Source

BellowsKV: One Latent for Many Layers, Expanded on Demand

Wed, 28 Oct 2026 08:00:00 +0800

In long-context inference, the largest memory cost is often the per-layer KV cache, the keys and values attention stores for every past token. BellowsKV observes that caches of adjacent layers differ, but the hidden states that produce them are similar. So it shares the input, not the cache. For each token, it concatenates the hidden states of n adjacent layers and multiplies them by a compressio…

Infra · Score 4.35 · Deep dive · 27 min · Jiamu Zhang, Hanjie Chen · Source

ZipFusion: Fusing Lossless KV Decompression with Attention for Efficient LLM Decoding

Wed, 28 Oct 2026 08:00:00 +0800

Most of the gain from lossless KV cache compression is eaten by decompression. Earlier methods restore the full cache on the GPU, so attention still reads uncompressed data. In ZipFusion, attention reads compressed data directly The format compresses only the exponent bits of 16-bit floats, which cluster in a narrow range, and keeps the fraction bits as is. Attention rebuilds the exponents in reg…

Infra · Score 4.25 · Deep dive · 27 min · Yuetao Chen, Peng Wang, Hong Xu · Source

Governing Enterprise AI Traffic at Scale: Design and Operation of a Policy-Driven, Multi-Vendor LLM Gateway on Envoy

Wed, 28 Oct 2026 08:00:00 +0800

When a company uses models from several vendors and every app integrates them directly, keys spread, switching models needs code changes, and spend goes unmanaged. PayPal's Cosmos gateway is built on the Envoy proxy. Slow decisions such as auth, budgets and policy run outside the proxy. Format translation and token counting run inside it, so no streamed chunk waits on an external call. The hard p…

Infra · Score 4.20 · Deep dive · 24 min · Mithun Sasidharan, Srinivasan Manoharan, Rupesh Thakur, Abhinav TK, Ashwin Hegde, Anand Gautam · Source

Prefetch-Steered Routing: Deterministic Cache-Aware Expert Selection for Offloaded Mixture-of-Experts Decoding

Wed, 28 Oct 2026 08:00:00 +0800

When a mixture-of-experts (MoE) model does not fit in GPU memory, single-request decoding is dominated by copying expert weights from host memory. Routing biased toward resident experts cuts copies. Combined with asynchronous prefetching, though, outputs depend on transfer timing, so offline evaluation no longer matches the deployed model. PSR routes on a deterministic logical cache. At each laye…

Infra · Score 4.10 · Siming Huang · Source

A new paradigm for inference acceleration: inside Volcano Engine EIC, a high-performance distributed KV cache

Wed, 28 Oct 2026 08:00:00 +0800

Many inference instances recomputing the same long prefix is a common waste in serving clusters. Volcano Engine's EIC (Elastic Instant Cache) is a distributed KV cache service for LLM inference. Its role is to keep the key-value cache computed during inference in a separate distributed cache layer. Multiple inference instances can then share and reuse it instead of recomputing long prefixes. Movi…

Infra · Score 3.80 · Source

3 Ways to Boost MoE Inference on NVIDIA DGX Spark

Wed, 28 Oct 2026 08:00:00 +0800

When decoding one request at a time, the slow part is often reading weights, not compute. NVIDIA optimized the mixture-of-experts model Ling-3.0-flash on DGX Spark. Profiling showed weight reads were the bottleneck, mainly in expert computation and the output-layer projection. So all three techniques read fewer weights or produce more tokens per step. They swap in a faster expert kernel, quantize…

Infra · Score 3.65 · Source

How NVIDIA Minerva Turns Telemetry Chaos Into Peak GPU Yield

Wed, 28 Oct 2026 08:00:00 +0800

In large-scale training, one GPU running a little hot or one network port quietly degrading can slow an entire synchronous job. Such gray failures often trigger no fixed threshold. NVIDIA's Minerva merges telemetry from GPUs, network, storage and jobs into a live knowledge graph. Machine learning models flag devices that behave differently from their peers, The graph then shows which alerts share…

Infra · Score 3.35 · Source

C 2 KV: Compressed and Composable KV Cache Reuse for Efficient LLM Inference

Tue, 27 Oct 2026 08:00:00 +0800

Retrieval-augmented generation stitches several retrieved documents into the context, and recomputing their KV caches from scratch every time is expensive, so some methods store each document's KV cache and concatenate them on use. But long documents have huge KV caches, so storing and loading them becomes the bottleneck, and naively compressing them before concatenation hurts accuracy badly. Thi…

Infra · Score 4.30 · Deep dive · 19 min · Chuheng Du, Junyi Chen, Hanlin Tang, Kan Liu, Tao Lan · Source

Where Rounding Happens Matters: Rounding-Location Alignment Stabilizes LLM Reinforcement Learning

Tue, 27 Oct 2026 08:00:00 +0800

LLM reinforcement learning usually generates answers with one inference engine and trains with another framework. Even with identical weights and BF16 on both sides, the numbers differ slightly, so an update meant to be on-policy quietly becomes off-policy and training can suddenly collapse. This paper from ByteDance Seed and Peking University finds that the two systems need not be aligned wholes…

Infra · Score 4.25 · Deep dive · 19 min · Yijing Ke, Tianle Cai, Ruosong Wang · Source

Qdrant 1.20 - Full-Text Search Improvements

Tue, 27 Oct 2026 08:00:00 +0800

Ranking keyword search results with BM25 in Qdrant used to mean turning every document and query into a sparse vector and tracking the average document length yourself, which drifts as data changes. Qdrant 1.20 computes BM25 directly on the full-text index: all statistics come from the data automatically, with no vectors to generate. Parameters can be tuned per query, and it combines with vector…

Infra · Score 4.20 · Deep dive · 19 min · Source

How NVIDIA Mission Control 2.5 Brings Autonomous Recovery to Kubernetes

Tue, 27 Oct 2026 08:00:00 +0800

Large training jobs that run across hundreds of nodes for days or weeks will eventually hit hardware faults, software errors and network timeouts, and manual recovery wastes a lot of compute. NVIDIA's cluster management software Mission Control extends its automatic recovery engine from Slurm to Kubernetes training jobs in version 2.5. It first classifies the failure from the job's context, then…

Infra · Score 3.90 · Source

NgramRL: Accelerating RL Post-Training for Structured Generation via Rollout-Aware N-gram Speculative Decoding

Tue, 27 Oct 2026 08:00:00 +0800

In LLM reinforcement learning post-training, generating answers often takes over 70% of training time. This paper notices that the answers a prompt received in recent training rounds closely resemble what it will produce now, so they make good drafts. NgramRL builds an n-gram draft table for each prompt from only its own recent answers. During generation, the last few tokens of the current answer…

Infra · Score 3.90 · Deep dive · 19 min · Nilesh Prasad Pandey, Jangseon Park, Jason Kong, Onat Gungor, Tajana Rosing · Source

KnitTrans: Coordinated Trajectory Transfer for Effective and Efficient LLM Reinforcement Learning

Tue, 27 Oct 2026 08:00:00 +0800

When LLM reinforcement learning runs generation and training on separate machines, finished trajectories often wait until a whole batch is complete before training, so the trained model drifts further from the version that generated them. Trajectory data is large, and mismatches between how it is moved and how training reads it cause repeated transfers and idle waiting. KnitTrans admits finished…

Infra · Score 3.90 · Bangjun Xiao, Chiheng Lou, Xiaodong Ji, Xiangwei Deng, Yikai Zhao, Gang Huang, Hailin Zhang · Source

Aquavit: Ascending Quantization for Communication-Efficient Vast-Scale Distributed Training

Tue, 27 Oct 2026 08:00:00 +0800

Training large models on huge GPU clusters is increasingly limited by communication. Frameworks like ZeRO++ compress traffic with a fixed quantization bit width: low bits give high throughput but leave a constant noise floor that hampers late-stage convergence, while high bits keep accuracy but slow training down. This paper hypothesizes that weight-quantization noise acts much like stochastic gr…

Infra · Score 3.75 · Hong Huang, Jiaxun Ye, Jinhai Yang, Wenjiao Feng, Zonghang Li · Source

Beyond Hotness: High-Resolution Memory Characterization of Cloud and LLM Workloads for Tiered Memory with ARM SPE

Tue, 27 Oct 2026 08:00:00 +0800

Memory demand from data analytics and LLM applications is growing fast, so data centers are adopting tiered memory that puts cold data on cheaper, slower memory. But deciding which data goes to which tier by access hotness alone is often imprecise. This paper uses the Statistical Profiling Extension (SPE) on ARM processors for high-resolution memory profiling of eight memory-intensive cloud and L…

Infra · Score 3.70 · Jie Ren, Ivy Peng · Source

Accordion: Elastic-Depth Inference via Fixed-Prefix Layer Looping

Tue, 27 Oct 2026 08:00:00 +0800

Serving one large model under different inference budgets is costly if each budget needs its own trained and maintained model. Accordion is a mixture-of-experts model that serves five inference budgets with one set of weights. A run of middle layers forms a loop window; each budget runs the first few layers of that window twice in a row, higher budgets repeat more layers, and the lowest budget is…

Infra · Score 3.65 · Yudong Zhang, Wei Ding, Kaili Zheng, Yuchen Ding, Zhenyu Zhang · Source

Tile-Based Decompression in Compiled GPU Engines

Tue, 27 Oct 2026 08:00:00 +0800

GPUs run analytical queries fast only when the data fits in GPU memory, which is shared among jobs and whose capacity per dollar has barely grown across generations. Compression saves memory and bandwidth, but existing approaches either tie decompression to one execution model or write decompressed columns back to GPU memory before the query runs. This paper makes decompression a standalone opera…

Infra · Score 3.35 · Antonio Boffa, Ananya Gupta, Hamish Nicholson, Anastasia Ailamaki · Source

KVStream: Adaptive Parallel Storage Management for KV Caching in Large-Scale LLM Serving

Mon, 26 Oct 2026 08:00:00 +0800

Long-context inference keeps reusing the same prefixes: system prompts, whole documents, code repositories and multi-turn session histories. Their KV cache no longer fits in GPU memory or host memory, so it has to live on disk and be read back on the next hit, which puts disk reads on the critical path of every inference step. Existing systems read only from the node-local NVMe, while the paralle…

Infra · Score 4.30 · Deep dive · 40 min · Avinash Maurya, Jaiaid Mobin, Jie Ye, M. Mustafa Rafique, Franck Cappello, Bogdan Nicolae · Source

Advancing Generative Recommendation Inference: our collaboration with NVIDIA on TensorRT LLM

Mon, 26 Oct 2026 08:00:00 +0800

By moving generative podcast recommendation off two separate inference engines and onto TensorRT-LLM together with NVIDIA, Spotify cut beam search cost by up to 54% and scoring cost by up to 7% on Qwen3-1.7B. Spotify recommends with semantic IDs, and a single online request needs three kinds of inference: beam search to generate candidates, scoring a given batch of candidates, and inserting the u…

Infra · Score 4.20 · Deep dive · 29 min · Source

BiLoCo: Binary Low-Rank Corrections for Post-Training Quantization

Mon, 26 Oct 2026 08:00:00 +0800

Quantizing a trained LLM straight to FP4 lets Blackwell GPUs use native FP4 matrix multiplies, but accuracy drops noticeably. The common fix, SVDQuant, stores an extra BF16 low-rank correction, which costs storage and competes with the main computation for the same tensor cores. This paper from the Massachusetts Institute of Technology proposes BiLoCo (binary low-rank correction). Every vector in…

Infra · Score 4.10 · Deep dive · 24 min · David Jin, Beshr IslamBouli, Tarushii Goel, Han Guo, Yoon Kim · Source

HyScale: Scaling Long-Context Training for Learned Sparse Attention

Mon, 26 Oct 2026 08:00:00 +0800

Learned sparse attention lets each query read only the k most relevant earlier tokens and is a popular architecture for million-token contexts. But the selection step that picks those k tokens scores every earlier token, and in training it becomes the bottleneck, taking nearly half of each step on long sequences in Megatron-LM. HyScale's key observation is that selection outputs only a set of ind…

Infra · Score 4.10 · Yangsheng Deng, Xinjing Huang, Yulong Wang, Yunzhuo Liu, Jun Hu, Shuyong Tang, Junchi Zeng, Mingji… · Source

GS-MoE: Training-Free MoE Compression via Groupwise Subspace Sharing and Pruning

Mon, 26 Oct 2026 08:00:00 +0800

Mixture-of-experts (MoE) models activate only a few experts per token, so compute is cheap, but every expert's parameters must stay in GPU memory, and memory is usually the first thing that blocks deployment. This paper proposes GS-MoE, which needs no retraining and compresses experts along two axes at once. First, similar experts are grouped, and each group shares one input and one output projec…

Infra · Score 4.10 · Deep dive · 26 min · Carson Cary, Yukai Jin, Yihao Huang, Zhankai Ye, Fangyi Li, Ting Hua, Xin Liu, Shangqian Gao · Source

Nested Sparsity for Scalable Pretraining and Continual Learning

Mon, 26 Oct 2026 08:00:00 +0800

Mixture-of-experts (MoE) models make only the feed-forward layer sparse, while attention still shares one set of parameters across all tokens and sees the full context. This paper proposes Nestor, which also splits attention into several independently parameterized attention modules called columns. A router picks a few columns per token, and each column computes attention only among tokens routed…

Infra · Score 3.70 · Tianzhe Chu, Druv Pai, Yi Ma · Source

Efficient Omni-modal Large Language Model Inference via Per-Query Modality Selection

Mon, 26 Oct 2026 08:00:00 +0800

Omni-modal LLMs can read text, hear audio and watch video together, but video is expensive. On the WorldSense benchmark, text and audio take about three thousand input tokens per question, and adding video raises that to about forty thousand. Many questions do not need the video at all. This paper leaves the model unchanged and decides, per question, which modalities to feed it, learning that cho…

Infra · Score 3.65 · Shuqi Liu, Yuzhou Cao, Luke Ong · Source

Tensormesh Launches Tensormesh Platform to Cut LLM Inference Costs by Reusing KV Cache at Enterprise Scale

Mon, 26 Oct 2026 08:00:00 +0800

LLM inference keeps recomputing the same context, wasting GPU time and delaying the first token. Tensormesh, the team behind the open-source KV cache project LMCache, has launched an enterprise product, Tensormesh Platform, that runs KV cache as a managed data layer. It caches automatically across GPU memory, host memory and storage, reuses both prompt prefixes and, through CacheBlend, chunks in…

Infra · Score 3.45 · Source

Kolibri: A Sovereign European Model on the Pareto Frontier

Mon, 26 Oct 2026 08:00:00 +0800

Regulated European sectors such as public administration, industry and aerospace need open models they can control and that handle German well. Kolibri is an English-German mixture-of-experts model with 78.1 billion total parameters and about 3.5 billion active per token, released as open weights. The team controls the whole process, from data and architecture to training infrastructure, post-tra…

Infra · Score 3.35 · Aleph Alpha, :, Moritz Althaus, Artur Baranowski, Michael Barlow, Fabien C. Y. Benureau, Felix Berk… · Source

人工智能平台 PAI：模型蒸馏（ModelDistill）

Mon, 26 Oct 2026 08:00:00 +0800

If you want to move a large model's ability into a cheaper small model without building the pipeline yourself, ModelDistill, the model distillation service on Alibaba Cloud's PAI platform, offers it as a no-code managed service. It uses black-box distillation: the teacher model generates high-quality training data, and the student model is trained on it with SFT, using LoRA, QLoRA or full fine-tu…

Infra · Score 2.90 · Source

ATTUNE: Agentic Throughput Tuning for LLM Training