INFRA · 2026-10-26
Infra — Paper Daily, 2026-10-26
The infrastructure track has 10 papers today, centered on three themes. First, KV cache storage during inference: KVStream reads from local disk and shared storage together, and Tensormesh launches an enterprise cache platform. Second, model compression: BiLoCo repairs FP4 accuracy with binary low-rank corrections, and GS-MoE compresses mixture-of-experts models without training. Third, lower training and serving cost, including HyScale for sparse-attention training and Spotify unifying three kinds of generative recommendation inference in TensorRT-LLM.
4.35
Long-context inference keeps reusing the same prefixes: system prompts, whole documents, code repositories and multi-turn session histories. Their KV cache no longer fits in GPU memory or host memory, so it has to live on disk and be read back on the next hit, which puts disk reads on the critical path of every inference step. Existing systems read only from the node-local NVMe, while the parallel file system that every cluster already mounts sits almost idle. This paper builds KVStream: every KV chunk is written to both the local disk and the shared storage. At read time both tiers read at once, and whichever finishes first claims the next batch. Plugged into LMCache, it reaches up to 1.49x LMCache's offline throughput and up to 6.1x lower mean TTFT under bursty traffic.
4.35
Spotify
NVIDIABy moving generative podcast recommendation off two separate inference engines and onto TensorRT-LLM together with NVIDIA, Spotify cut beam search cost by up to 54% and scoring cost by up to 7% on Qwen3-1.7B. Spotify recommends with semantic IDs, and a single online request needs three kinds of inference: beam search to generate candidates, scoring a given batch of candidates, and inserting the user vector into the prompt as one extra token. These used to be split across two inference engines, vLLM and SGLang, and vLLM's beam search could not saturate the GPU, so Spotify fell back to lower-tier cards. The two companies added the missing features to TensorRT-LLM and unified all three in one engine.
4.15
Quantizing a trained LLM straight to FP4 lets Blackwell GPUs use native FP4 matrix multiplies, but accuracy drops noticeably. The common fix, SVDQuant, stores an extra BF16 low-rank correction, which costs storage and competes with the main computation for the same tensor cores. This paper from the Massachusetts Institute of Technology proposes BiLoCo (binary low-rank correction). Every vector in the correction is replaced by entries of plus or minus one, so the correction needs only additions and subtractions and runs on idle CUDA cores alongside the matrix multiply. The same storage holds 16 times the rank with equal or better perplexity, and at batch size 1 the whole kernel is only 6% to 11% slower than plain FP4.
4.15
Mixture-of-experts (MoE) models activate only a few experts per token, so compute is cheap, but every expert's parameters must stay in GPU memory, and memory is usually the first thing that blocks deployment. This paper proposes GS-MoE, which needs no retraining and compresses experts along two axes at once. First, similar experts are grouped, and each group shares one input and one output projection while each expert keeps only a small private matrix. Second, intermediate neurons inside each expert that other neurons can stand in for are removed. On DeepSeek-MoE-16B with 60% of parameters removed, average accuracy on reasoning tasks stays at about 90% of the original model, versus 73% for the baseline D2-MoE.
4.15
Learned sparse attention lets each query read only the k most relevant earlier tokens and is a popular architecture for million-token contexts. But the selection step that picks those k tokens scores every earlier token, and in training it becomes the bottleneck, taking nearly half of each step on long sequences in Megatron-LM. HyScale's key observation is that selection outputs only a set of indices, so there is no need to store a full score table or to select again during recomputation. It finds the exact top k on the fly and reuses the forward-pass indices during recomputation. Training million-token sequences on 512 H20 GPUs, it runs 2.07x faster end to end than Megatron-LM.
3.70
Mixture-of-experts (MoE) models make only the feed-forward layer sparse, while attention still shares one set of parameters across all tokens and sees the full context. This paper proposes Nestor, which also splits attention into several independently parameterized attention modules called columns. A router picks a few columns per token, and each column computes attention only among tokens routed to it, with its own set of experts. Scaling-law experiments show that a standard MoE at equal sparsity needs 13% to 29% more training compute to reach Nestor's held-out loss, and the gap does not shrink as models grow. In continual training on new domains, it learns about as much as an MoE while forgetting less.
3.70
Omni-modal LLMs can read text, hear audio and watch video together, but video is expensive. On the WorldSense benchmark, text and audio take about three thousand input tokens per question, and adding video raises that to about forty thousand. Many questions do not need the video at all. This paper leaves the model unchanged and decides, per question, which modalities to feed it, learning that choice as a trade-off between accuracy and cost. The selector is a single linear layer that reads only the BGE-M3 embedding of the question text and decides whether to add video; its training objective is the error loss plus a weighted inference cost. On Qwen2.5-Omni, it matches or beats feeding every modality while using only about 41% of the input tokens and 39% of the latency.
3.50
TTensormesh
LLM inference keeps recomputing the same context, wasting GPU time and delaying the first token. Tensormesh, the team behind the open-source KV cache project LMCache, has launched an enterprise product, Tensormesh Platform, that runs KV cache as a managed data layer. It caches automatically across GPU memory, host memory and storage, reuses both prompt prefixes and, through CacheBlend, chunks in the middle of a prompt, which suits retrieval-augmented generation. Machines in a cluster can also fetch KV cache directly from each other. The platform deploys on the company's own Kubernetes cluster and adds production features such as data security, real-time monitoring, high availability, cache policies and multi-tenant isolation.
3.40
Aleph AlphaRegulated European sectors such as public administration, industry and aerospace need open models they can control and that handle German well. Kolibri is an English-German mixture-of-experts model with 78.1 billion total parameters and about 3.5 billion active per token, released as open weights. The team controls the whole process, from data and architecture to training infrastructure, post-training and evaluation, and built its own pipeline to curate and synthesize more than 2 trillion German tokens so that German makes up a large share of training data. Among the open models compared in the report, it sits on the quality and serving-cost frontier in both English and German, meaning the best quality at its cost.
2.90
AlibabaIf you want to move a large model's ability into a cheaper small model without building the pipeline yourself, ModelDistill, the model distillation service on Alibaba Cloud's PAI platform, offers it as a no-code managed service. It uses black-box distillation: the teacher model generates high-quality training data, and the student model is trained on it with SFT, using LoRA, QLoRA or full fine-tuning. Besides general templates, it offers scenario templates for instruction following, chain-of-thought reasoning, agent tool calling, text-to-video and coding agents. The documentation also warns that distillation always loses some quality, and for simple classification or text matching, fine-tuning a small model directly may be cheaper.