INFRA · 2026-10-30
Infra — Paper Daily, 2026-10-30
Today's infrastructure track is about one thing: making low-bit and sparse LLMs actually run fast and deploy cheaply. Humming compiles kernels for many quantization formats on the fly from shared components, while day-0 support for Mistral Large 4 and MiniMax-M3.1-Flash shows what NVFP4, sparse attention and speculative decoding deliver on large MoE models. vLLM TorchTPU brings the same serving stack to TPUs, and one paper warns that image embeddings stored in third-party vector databases can be turned back into the original images.
4.20
LEAD STORY · AAnt Group
AAnt Group
Teams serving large models should read this. It tackles a more basic problem in quantization: the hard part is often not squeezing weights into low bits, but that every extra format and every new GPU generation needs another kernel to write and maintain. Humming, open-sourced by Ant Group and the vLLM community, separates the description of a quantization format from how it executes on a given GPU, then compiles the needed kernel on the fly from shared CUDA components. The components are light, so compilation is fast: one W4A16 kernel on H200 takes 23.35 seconds with CUTLASS and 1.82 seconds with Humming. Maintenance cost scales with components, not format combinations, which suits inference teams supporting many quantization formats across several GPU generations.
3.80
MistralSwitching the trillion-parameter multimodal MoE model Mistral Large 4 to NVFP4 precision costs almost no quality, the most practical finding vLLM reported with its day-0 support. On multimodal, long-context, coding-agent and other evals it stays within 2.1 points of the FP8 version on every task, with a mean gap of only 0.3 points, while running faster and holding more KV cache. The model also ships its own DSpark draft model for speculative decoding, which gets about 2.9 tokens accepted per step and roughly doubles output throughput. NVFP4 plus the bundled draft model is the first combination to try when deploying such a large MoE, because neither needs extra training.
3.80
IInferact
Long-context decode is slow mainly because every step reads the whole KV cache from GPU memory. MiniMax-M3.1-Flash's attention reads only a small part of it: a lightweight indexer first splits the context into blocks of 128 tokens and scores each block, and main attention runs exactly over only the 16 highest-scoring blocks. Each token stores a small index vector that the indexer scores against the current query, and a block's score is the highest score inside it. Each step still reads the index vectors of the whole context, but they are one eighth the size of the full KV data, and full KV data is read only for the selected blocks. So at long context, indexer included, each step reads only about 12.5% of the memory dense attention reads. vLLM supported the model on day 0 and keeps both the KV cache and the MoE weights in NVFP4. For long-context agent serving, the savings come from reading less memory, and sparse attention and low-bit KV cache both do that.
3.60
IInferact
Google The new vLLM TorchTPU lets most models already in vLLM run on Google Cloud TPUs without changing model code. It is a vLLM platform plugin: the scheduler, KV cache manager and OpenAI-compatible server are all upstream vLLM, models are compiled to XLA through torch.compile, and the TPU-specific work sits in a set of hand-tuned Pallas kernels for attention, MoE and quantized matmul. All 6 of vLLM's parallelism modes can be combined on TPU; for large MoE models the authors usually get the best throughput by letting each chip compute its own attention while spreading experts across the whole TPU slice. It can also offload KV cache to host memory with zero copy. For teams already on vLLM, moving to TPU mostly costs installing a plugin and waiting for the first compile
3.05
Many teams hand their vector databases to third-party hosts on the grounds that embeddings are not raw data, so a leak would not matter much. This paper shows image embeddings are not safe: without knowing the embedding model's architecture or parameters, an attacker who can merely call the model or API can reconstruct high-fidelity original images from the embeddings. The method first trains a conditional diffusion model on images of the same kind as the victim's data, then feeds the image embedding in through cross-attention to guide generation. Earlier image embedding inversion attacks mostly assumed access to model parameters, which does not hold in outsourced settings, so this query-only setting is closer to the real threat. Image embeddings stored in a third-party vector database deserve the same protection as the images themselves
KDD 2026