INFRA · 2026-11-12

Infra — Paper Daily, 2026-11-12

Today's main thread is caching and scheduling in agent inference serving. One paper finds that a single traffic burst can leave an agent server that relies on prefix caching stuck at a low hit ratio for a long time. Recovery requires pausing some sessions at the gateway. ShadowPrefill uses idle compute to precompute context on the alternative model, easing the cost of switching models per step. TKNP, ADIndex and SlackTier lower the cost of long-context decode by changing where attention runs, the sparse attention index, and where large lookup tables are stored. ForeSpark and NVLinkOneSided cut small-batch waste in speculative decoding and MoE communication.

4.45

1. Bursts That Outlast Themselves: Persistent Degradation in Prefix-Cached Agent Serving

Serving agents depends on the prefix cache: on a hit each turn only computes the newly added content, and on a miss the whole context of tens of thousands of tokens is recomputed. The authors find that this can drop a server into a bad state it cannot leave. With the same base traffic, a few extra minutes of burst arrivals push the vLLM cache hit ratio down to about 10%, throughput halves, and the backlog keeps growing after the burst is over.

4.40

3. One-Sided Expert-Parallel All-to-All for Mixture-of-Experts Inference over NVLink

In MoE inference every layer runs two all-to-all exchanges: first it sends tokens to the GPUs that hold their experts, then it collects the experts' outputs back. During inference each GPU handles only a few tokens at a time. The time then goes to fixed costs such as GPUs waiting on and synchronizing with each other, not to moving data. The data is small enough to cross NVLink almost instantly, yet a round trip takes tens of microseconds.

4.25

4. SHADOWPREFILL: Keeping Your Next Model Warm for Per-Step Model Routing in Agentic LLM Serving

Routing an agent per step, with easy steps on a small model and hard ones on a large model, saves money. But every model switch carries an extra charge. A KV cache belongs to one model, so the model being switched to lacks the recent context and must compute it before emitting its first token. On long-context workloads such as coding agents, the switching step can take up to 3.2 times as long as usual.

4.15

5. Decode Token Parallelism: Memory-Efficient Distributed Attention Parallelism for Scalable LLM Decoding

In long-context, high-concurrency decode, dense layers and attention grow in cost differently. The MLP and projections process only newly generated tokens, so one read of the weights serves the whole batch. Attention, by contrast, must read every request's ever-growing KV cache. Existing parallel schemes each pay a price. Data parallelism copies the weights and eats into the memory left for KV cache. Once tensor parallelism spans nodes, its large all-reduce has to go over the slow network. Splitting the KV cache by sequence requires merging partial results across nodes. TKNP spreads one copy of the model across nodes in blocks of layers, runs each block's dense computation inside the node that holds it with tensor parallelism, and has every node compute attention only for its own requests against its local KV cache. Only the current token's activations cross the network. On vLLM, throughput is up to 1.7 times that of sequence splitting and up to 2.06 times that of data parallelism. Where the weights live and where attention runs can be decided separately.

4.15

6. ADIndex: Layer-Adaptive Sparse Attention Indexing via Group-Level Distillation

DeepSeek Sparse Attention (DSA) lets each query attend to only a selected subset of tokens, but the indexer that picks those tokens still scores every earlier token. With long contexts, the indexer alone can take more than half of prefill GPU time. The authors observe that neighboring queries pick heavily overlapping tokens, and that the amount of overlap differs from layer to layer. ADIndex lets a group of neighboring queries share one full scan. A small indexer distilled from the original one first picks a shared pool of candidate tokens for the whole group, and the original indexer then picks the final tokens for each query from that small pool. Each layer's group size and pool size are set separately by offline calibration. On two large models that use DSA, long-context task quality barely drops, and end-to-end serving is 1.54 times faster at a 128K context; with short contexts it can be slower. Sparse attention cut the cost of attention itself, and picking the tokens needs cutting too.

4.15

7. Placing LLM Lookup Tables by Per-Layer Prefetch Deadlines

DeepSeek's new model ships two large tables looked up by combinations of the last few tokens (Engram tables). They are too big for GPU memory, so they sit in host memory and take most of the serving process's memory. Which rows to read depends only on the input tokens, so reads can start at the beginning of each step alongside earlier layers. The authors find that what matters is how much time remains before the layer that reads the table. The later layer has more than a dozen layers in front of it, enough to finish a read from NVMe. The first layer has almost no margin and must stall if the read is late. SlackTier decides per layer which storage each table uses and who gets a small GPU cache, putting the later layer's table straight on NVMe and keeping the cache for the first layer. With both tables on NVMe, time per output token rises by at most 1.4%, and host memory falls from 212 GiB to 34 GiB. Where a table can live depends on how much time the layer reading it has left.

4.10

8. VeRL-Omni v0.3.0: Async Training and Broader Multimodal Post-Training

The new VeRL-Omni release puts generation, reward scoring, policy updates and weight synchronization for multimodal reinforcement learning into one pipeline, and lets more tasks run asynchronously: generation and training sit on different GPUs, joined by a buffer, so generation can run ahead of training. The most useful part of the post is a comparison. When training MiniCPM-o on the same machine with very short answers, splitting the GPUs into a generation group and a training group was about 9% slower per prompt. With answers stretched to 1024 tokens, the split setup was about 6.7% faster. The reason is that only when generation dominates does overlapping it with training save enough time to pay for the split. Async or sync depends on how much of each step generation takes. With short answers and training as the bottleneck, sharing one set of GPUs for generation and training remains the default choice.

3.75

9. Validating GPU Platforms with Runtime AI Inventory via the NVIDIA DSX AI Factory

NVIDIA

A GPU cluster's platform configuration can be version-locked and reproduced with recipes, but recipes do not cover which models and inference runtimes actually run on the cluster, and mutable image tags can leave two nodes running different code under the same tag. NVIDIA's AI Cluster Runtime now includes Google's open source k8s-aibom as a qualified component. It watches workloads on Kubernetes and produces an AI bill of materials for each AI workload it recognizes, recording which model is running and which inference runtime serves it, with each entry marked as declared, inferred or verified and pointing to its evidence. The controller itself is light: it averages 2.2 millicores of CPU and uses 28 to 34 MiB of memory. The bill records what a workload declared, and does not prove that it was scheduled or actually ran.

3.65

10. Manage Kubernetes Clusters for GPU Infrastructure with NVIDIA Kubernetes Engine

NVIDIA

AI factories that want to offer GPU capacity to customers as managed Kubernetes have to handle cluster provisioning, tenant isolation, upgrades and recovery themselves, and must also validate drivers and runtime software across GPU generations and Kubernetes versions. Two new NVIDIA open source projects split this work in two. NVKR builds node images, container images and packages from version-pinned sources and records every build input, so an image can be rebuilt exactly later. NVKE uses those artifacts to provision and manage customer clusters through a REST API, and each customer cluster has its own control plane and node pools. Customers just pick, by name, a release set the factory has published, without touching operating system images or boot configuration directly. The two projects can be used together or separately; both are in public preview, their APIs may change, and NVIDIA does not yet recommend them for production.