INFRA · 2026-11-12
Infra — Paper Daily, 2026-11-12
Today's main thread is caching and scheduling in agent inference serving. One paper finds that a single traffic burst can leave an agent server that relies on prefix caching stuck at a low hit ratio for a long time. Recovery requires pausing some sessions at the gateway. ShadowPrefill uses idle compute to precompute context on the alternative model, easing the cost of switching models per step. TKNP, ADIndex and SlackTier lower the cost of long-context decode by changing where attention runs, the sparse attention index, and where large lookup tables are stored. ForeSpark and NVLinkOneSided cut small-batch waste in speculative decoding and MoE communication.
4.45
Serving agents depends on the prefix cache: on a hit each turn only computes the newly added content, and on a miss the whole context of tens of thousands of tokens is recomputed. The authors find that this can drop a server into a bad state it cannot leave. With the same base traffic, a few extra minutes of burst arrivals push the vLLM cache hit ratio down to about 10%, throughput halves, and the backlog keeps growing after the burst is over.
4.45
A parallel draft model writes a whole block of draft tokens in one forward pass. Current systems write the full block first, then pick the most promising few for the target model to verify. The authors find this wasteful under high load: drafting takes up to 40% of each step, and most of what is drafted is thrown away without ever being verified.
4.40
In MoE inference every layer runs two all-to-all exchanges: first it sends tokens to the GPUs that hold their experts, then it collects the experts' outputs back. During inference each GPU handles only a few tokens at a time. The time then goes to fixed costs such as GPUs waiting on and synchronizing with each other, not to moving data. The data is small enough to cross NVLink almost instantly, yet a round trip takes tens of microseconds.
4.25
Routing an agent per step, with easy steps on a small model and hard ones on a large model, saves money. But every model switch carries an extra charge. A KV cache belongs to one model, so the model being switched to lacks the recent context and must compute it before emitting its first token. On long-context workloads such as coding agents, the switching step can take up to 3.2 times as long as usual.
4.15
In long-context, high-concurrency decode, dense layers and attention grow in cost differently. The MLP and projections process only newly generated tokens, so one read of the weights serves the whole batch. Attention, by contrast, must read every request's ever-growing KV cache. Existing parallel schemes each pay a price. Data parallelism copies the weights and eats into the memory left for KV cache. Once tensor parallelism spans nodes, its large all-reduce has to go over the slow network. Splitting the KV cache by sequence requires merging partial results across nodes. TKNP spreads one copy of the model across nodes in blocks of layers, runs each block's dense computation inside the node that holds it with tensor parallelism, and has every node compute attention only for its own requests against its local KV cache. Only the current token's activations cross the network. On vLLM, throughput is up to 1.7 times that of sequence splitting and up to 2.06 times that of data parallelism. Where the weights live and where attention runs can be decided separately.
4.15
DeepSeek Sparse Attention (DSA) lets each query attend to only a selected subset of tokens, but the indexer that picks those tokens still scores every earlier token. With long contexts, the indexer alone can take more than half of prefill GPU time. The authors observe that neighboring queries pick heavily overlapping tokens, and that the amount of overlap differs from layer to layer. ADIndex lets a group of neighboring queries share one full scan. A small indexer distilled from the original one first picks a shared pool of candidate tokens for the whole group, and the original indexer then picks the final tokens for each query from that small pool. Each layer's group size and pool size are set separately by offline calibration. On two large models that use DSA, long-context task quality barely drops, and end-to-end serving is 1.54 times faster at a 128K context; with short contexts it can be slower. Sparse attention cut the cost of attention itself, and picking the tokens needs cutting too.
4.15
DeepSeek's new model ships two large tables looked up by combinations of the last few tokens (Engram tables). They are too big for GPU memory, so they sit in host memory and take most of the serving process's memory. Which rows to read depends only on the input tokens, so reads can start at the beginning of each step alongside earlier layers. The authors find that what matters is how much time remains before the layer that reads the table. The later layer has more than a dozen layers in front of it, enough to finish a read from NVMe. The first layer has almost no margin and must stall if the read is late. SlackTier decides per layer which storage each table uses and who gets a small GPU cache, putting the later layer's table straight on NVMe and keeping the cache for the first layer. With both tables on NVMe, time per output token rises by at most 1.4%, and host memory falls from 212 GiB to 34 GiB. Where a table can live depends on how much time the layer reading it has left.
4.10
The new VeRL-Omni release puts generation, reward scoring, policy updates and weight synchronization for multimodal reinforcement learning into one pipeline, and lets more tasks run asynchronously: generation and training sit on different GPUs, joined by a buffer, so generation can run ahead of training. The most useful part of the post is a comparison. When training MiniCPM-o on the same machine with very short answers, splitting the GPUs into a generation group and a training group was about 9% slower per prompt. With answers stretched to 1024 tokens, the split setup was about 6.7% faster. The reason is that only when generation dominates does overlapping it with training save enough time to pay for the split. Async or sync depends on how much of each step generation takes. With short answers and training as the bottleneck, sharing one set of GPUs for generation and training remains the default choice.