INFRA · 2026-11-13
Infra — Paper Daily, 2026-11-13
Today's infra track is about where data should live and when to move it once GPU memory runs out. On the inference side, VerKV keeps the full KV cache in CPU memory and fetches from it only when the error exceeds a bound. SinkState finds that linear attention states are low-rank and can stay compressed throughout. Ostinato keeps the most frequently routed experts resident in GPU memory so that short prefills of local agents can use the GPU too. On the training side, OOM plans expert execution and memory allocation together on one machine, and TRANSIT lets multi-node training use half the GPUs without changing training code. Evolving Infra decides which KV to keep based on the context state of agents.
4.35
Vector retrieval often comes with a filter, such as finding nearest neighbors only within a few tags or categories. The hard part is that once a graph index like HNSW is filtered by label, many edges get cut, qualifying points can no longer reach each other, and the search gets stuck locally.
4.15
In speculative decoding, a drafter that drafts a whole block of tokens in one forward pass is fast, but it gives each position its own separate distribution. When the continuation has two equally plausible phrasings, a draft tree ranked by multiplying per-position probabilities splices them into sentences the target model would never say.
4.15
The linear attention layers in hybrid models compress history into a fixed-size state matrix, and every decoding step reads the whole matrix out and writes it back. With a large batch, this memory traffic takes most of the decoding time.
4.05
Agent workloads pose two problems for inference clusters. Some agents are prefill-heavy and others decode-heavy, and the mix drifts over time, so any fixed ratio of prefill to decode nodes loses out in some periods. Context compaction and sub-agents also change which KV is worth keeping: when a sub-agent returns after several minutes, almost the whole prompt can reuse earlier KV, while a request right after compaction can use little of it.
4.05
In long-context and agent workloads, the KV cache often exceeds GPU memory and has to live in CPU memory, and moving KV back to the GPU at every step slows inference down. Keeping only a compressed KV on the GPU avoids the transfers, but gives no error guarantee, and quality collapses at high compression. VerKV keeps two things on the GPU: a small set of tokens randomly sampled by importance, and a compact summary of the full KV used to quickly estimate how relevant each token is to the current query. At each step it first uses them to estimate the error of this attention computation, computes directly if the error is within the user's tolerance, and fetches the needed tokens from the CPU only when it is not. On RULER and LOFT, with only one eighth of the KV on the GPU, it keeps 96% of full quality on average and fetches only 2% to 23% of the cache per generated token. The point of random sampling is that the error can be estimated, so the system knows when it must go to the CPU.
4.05
With model state in CPU memory and weights streamed into the GPU layer by layer, a single multi-GPU machine can train a large MoE with full parameters. GPU memory then becomes a budget to divide, contested by resident expert parameters, retained activations and temporary compute space, and the split directly decides how fast a training step runs. OOM plans expert execution and memory allocation together, using calibrated cost models plus short trial runs to pick the fastest configuration between expert parallelism and computing all experts locally on each GPU. Change how memory is split and the faster execution path can flip, so the two decisions have to be made together. On four H100s it trains with 1M-token contexts, while FSDP2 and the other baselines run out of GPU memory at 512K. Its throughput averages 1.58 times that of CPU-offloaded FSDP2, while using the least host memory.
4.05
When a large MoE model runs an agent locally on a few consumer GPUs, the experts do not fit, and part of them must stay in CPU memory. Existing engines force a choice: either decode is fast but the short prefill after each tool output cannot use the experts in GPU memory, so the agent sits waiting for over two seconds, or prefill is fast and decode is noticeably slower. The authors observe that a short prefill touches many distinct experts, yet the routes its tokens actually take are concentrated: the 44% of experts used most in development sessions receive 84% of the routes in new sessions. Ostinato keeps these experts resident in GPU memory and computes them on the GPU in every forward pass, prefill or decode, while cold experts go to the CPU and the model's outputs stay unchanged. On DeepSeek-V4-Flash, it finishes whole agent sessions up to 2.5 times sooner than existing engines. Counted by routes instead of by distinct experts, even a short prefill is worth computing on the GPU.
4.05
Lazy image pulling on Kubernetes makes cold starts of model serving almost independent of model size, at the cost of writing the model's bytes into a node-level cache that has no quota and that the platform does not account for. Once the cache fills, Pods already serving start failing to read their model files while health checks all stay green. The authors show by experiment that neither of the two remedies an operator has works. A readiness check that reads a few sentinel files still reports healthy while 272 of 280 files are unreadable. Kubernetes sees the disk pressure yet frees not a single byte, and instead evicts a group of unrelated Pods, because its reclamation has no idea the cache exists. The problem can only be fixed inside the component that actually holds the bytes. The authors give the cache a hard byte budget triggered by actual disk usage, and when a cache write fails the data is still handed to the reader, just not kept. After the change every model file can be read, at 18.5% more completion time.
4.05
In MoE inference with expert parallelism, tokens are sent to experts on different GPUs through All-to-All, and there are already several ways to overlap this communication with computation, such as Comet, FlashMoE and DeepEP. The authors test them all under the same conditions on 103 production MoE shapes, input lengths from decode to long prefill, and four GPU platforms, and find that none wins everywhere, with even the best one fastest in only 80.6% of configurations. OverlapAdvisor uses one unified performance model to estimate how long each approach takes under a given configuration, accounting for how far real bandwidth and compute fall short of peak, and picks the fastest one when the server launches, without running benchmarks on the spot. In SGLang and vLLM, end-to-end throughput improves by up to 31.7%. The overlap strategy should be chosen by model shape, expert-parallel degree and input length, and sticking to one always loses on some configurations.
3.85
UIUC
IBMLLM training often adds machines only to get enough GPU memory, but more machines mean more participants in collective communication and more network congestion. TRANSIT goes the other way, using CPU memory to extend GPU memory so the same training job runs on fewer GPUs. It intercepts all GPU memory allocations in user space and redirects them to unified virtual memory, with no changes to the training code, framework, scheduler or driver. Relying on unified virtual memory's default page-fault migration would be slow, so TRANSIT keeps frequently reused activations and half-precision parameters in GPU memory, while rarely used data such as optimizer states is read by the GPU directly from host memory without being copied in. On an H100 cluster, it keeps over 90% of per-GPU throughput with half the GPUs, and its per-GPU throughput is up to 59% higher than ZeRO-Offload's. Fewer machines means fewer participants in communication, so network congestion matters less.