INFRA · 2026-11-04
Infra — Paper Daily, 2026-11-04
Today's systems track is about caching and queueing. Fairway, Penelope and a paper on agent serving that cannot recover after a traffic surge all ask the same kind of question. Once prefix caches are heavily reused, which request should run first, which segment should be evicted, and when should fewer new sessions be admitted? AttentionIndex speeds up the token-picking step of sparse attention by treating it as a nearest-neighbor search, and OctEx spends GPU memory to remove the round-after-round waiting in vector search. Scute lets one GPU serve many kinds of fine-tuning adapters in one batch. On the engineering side, there is low-bit quantized storage in Qdrant, prefix-aware routing in SageMaker, automatic training recovery from NVIDIA, and Google's experience hunting XLA compiler bugs.
4.45
Prefill queueing under a prefix cache can keep both the median latency low and the tail short. While the queue is short, serve cheap requests with long cache hits first; move a request forward the longer it has waited relative to its own prefill time; and cap how much work cheap requests may place ahead of the oldest request. This is what Fairway does. Agent and RAG requests often share a prefix, so the remaining work of requests in one queue can differ by about 100 times, and the serving order decides how long each request waits for its first token. In traffic spikes, Fairway's median time to first token is 0.82 seconds with a p99 close to first-come-first-served, while hit-leaning orderings let the p99 reach 11 to 50 seconds.
4.45
In learned sparse attention, as used by DeepSeek-V4 and GLM-5.3, each query attends to only k keys, but the small scoring network that picks those k keys still scores every key in the prefix. At one million tokens, that step takes more than half of prefill time. AttentionIndex sharply cuts this cost. During prefill it groups keys by similarity; a query first scores one representative per group, and only the members of the most promising groups go to the original scoring network for the final pick. At that length, its prefill throughput is 1.3 to 1.8 times that of scoring every key, and it misses fewer keys than HiSA, a block-based method at similar speed.
4.45
A ten-minute traffic surge can leave an agent serving engine with prefill and decode disaggregated unable to recover: after the surge ends and traffic returns to a level the engine handled before, goodput on SGLang stays about 88% lower over the following hour. Each agent call resends an ever-longer conversation to the model and relies on prefix caching to skip the recomputation. Behind it is a loop: too little cache causes recomputation, calls slow down, sessions end later, more context is live at once, and the cache falls even shorter. Based on this, the authors build Basin. It predicts the largest context each session will reach over its lifetime, reserves that amount from a total budget that adapts to load, and holds new sessions back when the budget runs out. It reaches 1.20 to 1.70 times the goodput of Continuum. This kind of failure shows almost no warning in utilization or queue length.
4.45
For high-recall vector search on GPUs, the fastest graph indexes such as CAGRA walk the graph round by round, each round waiting for the distances of the previous one, and higher recall needs more rounds. OctEx takes another route: it spends GPU memory on neighbor lists computed ahead of time. It makes a few hundred randomly projected low-dimensional copies of the data, each copy assigns vectors to the cells of a regular grid, and for every cell it stores the vectors closest to the cell center. A query reads the lists of its own cells in one step, then reads the stored neighbors of those candidates, in a fixed number of rounds instead of a long chain. On billion-vector datasets on four H100 GPUs, it is 1.46 to 2.05 times faster than CAGRA at high recall. Trading GPU memory for serial waiting at query time is the core trade-off of this paper.
4.35
When the KV cache runs out of room, vLLM and SGLang both evict from the end of cached sequences, to keep the opening prefix that many requests can share. But an agent sends its long history back every turn, and later tokens cost more to recompute, because each attends to every token before it. Penelope does the opposite: it still keeps the shared prefix, but evicts the rest of the history from the front and keeps the later part. When the request returns, only the missing segment is recomputed. On SGLang with real agent traces, mean time to first token drops by up to 60% with a GPU-only cache, and by up to 76% with CPU memory added as a second cache tier. Which segment to evict should depend on how much it costs to recompute, not only on how many requests share it.
4.35
vLLM and SGLang can batch requests for many LoRA adapters together, but they reject adapters from other fine-tuning methods such as DoRA, VeRA and OFT, so these cannot share a batch, and converting them to LoRA costs extra memory and can lose accuracy. That hurts multi-tenant platforms, which often keep one copy of a base model on a GPU and add a small adapter for each customer. Scute leaves a few places to fill in the LoRA computation. On the base-weight path, the input is first multiplied by a transform matrix, and the output is then scaled per column and shifted. In the low-rank term, the middle result and the output are each multiplied by a vector. An unused place just multiplies by one or adds zero. For example, OFT rotates the input and then runs the original layer, so its rotation matrix fills the input transform. Five methods can then be mixed exactly in one batch, with decode time only 5% to 17% above the same number of LoRA adapters. Rather than converting every adapter to LoRA, teach the LoRA batching kernel to understand them.
4.15
A compiler bug may raise no error at all and just shift the numbers quietly, making an expensive training run diverge weeks later or degrading live inference quality. Google describes how they hunt such problems in the XLA compiler for TPUs at scale. On one side, they fuzz with randomly generated programs and real production graphs, and let a swarm of AI agents write stress tests, reproduce failures and draft fixes. On the other side, to locate a fault they bisect the failing large graph step by step, down to the single failing operation inside a fused kernel. Since deployment, the process has driven fixes for more than 142 compiler bugs, with 103 more findings under review. Compiler correctness needs continuous production-scale testing, and regression suites alone fall far short.
3.80
AmazonMany LLM applications start their prompts with a large fixed block, such as a few thousand tokens of support rules, followed by one line from the user. Prefix caching on a single machine skips recomputing that block, but with many machines requests are spread at random, each one rarely sees the same opening twice, and the cache is barely used. Amazon SageMaker's new prefix-aware routing sends requests with the same opening to the same machine. When that machine is full, the request goes to an idle one instead, and scaling up or down moves only a small share of traffic. In AWS benchmarks, the cache hit rate rose from about 25% to over 80%, and median time to first token fell by up to 77%. With many instances, whether prefix caching pays off depends on routing.
Qdrant
NVIDIA