INFRA · 2026-10-28
Infra — Paper Daily, 2026-10-28
The infra track has 10 papers today, 6 with deep dives. The common thread is keeping LLM and retrieval services stable, cheap and trustworthy under tight resources. ZipFusion lets attention read a losslessly compressed KV cache directly, and BellowsKV lets several layers share one compressed input. Microsoft's HiLO moves degradation from the service level to the product level. PayPal's Cosmos gateway puts access, budgets and policy for many model vendors in one place. CoLoc decides where vector data should live, and TrustANN checks whether retrieved results can be trusted. Inference serving teams can start with ZipFusion and BellowsKV, and platform governance teams with Cosmos.
4.85
LEAD STORY ·
MicrosoftDuring traffic spikes, each service degrades on its own, so every service survives while the product's quality can collapse. Microsoft's HiLO moves degradation from the service level to the product level. It predefines a few validated product states. Each state sets a level for every service, such as dropping some features, reducing fan-out or using a cheaper model. At runtime services report load and a central controller picks one state. The unit of degradation should be the product, not the service. It went through 9 real traffic surges. The largest hit 2.38 times baseline peak, and Bing's relevance metric OneDCG, a normalized ranking-quality score, dropped at most 0.425 points, inside the 0.5-point operating limit.
4.50
A multimodal query first searches text vectors and then fetches the matching image vectors. Both nearest-neighbor searches are fast. The slow part is moving data when the two vectors sit in different storage tiers. CoLoc changes neither the vectors nor the index. It only decides which tier holds what, keeping vectors that are queried together in the same tier. It optimizes where data lives, not where to search. On a four-tier setup, cross-tier queries fell from 73.04% to 0, and end-to-end latency fell 24.8% at a half-multimodal mix, with recall unchanged.
4.50
Replicas in a distributed vector database keep results available but do not make them trustworthy. A compromised replica can return validly signed but tampered nearest neighbors. TrustANN adds middleware above the router. For each shard it greedily picks a set of replica replies that meets a set minimum size, scores high on pairwise similarity (candidate overlap plus rank agreement), and spans a set number of failure domains such as availability zones. If it falls short, it asks fresh independent replicas, and otherwise returns an explicit unverified status. Replicas being online does not make results trustworthy. In a multi-node test, tampered results accepted as trusted fell from 100% to 0, while p95 latency rose only 9.36% under normal load.
4.45
In long-context inference, the largest memory cost is often the per-layer KV cache, the keys and values attention stores for every past token. BellowsKV observes that caches of adjacent layers differ, but the hidden states that produce them are similar. So it shares the input, not the cache. For each token, it concatenates the hidden states of n adjacent layers and multiplies them by a compression matrix, giving one shared vector that replaces those layers' caches. Each layer has its own restore matrix that maps the shared vector back to its keys and values. Both are solved once offline from calibration data to minimize reconstruction error, with no training. Share the input, not the cache. On 3 models it keeps over 97% of accuracy at about 14x compression, with decoding speed close to uncompressed.
4.35
Most of the gain from lossless KV cache compression is eaten by decompression. Earlier methods restore the full cache on the GPU, so attention still reads uncompressed data. In ZipFusion, attention reads compressed data directly The format compresses only the exponent bits of 16-bit floats, which cluster in a narrow range, and keeps the fraction bits as is. Attention rebuilds the exponents in registers with a few integer adds and shifts, giving bit-identical values. Lossless compression can make decoding faster, not just cheaper to move. On an H100 the kernel is about 1.25x faster than FlashInfer, and at 8K context decode throughput beats uncompressed SGLang by 26.3%, bit for bit identical.
4.30
PayPalWhen a company uses models from several vendors and every app integrates them directly, keys spread, switching models needs code changes, and spend goes unmanaged. PayPal's Cosmos gateway is built on the Envoy proxy. Slow decisions such as auth, budgets and policy run outside the proxy. Format translation and token counting run inside it, so no streamed chunk waits on an external call. The hard part of a gateway is governance, not routing. It serves over 12,000 developers and about 200 billion tokens a day, adding only 4 ms of median latency in load tests.
4.20
When a mixture-of-experts (MoE) model does not fit in GPU memory, single-request decoding is dominated by copying expert weights from host memory. Routing biased toward resident experts cuts copies. Combined with asynchronous prefetching, though, outputs depend on transfer timing, so offline evaluation no longer matches the deployed model. PSR routes on a deterministic logical cache. At each layer, the current hidden state is fed to the next layer's router to predict its experts, and 1 or 2 of them enter the logical cache. Each layer's logical cache holds as many experts as there are GPU slots, and when full it evicts the expert with the lowest moving average of past router choices. In the next layer, cached experts get a fixed bonus added to their router scores. The cache depends only on token history, not on when copies finish. Physical copies follow it as prefetches and affect only speed. Make outputs deterministic first, then speed up with the cache. On an RTX 4090 with OLMoE, it decodes over 2.2x faster than exact routing with LRU eviction, with benchmark scores essentially unchanged.
3.85
ByteDanceMany inference instances recomputing the same long prefix is a common waste in serving clusters. Volcano Engine's EIC (Elastic Instant Cache) is a distributed KV cache service for LLM inference. Its role is to keep the key-value cache computed during inference in a separate distributed cache layer. Multiple inference instances can then share and reuse it instead of recomputing long prefixes. Moving the KV cache out of single-GPU memory into a shared storage service has become a clear direction for inference infrastructure. Teams running multi-instance serving, long context or multi-turn chat can use it as a product reference.
NSDI 2027
NVIDIA