2026-10-26 ~ 11-01 · Infra
Generative · Personalization · AI Search · Infra · Embodied AI
- 1. HiLO: Towards Product-Level Graceful Degradation with a Quality-Aware, Load-Aware Control Framework · 📖 Deep dive
During traffic spikes, each service degrades on its own, so every service survives while the product's quality can collapse. Microsoft's HiLO moves degradation from the service level to the product level…
Appeared in: 10-28
- 2. Cross-Modal Data Co-Locality for Heterogeneous Storage Systems · 📖 Deep dive
A multimodal query first searches text vectors and then fetches the matching image vectors. Both nearest-neighbor searches are fast. The slow part is moving data when the two vectors sit in different storage tiers…
Appeared in: 10-28
- 3. Failure-Resilient and Verifiable Distributed Approximate Nearest-Neighbor Retrieval · 📖 Deep dive
Replicas in a distributed vector database keep results available but do not make them trustworthy. A compromised replica can return validly signed but tampered nearest neighbors…
Appeared in: 10-28
- 4. Folding Attention Biases into Keys Makes Compacted KV Caches Run on Standard Kernels · 📖 Deep dive
KV cache compression can already cut a long document's cache to a tenth, but the strongest method, Attention Matching, adds a bias to every kept key, and vLLM has no slot for it…
Appeared in: 10-29
- 5. Let the Gate Decide: Per-Lookup Precision for Quantized Conditional Memory in LLMs · 📖 Deep dive
When you quantize a language model's n-gram memory tables to low bits, keeping the most frequently accessed rows precise does worse than picking rows at random. The reason is the gate…
Appeared in: 10-29
- 6. WAKMan: Workload Aware KVCache Manager · 📖 Deep dive
Which eviction policy vLLM's prefix cache should use depends on the traffic arriving right now. Agent requests come back within seconds with the same history, so the default LRU fits…
Appeared in: 10-29
- 7. BellowsKV: One Latent for Many Layers, Expanded on Demand · 📖 Deep dive
In long-context inference, the largest memory cost is often the per-layer KV cache, the keys and values attention stores for every past token. BellowsKV observes that caches of adjacent layers differ…
Appeared in: 10-28
- 8. Where Rounding Happens Matters: Rounding-Location Alignment Stabilizes LLM Reinforcement Learning · 📖 Deep dive
LLM reinforcement learning usually generates answers with one inference engine and trains with another framework. Even with identical weights and BF16 on both sides, the numbers differ slightly…
Appeared in: 10-27
- 9. KVStream: Adaptive Parallel Storage Management for KV Caching in Large-Scale LLM Serving · 📖 Deep dive
Long-context inference keeps reusing the same prefixes: system prompts, whole documents, code repositories and multi-turn session histories. Their KV cache no longer fits in GPU memory or host memory…
Appeared in: 10-26
- 10. C 2 KV: Compressed and Composable KV Cache Reuse for Efficient LLM Inference · 📖 Deep dive
Retrieval-augmented generation stitches several retrieved documents into the context, and recomputing their KV caches from scratch every time is expensive, so some methods store each document's KV cache and concatenate…
Appeared in: 10-27