INFRA · 2026-11-16
Infra — Paper Daily, 2026-11-16
Today's infrastructure track turns on one question: where the bottleneck really is, measured before anything is changed. Red Hat first checks whether the KV cache working set fits in CPU memory before deciding on a second storage tier. VecDBLens finds that vector database recall is often capped by the embedding rather than the index. SubGPU Pipelining and GC-MoE stagger and fill decode resources that sit idle in turns, inside one GPU and on consumer cards. SoScale picks FP4 scales by layer output. Olifant makes the draft in speculative decoding nearly free. Meituan uses unit tests and runtime logs to decide which AI review findings are real.
4.35
Once a vector database is outsourced to a third-party cloud, the provider may skip neighbors to save compute or even tamper with results, so the client needs a way to verify that a nearest-neighbor search is genuine. Existing methods have the cloud send back the traversal path on the graph index along with every raw vector visited on the way, and the client recomputes the distances itself.
4.25
Red HatLong-context agent requests resend the same prefix again and again, so the KV cache could be reused, but once GPU memory fills up it is thrown away and recomputed next time. Red Hat offloads the KV cache in OpenShift AI 3.5 to CPU memory, local NVMe, or shared CephFS, and gives a rule for judging a tier: it is worth using only when its p99 restore latency is below the prefill time a cache hit saves. On agent workloads averaging about 50 thousand input tokens, Nemotron-Ultra-253B throughput rose by up to 5.8 times. The difference depends on whether the working set fits in the CPU tier. When it does not, offloading to CPU memory alone gives only 3.3 times, and adding NVMe or CephFS nearly doubles that. When it fits, the CPU tier alone captures most of the gain. With multiple replicas under high concurrency, shared CephFS lets replicas reuse the same cache. Size the working set first, then decide whether a second storage tier is needed
4.20
In FP4 quantization each number has only 16 possible values, and each block of data gets one scale that decides where those grid points land. The default scales by the block's maximum absolute value, so outliers stretch the grid. In a real Qwen3-8B weight block, the authors find that the two outermost grid points receive no values at all, while the dense majority of values near zero get too little resolution.
4.20
When a vector database recalls poorly, the first instinct is to tune the index, but the ceiling on what an index can find is set by the embedding. On VisualNews with SigLIP2 and exact brute-force search, the authors find that only 2.04% of the nearest images by distance are the paired image in the labels. By contrast, graph indexes such as HNSW and CAGRA keep almost all of exact search's recall.
4.05
Multi-GPU programs often fail to scale because of data movement, yet existing tools cannot show clearly who moves what. NVIDIA's Nsight Systems only hints that there is traffic on NVLink, without showing which GPU issued a direct memory access or which object it touched, and it cannot show what actually moves beneath NCCL and NVSHMEM calls. Snoopie uses binary instrumentation on NVBit to track peer-to-peer transfers and communication library calls, maps every transfer to a source line and data object, and can zoom from the whole system down to single instructions and addresses. In one case study, a BFS program with ring communication showed unexpected traffic between GPU 1 and GPU 3, and the object view pointed to a flag pointer array allocated only on GPU 3, and fixing the allocation removed the extra transfers. The cost is heavy instrumentation overhead, especially on the NCCL case. It suits tracking down communication problems, not running permanently in production
4.05
LLM decode is a string of very short kernels, most of them limited by memory bandwidth. Running them one after another, each with the whole GPU, leaves compute and bandwidth alternately idle and busy, and many SMs sit unused while bandwidth is saturated. SubGPU Pipelining builds a pipeline inside one GPU. It cuts the operator graph into contiguous segments, binds each segment to its own disjoint group of SMs, and splits a batch into small batches that stream through in order, so work with complementary needs runs at the same time. Where to cut, how many SMs each segment gets, and how big the small batches are all affect one another, and the authors search them with a simulator that models shared-resource contention. In a simulated GPT-OSS 120B deployment on 32 B200s, at equal per-GPU throughput the per-user throughput is up to 2.2 times higher. Hardware tests validated only one two-segment configuration on a single B200, cutting decoder iteration time by 10.33%. A pipeline inside one GPU can stagger and fill resources that would otherwise sit idle in turns
4.05
Expert-parallel MoE decoding is latency bound: every layer must dispatch tokens to the GPUs, run grouped matrix multiplies over the experts, and combine the results, three steps in series, with very few tokens in flight on each GPU. The earlier GC-MoE fused the three steps into one persistent CUDA Graph on Hopper and datacenter Blackwell. This paper moves it to consumer Blackwell cards over PCIe, such as the RTX PRO 5000. These cards expose block-scaled matrix multiplies only through warp-level instructions and lack several hardware features of datacenter parts, so the original kernel cannot be used directly. The authors rewrite the whole operator around a warp-level multiply of FP4 weights by FP8 activations and redo the per-expert streaming dispatch and the return path. On eight RTX PRO 5000 cards it is on average 1.522 times faster than FlashInfer's comparable operator, wins all 18 decode test points, and passes every numerical check. The same fusion idea had to be almost rewritten from scratch for consumer hardware
4.05
Speculative decoding has a cheap draft model guess a few tokens, which a large model then checks in one forward pass. Once the verifier uses a KV cache it is already much faster, so the cost of the draft model can no longer be ignored. On consumer hardware the authors find that a TinyLlama draft has a high acceptance rate yet halves throughput and triples energy. Their Olifant draft is not a neural network at all but a memory-based classification tree that predicts the next token from the previous four, returns in under 0.3 ms on CPU, and needs no GPU and no backpropagation. With a KV-cached verifier it still delivers 1.25 times the throughput and the lowest energy in the whole study. They also find that the KV cache decouples speed from energy: longer drafts raise power draw during verification, so the most energy-efficient draft length is shorter than the fastest one. The faster the verifier, the closer to free the draft must be