INFRA · 2026-11-16

Infra — Paper Daily, 2026-11-16

Today's infrastructure track turns on one question: where the bottleneck really is, measured before anything is changed. Red Hat first checks whether the KV cache working set fits in CPU memory before deciding on a second storage tier. VecDBLens finds that vector database recall is often capped by the embedding rather than the index. SubGPU Pipelining and GC-MoE stagger and fill decode resources that sit idle in turns, inside one GPU and on consumer cards. SoScale picks FP4 scales by layer output. Olifant makes the draft in speculative decoding nearly free. Meituan uses unit tests and runtime logs to decide which AI review findings are real.

4.35

1. Authenticated Approximate Nearest Neighbor Search in Outsourced Vector Databases

Once a vector database is outsourced to a third-party cloud, the provider may skip neighbors to save compute or even tamper with results, so the client needs a way to verify that a nearest-neighbor search is genuine. Existing methods have the cloud send back the traversal path on the graph index along with every raw vector visited on the way, and the client recomputes the distances itself.

4.25

2. Benchmarking tiered KV cache offloading on H100 GPUs in OpenShift AI 3.5

Red Hat

Long-context agent requests resend the same prefix again and again, so the KV cache could be reused, but once GPU memory fills up it is thrown away and recomputed next time. Red Hat offloads the KV cache in OpenShift AI 3.5 to CPU memory, local NVMe, or shared CephFS, and gives a rule for judging a tier: it is worth using only when its p99 restore latency is below the prefill time a cache hit saves. On agent workloads averaging about 50 thousand input tokens, Nemotron-Ultra-253B throughput rose by up to 5.8 times. The difference depends on whether the working set fits in the CPU tier. When it does not, offloading to CPU memory alone gives only 3.3 times, and adding NVMe or CephFS nearly doubles that. When it fits, the CPU tier alone captures most of the gain. With multiple replicas under high concurrency, shared CephFS lets replicas reuse the same cache. Size the working set first, then decide whether a second storage tier is needed

4.20

3. SoScale: Second-Order Scale Search for FP4 Quantization

In FP4 quantization each number has only 16 possible values, and each block of data gets one scale that decides where those grid points land. The default scales by the block's maximum absolute value, so outliers stretch the grid. In a real Qwen3-8B weight block, the authors find that the two outermost grid points receive no values at all, while the dense majority of values near zero get too little resolution.

4.20

4. VecDBLens: An Evaluatology-Based Framework for Vector Database Diagnosis

When a vector database recalls poorly, the first instinct is to tune the index, but the ceiling on what an index can find is set by the embedding. On VisualNews with SigLIP2 and exact brute-force search, the authors find that only 2.04% of the nearest images by distance are the paired image in the labels. By contrast, graph indexes such as HNSW and CAGRA keep almost all of exact search's recall.

4.05

5. Snoopie: A Multi-GPU Communication Profiler and Visualizer

Multi-GPU programs often fail to scale because of data movement, yet existing tools cannot show clearly who moves what. NVIDIA's Nsight Systems only hints that there is traffic on NVLink, without showing which GPU issued a direct memory access or which object it touched, and it cannot show what actually moves beneath NCCL and NVSHMEM calls. Snoopie uses binary instrumentation on NVBit to track peer-to-peer transfers and communication library calls, maps every transfer to a source line and data object, and can zoom from the whole system down to single instructions and addresses. In one case study, a BFS program with ring communication showed unexpected traffic between GPU 1 and GPU 3, and the object view pointed to a flag pointer array allocated only on GPU 3, and fixing the allocation removed the extra transfers. The cost is heavy instrumentation overhead, especially on the NCCL case. It suits tracking down communication problems, not running permanently in production

4.05

6. SubGPU Pipelining: Intra-GPU Fine-Grained Spatial Parallelism for LLM Inference

LLM decode is a string of very short kernels, most of them limited by memory bandwidth. Running them one after another, each with the whole GPU, leaves compute and bandwidth alternately idle and busy, and many SMs sit unused while bandwidth is saturated. SubGPU Pipelining builds a pipeline inside one GPU. It cuts the operator graph into contiguous segments, binds each segment to its own disjoint group of SMs, and splits a batch into small batches that stream through in order, so work with complementary needs runs at the same time. Where to cut, how many SMs each segment gets, and how big the small batches are all affect one another, and the authors search them with a simulator that models shared-resource contention. In a simulated GPT-OSS 120B deployment on 32 B200s, at equal per-GPU throughput the per-user throughput is up to 2.2 times higher. Hardware tests validated only one two-segment configuration on a single B200, cutting decoder iteration time by 10.33%. A pipeline inside one GPU can stagger and fill resources that would otherwise sit idle in turns

4.05

7. GC-MoE on SM120: Overlapping Expert-Parallel Token Dispatch with Warp-Level FP8xFP4 Block-Scaled MMA on Consumer Blackwell

Expert-parallel MoE decoding is latency bound: every layer must dispatch tokens to the GPUs, run grouped matrix multiplies over the experts, and combine the results, three steps in series, with very few tokens in flight on each GPU. The earlier GC-MoE fused the three steps into one persistent CUDA Graph on Hopper and datacenter Blackwell. This paper moves it to consumer Blackwell cards over PCIe, such as the RTX PRO 5000. These cards expose block-scaled matrix multiplies only through warp-level instructions and lack several hardware features of datacenter parts, so the original kernel cannot be used directly. The authors rewrite the whole operator around a warp-level multiply of FP4 weights by FP8 activations and redo the per-expert streaming dispatch and the return path. On eight RTX PRO 5000 cards it is on average 1.522 times faster than FlashInfer's comparable operator, wins all 18 decode test points, and passes every numerical check. The same fusion idea had to be almost rewritten from scratch for consumer hardware

4.05

8. Towards Green Text Generation: Memory-Based Speculative Decoding

Speculative decoding has a cheap draft model guess a few tokens, which a large model then checks in one forward pass. Once the verifier uses a KV cache it is already much faster, so the cost of the draft model can no longer be ignored. On consumer hardware the authors find that a TinyLlama draft has a high acceptance rate yet halves throughput and triples energy. Their Olifant draft is not a neural network at all but a memory-based classification tree that predicts the next token from the previous four, returns in under 0.3 ms on CPU, and needs no GPU and no backpropagation. With a KV-cached verifier it still delivers 1.25 times the throughput and the lowest energy in the whole study. They also find that the KV cache decouples speed from energy: longer drafts raise power draw during verification, so the most energy-efficient draft length is shorter than the fastest one. The faster the verifier, the closer to free the draft must be

3.90

9. When One GPU Isn’t Enough: Serving GLM-5.3 Across Two NVIDIA DGX Stations with vLLM

NVIDIA

The NVFP4 version of GLM-5.3 is a large MoE model in which each token activates only a small share of the parameters, but each token may be routed to different experts, so all weights must stay in GPU memory. The full weights are about 464 GB, and a single GB300 has only 252 GB of memory, so they do not fit. This NVIDIA tutorial links two DGX Stations directly over RDMA, splits weights and compute in half with tensor parallelism, serves one OpenAI-compatible endpoint through vLLM, and turns on multi-token prediction, an FP8 KV cache, and prefix caching. Under a coding benchmark, total throughput peaks at concurrency 16 and falls at concurrency 32, where mean request latency stretches to seven and a half minutes and each user gets slower and slower. When running a huge model locally, more concurrency is not always better, so benchmark first to find the turning point

2.70

10. How to Use AI Coding in a System That Must Not Fail: Practice and Limits in Distributed Storage

Meituan

Meituan's distributed storage team shares what it learned using AI to write code in a system that must not fail. They start with a simple calculation: even if AI code has the same defects per thousand lines as human code, shipping more code means more defects in absolute terms. AI review is good at finding inconsistencies across files and will even build malformed inputs to reproduce a crash, but it cannot see what lies outside the code, such as manual operations or deliberate designs like letting the process exit rather than writing dirty data. The team turned each of the 117 suspected issues from one full-repository review into a unit test. An issue counted as confirmed only if its test failed on the main branch, and 36 were confirmed, 3 were refuted, and the rest could only be confirmed by reading source or not verified yet. When they gave runtime logs to the AI to map back to source, every defect it found was confirmed by people. AI review only finds candidates, and tests and logs decide which are real