INFRA · 2026-10-25
Infra — Paper Daily, 2026-10-25
Amazon’s coding agent retunes training recipes, user-space rebuilds GPU Direct Storage, and native KV reduction plus stratified sparse attention land—today’s infra thread moves throughput, I/O control paths, and cache/attention cost from fixed-knob search into editable recipes, certified screening, and lifetime engineering.
4.55
LEAD STORY ·
AmazonAmazon LLM training throughput hinges on the recipe—sharding, memory–compute trade-offs, kernels and overlap—and that recipe changes with every model, accelerator and framework. Fixed-knob tuners only search a preset space and check that a job runs, so they miss levers outside that space and can silently change training semantics. Amazon’s ATTUNE is a coding agent that retunes any working recipe unattended inside a GPU-hour budget. On H100/H200/B200 it lifts throughput 1.7× on average and beats four of NVIDIA AutoModel’s published recipes by 6–17%; on held-out models it improves recipes written for another setting by up to 2.6× (MiniMax-M2). Throughput gaps often live outside the fixed knob space—letting a coding agent edit the recipe beats grid search.
4.50
NVIDIA GDS moves NVMe↔GPU over PCIe P2P DMA, but still routes every I/O control path through the kernel, imposes total order on the stream path, and ships a closed user-space engine. uGDS is a user-space kit: on a Samsung 990 PRO it reaches up to 12.12× GDS batch-read and 30.69× write bandwidth, and cuts 4-KiB sync write latency 84.1% with two issuing threads. On one stream GDS keeps only 40.8% of its sync 16-MiB read bandwidth; submission calls can eat 99.6% of wall time. GPU Direct Storage bottlenecks are often control-path and ordering semantics, not just DMA bandwidth.
4.40
Head-dimension KV compression often reconstructs full-dimensional tensors at inference and adds decode FLOPs. RACE calibrates pretrained attention into a natively reduced form: shared Q/K projections that stay RoPE-aligned, compressed by query–key activation energy, cutting both KV storage and attention compute. At 50% KV retention and 256K context it cuts total inference FLOPs 47.8%, lifts throughput 1.79×, TTFT −43.2%, and decode latency −28.1%. Head-dim compression must be native reduction, not reconstruct-at-decode—or saved memory becomes a FLOPs tax.
4.40
On long agent trajectories KV-cache I/O and storage dominate cost, and editing the prefix invalidates later cache and forces re-prefill. This paper shifts from rewriting history text to engineering cache lifetime and eviction. Versus threshold-triggered summarization, a simple tool-response drop policy cuts average cost 27.7% and lifts benchmark scores 1.7 points; with AKV it reaches ~1.2× output throughput and ~1.5× better P95 latency; an RSI policy saves another 7.1% on held-out tasks. KV reads can be 82.4% of API cost. Agent cost often turns on not invalidating cache, not only on shorter summaries.
4.40
Long-context decode is bottlenecked by a growing KV cache; sparse attention’s deterministic top-k drops residual mass, sampling adds variance, and hybrids still lag dense. Stratified Sparse Attention (SSA) allocates the sparse budget by strata to close the sparse–dense gap at extreme sparsity: at 50× sparsity it lifts the strongest practical baseline from 81.8%→84.2% at 32K and 63.8%→69.3% at 128K, recovering 94% and 85% of dense performance; decode vs FlashInfer reaches up to 2.96× at 1M context. At extreme sparsity, stratify to recover residual attention mass—do not only chase top-k.
4.20
NVIDIATensorRT 11.1’s Global Performance Tuner does automatic, accuracy-aware end-to-end tuning with per-model optimization routes. Part 2 spells out the algorithm and architecture on Part 1’s notation, plus best practices: when to enable global tuning and how to read accuracy–performance trade-offs across build routes. Compile scheduling is not “pick the fastest kernel”—it is a global, accuracy-gated search over graph transforms, fusion, impls, and quantization.
4.15
Learned PTQ for text embedders usually conflates how hard to calibrate with whether to calibrate at all. Across e5-base-v2, BGE-M3 and EmbeddingGemma-300m at 4-bit weights (161 runs), the same procedure ranges from a gain to catastrophic degradation; the driver is update dose (functional change). In-domain NDCG@10 swings from +1.47 to −71.53. The paper adds dose-driven collapse diagnosis and validation-gated abstention: refuse the calibration when dose is too high. For embedding PTQ, ask dose before “calibrate or not”—too much dose can destroy retrieval metrics.
4.15
In multi-turn chat and agent workloads, KV writes to SSD can exceed drive capacity and shrink flash lifetime from years to months. Gandalf targets that write amplification for KV offload: gate what lands on disk, merge and tier writes, and stretch flash lifetime while keeping the capacity win. When KV spills to SSD the first constraint is often lifetime, not bandwidth—cut writes before celebrating offload.
4.15
Watchlist face match, fraud pattern match, and medical image lookup need neighbor recall 0.999–0.9999, yet approximate indexes cannot say which queries lost a true neighbor. IVF raises recall by probing more lists but still scores every probed vector at full precision. Lychee adds certified screening of inverted lists: skip candidates that cannot enter the top while keeping high recall, at cost far below exact search over billions. Spend high-recall budget on provably not reading candidates, not on blindly probing more lists.
4.05
NVIDIA Deploying a net is more than op→GPU kernel: TensorRT’s engine build makes interdependent scheduling choices over graph-IR transforms, fusion, impl selection, memory placement, and quantization—many answers are legal. Part 1 frames the Global Performance Tuner inside that compile problem and why a single build route is not enough. Engine build is constrained compile scheduling; single-route search misses global optima across decisions.