INFRA · 2026-11-06
Infra — Paper Daily, 2026-11-06
Saving GPU memory does not automatically accelerate inference. GAAC reconstructs KV during attention reads, while FlashOffloading overlaps transfers with computation. FastAFD shows that once expert GPUs count too, disaggregation’s throughput advantage can be small. Ask how saved capacity becomes full-step gains, and whether the comparison charges all extra resources.
4.40
Compressing KV saves memory but need not accelerate decoding: decompressing the entire cache first still moves the original volume of data. GAAC retains BF16 signs and mantissas, storing exponent differences from calibrated bases. Eight KV elements form a group in attention-access order. If a difference does not fit, the whole group uses wider offset codes, or raw exponents when necessary. Adding the base reconstructs KV bit-for-bit. Attention reconstructs small tiles as it reads them instead of expanding the entire cache first. On DGX Spark, resident KV memory fell by about 26%, and equal-batch full decode-step speed reached 1.04–1.15 times the baseline. Benefits depend on KV reads becoming the bottleneck.
4.40
Keys and values need not both be stored in full: within an attention head, much of V is predictable from K. LowRAM fits a small network per head to the current context, reconstructing V from K and retaining corrections only for tokens with the largest errors. Cross-layer low-rank decomposition compresses K. On Mistral in LongBench, KV storage fell to one quarter of the original footprint while retaining 97.9% of the baseline score. This trades fitting overhead for storage; its value also depends on how often the long context is reused.
4.40
Low-bit quantization saves GPU memory but loses accuracy. StrataFlux fits a compensation matrix on calibration data to recover projection quantization error. Earlier-layer state is normalized, and component indices ranked by descending absolute magnitude select correction-matrix columns with matching indices for prefetching. Each fetched column multiplies the current input component with that same index; summed products correct the low-bit projection. For YAQA two-bit Llama-3.1-8B-Instruct on RTX 5090, WikiText-2 perplexity fell from 9.46 to 8.40 while empty-context decode retained 95.4% of the quantized base’s throughput. This recovers some accuracy at a small speed cost, not losslessly.
4.40
Changing other requests in a batch can change greedy output for an identical prompt. BlockLock-Rows fixes batch size and row width, keeping the protected request in the same row and position while all requests still share prefill and decode. Single-GPU experiments paired runs of an identical prompt with different companions: ordinary padded batching achieved 52.59% exact sequence agreement, versus 100% for this scheme. Operators must themselves be repeat-stable at a fixed shape. Scheduling and padding buy reproducibility, not equality across different batch sizes or GPUs; widely varying prompt lengths also increase padding cost.
4.40
Moving experts to a separate GPU pool frees space for KV on attention GPUs and overlaps one microbatch’s expert computation with another’s attention. But expert GPUs must also count as a cost. FastAFD rechecks the trade-off on GB200 NVL72 against vLLM with expert-parallel width tuned per workload. Charging every allocated GPU, including experts, standard-decode throughput per GPU reaches only 0.99–1.17 times the control. Whether disaggregation pays depends on whether larger batches and overlapped work repay the GPUs assigned to experts; comparison against untuned narrow parallelism cannot establish a universal advantage.
4.40
In multi-GPU training, permanently assigning some compute units to transfers and others to matrix multiplication can leave one side idle while the other remains busy. KOSMOS puts both kinds of work into one persistent kernel, coordinating workgroups through queues and data-readiness flags. Groups that finish transfers can join computation; during expert-output combine they can switch back and forth. In single-node AMD experiments, MoE proxy-training throughput reached 1.29–1.83 times the DeepEP control. The change is how communication and computation share work during execution. The current implementation requires intranode transport, so these gains cannot be extrapolated to multinode runs.
4.40
In multi-GPU decoding, waiting for every GPU to confirm readiness can cost more than a small transfer itself. Flash-Comm’s primary path pre-fills receive storage with sentinel values that arriving data overwrites. Receivers process a required block once it is ready rather than first waiting for a group-wide barrier. When GPUs independently perform a reduction, they still accumulate in a fixed GPU order to avoid divergent results. In DeepSeek decoding tests on eight B300 GPUs, TPOT fell by 1.7–6.8%. The intervention optimises communication, not the remaining model computation; microbenchmark reductions cannot be treated as proportional full-step acceleration.
4.35
Moving KV into host memory admits more requests but can lose those gains to transfers. FlashOffloading loads historical KV from host memory onto the device layer by layer during prefill and fetches only the KV entries selected by the model’s sparse indexer during decoding. Token selection also supplies cache positions, and transfers overlap computation. In Ascend-based GLM tests at 128K input length, a larger running batch delivered 1.91 times the control’s output throughput with mean TPOT of 26 ms. Gains come from capacity-enabled concurrency and reduced transfer overhead, not faster individual requests; the decode path also requires native sparse selection.
4.30
Once vectors reside on SSD, the number of raw vectors read for refinement directly affects search throughput. Recast retains inverted lists and quantized codes in memory, estimating distances and error ranges before reading only candidates that can still enter the leading results. It does not fetch the same fixed candidate count for every query. In static tests at a specified memory budget and 0.99 recall, query throughput reached 1.56–2.01 times the strongest control. Disk-read budgets adapt to query difficulty, but interval filtering cannot recover neighbours missed when inverted lists were selected.
3.90
NVIDIAHealthy node checks can still leave distributed training held back by one slow GPU. NVIDIA DSX OS connects versioned configuration, workload certification and post-repair revalidation. NVCRE runs training and NCCL tests, but sets no performance thresholds by default: starting a test does not establish adequate bandwidth. The example’s 300 GB/s is a configured threshold, not a measured speedup. With lifecycle management enabled, NVSentinel coordinates validation and removes scheduling gates from repaired nodes after tests pass. Operators must define acceptance criteria and incorporate revalidation into recovery.