4.55
LEAD STORY ·
AmazonAmazon Move a large model to a different GPU or framework version, and the training recipe that used to be fastest often no longer is. The recipe settles how the model is split across GPUs, which low-level routines to use, and how GPU data transfers overlap with computation. ATTUNE lets an AI agent able to write code and run commands tune recipes, given a recipe that runs and a budget of GPU-hours; it tries things and analyses bottlenecks on its own. But the real point of the paper lies outside the agent, in a correctness gate run by fixed code the agent cannot change. The model, data, batch size and numeric precision are locked, and any change that would quietly alter what is being trained, such as dropping overflow tokens, is rejected before launch. After fine-tuning jobs for three models switched to the tuned recipes, training ran on average 1.7 times as fast as before, and up to 3.9 times.
4.50
When NVIDIA's GDS reads data from SSD straight into GPU memory, the authors measured that 99.6% of the time for asynchronous reads was stuck at the step of submitting requests. Large-model inference services store the KV cache they have already computed on SSDs, so that the next request with the same opening can read it back instead of recomputing, and the speed of this read feeds directly into time-to-first-token. uGDS moves the job of telling the SSD what to do out of the operating-system kernel and into an ordinary program. Data still goes straight from SSD to GPU; what changes is who gives the orders. uGDS's batched reads are up to 12 times as fast as GDS, and plugged into an open-source cache system, the median time-to-first-token for long prompts fell from 1.9 seconds to 0.7 seconds.
4.40
The KV cache a large model keeps in GPU memory grows with the text, and one way to compress it is to make the vector stored for each word shorter, but most existing methods only save storage. They store short vectors, yet at compute time must first restore them to full length, so computation balloons. RACE finds a way to shorten vectors without restoring them. Position encoding rotates each pair of adjacent numbers in a vector together, so as long as whole pairs are kept, the short vector can still be rotated directly. The pairs kept are those where queries and keys have the largest sum of squares on a small batch of text, and that choice is written directly into the model weights. On Llama-3.2-3B, with vectors halved and an input of 256,000 tokens, overall throughput rises to 1.79 times and time-to-first-token falls 43.2%.
4.40
When an agent goes back and forth with tools such as search, web pages and code execution for hundreds of turns, it pays a recurring cost. On every turn the model re-reads the KV cache of everything that came before, and reading this old cache accounts for 82.4% of total inference cost. The usual way to save money is to delete or summarize old messages, but any edit to earlier text invalidates all the cache after it, so everything from the edit point onward has to be recomputed. The authors work at a different layer: leave the text history untouched and drop only selected entries from the cache, while the remaining entries are reused as they are. A simple policy of keeping the cache of only the 12 most recent tool outputs beats the common approach of summarizing at the length limit, with a pass rate 1.7 percentage points higher on average across four agent benchmarks and 27.7% lower average cost.
4.40
In long-context decoding, every generated token reads the whole KV cache from HBM, so the longer the context, the slower attention gets. Sparse attention reads only a small subset of tokens, but at 50× sparsity existing methods fall far short of dense attention. The idea of SSA is that high-scoring tokens are always kept, the remaining long tail is sampled by proxy score, and each sampled token is up-weighted by its probability of being sampled. The proxy score is a quick attention estimate from compressed keys. After softmax, all probabilities are multiplied by one common factor, and tokens that reach one are kept outright. The factor is chosen so that the number kept plus the scaled probabilities of the other tokens equals the budget exactly. The rest of the budget is sampled with replacement by that probability, a token drawn more than once counts once, and its weight is divided by its chance of being drawn at least once. At the same sparsity, on AIME 2025 with Qwen3.5-9B, SSA scores 59.8 versus 36.7 for PQCache.
4.20
NVIDIAWhen TensorRT compiles a model, the best choice of how to fuse operators and where to quantize depends on the model, the tensor shapes and the GPU, so one optimization step that looks effective does not mean the whole model gets faster. For example, several independent operators that sit side by side can be fused into one GPU kernel, or operators that follow one another can be fused into one. NVIDIA's approach turns these options into switches that can be combined, builds and measures an engine for each, rules out those that miss the accuracy bar, and keeps the fastest. Across ten workloads, the tuned engines are 1.14 to 1.63 times faster than the default build, so the default build often has plenty of room to get faster.
4.15
Calibrating a text embedding model after compressing it to 4 bits can make it better, or can wreck it outright. On EmbeddingGemma, changing nothing but the learning rate swings the ranking metric NDCG@10 from a gain of 1.47 points to a crash of 71.53 points. Whether it collapses depends mainly on how far calibration pushes the model away from its original state. The authors' rule is to test every calibration on held-aside queries first and adopt it only if it is statistically better. In the experiments this rule never picked a harmful result, at the cost of missing a few helpful updates.
4.15
When the attention cache lives on SSDs, you have to count write volume as well as hit rate, because SSDs wear out from heavy writing. Deciding what to write based on access history costs hit rate, so the authors decide by which kind of attention layer the cache comes from. Linear-attention state is used up quickly and stays in memory; full-attention cache will be reused again and again later, so only it is written to SSD. Replaying a Claude Code usage trace, write volume drops from 153TB to 92TB with almost the same hit rate; across all test traces, expected SSD lifetime extends 7.3 times on average.
4.15
When recall must be close to 100%, inverted indexes probe many lists and read full-precision vectors one by one. Ruling out most vectors first with a distance lower bound that never overestimates, and reading only the few left, is much faster with exactly the same result. Each vector is projected in advance onto the dataset's directions of largest variance, plus one extra coordinate holding the length of what the projection leaves out, and distances in these short coordinates never exceed the true distance. The upper limit comes from anchors, stored vectors taken at a fixed interval from the probed lists. Since they are themselves candidates, the k-th smallest of the query's true distances to them is never below the true k-th nearest-neighbour distance. Vectors whose lower bound already exceeds this limit are excluded without being read, and the rest are checked against full vectors in increasing order of bound. At equal recall it is 9.7 to 117 times faster than IVF-Flat.
4.05
NVIDIA The first part of NVIDIA's series on the TensorRT global tuner, covering only the idea: instead of following one fixed path at build time, try many build configurations, measure each whole engine, and pick the fastest among those that meet the accuracy bar. The post has no measured results; for speedups, see part two.