ATTUNE: Agentic Throughput Tuning for LLM Training

Before training a large model, engineers have to settle on a "training recipe": how the model is split across tens or hundreds of GPUs, how much computation to trade for memory when memory runs short, which low-level compute routines to use, and how data transfers between GPUs overlap with computation. Move the same model to a different GPU or framework version, and the recipe that used to be fastest often no longer is. Amazon built a system called ATTUNE that lets an AI agent able to write code and run commands tune recipes: give it a recipe that runs and a budget of GPU-hours, and it tries things, analyses bottlenecks and keeps going on its own, with no one watching. But the real point of the paper lies outside the agent: a "correctness gate" run by fixed code the agent cannot change, which rejects before launch any change that would quietly alter what is being trained. After Amazon's fine-tuning jobs for three in-house models switched to the tuned recipes, training ran on average 1.7 times as fast as before, and up to 3.9 times.

1. The problem: existing auto-tuners either cannot find the answer or "break" the training

A training recipe has many "levers" that can be adjusted. The most important is how the model is split: each layer's matrices can be cut up across several cards, different layers can be given to different cards that pass work along like a pipeline, or the data can be split so that several groups of cards each compute one share. There is more: when memory is short, whether to discard intermediate results and recompute them when needed (trading compute for memory); whether to park some data in CPU memory; which version of the attention routine to use.

Existing auto-tuners fall roughly into three kinds. The first picks from a pre-listed set of options using a formula that estimates performance. The second is "black-box" search, such as Bayesian optimisation: it does not understand how the system works, it just runs trial after trial and decides what to try next from the results, but each trial is expensive. The third is the recent kind of large-model tuner, such as OptiCo, which can read performance analyses but likewise only adjusts a fixed list of knobs.

Two observations by the authors shaped ATTUNE's design.

First, part of the speed a stock recipe leaves on the table is hidden in combinations of levers that only help when changed together, and part in levers that the fixed list does not include at all, such as switching to a different routine for distributing data among cards.

Second, some settings quietly change what is being trained, while training still finishes without errors. Two examples. In "mixture-of-experts" models, each input word is sent to a few "expert" sub-networks. A common speed-up parameter caps how many words each expert handles, and words over the cap are simply dropped and not trained on. Another parameter, judging by its name, only "sends gradients at lower precision", but in fact it also makes gradient accumulation low-precision. A tuner that only checks "does it run" would hand both settings over as "optimisations", and the trained model would differ from the original.

2. Method: the agent searches, fixed code judges

One tuning job is called a "campaign": one fixed model, one cluster, a fixed number of GPUs. Every candidate recipe trains exactly the same thing: model, data, length of each example, examples per step and numeric precision stay the same, and every run even reads the same data in the same order. Each trial runs a fixed number of steps; the first few are warm-up and not scored, and after that the score is the average number of words each GPU processes per second.

Speed measurements are noisy, so first one has to decide "how much faster counts as really faster". The original recipe runs at least 5 times on different machines to get its average speed and its standard deviation as a share of that average, $\sigma$. The noise band is the larger of 3% and $3\sigma$; a new candidate must beat the current fastest by more than this band to replace it. That way a run that happened to be fast is not mistaken for a real improvement.

The agent uses only ordinary tools: reading and writing files (including the training framework's source code), running commands, and calling the cluster's interface. A written "tuning code of practice" sets it six rules; the three most important are:

• before every launch, write down a hypothesis: which bottleneck to ease, why it should work, and how much faster it should be;
• change only one lever at a time, or one set of splitting choices that must change together, so the result can be attributed;
• before using a parameter, read its implementation in the framework source to see whether it quietly changes other parameters too, and record the code location in the ledger.

When the bottleneck is unclear, the agent runs a performance analysis: a fixed analysis program splits each training step's time into computation, data transfer between GPUs, work on the CPU side, and idle waiting, telling the agent where time is mainly being wasted.

Spending is policed by fixed code. An accounting script counts from the moment the cluster allocates machines until they are released, including failed runs and runs used for checks, and nothing launches if the budget is not enough. Re-running a failed recipe unchanged is also refused, forcing the agent to find the cause before changing it.

The correctness gate has two steps, and it is the heart of the system.

The first step happens before launch: compare all parameters and environment variables that finally take effect in the candidate and in the original recipe. Final values are compared because one parameter may quietly change others. Every change is looked up in a register that the agent cannot edit. Changes that alter what is trained (data, examples per step, optimiser, loss function, routing, numeric precision), or that trigger known software defects, are rejected outright. Changes that alter only "how it is computed" and not "what is computed", such as a different way of placing data on GPUs, are passed outright. Anything not in the register is passed on to the second step and written into the report for a person to review.

The second step happens after launch: compare the loss at the first training step, i.e. how wrong the model is on the first batch of data. At the first step the weights have just been loaded and not yet updated, so the candidate and the original face the same weights and the same batch, and their losses should be almost identical. Legitimate changes only cause tiny differences from a different order of computation, whereas weights that were loaded or split wrongly make the loss change several-fold. So the rule is that the first-step loss must be within 5% of the original's, and no loss in the whole trial may be infinite or invalid. Hoot says▾

Hoot saysWhat each gate stops, and what the first-step loss cannot seeClose ✕

The first-step loss is computed before any weight update: same weights, same batch, so legal changes differ only in order of computation. It catches wrong loading or splitting but cannot see changes that act only at update time, such as the optimiser; those must be rejected by the register before launch.

Three illustrative candidates (original first-step loss 2.3)

A: new data layout on GPUs

register passes it; loss 2.302, 0.09% off → passes

B: a harmless-looking flag

final values show examples per step changed → rejected before launch

C: a split flag not in the register

let through; weights split wrong, loss 4.6, 100% off → rejected

B is why final values are compared: the changed item alone looks like it does not touch what is trained.

Blind spots of the first-step loss

why it is comparable

no update yet, so both compute the same thing

blind: optimiser, learning rate

computed before the update, so unchanged → only the register can reject them

blind: numeric issues that appear later

covered by "no infinite or invalid loss in the whole trial"

The two gates complement each other: one checks what is computed, the other whether it is computed right.

The three candidates and their losses on the left are illustrative; the blind spots on the right follow from "no update has happened at step one" and are not listed by the paper. Source: paper §2: two-step correctness gate: before launch, compare final effective parameters and env vars against a register the agent cannot edit; after launch, first-step loss within 5% of the original, and no infinite or invalid loss in the trial

To make it truly unattended, the authors added several hooks around Claude Code: the agent cannot finish until the budget is spent or it has written down a reason to stop, with evidence; when it wants to ask a person something, it instead "takes the default and records the assumption"; unknown destructive commands are blocked. Every run is written into an append-only ledger, so the system can recover even after a crash. When a campaign ends, the agent writes down lessons, which go into the lesson library for next time after human review.

3. Result one: faster than the strongest public recipes

The authors first reproduced NVIDIA's four official benchmark recipes unchanged on their own cluster; three came within 2% of NVIDIA's published speed, showing that the starting point was fair. From those recipes ATTUNE gained another 7% to 26%, putting it 6% to 17% above NVIDIA's published figures. The authors then took four models not seen during development and tuned them starting from public recipes written by others.

How much speed can be gained depends on how far the original recipe is from the current setting. MiniMax-M2's original recipe was written for reinforcement learning on 128 cards and kept the optimiser's data in CPU memory to save GPU memory; on 64 cards doing ordinary training, that trick slows things down instead. ATTUNE turned it off and changed how the pipeline was split, raising the speed to 2.6 times the original. GLM-4.5's recipe was written for H100s; on B200s the diagnosis found the bottleneck had become transferring data between machines and launching many small compute tasks. ATTUNE therefore shrank expert parallelism so that each expert group sits on the GPUs of one machine. Every token's experts are then local, and the data exchange between experts no longer crosses machines; the routing rule and the model are unchanged, only the placement of experts on GPUs. The fastest campaign also cut tensor parallelism to 1. Llama-3.1-70B switched to a faster attention routine, and computation came to take 98.5% of each step, with almost no idle waiting. The Mixtral and 405B recipes had already been tuned by the vendor for the same hardware, leaving little to gain; Mixtral's best candidate stayed within the noise, so by the rules the original recipe was kept.

4. Result two: with the same budget, it finds better recipes than other tuners

The authors gave every tuning method the same budget: 30 times the GPU-hours of one trial of the original recipe, with the same gate and the same release rules, and 3 campaigns per model. On GLM-4.5, ATTUNE gained 33.9% on average using only 669 GPU-hours and 6.4 hours; OptiCo gained nothing yet spent 1,407 GPU-hours; random search gained 7.8%; Bayesian optimisation only 1.8%. On MiniMax-M2, ATTUNE gained 164.2%, the best rival 55.5%. This shows ATTUNE not only found faster recipes but also spent less on the tuning itself.

Most interesting is why OptiCo failed. The way of splitting the model that ATTUNE found on GLM-4.5 is actually within the 11 knobs OptiCo can adjust: rebuilt from those 11 knobs alone, it runs within 2% of ATTUNE's final recipe, but OptiCo did not find it: it only tried the relevant setting with shallower pipelines, and its data exchange between experts always crossed machines. The answer was inside the black-box search space, but without diagnosis to point the way, most attempts either ran out of memory or were slower than the current best and got discarded. This shows that ATTUNE's advantage comes mainly from "look at the bottleneck before acting", not merely from having more things it can change.

The authors also removed ATTUNE's parts one at a time to see the effect. Without the code of practice and lesson library, the speed-ups on the two test models fell from 88.0% to 66.9% and from 49.4% to 42.9%, and half the trials failed to launch properly. Allowed only OptiCo's 11 knobs, GLM-4.7-Flash got just 18.1%, because its biggest gain came from turning off "keep optimiser data in CPU memory", which is not on that list.

5. Can the gate itself be trusted

The authors tested the gate specifically. All 460 legitimate runs were passed, and every configuration error was rejected before launch. They then deliberately planted 6 kinds of errors in the code: the 2 that change the model's forward computation raised the first-step loss by over 150% and were caught. Errors that only act when weights are updated, such as leaving out part of the gradient summation, cannot be seen at the first step at all. Shifting the labels by one word when computing the loss changed it by only 2%, about the same as the largest legitimate difference. These slipped through.

The gate also proved its worth in real use. On Nemotron-3-Nano, the configuration check blocked 19 pipeline splits with "unequal numbers of layers per stage". One of them happened to be the fastest of all trials, but it loaded other layers' weights into the wrong layers, and its first-step loss was 215% to 768% higher than normal. Without the gate, this "fastest recipe" would have been handed over.

6. Limitations and practical meaning

The authors list three limitations. First, the gate only looks at the configuration and the first step, so errors that act only when weights are updated go unnoticed, and a full training run is still needed for validation before real use. Second, the lesson library comes from past tuning, and GLM-4.5 belongs to the same family as a model used during development, so some of the gain may come from "having seen something similar". Without the lesson library, however, GLM-4.5's result was within the noise of the full system. Third, ATTUNE only tunes configuration the framework exposes; it does not change low-level compute routines or code, and does not cover the part of reinforcement learning where the model generates samples. There is one more lesson: when the authors went back and checked the lesson library, about one fifth of the entries were wrong, some holding only for the job on which they were first measured.

For training-platform teams, the most useful parts to borrow are the gate and the accounting, not the agent itself. Whether tuning is done by a person or an agent, you can first build a set of checks: compare the parameters that finally take effect, look them up in an uneditable register of "changes that alter what is trained", then compare the first-step loss. That keeps parameters which quietly change training, like "drop words over the cap" or "accumulate gradients at low precision", out before launch. Budgets should count from machine allocation to release, failed runs included. If your recipe was carried over from other hardware or another task (such as reinforcement learning), auto-tuning pays off most; if you use a vendor recipe tuned for the same hardware, expect much less, as in the paper such recipes gained only 0% to 26%. The lesson library should be checked against the original records regularly, keeping only entries the model does not already know or that correct its wrong intuitions.