Contents · 6 sections
1Where the idle time comes from under synchronous colocation2The barrier can sit in front of the weight update3Walkthrough: one step on 8×H2004How much faster end to end5What this means for people who build on-policy RL training systems6LimitationsSHARE
The GPUs that finish first just sit and wait for the longest problems: how TIDE reclaims that idle timeYou are running one on-policy reinforcement-learning step on eight H200s. The model first answers a batch of math problems with the current weights, then those answers are used to update the model. On-policy here means the training samples must come from the weights at this moment.
The eight GPUs share the same hardware: they all do inference first, and only after every request finishes do they all switch to training. This setup, where inference and training take turns on the same GPUs, is called colocation. Generating a full answer with the current weights is one rollout. The problems vary wildly in length. DAPO-Math is a math-reasoning task: a single answer can run up to 32,000 tokens, the longest one observed in the paper is 28,000, and the median is only 6,000.
So the GPUs that finish first just sit idle. The paper looks at one sampled step: under colocated execution, 20.8% of that step's GPU time is idle, and the step takes 746 seconds. The GPUs still decoding do not have it easy either. As more requests finish, the remaining batch shrinks, and throughput falls with it.
This paper is from the University of Washington, Brown University, the University of California, Berkeley, and New York University. It targets the waste in synchronous, colocated on-policy RL, where long-tail rollouts free GPUs that still cannot be used. The existing approach inserts a barrier between inference and training: the whole cluster starts training only after every rollout has finished, which keeps training samples on-policy. TIDE, a training system, moves that barrier to just before the weight update. Completed rollouts can start computing gradients, as long as the optimizer step and whole-batch operations such as gradient clipping wait until every gradient has been accumulated. It does two things: it repurposes freed GPUs for training, and it migrates unfinished requests onto fewer inference engines so the decode batch can grow again. On Text2SQL and DAPO-Math, from 8×H200 to 4×L40S, TIDE is up to 1.36× faster than synchronous colocation, and its of-optimal share reaches 77.2% to 97.4%.
To stay on-policy, the barrier is only needed before the weight update; the evaluation is on single-node machines.
Where the idle time comes from under synchronous colocation
The paper runs 20 DAPO-Math steps on 8×H200, and idle time shows up in every step. The first inference engine to finish and the last one are far apart. GPUs that are already done can only watch the remaining long-tail requests, and they cannot enter training.

Idleness is only half of the waste. The other half sits on engines that are still working: as requests finish, the decode batch shrinks, and per-GPU throughput drops well below its peak. The paper calls this decode inefficiency.
The two losses add up. On Text2SQL with 8×H200, idle time is 30.0% of total run time, inefficient decoding adds another 9.8%, and phase-switching overhead adds 3.6%. On DAPO-Math with the same hardware, idle time is 15.2%, inefficient decoding 8.4%, and phase switching 1.3%. That optimum is the GPU time obtained by assuming training is already saturated, measuring inference while a single GPU keeps the batch as large as possible, then adding the actual training GPU time and the actual weight-sync time. The of-optimal share is this optimal GPU time divided by the step's actual total GPU time. The paper adds the three losses and this share so that they sum to 100%, which leaves 56.6% for Text2SQL and 75.1% for DAPO-Math.
That means the barrier keeps training on-policy, and it also locks both the finished GPUs and the shrinking batches inside the inference phase.
The barrier can sit in front of the weight update
If completed rollouts start computing gradients early, does that break on-policy training? As long as the optimizer has not stepped, the weights have not changed. TIDE defers the optimizer step, and nonlinear whole-batch operations such as gradient clipping, until all gradients have been accumulated. Training can start early; the weight update still waits until every sample has arrived.
Repurposing freed GPUs for training right away is not enough. Engines that finish hand over their GPUs, but the remaining engines still chew on the tail with a tiny batch. If trainers simply grab whatever work is ready, late long samples also pile up on the same replica. So TIDE does a second thing: it migrates unfinished requests onto fewer inference engines. When a migration fires, the system preempts in-flight rollouts and sends only the tokens already generated to the least-loaded remaining engine. It does not move the KV cache, the intermediate state stored for tokens that have already been generated. The paper says this is simpler and matches the performance of moving the KV cache, but it does not describe how the receiving engine restores decoding state, and it does not report that cost on its own. The remaining engines get a larger batch again, and the vacated GPUs can switch to training.
The moment to migrate is set by a threshold B. When the number of remaining rollouts on an inference engine falls below B, TIDE moves those requests to the least-loaded engine and repurposes the vacated GPUs for training. If B is too large, trainers starve because not enough samples have arrived yet. If B is too small, inference spends too long at a small batch. The threshold is adjusted according to whether trainers starved, and a KV-cache capacity check can veto a migration inside the step. Hoot says
The body walked three migrations in one 8×H200 step, but not how the threshold is updated. The paper starts from B=8 by default and adds or subtracts 8 after each step. The KV-cache percentages below are illustrative, so the in-step veto can be worked through.
remaining rollout count falls below the threshold → check source and destination KV caches → veto: skip this migration attempt and lower B; later attempts still use the new threshold and the capacity check / pass: migrate and switch GPUs to training → at step end, adjust the threshold for starvation
- remaining rollout count → threshold BA check fires only when the remaining rollouts on an inference engine fall to B or below. The first step starts at B=8 by default, because migrating too early starves training, which is worse than migrating too late.8 remaining ≤ B=8, the check fires
- source KV + destination KV → vetoInside the step, TIDE first adds the KV-cache occupancy of the source and destination engines, and if the sum is over 100%, it skips this migration attempt and decreases B by one step, so the destination is not filled to overflow. Later engines can still be checked against the new B and the capacity rule.60% + 50% = 110% > 100%, so B=8-8=0 and this migration attempt is skipped
- whether training starved → B for the next stepThe threshold is adjusted again at the end of the step: if any training GPU starved, B decreases by 8; otherwise it increases by 8. This step had no starvation, and the veto already set B to 0, so the next step starts from 8 again.0+8=8
- KV sum at most 100% → migrate and switch to trainingIn the next step the remaining count hits 8 again. The two engines sit at 40% and 50%, which add up to 90% and stay at or below 100%, so the requests move to the least-loaded engine and the vacated GPUs switch to training.40%+50%=90% ≤ 100%, so the migration happens
- no starvation → threshold risesAt the end of the step there is still no training starvation, so B increases by 8 to 16. The next step will fire the check once 16 rollouts remain, which is earlier.8+8=16
Paper Figure 6 compares fixed thresholds on one DAPO-Math step: migrating too early leaves trainers idle, and migrating too late leaves inference underused. Starting conservatively at 8 is how the controller first steers away from migrating too early.

Source
paper §4.1.1, §4.1.2, Algorithm 1, and §6.1 (defaults B0=8, ΔB=8)Migration also has to wait until the training backend can actually use the freed GPUs. On memory-rich 8×H200, the paper uses Megatron with tensor parallelism TP=2: a training engine needs two GPUs tied together, so freeing a single GPU cannot form a training group. On memory-tight 4×H100 and 4×L40S, it uses FSDP, data parallelism that shards model parameters across GPUs. Adding a GPU requires re-sharding, so nearby migrations can be bundled into one reconfiguration and the system pauses fewer times.
The training side is dispatched by a Train Router. At the beginning of a step it sends 64 rollouts at a time; once fewer remain, it sends at most one fourth of what is left. Using small batches all the way through is slower, because each batch has fixed overhead. Ignoring load makes late long samples pile up on one replica.
Walkthrough: one step on 8×H200
The numbers below are from Appendix A of the paper. The task is DAPO-Math on 8×H200. Each inference engine uses one GPU, so there are eight engines; training uses TP=2, which gives four data-parallel groups. As the opening already showed, under colocation this step spends 20.8% of GPU time idle and takes 746 seconds.
First, watch how colocation finishes the step. All eight GPUs do rollouts first. Whoever finishes early sits idle until the last engine is done, then they all switch to training. The finished GPUs accomplish nothing in that window; the figure marks this step as 79% utilized.
Second, TIDE fires the first migration at 249 seconds, moving 80 unfinished requests to the least-loaded engines at that moment, and switches the vacated GPUs to training. Because of TP=2, it has to free a set of GPUs that can join training immediately; a single GPU cannot form a group. Those GPUs can then train on rollouts that are already complete, and the weights still wait until every gradient has been accumulated.
Third, it moves 80 more requests at 271 seconds, and 88 more at 341 seconds. The tail is now decoded on fewer engines, at a larger batch. Decode throughput per still-active engine rises from 2,766 to 2,897 tokens per second. The tail is no longer chewed through by eight engines that each have only a few requests left.
Fourth, idle time therefore falls to 3.2% of GPU time. Counting all GPU time spent in inference mode, rollout cost drops from 1.14 GPU-hours to 0.80 GPU-hours. The step falls from 746 seconds to 595 seconds, which the figure labels 1.26×. A large share of the remaining idle time comes from training groups waiting for the slowest samples to finish.

These numbers show that, in this step, repurposing and migration work together: moving requests both raises the decode batch on the engines that remain and lets vacated GPUs enter training earlier. They cannot show how much each change contributes on its own, because the two are not separated here.
The paper also reports statistics over the full 20 steps, showing that idle time drops from 15.2% to 4.8%, and GPU time in active decoding drops from 15.48 GPU-hours to 14.82 GPU-hours, a 4.3% decrease. Those figures come from the full run; they are not derived from the single step above.
The paper runs a separate ablation on DAPO-Math, adding the mechanisms one at a time. Repurposing means an inference engine hands its GPUs to training as soon as its rollouts finish. Load balancing is done by the Train Router described above. Migration consolidates the tail onto fewer engines once the remaining request count falls below B. The KV veto is an in-step safeguard: if the KV-cache occupancy of the source engine and the destination engine would add up to more than 100%, the migration is skipped for now. The ablation sampled a fresh set of rollouts with different lengths, so the seconds in the table cannot be compared directly with the 746-second and 595-second step above.
| Added to the row above | 8×H200 sec/step | 4×H100 sec/step |
|---|---|---|
| Colocated | 689.1 (1.00×) | 1142.6 (1.00×) |
| Repurposing | 666.8 (1.03×) | 1025.5 (1.11×) |
| Training-side load balancing | 617.9 (1.12×) | (already in the row above) |
| Migration | 572.5 (1.20×) | 1052.4 (1.09×) |
| KV veto | 576.9 (1.19×) | 1002.9 (1.14×) |
(Numbers from paper Table 3. The 4×H100 run uses FSDP, whose dispatch already does centralized bin-packing, so the repurposing row already includes load balancing.)
On 8×H200, repurposing alone barely reduces step time, because large batches at the end of the step unbalance training load; adding the Train Router's load balancing reaches 1.12×. Adding migration then cuts step time from 617.9 seconds to 572.5 seconds, reaching 1.20×. That shows consolidating the tail onto fewer engines frees GPUs earlier than waiting for each engine to drain on its own. On 8×H200 the engines have ample memory, so adding the KV veto brings almost no gain: 1.19× is about the same as 1.20×. On the tighter 4×H100, migration without the veto is slower than load balancing alone, 1.09× versus 1.11×; with the veto it reaches 1.14×. This table can show which way each mechanism pushes the two hardware setups. It cannot be read as a breakdown of the 746-second and 595-second end-to-end step.
How much faster end to end
Relative to synchronous colocation, TIDE reaches 1.36× on Text2SQL and 1.16× on DAPO-Math with 8×H200. On the memory-constrained 4×L40S (Text2SQL) and 4×H100 (DAPO-Math) setups, the speedups are 1.21× and 1.17×. Text2SQL is a multi-turn tool-use task: one rollout has 1 to 5 turns, each at most 3,000 tokens, with SQL queries in between, so lengths are more spread out. That is why colocated idle time on 8×H200 is 30.0%, versus 15.2% for DAPO-Math, and there is more idle time to reclaim.

The of-optimal share rises too. The definition is the same as above: the share is optimal GPU time divided by the step's actual total GPU time, and the three losses plus this share sum to 100%, so colocation is left with 56.6% on Text2SQL with 8×H200 and 75.1% on DAPO-Math. TIDE raises that share:
| Task and hardware | Colocated | TIDE |
|---|---|---|
| DAPO-Math, 8×H200 | 75.1% | 87.0% |
| DAPO-Math, 4×H100 | 83.2% | 97.4% |
| Text2SQL, 8×H200 | 56.6% | 77.2% |
| Text2SQL, 4×L40S | 78.4% | 95.1% |
(Numbers from paper Table 2. The paper's text describes TIDE's share as ranging from 77.2% to 97%.)
The paper also compares against two other systems. RollPacker also repurposes freed GPUs for training, but it does one coarse migration of exactly half the GPUs, which often leaves training or rollout underutilized. TLT is a synchronous RL system that uses speculative decoding to speed up long-tail rollouts. To show the best speedup speculation could offer, the draft model was trained offline, and the speedup is reported against TLT's own run with speculation turned off. Speculative decoding speeds up tail decoding by 64% on Text2SQL and only breaks even on DAPO-Math. After phase-switch overhead and the draft model's memory footprint, all four evaluated runs are slower than TLT's own colocated version.
Looking step by step, TIDE is faster in all 20 DAPO-Math steps and in 91% of Text2SQL steps on 8×H200. On Text2SQL with 4×L40S, some steps are flat or slower, because FSDP has to reconfigure when GPUs are added, the steps are shorter, and the reconfiguration cost shows up when there are fewer stragglers. In aggregate it is still faster than colocation.
What this means for people who build on-policy RL training systems
To stay on-policy, you do not have to wait for every rollout to finish before training starts; placing the barrier before the optimizer step is enough.
If you already run synchronous RL in a colocated system, measure two accounts: how long finished GPUs sit idle, and how much decode throughput drops after the batch shrinks. Watching only the idle time misses the slowdown on GPUs that are still working.
If training is to start early, the optimizer step and gradient clipping should wait until all gradients have been accumulated. Do not migrate by cutting away half the GPUs in one shot. When memory is tight, first check whether the source and destination KV caches would add up to more than 100%; if they would, skip that migration, or the destination can overflow and step time can rise rather than fall. On the training side, shrink batches toward the end of the step so late long samples do not pile up on one replica.
Limitations
TIDE-FSDP's reconfiguration when GPUs are added adds 2% to 3% overhead overall.
The paper expects less systems-level benefit in fully asynchronous RL, where GPUs on both the inference and training sides can already be kept busier. Disaggregated deployments and one-step-overlap async policies could borrow the idea of flexibly splitting the pool; those settings were not evaluated.
Speculative decoding can be used together with migration, especially once migration is no longer worthwhile and the batch is already small. The paper does not measure the two together.
How did this read for you?
Hoot made a note. Thanks for the feedback.
One tap, no sign-in; tap again to undo. Only anonymous counts are kept, never who you are.
Discussion
Add experiment details, reproduction notes or a different take. Requires a GitHub sign-in.
Loading the discussion…