Contents · 10 sections
1The six steps CTR performs before training2A task-specific processing program decides which timings can move3After shifting, the same motion is paired with more kinds of opposite-arm progress4Foreground composition builds the main camera view, and IdleMask masks artificial waits5The probe experiment: dependence on the opposite arm drops in independent phases6Starting from a different progress state, worst-case success rises from 56% to 76%7CTR holds up under pauses and progress combinations absent from training8Where cooperation is required, violations drop to 0% and 4%9Does the gain come from rescheduling itself, or from the composed views and IdleMask?10Under normal conditions 19 of 24 combinations improve, and Place Dual Shoes does notSHARE
Bimanual robot policies learn a bad habit, "if the other hand has not moved, this hand waits too." This paper traces the cause to overly uniform timing in demonstrations (recordings of a person operating the robot), and its fix adds no new demonstrations: before training, it re-staggers the recorded motions in time within the range the task allows.
Bimanual imitation learning works like this. A person operates two robot arms through a task several times, the system records the camera views and the motions of both hands, and a policy model then predicts the next motions of both hands from the current views and the joint angles of each arm. The article uses one illustrative scene throughout: the robot must put an object into a drawer, the right hand pulls the drawer open, and the left hand picks up the object and places it inside. The task has only one hard requirement, which is that the drawer must already be open before the left hand places the object. When the left hand picks up the object has nothing to do with whether the drawer is open.
The difficulty is that a demonstration records "what to do" and "when it happened to be done this time" on the same timeline. Suppose the demonstrator always opens the drawer with the right hand first and only then grabs with the left (illustrative). Then in every demonstration, the drawer is already open when the left hand starts grabbing. For the model, "the drawer is open" becomes a perfect signal for predicting "the left hand should grab now," and using it never makes the training loss worse. At deployment, if the right hand is a little slow, the left hand may simply stall. The authors call this spurious cross-arm coupling: the two hands could work independently, yet one hand's motion has come to depend on the other hand's progress.
The most direct remedy is to record more demonstrations. But if the same habit is recorded another 1000 times (illustrative), the drawer is still open whenever the left hand starts grabbing, and the new data still cannot tell whether this order is a task requirement or the demonstrator's habit. A slightly better approach mixes three schedules in training: right hand first, left hand first (one hand finishes its part before the other starts), and both hands starting together. The paper calls this baseline Mixed. In the drawer example, Mixed provides only three time gaps, so an intermediate state such as "the left hand is a quarter into its grab while the right hand has pulled the drawer two thirds open" never appears in training.
Uniform timing is common in public datasets. The authors examined four bimanual datasets, RoboTwin 2.0, RoboCOIN, DexMimicGen, and BiCoord. Of 716 tasks, 266 (37.2%) contain independent motions whose order could be swapped, meaning their schedule could change, yet only 3.4% of those 266 tasks show more than one admissible schedule in the data. This means most of the possible schedules were never recorded, so a model trained directly on existing datasets rarely sees them.
The paper's CTR (Constraint-preserving Temporal Recombination) changes the training data: the model architecture stays the same and no demonstrations are added. Before training, CTR cuts each hand's recording into segments at the detected interactions. Each segment keeps its content and duration, and only its start time changes, within the range the task allows. In the drawer example, the left hand's grab sometimes starts before the drawer is opened and sometimes runs alongside the drawer pull, while the left hand's placement always comes after the drawer is open.
The most intuitive numbers come from the Sort Letters task on a real robot, where the two hands each sort different letters. The authors trained the vision-language-action model π0.5 (it takes camera views and a text instruction as input and outputs motions), then deliberately paused one hand for 4 seconds partway through execution. When trained only on "both hands start together" data, the other hand's completion was delayed by 2.56 seconds; when trained on CTR data, it was delayed by only 0.25 seconds. This means that on a task where the two hands do not depend on each other, the CTR-trained policy mostly stops making one hand wait for the other.
The six steps CTR performs before training
All of CTR's work happens before training. At deployment the policy still receives ordinary camera views and joint states and cannot read any constraint annotations. The table below traces one demonstration through the drawer example.
| Step | Input | Operation | Output | When |
|---|---|---|---|---|
| Generate the processing program | Task description | GPT-6 Astra (a large model from OpenAI) writes a video-processing program specific to the task | Processing program | At task setup, once per task |
| Segment and find constraints | One demonstration | The program detects interactions, cuts each hand into ordered segments, and lists the required cross-arm relations | Segment table and relation table | During data construction |
| Sample a new schedule | Segment table and relation table | Keep the order within each hand, freeze the time gaps of constrained segments, and move the other segments randomly | New start time for each segment | During data construction |
| Recombine the data | New start times and original recording | Each hand reads its original clips at the new times, and the main camera view is built by foreground composition | One new demonstration | During data construction |
| Mask artificial waits | New start times | Mark the waits inserted by the shifting; motions at those positions carry no imitation loss | IdleMask | Generated during data construction, applied during training |
| Train | New demonstrations and IdleMask | Train the existing DP (Diffusion Policy), ACT (explained below), or π0.5 as usual | Policy model | During training |

A task-specific processing program decides which timings can move
The second step in the table decides whether the method is safe. "The drawer must open first" and "the demonstrator habitually moves the right hand first" look identical on the timeline, and the recorded timing alone cannot separate them, so the decision must come from task knowledge outside the timing. CTR's task knowledge comes from the processing program GPT-6 Astra writes at task setup. GPT-6 Astra is called only once per task, after which this program processes all the videos; GPT-6 Astra does not label individual videos.
In the drawer example, the processing program would produce something like the following for a 10 second demonstration. The numbers and format are illustrative, and only the segments where the hands are actually working are listed:
Right hand segments: open drawer [0, 3]
Left hand segments: grab object [3, 7], place [7, 10]
Required relation: place must come after open drawer; the start time gap between the two segments is fixed at 7 seconds
CTR does not change the start time gap that the two constrained segments had in the original recording (7 seconds here), so open drawer and place can only move together. In this demonstration the only segment CTR can move on its own is the left hand's grab. The grab keeps its content and its 4 second duration, and its start time can move from second 3 to second 0, 1, or 2, because it only needs to finish before the placement at second 7. The paper's implementation samples this offset uniformly within the admissible range.
Why not move every segment at random? If the placement could also move freely, CTR might create a demonstration where "place comes before the drawer opens," and the model would learn to push the object into a closed drawer.
CTR is even more conservative than "only require the order": the two constrained segments keep even their time gap. In the illustration, keeping the drawer opening fixed and starting the placement 1 second later would also be legal, yet CTR does not sample that schedule. It gives up some legal schedules so that the recombined data is sure not to violate task constraints.
If the segmenting and constraint-finding step goes wrong, the error goes straight into the training data. If the processing program detects the wrong interactions, cuts segments at the wrong boundaries, or writes the wrong relation logic, CTR will create invalid recombinations. So CTR also adds task-specific geometry, contact, and observation-consistency checks as conservative filters. For example, if the new combination of the two hands' poses after shifting cannot be confirmed collision-free, the schedule is discarded.
After shifting, the same motion is paired with more kinds of opposite-arm progress
Focus on one short motion, "the first second of the left hand's grab." In the four schedules below, this left hand motion is exactly the same. Only the right hand's progress at the same moment differs (illustrative). Progress is the fraction of the 3 second drawer segment that has elapsed, where 0 means it has not started and 1 means it is finished:
| Schedule (illustrative) | Left grab starts at | Right drawer opening starts at | Right progress when the left hand is 1 second into its grab |
|---|---|---|---|
| Right first (original recording) | second 3 | second 0 | 1 |
| Left first | second 0 | second 4 | 0 |
| Both start together | second 0 | second 0 | 1/3 |
| One schedule CTR sampled | second 1 | second 0 | 2/3 |
The first three rows correspond to the three schedules of Mixed. In the illustration each schedule has only one fixed time gap, so the same left hand motion has only been paired with three right hand progress values. CTR can take other values in between within the admissible range, such as the 2/3 in the fourth row. My understanding is that once the same left hand motion is paired with many right hand progress values, the right hand's progress can hardly help the model predict the left hand's next move, and the model cannot lower its loss much through this shortcut.
The paper calls the pair (left hand progress, right hand progress) the bimanual progress state, referred to below as the progress combination. The experiments compare five training schedules. L→R is one serial order with the left hand first (one hand finishes its part before the other starts). Serial contains both serial orders, left then right and right then left, Concurrent has both hands start together, Mixed is the two serial orders plus the concurrent start, and CTR samples relative timing across the admissible range.

Foreground composition builds the main camera view, and IdleMask masks artificial waits
First, the views. Each hand's joint data and wrist camera views can be read from the original recording at the new times. The main camera, however, films both hands at once, and the view needed after shifting simply does not exist in the original video.
Illustration: when CTR samples "the grab starts at second 1," at second 2 of the new timeline the left hand needs the content of original second 4 and the right hand needs the content of original second 2. At original second 2 the left hand has not yet started grabbing, and at second 4 the drawer is already fully open, so no single frame satisfies both sides.
CTR's solution is to use foreground segmentation to cut out the two hands and the relevant objects and paste each onto the scene background at its own original time. The composed image only approximates a real simultaneous scene, so contact-sensitive segments and coupled segments (for example, both hands acting on the same object) are protected and do not take part in rescheduling. The left hand grasping the object on its own involves only that hand and the object, so it is a segment that can be shifted on its own.
Next, the waits. In the schedule above, the left hand does nothing from second 0 to 1 and from second 5 to 7, waiting 3 seconds in total (illustrative). Both gaps come from the recombined schedule: one before the grasp, one between the grasp and the placing. The demonstrator never waited on this schedule; these waits are produced by the splicing. If the imitation loss were computed as usual, 30% of the left hand's motion labels would be teaching the model to stand still.
IdleMask masks the motion dimensions and time steps that correspond to these waits inserted by recombination, so the imitation loss is zero at those positions. The masking applies only to artificially inserted waits. The right hand's hold from second 3 to 10 after opening the drawer is a final hold, and holds and support the task requires are treated the same way; all of these still count toward supervision. Hoot says
- constrained segment pair: relative offset frozen
In columns t3 and t4 the working left arm is still trained, and only the right arm's two rows are masked. The right arm is also still at t6 and t7, but that holding was in the recording, so it stays supervised.
- actions that should be supervised but are lost
This grid is the misreading, and the paper does what the other grid shows. Dropping a frame whenever one arm waits also removes supervision for the other arm's valid actions.
- action from a recorded segment: counted
- required or terminal holding: counted
- artificial wait: loss zeroed
- Count the cells (illustrative): 4 coordinates × 10 timesteps = 40 cells. Each arm has 3 artificial-wait steps and 2 coordinates, so 3×2×2=12 cells are zeroed and the other 28 are counted, and the loss denominator also counts only those 28.
- Dropping whole frames: in each of the first 5 steps one arm is waiting, so 5×4=20 cells go, 8 more than should be masked, and those 8 are exactly the cube grasp and the drawer opening.
- Masking everything that is still: the two arms have 6 holding steps in total, so 6×2=12 more cells lose supervision, and 2×2=4 of them record the right arm staying still while the left arm places the object.
Illustrative: one retimed drawer-task timeline with 10 timesteps and only 2 action coordinates drawn per arm; the segments and step counts are illustrative. While an arm waits, its state, wrist image, main-view foreground, and action label are all read at one frozen recorded moment, so it holds a recorded pose. This timeline is only one sample from the admissible set Ω(τ): it must satisfy the constraint annotations and then pass geometric, contact, and observation-consistency checks, or it is rejected. The required-versus-reschedulable logic lives in the program GPT-6 Astra generates from the task description, and neither is queried at deployment.
Source
paper §4.2–§4.3; Appendix F algorithm (program generated from task description ξ); F.3 Eq. 16 (during waits the source-time map stays at the first segment's start or the previous segment's end); F.5 Eqs. 18–19 (mask indexed by batch element, prediction step, arm, and action coordinate; the denominator counts only unmasked terms); Appendix G (waits hold a recorded pose; geometric, contact, and observation-consistency checks). The timeline, coordinate count, and cell counts are illustrativeThe probe experiment: dependence on the opposite arm drops in independent phases
The first test measures motions directly. The authors ran a probe experiment on the real robot Deliver Water task, which has three phases: when placing the bottle the two hands are independent, when pouring water they must be synchronized, and when returning to standby they are independent again. The probe fixes one hand's state and observations, changes only the opposite side's state and observations, and measures how much this hand's predicted motion changes. A larger value means this hand's motion is more influenced by the opposite side.
ACT (Action Chunking with Transformers, an imitation learning policy that predicts a whole chunk of motion at once) trained on Concurrent data has sensitivities of 13.28, 5.82, and 9.77 in the place, pour, and return phases. This means that in the two phases that do not need to track the opposite side, the model is actually more influenced by it than in the pouring phase, which requires synchronization.
ACT trained on CTR data scores 2.79, 4.19, and 1.07 in the three phases. The two independent phases are pushed very low. The pouring phase also drops from 5.82 to 4.19, but it stays higher than CTR's own two independent phases. This means CTR mainly lowers dependence in the independent phases, while the synchronized phase keeps a stronger response than the independent ones. This probe only shows how motion predictions change; task success is covered in the next section.

Starting from a different progress state, worst-case success rises from 56% to 76%
The probe in the previous section shows that the policy's motion predictions change with the opposite side's progress. If the progress combination at the start is rare in training, this dependence may make the task fail. The authors tested six starting states on the real robot Sort Letters task with π0.5. The parentheses in the table header are (left hand progress, right hand progress), where 1 means this hand has finished its independent part.
| Training data | C00 standard start (0, 0) | Cmid both halfway (0.5, 0.5) | L-lead (0.75, 0.25) | R-lead (0.25, 0.75) | CL (1, 0) | CR (0, 1) | Worst success rate |
|---|---|---|---|---|---|---|---|
| L→R | 90 | 82 | 36 | 72 | 92 | 98 | 36 |
| Serial | 92 | 0 | 18 | 50 | 96 | 88 | 0 |
| Concurrent | 92 | 92 | 54 | 52 | 42 | 28 | 28 |
| Mixed | 74 | 94 | 58 | 56 | 96 | 74 | 56 |
| CTR | 94 | 94 | 76 | 96 | 100 | 100 | 76 |
The table shows success rates (%). Each cell has 50 trials, so one success or failure is 2 percentage points.
L→R, Serial, and Concurrent all reach 90% to 92% from the standard start, yet their worst states fall to 36%, 0%, and 28%. This means the success rate from the standard start cannot reveal the problem. Serial has seen both serial orders, yet it never succeeds at Cmid, where both hands are halfway, which means seeing two orders is not the same as seeing the progress combinations in between.
Mixed raises the worst success rate to 56% and CTR raises it to 76%, which means covering more schedules helps. Comparing the two, R-lead, where the right hand leads, rises from 56% to 96%, CR (the last state in the table, right hand finished, left hand not started) rises from 74% to 100%, and both score 94% on Cmid. This means mixing three fixed schedules is not enough; once the intermediate schedules beyond those three are covered, success at the worst starting state is higher.
On the simulated Put Object Cabinet (an object-into-cabinet task of the same kind as the drawer scene) the gain is smaller, because the worst success rate rises from 48% for Mixed to 56% for CTR. On this task Concurrent scores 98% at Cmid, yet 0% on all four states where the two hands' progress is asymmetric. This means a policy that has only seen concurrent starts may fail completely once the progress of the two hands is out of sync.
CTR holds up under pauses and progress combinations absent from training
The six starting states might happen to fall within CTR's training coverage, so the authors ran two more tests that do not appear in training.
The first is a pause during execution. On Sort Letters, the authors pause one hand externally and measure how much longer the other hand takes than without the pause. Because the other hand's task requirements have not changed, ideally it should not slow down at all. With a 4 second pause, L→R is delayed by 5.12 seconds, Concurrent by 2.56 seconds, and CTR by 0.25 seconds.
The authors also fit a straight line of the extra time against pause length over four pause lengths from 1 to 4 seconds. A slope near 1 means the opposite side slows down by as long as the pause, and a slope near 0 means it is barely affected. The three slopes are 1.232, 0.758, and −0.042. The L→R slope is above 1, which means its policy passes the pause fully to the other hand and even slightly amplifies it; the CTR slope is slightly below 0, which can be read as passing almost nothing. Back in the drawer example, this is like the right hand getting stuck for 4 seconds while pulling the drawer, with the left hand's grab barely delayed (illustrative).

The second test is stricter, because the progress combinations created by a pause may still have been seen by CTR in training. On Sort Letters, the authors carve out one progress region: combinations with left hand progress from 0.20 to 0.30 and right hand progress from 0.70 to 0.80. Each of these single-hand progress values was seen in training; only the combination was not. The CTR-hole version deletes the training samples that fall in this region, CTR-full deletes nothing, and the three schedules of Mixed never reach this region in the first place.
Starting from within this region, CTR-hole succeeds 74% of the time, Mixed 58%, and CTR-full 76%. This means that after deleting this region, CTR's success rate drops by only one of 50 trials compared with not deleting it, and it stays above Mixed, so the benefit does not rely entirely on having seen the test states in advance. From the standard start the three score 92%, 74%, and 94%, so CTR-hole did not get noticeably worse from losing this data either.
Where cooperation is required, violations drop to 0% and 4%
The biggest risk of scattering the timing is scattering the dependencies that are truly needed along with it. The authors tested this directly with π0.5 on two real robot tasks.
The first task is Put Object Cabinet on the real robot, of the same kind as the drawer scene. The authors close the drawer before the left hand places the object; the correct behavior is then to wait for the drawer to reopen, and forcing the placement counts as a violation. The training data of the L→R baseline never contained waiting in front of the drawer, and it violated the constraint in 98% of trials; the CTR-trained policy waited in front of the drawer every time, a violation rate of 0%.
In the second task, Organize Hardware, the two hands handle different hardware parts but share one workspace, and while one hand occupies it the other may not enter. The violation rate drops from 36% for the Concurrent baseline to 4% for CTR. The two tasks use different baselines, so they cannot be combined into one total.
The cost falls on release latency, meaning the time between a condition being restored and the arm starting to move. CTR goes from the baseline's 0.01 seconds to 0.27 seconds on the drawer task, and from 0.09 seconds to 0.15 seconds on the workspace task.
This means that while CTR pushes violations down, it starts a few tenths of a second later after a condition is restored. Back to the segment table above: in every recombined sample, the placement always comes after the drawer opens, and this relation is never shuffled. Together with the Deliver Water probe results, one can say that the response to the opposite side dropped in independent phases while the response remains where coordination is needed.
Does the gain come from rescheduling itself, or from the composed views and IdleMask?
CTR introduces three things at once, rescheduling, main camera composition, and IdleMask, and the authors tested the latter two separately.
When only main camera composition is added to L→R, with no rescheduling at all, the success rate on Sort Letters is 90%, the same as without it. The simulated Scan Object task requires the two hands to hold a scanning pose together, and its success rate goes from 68% to 62%. This means the composed views alone cannot explain the gains on these two tasks.
Removing only IdleMask from CTR takes Sort Letters from 94% to 92%, one trial out of 50. Scan Object goes from 70% to 56%, a clear drop. This means masking supervision on artificial waits also contributes, especially on Scan Object.
Under normal conditions 19 of 24 combinations improve, and Place Dual Shoes does not
The last question is whether CTR improves things under ordinary execution with no interference. The authors compared it with Serial on 8 tasks and 3 model families, and 19 of the 24 task-model combinations improved. This number needs context: DP and ACT are close to 0 on many tasks to begin with, for example on Organize Hardware DP goes from 0% to 2%, just one more success, and ACT stays at 0%. Gains like these say little.
The π0.5 gains are more telling: simulated Put Object Cabinet goes from 36% to 66%, and real robot Deliver Water from 78% to 92%. This means that when the model itself can complete the task, changing the training schedule also improves success under normal conditions. On Place Dual Shoes (a task where the two hands place shoes with overlapping placement areas) CTR shows no gain. These 19 gains are relative to Serial, not to Mixed.
Back to the drawer: CTR only changes "when the left hand starts grabbing," and the requirement "the drawer must be open before placing" is always kept. This depends on a person or program first writing the task constraints correctly. If your demonstrations also follow a single habitual order, the success rate from the standard start will not reveal the problem; starting from a different progress combination, or pausing one hand and measuring how much the other slows down, will quickly show whether the policy has learned this shortcut.
How did this read for you?
Hoot made a note. Thanks for the feedback.
One tap, no sign-in; tap again to undo. Only anonymous counts are kept, never who you are.
Related papersMore from Paper Daily
Embodied · 2026-11-03Contact Imitation Interface: Transferring Dexterous Skills by Showing How Contact UnfoldsTeaching a robot hand from human demonstrations is hard because human and robot hands have different shapes. If the conversion only tries to make the robot hand look like the human hand, the robot's fingertips often touch the object in different places than the human did, and grasps and pushes fail. CIMI puts contact first. During demonstrations it records the contact surfaces between fingers and…
Embodied · 2026-11-02UMI-Live for In-The-Wild Bimanual Robotics Data CollectionMost of the training data came from handheld grippers, collected at a different place and under different lighting than the test scene, with only a small share of real-robot teleoperation. Fine-tuned on this mix, pi0.5 succeeded in all 10 trials of a bimanual task that puts oranges into a bowl. The demonstrations come from UMI-Live, an open-source bimanual handheld data-collection system released…
Embodied · 2026-10-27Coupling Execution Horizon and Sampling Depth in Flow-Matching VLA PoliciesIn flow-matching vision-language-action models, solver steps and execution horizon turn out to be coupled, and the best setting can reverse. These models start from random noise and use a numerical solver to push it step by step into a smooth action chunk. Deployment usually tunes the two separately: the horizon amortizes inference latency, while solver steps stay high for accuracy. Under frequen…
Discussion
Add experiment details, reproduction notes or a different take. Requires a GitHub sign-in.
Loading the discussion…