AIRoA-0: Long-Horizon, Precise, Cross-Embodiment Manipulation from Handheld Data

Yueh-Hua Wu, Kei Ota, Hanlong Li, Hayata Yuasa, Jason Wang, Kengo Ikeuchi, Kensuke Chiyoma, Makoto Sugiura, Masafumi Takahashi, Masatoshi Tateno, Naoaki Kanazawa, Nicola Scianca, Petr Khrapchenkov et al.
33 minread4.65scoreNo. 111-05 Embodied AI issue
Contents · 9 sections1What problem is it actually solving?2The pipeline: four parts, one job each3How actions are represented: only the gripper counts4How can it do long tasks without seeing the whole scene?5How does inference keep motion smooth and fast?6What the results show7Limitations as I see them8How a robotics team could try it9Verdict
Rel 5Nov 4Prac 5Evid 4Time 5

Worth a close read for teams building robot data and policies: it answers a more basic question, whether demonstrations recorded by people alone, with no robot data, can train a usable bimanual manipulation model. Robot imitation learning usually starts with teleoperating a robot to collect data; AIRoA-0 collects no robot data at all. It is trained only on 16,151 hours of bimanual demonstrations that people recorded with a handheld gripper, and the robot shows up only at test time. At a site where none of its training data was recorded, using just one camera on each wrist, it sorts small parts, swaps the mop pads and brush of a robot vacuum, folds a towel and more, averaging 78% success over ten tasks. Post-trained on the same task demonstrations, the best of four other vision-language-action models (VLAs) reaches only 27%. Data collection no longer has to be tied to one robot: any robot that mounts the same gripper can run the model.

What problem is it actually solving?

Teleoperation data scales with the number of robots: a lab can only collect as much as its robots and operators allow. A handheld gripper turns this around. The paper uses YUBI, the team's own open-source handheld gripper. A person holds one in each hand and does the task as usual; each gripper has a wrist camera and records its position in space and how far it is open. Later the same gripper is mounted on a robot arm as its end effector, so what the robot sees and does matches what the person recorded.

The catch is that this data is hard to learn from. Here is what each difficulty looks like in the data and at deployment (illustrative):

DifficultyWhat it looks like in the dataWhat happens at deployment
Only people actNo clip shows a robot failing and recoveringThe robot meets surprises with no reference
Only wrist camerasTwo partial views, never the whole tableOne view can match several steps of a task
Many people, many placesOver a hundred tasks, varied backgroundsThe test site is always unseen

The paper's approach is to handle each difficulty at the layer of the system where it arises.

The pipeline: four parts, one job each

A-0 keeps the π0.5 backbone: a SigLIP image encoder, a Gemma 2B vision-language backbone and a 300M action module trained with flow matching, initialised from π0.5's public weights. On top of it there are four parts:

PartWhat it doesWhich difficulty it addresses
Data processingCleaning such as filtering and outlier detectionCleans data from many operators before training
Pre-trainingVisual budget gate plus subtask predictionBackground clutter, no global view
Post-trainingStage prediction plus memory of recent motionNot knowing which step of a long task it is on
InferenceSmoothed stage prediction plus streaming decodingSmooth motion that still reacts fast

The paper's model figure below puts these parts together; the sections after it follow the data flow.

The A-0 model: white boxes are the original π0.5 parts, light blue boxes were added in pre-training, dark blue boxes in post-training. The two wrist views pass through the visual budget gate before the backbone; the stage head reads the backbone's output and predicts the current stage; the last ten gripper states are compressed by the memory module and also enter the backbone.
The A-0 model: white boxes are the original π0.5 parts, light blue boxes were added in pre-training, dark blue boxes in post-training. The two wrist views pass through the visual budget gate before the backbone; the stage head reads the backbone's output and predicts the current stage; the last ten gripper states are compressed by the memory module and also enter the backbone.

How actions are represented: only the gripper counts

Why single out the action representation? Because the gripper is the only thing a person and a robot share. A person has no robot base and no joint angles. A-0 describes state and action only through the grippers. Besides the two wrist views and the instruction, the model reads a gripper state: the pose of one gripper relative to the other, plus the openings of both grippers, 11 numbers in all. Each action is the pose a gripper should reach next, written relative to where the gripper was when the frame was taken.

An example (illustrative): when the frame is captured the left gripper sits somewhere, and the model outputs a chunk of future actions. The first says "1 cm forward from the current pose", the fifth says "4 cm forward and rotated 10 degrees". Every action is a target relative to the same starting point rather than a small increment on the previous one, so errors do not pile up along the chunk. And because no base or joint appears anywhere, the weights post-trained for a task run unchanged on a bimanual Franka, the TX-G2 humanoid, the open-source OpenArm and a pair of UR (Universal Robots) industrial arms. The paper does not explain how, under the streaming decoding described below, targets in the chunk still awaiting execution are re-expressed from the newest pose when the next frame is captured.

How can it do long tasks without seeing the whole scene?

Visual budget: the model learns where to look

Most of what a wrist camera captures is irrelevant to the task, and the irrelevant part is exactly what changes from site to site. Before each image token enters the backbone, A-0 applies a gate: a small network scores the token against the task instruction and outputs a keep probability, and the token is scaled by it. A budget loss keeps the average keep probability over an image close to a target, which is 0.9 in post-training.

The budget only fixes the average; which regions keep more weight is taught entirely by the action loss, with no labels saying where to look. Across 25 subtasks the gate keeps the grippers and the objects being handled while the table, background and rest of the room fade.

Post-training uses the gate a second time. When the action module computes attention, a bias is added to its score for each image token, and the bias is derived from that token's keep probability. With a keep probability of 1 the bias is 0 and the score is unchanged; with a keep probability near 0 the bias is near −4, so the token gets far less weight after the softmax. Both uses are on at run time.

Visual budget gate weights on four subtasks: each tile shows the left and right wrist views; contours mark regions with high keep probability and faded areas are down-weighted. Weight concentrates on the grippers and the objects being handled.
Visual budget gate weights on four subtasks: each tile shows the left and right wrist views; contours mark regions with high keep probability and faded areas are down-weighted. Weight concentrates on the grippers and the objects being handled.

Two levels of language and stage prediction

Every demonstration carries two levels of text: a task instruction for the whole job ("Replace the parts of the robot vacuum cleaner") and a subtask instruction for the current few seconds ("Remove the first used mop pad and place it into the box"). The corpus has 10.65 million subtask segments, 5.5 seconds each on average.

In pre-training each sample is paired with one of the two, the task or the subtask instruction, so the model can follow one sentence for a whole job or step-by-step commands. When it gets the task instruction, the backbone must also predict the current subtask as text. That prediction is used only in training and is never fed to the action module. Its job is to force the backbone to encode where the task stands, since the action module reads the same representation.

Post-training goes further. A long task is split into stages, each a run of consecutive subtasks. Each stage has a written description. Parts kitting, for example, has three stages, and the first one reads "Place the fingers, flaps and gears in cells 1, 2 and 3". Post-training adds a stage head. It reads the backbone's representation of the current views and predicts which stage the task is in. The description of the predicted stage is handed to the action module as text. This is the same form as the subtask instructions it learned to follow in pre-training. In the paper's model figure, this link is labelled "stage as text". The paper does not say where the text of the stage prediction enters the action module. It says only that each query encodes the observation once. Labels come from the subtask annotations already in the data. It is needed because two wrist views often fit several steps of a task; without naming the step, the action module has to guess inside its action distribution.

Hoot saysPre-training vs post-training: where the action module's sentence comes fromClose ✕
Post-training lets the model say for itself which step it is on. The action module still hears one sentence, in the same form as in pre-training.

Pre-training

action module hears
the task or subtask instruction written in the data
who supplies it
the data annotation
extra job of the backbone
given a task instruction, predict the current subtask text, used only in training

Purpose: force the backbone to encode how far the task has got.

Post-training

action module hears
the description of the predicted stage
who supplies it
the stage head, reading the backbone's representation of the current views
training labels
subtask annotations already in the demonstrations, no extra labelling

Purpose: tell the action module the step when the two wrist views cannot.

Left is pre-training, right is post-training. The action module always follows one written sentence. What changes is who supplies it. In post-training the stage head decides by itself, so a long task needs no step-by-step prompting.

Sourcepaper §5.1 and Fig. 7 (stage as text)

Post-training also adds a memory of recent motion. The memory holds the last ten gripper states, the same 11 numbers as above. During recording they come from tracking the handheld grippers; on the robot they are measured values. Each state is tagged with its age. Four learned queries, conditioned on the instruction and the current state, attend over these ten states and compress the whole history into four tokens in the prompt. Why record time rather than step count? Robots query a policy at different rates. Training samples the history at 5 to 15 Hz, so one policy works at any query rate in that range. A quarter of the time the memory is withheld so the model can still act without it.

How does inference keep motion smooth and fast?

The stage is predicted afresh at every query, and one wrong prediction would send the action module to another step. So inference takes an exponential moving average of the stage probabilities and uses the most likely stage. A one-off error barely moves the average, while a real transition is followed.

Actions are produced by streaming decoding, an approach taken from FlashVLA. Standard flow matching starts each chunk from pure noise and runs the action module ten times. Streaming decoding keeps a buffer of chunks at different noise levels; at each query the action module removes part of the noise from every chunk in the buffer using the newest frame. The front chunk loses its last bit of noise and goes to the robot's controller, which follows the chunk at its own rate, and fresh noise enters at the back. The paper does not say how buffered chunks are converted to the newest gripper pose; it says only that the runtime harness keeps them consistent.

In FlashVLA the noise levels of the chunks are evenly spaced, so the size of each step is set by the number of slots. With few slots, the last step before execution removes too much noise and seams appear between consecutive chunks. With many slots, each chunk is settled on frames taken long before it runs, and the policy reacts slowly when the scene changes.

A-0 makes the steps uneven: the closer a chunk is to execution, the smaller its correction. It uses two slots with noise levels 0.1 and 1. A new chunk takes one big step from pure noise down to 0.1, and just before it runs it is corrected from 0.1 to 0 on the newest frame.

Each query is also cheaper. By the paper's calculation, in steady state each query encodes the observation once and runs the action module twice, against ten times for standard decoding. A query takes about 60 ms on an H200.

What the results show

The main test runs on a bimanual Franka at a site with no training data. The table reports each task's average success rate (%) over repeated trials. The four baselines start from their public weights, are post-trained on the same demonstrations, and report their best result over many settings per task; A-0 uses one recipe for all ten.

TaskPost-training data (h)A-0π0.5XR-1GR00T N1.7MolmoAct2
Towel folding3.373222108
Letter blocks3.484130240
Orange pyramid3.4902521328
Parts kitting4.0833013712
Colour sorting4.19745392019
Robot vacuum5.37311413
Fuel bracket6.39334334711
Pencil case8.87336311523
Cup pyramid17.98071112336
Earbuds25.8301318103
Ten-task average7827231614

A-0 is best on every task. The gap is especially clear on multi-minute tasks: on servicing the robot vacuum, the best baseline gets only 14% and A-0 73%. This is the result of the whole system; the table does not test the parts separately.

Post-training data per task is small: six of the ten tasks use three to six hours. That means that once the model is pre-trained, adding a new task takes only a few hours of handheld demonstrations.

Four moments of one autonomous robot-vacuum run: removing an old pad, fetching a new one, both new pads on, pulling out the brush. The picture covers two minutes of one autonomous run, with a single instruction given at the start.
Four moments of one autonomous robot-vacuum run: removing an old pad, fetching a new one, both new pads on, pulling out the brush. The picture covers two minutes of one autonomous run, with a single instruction given at the start.

No training demonstration shows a robot facing a disturbance, yet A-0 acts on what it sees. During colour sorting, a person pushes the bins aside while the arm carries a green part, leaving the red bin under the gripper. The arm moves over to the green bin in one continuous motion and drops the part there. It places parts by where the bins appear in the image, not where they used to be, and non-uniform streaming decoding keeps the motion continuous each time a query brings a new frame.

Bins pushed during colour sorting: carrying, bins pushed, redirecting to the green bin, placed, in about 5 seconds.
Bins pushed during colour sorting: carrying, bins pushed, redirecting to the green bin, placed, in about 5 seconds.

The most direct evidence on whether handheld data can win through scale is the data-scaling experiment. The authors pre-train on all, half, a quarter, an eighth and a sixteenth of the data, take each run's checkpoint at 100,000 steps, and post-train it on the same 16 task sets. None of those sets' demonstrations was used in any pre-training run, and every run is post-trained with identical settings. Models with more pre-training data keep a lower loss after post-training, and the gap does not close. Loss follows a power law in hours of pre-training data: from about 1,000 to 16,000 hours, each doubling lowers post-training loss by about 2.4%. Recording more handheld data keeps making the model better, which is what this approach depends on.

Limitations as I see them

First, the comparison is not fully clean. Baselines were tuned to their best settings, which works against A-0, but A-0 is pre-trained on the same handheld corpus while the baselines were pre-trained elsewhere, so the table cannot separate data from method. The success table compares only full systems, with no single-module ablation.

Second, the scaling law is about loss, not success. The 2.4% is held-out action loss; the scaling experiment reports no matching real-robot success rates.

Third, the other robots get showcases, not numbers. TX-G2 is shown on 4 tasks, OpenArm on 3 and the UR arms on 2, but success rates were measured only on the Franka. How much success drops on these three robots is unknown.

How a robotics team could try it

  1. First mount the YUBI gripper and wrist camera on your own arm as its end effector, since A-0 can only drive robots set up this way. Then take the public weights and recipe, record a three-to-six-hour task with handheld grippers at your own site and post-train; one eight-GPU H200 node takes under four hours. Check whether success approaches similar tasks in the paper.
  2. Record the same task by teleoperation as a control, to see how far handheld data falls short on your arm.
  3. Try stage prediction on a long task: cut stages from the subtask labels and compare where the policy gets stuck with and without the stage head.

Verdict

DimensionAssessment
Problem importanceHigh. Data collection is the biggest bottleneck in embodied AI, and this shows usable data without robots
NoveltyMedium. Most parts come from prior work; the contribution is a system that handles long tasks
EvidenceMedium-high. Ten real tasks with success rates, but only on the Franka; the other three robots are only showcased, and there is no module ablation
Engineering costMedium. Data and weights are open; post-training takes four hours on one eight-GPU node
Worth tryingYes. Teams with handheld grippers can try the recipe directly

The key lesson: once state and action are defined on the tool that people and robots share, data recorded by people becomes robot training data directly, and its scale follows the number of people recording, not the number of robots.

How did this read for you?

One tap, no sign-in; tap again to undo. Only anonymous counts are kept, never who you are.

Discussion

Add experiment details, reproduction notes or a different take. Requires a GitHub sign-in.

Loading the discussion…