Zero-HLM: Generalizable Humanoid Loco-Manipulation without Real Robot Data

Teaching a humanoid a long task such as walking to a table, picking up a banana and putting it in a basket usually takes a hundred or more teleoperated real-robot demonstrations, and moving the table often means collecting them again. Zero-HLM builds all of its data in simulation instead. It scans a real room with a smartphone, rebuilds it as a simulated scene that looks like a photo and supports physical interaction, and rewrites 15 teleoperated simulation demonstrations per task into a thousand or more new ones. Each new demonstration randomly moves the robot's starting point and the positions of the object and placement target, the hand trajectory moves along with the target, and only those that succeed in simulation are kept. Fine-tuning NVIDIA's robot foundation model GR00T N1.7 on this data, with no real-robot data at all, and deploying it directly on a Unitree G1 humanoid averages 66.7% success over three tasks, above the 46.7% from training on 100 real demonstrations. At positions the real demonstrations never covered, success rises from 23% to 69%.

Robot-learning teams that care about data cost should read this one. It runs a clean comparison on whole-body humanoid loco-manipulation: the model, the controller and the number of evaluation trials stay the same, and only the source of training data changes. I think the two diagnostic experiments are worth more than the reconstruction pipeline. Action labels generated in simulation are mostly usable as they are, and the remaining gap is mainly in the images. When the low-level controller drifts during execution, moving the scene to follow the robot works better than changing the motion. Keep in mind that every experiment uses the same reconstructed scene, each expanded position is tested only five times, and absolute success is still not high.

What problem is it actually solving?

Start with a concrete scenario (illustrative). A team wears a headset in a living room and teleoperates 100 real demonstrations of "put the banana in the basket", with the start, the banana and the basket at fixed reference positions. The trained policy is then tested under other conditions:

The paper's real numbers confirm this. A policy trained on 100 real demonstrations succeeds only 22.9% of the time on Task A when the start changes, and 20% when it must put a bitter melon on a white plate. Covering these variations means teleoperating again at every new position, and on a humanoid one demonstration takes more than half a minute with someone watching.

Simulation is the usual answer to data volume, but humanoids add two hurdles. First, the robot walks around, and its head camera sees the scene from different positions and heights, so the whole room has to be rebuilt realistically; a tabletop reconstruction is not enough. The room also has to support physical interaction: drawers must open and objects must rest stably. Second, a humanoid executes motion through a whole-body tracking controller that must handle balance and contact at once. Given a reference motion, it will not reproduce it frame by frame. A few centimetres of drift at the feet and the hand can no longer reach the object.

The controller used here is SONIC, an existing whole-body motion tracking controller for humanoids. It encodes a short reference motion into a 64-dimensional vector and decodes that vector into joint targets. The trained policy also outputs this 64-dimensional vector plus open and close commands for the two hands, 66 dimensions in total, and at deployment the frozen SONIC decoder executes it.

The pipeline

The key split is this: images come from Gaussian reconstruction and physics from the mesh, and only once the two are aligned is the data both realistic and executable. The parts follow.

Step one: moving the room into simulation

Gaussian Splatting represents a scene with many small coloured ellipsoids and renders close to a photo, but a mesh exported from it has bumpy surfaces, and using it for collision makes objects snag or pass through each other. Zero-HLM has a multimodal agent build a clean mesh scene in Blender. It first places furniture according to a rough mesh exported from the Gaussian reconstruction. It then renders the Gaussian version and the mesh version from the same camera, compares the two images, and adjusts sizes and placements one by one. After import into IsaacLab, the agent configures colliders, cabinet-door joints, mass and friction, and runs tests such as opening a cabinet door and dropping an object.

With a handheld LiDAR scan as ground truth, points are sampled on both surfaces and the average distance from each point to the nearest point on the other surface is computed (Chamfer distance); smaller means the mesh fits the real room better. With Gaussian guidance the distance is 5.517 cm, and without it 11.689 cm. This shows that letting the agent compare against Gaussian renders more than halves the geometric error. The whole reconstruction took 7 hours 13 minutes, so the cost of one scene is acceptable.

Training images are composed of two parts: the background is rendered from the Gaussians, the robot and the manipulated objects by the simulator, and the two are merged by depth. This helps grasping the most. Each policy is trained on 100 simulation demonstrations, so the comparison isolates the images. Switching to Gaussian images raises real-robot picking from 3/20 to 10/20 and the full task from 3/20 to 7/20, while walking only goes from 18/20 to 20/20. This shows that photorealism matters mostly for the part that touches objects; walking was never very sensitive to images.

Step two: turning 15 demonstrations into thousands

Each generation first randomly moves the robot start (within a 30 by 30 cm region) near its reference position, along with the objects and goals. It then splits the source demonstration into walking, manipulation and leg-interaction segments and handles each kind separately.

Walking segments get a collision-free route from A*, and SONIC's own motion planner turns it into whole-body walking, with speed sampled between 0.5 and 0.7 m/s. Manipulation segments keep the body still and move only the hand trajectory to the new position, keeping the pose of the hand relative to the object unchanged:
$${\tilde{T}}_{e,t}=T_o(T_o^{\mathrm{src}}{)}^{-1}{T}_{e,t}^{\mathrm{src}}$$
$T_o^{\mathrm{src}}$ and $T_o$ are the object's poses in the source demonstration and in the new scene. $T_{e,t}^{\mathrm{src}}$ is the hand pose at frame $t$ of the source demonstration. The resulting ${\tilde{T}}_{e,t}$ is converted into arm joint angles by inverse kinematics. The intuition: express the hand in a frame centred on the banana, then put it back at the new banana's position. For example (illustrative), in the source demonstration the banana sits at (0, 0) on the table and the hand is 8 cm above it while grasping. In the new scene the banana is moved to (12, −5) and rotated by 15°. The hand trajectory then shifts as a whole to 8 cm above (12, −5) and rotates by 15°, grasping exactly as in the source. Leg segments, such as closing a door with a foot, keep the lower-body motion from the demonstration.

Transitions between segments need care too. The hand starts from where the previous segment ended, and the offset decays smoothly to zero over time so that it merges into the new trajectory. When positions move farther, the transition time is stretched according to the hand speed in the source demonstration.

Why correct from the execution result?

A reference motion can be kinematically perfect and still drift when SONIC executes it. Suppose the plan stops the robot 40 cm in front of the table, facing the banana, but it actually stops 6 cm to the left with a 3° heading error (illustrative). The hand reaches out as planned and closes on nothing.

Zero-HLM exploits a convenience of simulation: when the robot drifts, move the scene to follow it. Based on the robot's actual position and heading, it applies the same planar translation and rotation to the relevant furniture and objects. The relation between the robot, the table and the banana is then restored to the plan, and the motion is executed again. The policy only sees the final images and actions, so the resulting trajectory is still a self-consistent valid demonstration. The hand still has local errors when grasping and placing. At the moment of grasp or release, the method measures how far the hand's actual position is from its reference position. It subtracts the drift of the successful source demonstration at the same moment to get the extra planar drift of this attempt. It then translates the object or placement target by the same distance in the same direction. For example, if the hand drifts 3 cm to the right this time and the source drifted 1 cm to the right (illustrative), the banana moves 2 cm to the right. Height errors cannot be fixed by moving the table, so the wrist height in the reference motion is adjusted instead.

The ablation on Task A compares each correction, measured by generation success rate:

Moving the scene to the robot's position contributes the most; without it, success drops to about a quarter. This shows that data generation fails mainly not because motion planning is wrong but because of position errors during controller execution.

Results

The main experiment uses the reference configuration, with 20 real trials per task. The three tasks are: walk over, pick the banana and put it in the basket (A); squat, pick up a hat, carry it and place it on a cabinet (B); carry a fruit basket with both hands, close a cabinet door blocking the way with a foot, and place the basket on a stool (C). The table counts full-task successes:

At 100 demonstrations each, simulation data and real data are nearly tied; at 800 simulation demonstrations, simulation clearly wins. This shows that each simulated demonstration is already close to a real one in quality, and the advantage comes from adding volume cheaply.

On Task A the gains from more data stop around 500 demonstrations, and 800 brings nothing more, which shows that adding data only near the reference positions saturates quickly.

The generalization experiment shows the value of coverage more clearly. The authors test five variations the real demonstrations never covered. They are a new start for Task A, a new start for Task B, a new picking position, a new placement position, and a new object and goal. The simulation-trained policy now also includes generated data at these shifted positions. Over 100 trials it succeeds 69 times in total, against only 23 for the policy trained on real data. This shows that the real value of simulation data is covering new positions cheaply. With the bitter melon and the white plate, the simulation-trained policy succeeds in all 5 trials and the real-trained one only once.

On collection speed, gathering 50 successful demonstrations takes 1.35, 1.02 and 1.68 hours of real teleoperation for the three tasks. Zero-HLM needs 0.56, 0.63 and 1.19 hours, so it is 1.8 times faster on average. The GPT-6 Astra baseline gives a general agent the same scene, demonstrations and control interface and lets it write its own code to generate data. It needs more than 5 hours on Tasks A and B, and on Task C it ran for 8 hours without a single success. Its trajectories often pause and retry, and the policy trained on them hesitates on the real robot too. This shows that "let an LLM agent generate the data itself" cannot yet replace a purpose-built generation pipeline.

Where is the gap: actions or images?

The authors use a small experiment to separate actions from images. It uses only the banana-picking segment: the 25 real demonstrations all have the banana in the lower half of the table, and the 25 simulation demonstrations all in the upper half. Training on either one alone, the policy never succeeds in the half it has not seen.

Mixing the two directly, the policy succeeds only 2 out of 5 times in the upper half, worse than with simulation data alone, so wider coverage by itself is not enough. The third approach replays those 25 simulation trajectories on the real robot, swaps in the images the real robot captured, keeps the action labels from simulation, and trains on them together with the original 25 real demonstrations. That is still 50 demonstrations with the same action labels, and the policy succeeds in all 5 trials in both halves.

This experiment shows two things. Action labels generated in simulation can be used on the real robot directly, and direct mixing works poorly because the images of the two datasets are inconsistent. Co-training on real and simulation data agrees: 100 real demonstrations plus 800 simulated ones reach 53.3% average success, below the 66.7% from 800 simulated demonstrations alone.

Another finding is that simulation evaluation and real evaluation rank policies differently. On Task A, the real-trained policy succeeds 40% of the time on the robot and 31% in simulation, and the simulation-trained policy 35% on the robot and 39% in simulation. The absolute success rates are similar on both sides, but the conclusion about which policy is better flips, so the simulation benchmark released with the paper can estimate task difficulty but cannot rank policies.

The limitations as I see them

First, there are too few scenes and tasks. All real experiments take place in one reconstructed room, the objects in the three tasks are largely fixed, and cross-scene generalization is not tested at all; the authors list multiple scenes as future work.

Second, the generalization samples are small. Each expanded position is tested only five times, and the new object and goal setting has only 5 trials in total. The gap of 69% against 23% is large enough, so the overall conclusion holds, but individual numbers will fluctuate a lot; for example, 50% against 30% for the new placement position comes from just 10 trials.

Third, absolute success is still not high, and generation wastes a lot. Even with the most simulation data, Task C succeeds only 13/20. During generation only 25% of Task C attempts succeed, which means most simulation compute goes into failed trajectories. The method also depends on a single controller, SONIC, and on 15 teleoperated simulation demonstrations per task; how it works with another controller is not verified.

How a robotics team could try it

1. Start with the cheapest diagnosis: take a segment of an existing task, replay simulation trajectories on the real robot with real images swapped in, and check whether the action labels work directly. If they do, simulation data generation is worth the investment, and the next step should focus on images.
2. Then rebuild a single scene: scan a room where the robot will actually be deployed with a smartphone, split the work as "Gaussians for images, mesh for physics", and spot-check key surface dimensions with LiDAR or a tape measure, keeping errors within a few centimetres before going further.
3. When generating data, implement the correction that moves the scene to the robot's actual position first. It contributes the most in the ablation and is the simplest to build; then see whether the generation success rate rises clearly.
4. Evaluate on the real robot at the reference configuration and at a few shifted positions, with more trials per position, and do not draw conclusions from the simulation benchmark ranking alone.

One-line verdict

For whole-body humanoid loco-manipulation, simulation data can already replace real demonstrations in volume and coverage; the real bottleneck is whether the images match the real robot's.