OpenJaws: An Open Hardware UMI Framework with Stereo Localization for Generalist Manipulation

Collecting robot demonstrations in the wild with a handheld gripper (UMI) is now common, but how good the action labels are depends on one step: recovering the gripper trajectory from the wrist-camera video. The original UMI uses a monocular camera plus an IMU (inertial measurement unit); a single camera cannot see true scale, so it also needs a pre-built map and a fiducial marker to stay stable. OpenJaws swaps in a stereo camera that costs about a hundred dollars. The distance between the two lenses is calibrated, so triangulation gives true scale directly. On a new benchmark with ground truth, comparing only segments both methods can track, the stereo trajectory error is about 2.9 mm versus 18.4 mm for the original pipeline, which also tracks only 37% of frames on average. The whole gripper costs about $400, and all hardware and software are open source.

Teams that collect robot data or train manipulation policies should read this paper, because it turns "how accurate are my action labels?" into something you can measure for the first time. The core idea in one sentence: a calibrated stereo camera supplies true scale, open-source SLAM (simultaneous localization and mapping) or SfM (recovering camera poses and scene structure from a set of images) tools recover the trajectory, and one gripper with one standard flange interface handles both collection and deployment, removing the mapping run, the marker and re-calibration. I think the most valuable parts are the ground-truth benchmark and the controlled experiment in which raising localization error from about 10 mm to over 30 mm drops policy success from 90% to 30%. The hardware itself is not very novel; its strength is that it is open and easy.

What problem is it actually solving?

First, what UMI is. It is a gripper a person holds in the hand, with a camera on the wrist. The person grasps, inserts and pulls things directly in a kitchen or warehouse, with no robot present. Afterwards, the gripper's position and orientation in every frame are computed from the video and cut into short chunks; within each chunk, the pose change of later frames relative to the chunk's starting frame is the action label for policy training. So errors in the trajectory flow into the action labels. Because labels are relative to each chunk's start, slow absolute drift of the whole trajectory partly cancels out; a scale error (say, 10 cm read as 18 cm) stays in the labels as is.

A concrete example (illustrative). A person reaches the gripper 10 cm forward from the table edge to grab a block. Three ways of recovering the trajectory give different labels:

The 18 cm in the first row is not an arbitrary ratio. In this setting, the paper's learned monocular method ViPE (a neural network that estimates camera pose from video) needs its whole trajectory shrunk to about 0.556 times to match ground truth, meaning it overestimates distances by nearly a factor of two. The shape of the trajectory is fine, but a robot following that label would overshoot.

Monocular plus IMU can recover scale in theory but is fragile in practice. The original UMI authors themselves found that standard ORB-SLAM3 (a classic open-source SLAM system) initializes slowly under fast handheld motion and often produces no poses for a whole recording. Their fix was to record a separate mapping video first, with a marker of known size in the scene during mapping to pin down scale and initialization for mono+IMU; the marker is not needed when collecting demonstrations; the paper calls this modified pipeline UMI-SLAM. The problem is that the scene at mapping time often differs from the scene during manipulation, and once objects move, the map is stale.

The overall pipeline

The OpenJaws pipeline is short and splits into five steps along the data flow:

1. Collection: a person demonstrates with the gripper while left and right wide-angle images, gripper opening and IMU readings are recorded.
2. Preprocessing: fisheye distortion is removed, then the left and right images are rectified onto the same horizontal lines so matching points can be searched row by row.
3. Masking: SAM (Meta's general image segmentation model) segments the gripper fingers so localization ignores those regions. The fingers move, and matching them as background features would corrupt the camera's own motion estimate.
4. Localization: the rectified stereo images go to off-the-shelf open-source tools such as COLMAP (offline SfM reconstruction), ORB-SLAM3 or cuVSLAM (NVIDIA's GPU-accelerated visual SLAM), producing a camera trajectory with true scale.
5. Training and deployment: the trajectory is cut into action chunks, each expressed as the gripper's pose change relative to the chunk's starting moment, to train Diffusion Policy. At deployment the same gripper is mounted directly on the robot arm's flange. For the relative end-effector action labels, the paper does not say how the fixed camera-to-TCP transform is obtained or how gripper opening enters them. The paper also does not say what the Diffusion Policy observes at deployment, how relative actions become robot execution targets, or whether stereo localization runs online; it only says it uses the same Diffusion Policy and relative end-effector action space as the original UMI.

The key division of labor is: hardware supplies the scale, open-source algorithms supply the trajectory. The stereo camera's lens spacing is a calibrated, known quantity; the algorithm matches the same point in the left and right images and triangulates it to get true distance, then localizes and optimizes to chain frames into a trajectory. The parts are explained below.

Why does stereo give scale directly?

A single camera only captures directions. A 1 cm block 10 cm away and a 10 cm box 1 m away can look the same size in the image. When two cameras a known distance apart shoot together, the same point appears shifted by some pixels between the two images; this shift is called disparity. The closer the point, the larger the disparity.
$$Z=\frac{f\cdot B}{d}$$
Here $Z$ is the distance from the point to the camera, $f$ the focal length in pixels, $B$ the spacing between the two lenses (the baseline), and $d$ the disparity. The intuition: the baseline is a built-in ruler, and disparity tells you how long that ruler looks in the image.

Here is a worked example, with illustrative focal length and disparity. OpenJaws' calibrated baseline is 59.7 mm. Suppose the focal length is 300 pixels and a point has a disparity of 30 pixels; its distance is about 300 × 59.7 ÷ 30 ≈ 597 mm. Because $B$ is a real length, $Z$ is a real length too, with no IMU needed. Hoot says▾

Hoot saysA near small block and a far big box: mono can't tell, stereo canClose ✕

Mono only measures size ÷ distance, so both objects are 30 pixels; stereo adds a 59.7 mm ruler, the disparities differ 10×, and distance is fixed.

Mono: one image only

Block in the image

300 × 10 ÷ 100 = 30 px

Box in the image

300 × 100 ÷ 1000 = 30 px

Distance it can infer

Ambiguous: 10× bigger and 10× farther looks the same

So a mono trajectory has shape only; scale needs an IMU or a marker.

Stereo: two images + known 59.7 mm baseline

Block disparity

300 × 59.7 ÷ 100 ≈ 179.1 px

Box disparity

300 × 59.7 ÷ 1000 ≈ 17.9 px

Distance it can infer

Resolved: 100 mm vs 1000 mm

The baseline is a real length, so the distance is too.

Illustrative: focal length 300 px, a 10 mm block at 100 mm, a 100 mm box at 1000 mm; baseline is the paper's calibrated value. Source: projection: pixel size = f × object size ÷ Z; disparity d = f × B ÷ Z (formula in the text); B = 59.7 mm from the paper's calibration

This also shows why baseline calibration matters. $Z$ is proportional to $B$, so a baseline that is 1% wrong scales the whole trajectory up or down by 1%. The authors ran a sensitivity test: without re-rectifying images, they shifted the baseline value given to cuVSLAM by −5 to +5 mm. Mean translational error is about 3 mm near the calibrated value and rises to 6 mm and 8 mm at −5 and +5 mm. Rotational error changes by only 0.1 degrees across the whole range. The baseline affects only the size of the trajectory, not its shape, consistent with the formula above.

Hardware: why use the same gripper for collection and deployment?

The original UMI collects data with a handheld gripper but deploys with a commercial electric gripper (WSG-50) fitted with matching fingers. The two grippers must agree on stroke, finger shape and the offset from camera to gripper center; any mismatch means what the policy learned about how wide to open and where to reach no longer matches the real robot. The paper calls this the UMI-to-real gap and notes that the best-performing UMI stacks are not open source.

OpenJaws builds a single gripper that can be held in the hand for collection and bolted directly onto a robot arm. Each choice has a reason:

• The interface follows the ISO (International Organization for Standardization) 9409-1-50-4-M6 flange standard. Many collaborative arms use this end flange, so the gripper screws on and works, and the camera-to-gripper-center offset is identical during collection and deployment, so no re-calibration is needed.
• Only off-the-shelf modules: a Dynamixel servo, an Arduino-family controller, an IMU shield and a stereo camera, with all structural parts 3D printed. The authors deliberately avoid a custom circuit board, because anyone reproducing it would first have to get a board fabricated.
• A global-shutter camera. Handheld motion is fast, and a rolling shutter would skew the image so feature points no longer line up.
• Fingers printed in TPU (thermoplastic polyurethane, a rubber-like soft plastic) give some natural compliance when grasping.

At US retail prices in October 2026 the bill of materials is about $399 and the gripper weighs 0.408 kg; the original UMI, including its deployment gripper, is about $5,389 and 0.650 kg. The gap shows that most of the cost of collection hardware sat in the deployment gripper and dedicated localization, not in collection itself.

The benchmark: how do you know the trajectory is right?

There was no localization benchmark built for the UMI setting; SLAM benchmarks for driving and indoor reconstruction differ greatly in motion and distance scale from handheld manipulation. The authors therefore built UMI-Loc-Bench-v1: 9 tasks with 10 handheld demonstrations each, 90 in total, including block stacking, gear insertion, wrench rotation, retrieving from a shelf and a bin, inserting a flash drive, placing a pen and picking candy.

Ground truth comes from a fixed cube in the scene carrying ArUco markers (printed QR-like patterns from which a camera can compute its exact pose relative to the cube). For fairness, the cube is masked out in both image streams before evaluation, so the tested methods cannot use it for tracking.

The error metric is ATE (absolute trajectory error): the estimated trajectory is first rotated and translated as a whole to align with ground truth, with no scaling allowed, and then the per-frame position error is computed. Disallowing scaling is crucial, because a wrong scale is exactly the problem to catch. Two more quantities are reported: coverage, the fraction of frames in a demonstration that get a pose, and processing time per demonstration.

Results: how much better is stereo?

Start with the fairest comparison. The original UMI-SLAM tracks only some frames, so the authors restrict every method to the segments UMI-SLAM tracked, 27 demonstrations in total. All methods are scored on the same segments. The last column is the fraction of frames with a pose over the whole benchmark, except the ViPE row, which counts whether whole demonstrations complete. So the error columns only show who is more accurate on the segments UMI-SLAM can track; the coverage column shows who can track whole demonstrations.

The table shows three things.

First, stereo is both accurate and stable. COLMAP's error is about one sixth of UMI-SLAM's, with no mapping run or marker, and almost every frame gets a pose.

Second, the IMU cuts both ways here. For the same ORB-SLAM3, adding the IMU lowers error from 5.1 mm to 2.0 mm, but coverage falls from 100% to 18%, losing track most of the time. The reason is that handheld manipulation has a tiny workspace and abrupt motion, so IMU initialization and parameters need dedicated tuning. A trajectory that is locally more precise but mostly missing is useless for data collection.

Third, the learned monocular method gets the shape but not the scale. ViPE completes 89 of 90 demonstrations and, after scaling with ground truth, has only 3.5 mm error, comparable to stereo; unscaled, it is 48.2 mm. The authors explain that ViPE was trained mainly on outdoor and room-scale data, while a wrist camera sits only tens of centimeters from objects, outside what it has seen.

Now the full trajectories. Among pure stereo methods, COLMAP is most accurate (5.2 mm, 0.8°), ORB-SLAM3 is 7.4 mm, and cuVSLAM 12.4 mm. But COLMAP needs 836 seconds per demonstration and cuVSLAM only 34.8 seconds, because COLMAP ends with a global optimization that jointly adjusts all frames. The authors use cuVSLAM for policy training, trading a bit over twice the error for more than twenty times the speed. For teams processing thousands of demonstrations, that trade-off is reasonable.

Does trajectory accuracy really affect the policy?

The authors train two tasks with the same gripper on a UR10 (a six-axis collaborative arm from Universal Robots): 200 demonstrations for block stacking and 50 for pick-and-place, localized with cuVSLAM, with Diffusion Policy and actions again expressed as gripper pose changes relative to the start of each chunk. Each task is tested 10 times under varied initial conditions and succeeds 9 times. The one pick-and-place failure was grasp slippage; the one stacking failure was dropping the block midway. At deployment the gripper is screwed straight onto the arm with no re-calibration.

The controlled experiment is more interesting. The authors inject drift into the training trajectories: every 10 to 30 frames, with probability 8%, a random offset of up to 10 cm is added to the current and all later frames, imitating SLAM jumping once and never recovering. Training-set translational error rises from about 10 mm to over 30 mm, and policy success falls from 90% to 30%, with failures mostly from slipping off-center grasps or imprecise placements that hit the scene.

This shows that a relative action representation does not fully absorb localization error. Many assume UMI actions only look at relative motion over a short window, so absolute drift does not matter; but once drift happens in the middle of an action chunk, that chunk's relative motion is itself wrong.

The authors also demonstrate turning demonstrations into simulation scenes: an LLM agent is given a gear-insertion demonstration plus models of the gripper and parts, builds the simulated scene itself and replays the demonstration, succeeding for 7 of 10 demonstrations. This covers only one task, so I treat it as a side possibility rather than a main contribution.

Limitations as I see them

First, the policy validation is thin. Only two tasks were tested, 10 trials each, on one arm with one policy. The direction of the 90% versus 30% contrast is clear, but because there are only 10 trials, one or two more successes or failures would move the rate by 10 to 20 points.

Second, the injected noise is rather extreme. Step offsets of up to 10 cm look more like failed relocalization after losing track than the slow drift common in real SLAM. The experiment proves that very poor localization hurts the policy, but does not answer how much the gap between 2.9 mm and 9.9 mm matters, which is exactly what you want to know when choosing between COLMAP and cuVSLAM.

Third, it is single-arm only. Two arms would require localizing two independently moving stereo rigs in one coordinate frame, which the authors leave for future work. The benchmark also has only 90 demonstrations over 9 tasks, so scene diversity is limited.

How a robotics team could try it

1. Measure your current setup first. If you already collect with monocular UMI, follow the UMI-Loc-Bench-v1 recipe: place an ArUco cube in your own scenes and measure your trajectories' error and coverage. If error is in the teens of millimeters and coverage below half, switching is worth it.
2. Change only the camera. Swap the wrist camera for a global-shutter stereo pair, batch-process with cuVSLAM, and compare coverage and error on the same tasks. The test is simple: does coverage approach the full recording, and does error drop to single-digit millimeters?
3. Then change the gripper interface. If your arm has an ISO flange, try using one gripper for collection and deployment and see whether the pre-deployment calibration step disappears.
4. Finally, run your own noise experiment. Add noise of different sizes to trajectories on your own tasks, plot localization error against policy success, and use that to decide between fast cuVSLAM and accurate COLMAP for offline processing.

Bottom line

Action-label quality is the ceiling of handheld data collection, and getting true scale from a calibrated stereo camera is simpler and more reliable than stacking an IMU, a pre-built map and a marker on a monocular pipeline.