On Agentic Manipulation Policies

Chen Chen, Omid Bagherieh, Naveen Nikamanth Appiah Balaji, Jesse T. Gonzalez, Negin Heravi, Fahad Islam, Xueting Li, Jean-Baptiste Regli, Kejia Ren, Petr Švec, David J. Yoon, Matthias Müller et al.
41 minread4.30scoreNo. 211-10 Embodied AI issue
Contents · 8 sections1My take2Why earlier routes fell short3How AMP1 lets the model control the arm directly4Strategies nobody taught it5Evidence: real robot and four simulation benchmarks6Does the gain come from the model or the interface?7Adding memory: learning from experience8Limitations
Rel 5Nov 4Prac 4Evid 4Time 4

Without robot demonstrations or task-specific training, can a generalist AI model that writes its own code and commands a robot arm directly do better than models trained specifically on robot demonstrations? This Apple paper answers on a real arm, with the same 12 tasks, 9 trials each, 108 trials in total. The VLA (vision-language-action model, which maps images and an instruction directly to actions) π0.5 succeeded 30 times. The representative WAM (world-action model, which predicts both how the scene will change and the actions), the model FLUX 3 Action, succeeded 49 times. By contrast, Claude Opus 5.5 controlling the arm directly through an interface called AMP1 succeeded 104 times. The gap is widest on tasks like "Put all the lemons and the orange on the plate": both specialist models failed all 9 trials, AMP1 succeeded in all 9. This is not just a result on tasks the authors chose: on the public RoboLab-120 benchmark (120 tasks, 1,200 episodes per instruction phrasing), AMP1 with Gemini 4 Argon succeeds 85.6% of the time (82.8% with Opus 5.5), while the best VLA and WAM on the leaderboard reach 49.2% and 42.9%. And AMP1 gives the model no ready-made grasping or placing skills; how to grasp is computed entirely by the model. The price is speed: on successful real-robot trials, AMP1 spends on average 69 seconds executing motion, the same order as the two baselines (62 and 44 seconds), but it also waits 211 seconds for the model to think.

My take

Robot-policy people should read this carefully, not for a new module but because it systematically validates one route: keep a generalist model in the control loop, looking at the result of each step before deciding the next. The authors call this an . Simulation, real robot and four benchmarks all point the same way. The authors also run controlled experiments that show where the gains come from: changes to the AMP1 interface matter far less than swapping the model. But interface design is not irrelevant: with the same Opus 5.5 on the same 30 RoboLab tasks, Astranomy, another interface that controls in small steps and does not let the model write code, reaches only 25.3% against AMP1's 84.7%. Open questions remain on speed and cost, and on an asymmetry in the real-robot trials: AMP1 can recover from errors and the baselines cannot (without recovery, AMP1 still gets 94/108, with failures rising from 4 to 14).

Why earlier routes fell short

The mainstream in robot manipulation has been specialist policies: VLAs or WAMs trained on large amounts of robot demonstrations, taking images and an instruction and outputting joint actions. They are fast, but they slip when the phrasing changes: on RoboLab-120, with vague instructions, FLUX 3 Action drops from 42.9% to 28.4% and π0.5 from 28.0% to 15.2%.

Generalist models had mainly been used in two ways. One has the model select and sequence ready-made skills ("grasp", "place"); however clever the model, it can only do what the skill library allows. The other has the model write an entire control program before execution (code-as-policies); the program can read sensors, but the model is absent during execution and cannot change course if the strategy is wrong. AMP is the third way: the model stays in the loop, looks at the result after each segment of motion and then decides the next one (it cannot change a segment while it is executing). This route used to fail mainly because large models' spatial understanding was too weak; the authors' judgment is that frontier models' spatial reasoning has now improved enough to try.

How AMP1 lets the model control the arm directly

AMP1 Figure 6: the generalist AI model writes code, via tool calls, in a persistent Python interpreter; the code calls the harness's perception (list objects, show images, draw 3D annotations), sensing (instruction, RGB, depth, point cloud) and motion (preview waypoints, collision check, execute) functions; skills such as grasping and scene reasoning are implemented by the model itself in the interpreter.
AMP1 Figure 6: the generalist AI model writes code, via tool calls, in a persistent Python interpreter; the code calls the harness's perception (list objects, show images, draw 3D annotations), sensing (instruction, RGB, depth, point cloud) and motion (preview waypoints, collision check, execute) functions; skills such as grasping and scene reasoning are implemented by the model itself in the interpreter.

AMP1 is a , not a new model. During a task the model repeats three things:

Hoot saysOne round of write waypoints, get refused, revise, execute (illustrative)Close ✕
The model only sets gripper targets; the interface works out how the joints turn. When a path fails the checks, the interface sends it back with the reason; the model revises it, and checks the result after each segment.
The model outputs three gripper waypoints and hands them to the interfacemodellooks, computes graspwaypointsposition·orientation·time·open/closeinterfacereceives, checks first
The model outputs three gripper waypoints and hands them to the interfacemodellooks, computes graspwaypointsposition·orientation·time·open/closeinterfacereceives, checks first
1/3 · ① The model writes waypoints: only where the gripper goes, when, open or closedIn code, the model computes a sequence of gripper targets (illustrative: above the cube, then down, then close). It writes no joint angles; how the joints turn is the interface's job.
The interface refuses the path and returns the reason to the modelinterfaceinterpolate + solve jointsrefuseddescent too fastmodelreads why, slows that segment
The interface refuses the path and returns the reason to the modelinterfaceinterpolate + solve jointsrefuseddescent too fastmodelreads why, slows that segment
2/3 · ② The interface checks: unreachable or too fast is sent back with the reasonThe interface interpolates between waypoints and solves joint angles step by step; if a segment is unreachable or a joint would move too fast, it does not execute and returns the reason and the affected steps (illustrative: the descent was given too little time). Collision checks are up to the model to call.
The interface executes; the model observes and writes the next segmentinterfacesends joint targetsarmexecutes this segmentmodellooks, reads pose, next segment
The interface executes; the model observes and writes the next segmentinterfacesends joint targetsarmexecutes this segmentmodellooks, reads pose, next segment
3/3 · ③ Once it passes it runs; the model checks the result and plans the next segmentAfter the revised path passes, the interface sends the joint targets to the arm step by step. The model cannot change this segment mid-execution; afterwards it looks at the cameras and reads the pose to see whether the cube is held, then writes the next waypoints.
1/3

The cube grasp and the too-fast descent are illustrative; the paper gives no such step-by-step example. The division of labor (the model gives timed gripper waypoints; the interface interpolates, solves joint angles, checks reachability and speed and returns reasons; collision checks are called by the model; no changes mid-execution) is from the text.

Sourcepaper §4A.3 motion interface and §2D.2 basic harness: waypoint = gripper pose, arrival time, gripper command; interpolation + damped least-squares IK; refused with reason and affected steps if tolerances or joint speed limits are exceeded; collision checks called by the model
  1. It decides what to look at. The model calls functions to read the instruction, fetch a camera's color and depth images, read camera calibration, and read the arm's current pose. The interface also fuses the cameras' depth into one point cloud (calibration tells it where each camera sits and points in the robot's coordinate frame), splits it into individual objects, gives each an ID, color and size, and reports its center and bounding box in the robot's frame. So any point the model takes from an image or an object can be converted to a 3D position in the robot's frame. Images enter the context only when the model chooses to display them.
  2. It writes code to compute motion. The model outputs a sequence of timed : each states at what time the gripper center should reach which position in the robot's frame, with what orientation, and whether the gripper is open or closed. The interface interpolates smoothly between neighboring waypoints, converts each control step into joint angles, and sends them to the arm. Where to grasp, from which direction to approach and where to release are all computed by the model; the interface has no function that "picks a grasp for you".
  3. It checks the result and revises. The interface first checks the path: if it is unreachable or too fast, it refuses to execute and tells the model why and at which time steps; collision checking is left to the model to call. After execution the model looks at the images again and reads the pose to judge whether it worked.

All code runs in an interpreter that is never restarted during the task, so functions written and images saved earlier can be reused later; but each new task starts from a blank slate and the skills are rewritten. The paper counts the "skills" Opus 5.5 repeatedly wrote for itself across 1,200 RoboLab simulation episodes. The percentages are shares of episodes in which each appears: measuring a container before placing into it (66%), aligning the grasp with the object's axes (61%), reading the reason when a path is refused and fixing it (40%), diagnosing the cause of a failed motion before retrying (39%).

Strategies nobody taught it

AMP1 Figure 3: strategies AMP1 devised during execution; A–C on the real robot, D in simulation. A: the target is hidden, so it moves the occluder to look; B: the bowl cannot be grasped, so it moves the plate against the bin and pushes the bowl off onto the plate; C: the bottle is out of reach, so it tips it toward itself with a wooden spoon and grasps it by the cap; D: the pomegranate is too wide, so it presses it against the crate wall and rolls it up. E: skills the model writes for itself and the share of 1,200 RoboLab episodes in which each appears.
AMP1 Figure 3: strategies AMP1 devised during execution; A–C on the real robot, D in simulation. A: the target is hidden, so it moves the occluder to look; B: the bowl cannot be grasped, so it moves the plate against the bin and pushes the bowl off onto the plate; C: the bottle is out of reach, so it tips it toward itself with a wooden spoon and grasps it by the cap; D: the pomegranate is too wide, so it presses it against the crate wall and rolls it up. E: skills the model writes for itself and the share of 1,200 RoboLab episodes in which each appears.

Because the interface offers only the lowest-level "look" and "move", when the model meets something it cannot grasp or cannot see, it has to find its own way. The easiest example to follow is the real-robot task "Put the bowl on the plate": the bowl lies upside down on a bin, wider than the gripper can open, with no edge for the fingers to slip under. In one trial, AMP1 did not force a grasp; it grasped the plate by its rim and slid it up against the side of the bin, then pushed the bowl across the bin with the open gripper until it fell off the edge onto the plate, and the bowl was never held. In other trials it took another approach: push the bowl to the edge of the bin so part of it overhangs, then grasp the overhanging part. AMP1 succeeded 7/9 on this task, π0.5 2/9, FLUX 3 Action 0/9.

There are more. When the target is hidden behind something, it pushes aside likely occluders one at a time. In simulation, when a pomegranate is wider than the gripper's 85-millimeter maximum opening, it braces it against a heavier object and slides the gripper underneath to lift it like a forklift, or presses it against the crate wall and rolls it up (Fig. 3D). When a bottle is out of reach, it first uses inverse kinematics to work out how far short it is, then finds a long-enough wooden spoon on the table to pull it closer. On the real robot, when a motion misses, it compares "where the gripper was commanded to go" with "where the gripper actually appears in the image", infers a camera calibration error, and corrects it in later motions. None of this is in the prompt or provided as a predefined skill.

Evidence: real robot and four simulation benchmarks

The real-robot experiments use a Franka arm with three cameras, two at the shoulders and one on the wrist (depth is estimated from the stereo images by a stereo-matching model), with 3 trials on each of three platforms that differ in background, calibration and cabling. Every method starts from the same object layout, the execution time limit is 5 minutes (model thinking time excluded), and grasping and placing an uninstructed object counts as failure. π0.5 uses only the left-shoulder and wrist cameras, as in its training; FLUX 3 Action and AMP1 also use the right-shoulder camera.

Task (selected)π0.5FLUX 3 ActionAMP1
Put all the lemons and the orange on the plate0/90/99/9
Put all the tools except the hammer in the right bin3/92/99/9
Use the spatula to push the four cubes into a straight, aligned row0/90/99/9
Cover each cube with a cup, then uncover the yellow cube0/90/99/9
All 12 tasks30/108 (27.8%)49/108 (45.4%)104/108 (96.3%)

The two baselines often grasp the wrong object, even the hammer the instruction explicitly excludes; facing an object they cannot grasp, they keep repeating an infeasible grasp until time runs out. Of AMP1's 4 failures, two were cubes landing outside the plate while pouring, and two were a bowl rolling off the table. Its main trouble on the real robot is collisions: motion planning depends on camera calibration, and the robot's geometry is simplified. When a collision triggers the arm's protective stop, AMP1 can call a function to request recovery and continue; the baselines are scored as failed on such errors. Without recovery, AMP1 gets 94/108 (87.0%), still far above the baselines.

In simulation, on RoboLab-120 (120 language-specified manipulation tasks, 10 episodes each), AMP1 succeeds 85.6% of the time with Gemini 4 Argon and 82.8% with Opus 5.5; the best VLA and WAM on the leaderboard are at 49.2% and 42.9%, and the best existing entry on the leaderboard, another agent built on a large model, is at 57.1%. With vague or more specific phrasings of the instruction, AMP1's success rate moves by only about 3 points. On CaP-Gym's three suites, AMP1 makes a single attempt without practicing on the tasks (the LIBERO-PRO and robosuite numbers use Gemini 4 Argon): LIBERO-PRO 95.2%, all 700 robosuite episodes (including two bimanual tasks), and 48/50 on the two BEHAVIOR-1K mobile manipulation tasks. The few rivals scoring higher all practiced their programs or skills on the evaluated tasks. AMP1's failures on LIBERO-Goal, one part of LIBERO-PRO, were almost all cases where it believed it had finished; after reviewing them by hand, the authors judge that these results mostly match a human reading of the instruction and are only inconsistent with the benchmark's checking rules (for example, which side of the stove counts as its "front").

Does the gain come from the model or the interface?

AMP1 Figure 5: controlled experiments on RoboLab-120. A: success rates of 13 models with less and more reasoning; B: basic harness versus an assistive harness with ready-made pick-and-place routines; C: removing perception tools, depth and camera calibration step by step, then replacing images with text descriptions, plotted against model calls per episode; D: Opus 5.5 with only some of the cameras.
AMP1 Figure 5: controlled experiments on RoboLab-120. A: success rates of 13 models with less and more reasoning; B: basic harness versus an assistive harness with ready-made pick-and-place routines; C: removing perception tools, depth and camera calibration step by step, then replacing images with text descriptions, plotted against model calls per episode; D: Opus 5.5 with only some of the cameras.

With the interface fixed and only the model swapped, 13 models on RoboLab-120 range from 12.8% (Qwen3.6-35B-A3B) to 85.6% (Gemini 4 Argon), which shows that the model is the first thing that decides success. Within the Claude family, Opus 5.5 gets 82.8%, the smaller Haiku 5.5 51.6%, and the older Opus 4.5 35.2%. Every model was run at a low and a high reasoning-effort setting, with everything else unchanged; raising it helps every model, Opus 5.5 by 14.6 points. None of the changes to the AMP1 interface matters as much as swapping the model.

The most interesting interface comparison is an "assistive interface": the model can request ranked grasp and placement poses, a motion planner executes collision-free picks and placements, and if a grasp fails it automatically retries with another; the model can still fall back to the basic interface at any time. The result: the four strongest models under the basic interface all do worse with assistance, Opus 5.5 dropping from 82.8% to 75.3%; the six weakest all do better, the weakest, Qwen3.6-35B-A3B, by 17.6 points. The authors' explanation is that failure information gets hidden: the assistive routines retry internally and only report "succeeded" or "failed", not what went wrong. Weak models' own picks and placements succeed less often than these routines, so the routines help; strong models could otherwise see the result of every step and correct it in time, and the routines block their view.

On perception, the authors remove inputs step by step. Opus 5.5's success rate and model calls per task are: full input (color images, depth, calibration, object segmentation tools) 82.8%, 34.8 calls; without segmentation tools 81.9%, 38.7; also without depth 81.1%, 46.3; also without calibration, color images only, 75.8%, 53.5. It rebuilds the missing information itself. Without depth, it finds the same point in zoomed crops from different cameras and triangulates its 3D position with the known calibration. Without calibration as well, it moves the wrist camera with the arm to several known gripper poses (the robot knows its own gripper pose), takes pictures, matches feature points, and fits the focal length, principal point and the camera's pose relative to the gripper by least squares. Weak models cannot do this: with color images only, Haiku 5.5 gets just 10.7%. A final condition replaces images with text descriptions (written by a second instance of the same model after looking at the images); Opus 5.5 falls from 75.8% with color images only to 36.8%, because text cannot convey geometric details such as "is the gripper aligned with the object".

Adding memory: learning from experience

All the results so far start each task from scratch. The authors then give AMP1 a memory folder kept across episodes (called AMP1+), with no format imposed; the model usually keeps a Markdown file of notes (what worked, what failed, useful parameters) and a Python code library, and later tasks read the notes and call the code directly, while the model weights never change. The experiments run on RoboDojo: 42 bimanual tasks, color images only, no depth or calibration; each task comes with a scene description and a scoring rubric.

The first experiment practices each task separately, each yielding its own memory. During practice, each round runs 4 episodes in parallel and the benchmark reports only success or failure; after a round, one of the instances reviews all the records, outcomes and memory versions from every round so far, updates the memory and decides whether to keep practicing, for at most 12 rounds. The paper's example is a Mahjong task: take a tile from the stack and stand it at the right end of a row; the model is not told the tile must stand the right way up, but the scoring requires it. In round 2 one episode succeeded and one failed. The model enlarged the last frame of each and saw that in the success the tile's red symbol was at the bottom, like its neighbors', and in the failure at the top. Tracing the action records, it found the two episodes had picked up the face-down tile from opposite ends, and recorded which end is the tile's top. After one more round, in which it learned to open the gripper before reaching, the task went from 10% to 100% on unseen layouts.

Overall, practice raises the success rate (by the leaderboard's method: average within each of 5 task categories, then across categories) from 26.4% to 84.1%, above the leaderboard's top entry at 68.9% (note this is no longer zero-shot; it comes after practicing on this benchmark). By round 2, 39 of the 42 tasks have a code library in memory, and model calls on successful episodes halve. One caveat: each task was practiced independently 5 times in full (up to 12 rounds each, each yielding its own memory); the memory with the highest success rate on a second set of layouts was chosen and then evaluated on a third set of unseen layouts.

Another experiment does not practice the same task repeatedly: starting from empty memory, the agent does each of the 42 tasks once in random order, updating its memory after each based on success or failure, and is instructed to keep only general lessons useful for any task. After one pass, evaluating with the final memory on all 42 tasks lifts the success rate from 26.4% without memory to 57.6% (a single run with a single ordering). The authors also evaluate after the 10th, 20th and 30th tasks: after only the first 10 tasks, the last 12 tasks, not yet attempted, rise from 24% to 47%, which shows that much of what it records is general knowledge, such as this robot's coordinate frames, its motion limits, and how to estimate distance without depth.

Limitations

  • Slow. On the real robot, a successful AMP1 trial takes 280 seconds on average, 211 of them waiting for the model to think; π0.5 and FLUX 3 Action average 62 and 44 seconds on their successes (these averages cover only each method's successful trials, so the task mix differs). In simulation the clock pauses while the model thinks, and the time limit counts only execution. The paper frames it this way: AMPs are reliable but slow, VLAs/WAMs fast but unreliable.
  • Evaluations are mostly quasi-static. The model cannot change a path while it executes, so situations that need an immediate response, such as a moving target or a slipping object, remain a weak spot.
  • Comparison conditions are not fully equal. On the real robot only AMP1 has error recovery; AMP1 also uses depth and calibration, whereas typical VLAs use only color images (though with color images only, Opus 5.5 still gets 75.8%). The authors attribute a sizable share of RoboLab-120 failures to the benchmark itself (object shapes or weights unsuited to a parallel-jaw gripper, flaky success checks); that is their own analysis.

The authors' way forward: the cost of a given capability level falls by roughly an order of magnitude per year; code kept in memory can be reused to cut model calls; and AMP execution traces could be used to distill smaller, faster policies.

QuestionVerdict
Can a generalist model control an arm directly, without training or provided skills?Yes: 104/108 on the real robot, leading or near the best on all four simulation benchmarks
Where does the gain mainly come from?Model capability; changes to the AMP1 interface matter far less than swapping the model
Do ready-made pick-and-place routines help?They help weak models and hurt strong ones, because they hide the details of failures
Can it learn from experience?Yes: 26.4% → 84.1% on RoboDojo, without changing weights
Can it be deployed now?The authors think not yet: each success waits about 3.5 minutes for the model to think, and the evaluations are mostly quasi-static

How did this read for you?

One tap, no sign-in; tap again to undo. Only anonymous counts are kept, never who you are.

Discussion

Add experiment details, reproduction notes or a different take. Requires a GitHub sign-in.

Loading the discussion…