The tables, studies and protocol that the main page states in one line each.
Two arms share one coordinate frame, and the agent decides how to use them. It watches an overhead demonstration video and reproduces the folds: first the near long edge onto the far long edge, to make a doubled strip, and then both short ends inward. Two configurations were evaluated with five trials each. In sequential folding the demonstration folds the right short end first and then the left. In simultaneous folding the demonstration brings both short ends inward together. The agent chooses the grasp points, which arm does what and when, and may stage a placement to keep the arms clear of each other. The towel changes shape during grasping and folding.
Success criterionSuccess requires both inward folds with reasonably aligned edges, verified in an overhead image after both grippers open and both arms withdraw. Small wrinkles and creases are allowed, and the final position on the table is not restricted.
Sequential 5 of 5, at a mean 50.8 minutes and USD 24.14, among the highest in the suite. Simultaneous 3 of 5, at a mean 21.4 minutes and USD 7.46, the lowest success rate of the eight configurations.
The agent watches one human demonstration video and then assembles eight 3D-printed parts into four pairs in the demonstrated order: a ridged sleeve over a broad round base, a hexagonal nut over a short post, a smooth sleeve over a stepped cylinder, and a long pin into a hollow tube. Nothing tells the agent which part mates with which; it has to read the relationships and the insertion motions out of the video and out of the scene in front of it. The parts are printed from AutoMate meshes, with a nominal radial clearance as small as 0.74 mm, requiring AGP to align and insert the parts precisely.
Success criterionSuccess requires all four pairs to be assembled at the end, with pairs that were finished earlier still intact, and everything stable after the gripper opens and the arm withdraws.
8 of 10 trials succeeded. Successful trials took a mean of 37.2 minutes, ranging from 16.0 to 59.1, with mean usage of 13.02 million tokens and mean inference cost of USD 16.62.
| Pair | Hole diameter (mm) | Shaft diameter (mm) | Radial clearance (mm) |
|---|---|---|---|
| Ridged sleeve and broad round base | 37.41 | 35.86 | 0.78 |
| Hexagonal nut and short post | 28.18 | 25.80 | 1.19 |
| Smooth sleeve and stepped cylinder | 42.86 | 37.20 | 2.83 |
| Long pin and hollow tube | 18.01 | 16.54 | 0.74 |
Scroll the table sideways for the rest of the columns.
Nominal print dimensions as recorded by the authors, in demonstration order. Radial clearance is half the diametral clearance, computed from the two columns beside it and rounded once, and gives the transverse offset available when the two axes are parallel. What the parts actually tolerate also depends on printing accuracy and on angular alignment.
The agent gets six cubes on a table and a single photograph of the structure it should build. It has to work out which cube is which, how they support each other, and an order of placements that keeps the partial structure standing. Three goal structures were evaluated with ten trials each: a pyramid with three blue cubes in the base, two grey cubes bridging the base seams and one grey cube at the peak; two towers whose colour orders run blue, grey, blue and grey, blue, grey from the bottom; and a six block tower alternating grey and blue above a grey base that stays where it is.
Success criterionSuccess requires the complete target structure, still standing after the gripper opens and the arm withdraws.
Pyramid succeeded in 10 of 10 trials at a mean of 21.6 minutes, two towers in 10 of 10 at 20.6 minutes, and the six block tower in 9 of 10 at 28.2 minutes. Together, 29 of 30 block construction trials succeeded.
Six dice lie on the table with different faces upward. The agent receives a language instruction to turn them so that the upward face matches the goal, together with a goal photograph that fixes which face that is. The initial and target faces are balanced across predefined conditions. Nothing restricts where the dice end up or how they are rotated in the table plane, so the agent has to choose a grasp for each die that makes the required rotation reachable, and then release it so that it settles on the intended face.
Success criterionSuccess requires all six dice to show the requested number after placement. The task prompt leaves the dice positions and in-plane rotations unspecified.
10 of 10 trials succeeded. Successful trials took a mean of 37.9 minutes, ranging from 22.6 to 56.0, with mean usage of 17.08 million tokens and mean inference cost of USD 21.07.
The agent receives a text instruction and a goal photograph specifying the requested face.
A language instruction asks the agent to pick up a potato-shaped object and throw it into a stationary bowl. The required motion is a circular swing driven by one joint, with the other five joint targets held fixed and the gripper opening during the swing, so the outcome depends on the relationship between arm motion and release timing. This configuration uses a separate runtime with timed motion programs: the agent specifies joint offsets at time knots and a schedule for the gripper, and the executor interpolates the commands on a 50 Hz grid.
Success criterionSuccess requires the object to separate from the moving hand, travel through the air, and come to rest inside the target region, which here is the bowl.
Two recorded sessions, both successful, at a mean 22.8 minutes, from 22.2 to 23.3, at an average of 6.26 million tokens and USD 8.94. The throw itself completes an average of 13.9 minutes after the task is delivered, so about 40% of that time is the analysis and verification that follow it.
The two sessions were selected rather than drawn from a planned batch. They document successful throws but do not establish a success rate. In both sessions, the object was released during the swing and came to rest in the bowl. Occlusion leaves the rising part of the flight and the apex unresolved.
Open this image at full size, 1200 by 338 pixels
Eight configurations, five trials each. Three of them do badly, and they stay in the table.
The comparison uses a smaller assembly task: four parts forming two pairs, a hexagonal ring over a short post and a circular sleeve over a stepped cylinder, learned from one human demonstration. Every configuration receives the same task references, the same robot interface and the same time and request budgets. Success requires both pairs to remain assembled after release and arm withdrawal.
Scroll the table sideways for the rest of the columns.
| Configuration | Agent | Success | Time (min), mean [min, max] | Tokens (M), mean [min, max] | Cost (USD), mean [min, max] | Per-trial completion time (min) |
|---|---|---|---|---|---|---|
| GPT-6 Astra · low effort | Codex CLI | 5/5 | 9.9 [6.0, 19.0] | 3.69 [1.44, 7.72] | 4.79 [2.15, 9.56] | 6.0, 6.3, 7.4, 10.9, 19.0 |
| GPT-6 Astra · medium effort | Codex CLI | 5/5 | 9.2 [6.3, 15.4] | 3.02 [1.84, 5.80] | 4.09 [2.76, 7.48] | 6.3, 7.6, 7.6, 8.8, 15.4 |
| GPT-6 Astra · high effort | Codex CLI | 5/5 | 9.2 [7.9, 10.6] | 3.30 [2.58, 4.02] | 4.47 [3.72, 5.19] | 7.9, 8.9, 9.0, 9.7, 10.6 |
| GPT-5.6 Sol | Codex CLI | 5/5 | 14.2 [9.1, 19.3] | 7.13 [4.12, 9.72] | 3.94 [2.40, 5.30] | 9.1, 12.3, 15.2, 15.5, 19.3 |
| GPT-5.6 Terra | Codex CLI | 1/5 | 19.0 [19.0, 19.0] | 6.28 [6.28, 6.28] | 1.98 [1.98, 1.98] | 14.4 (failed), 19.0, 21.4 (failed), 22.1 (failed), 30.4 (failed) |
| GPT-5.6 Luna | Codex CLI | 0/5 | — | — | — | — |
| Claude Opus 5 | Claude Code | 5/5 | 22.2 [14.4, 31.9] | 12.40 [9.05, 16.72] | 9.75 [6.94, 13.03] | 14.4, 19.2, 20.0, 25.5, 31.9 |
| Claude Fable 5.1 | Claude Code | 3/5 | 27.7 [18.3, 39.8] | 9.27 [6.87, 11.10] | 11.30 [8.33, 14.36] | 18.3, 25.2, 39.8 |
Means, minima and maxima are over successful trials only. GPT-5.6 Luna has no successful trial, and seven failed trials have no usable resource record.
Source: the pinned per-trial two pair assembly records used for model comparison.
Two pair assembly example (5/5). GPT-6 Astra at high effort in Codex CLI, single arm. 7.9 min real at 8x. The other arm runs a separate trial outside this crop.
Two pair assembly example (5/5). GPT-5.6 Sol at high effort in Codex CLI, single arm. 9.1 min real at 8x. Nothing else is running: four parts, one arm.
Two pair assembly example (5/5). Claude Opus 5 at high effort in Claude Code, single arm. 14.4 min real at 8x. No failed commands; its report notes that the steep overhead view made the upright posts look like they were lying down, and that it checked with wrist depth.
The other arm on the rail sometimes runs a separate trial at the same time. Where that is visible in a clip, the caption says so and the frame is cropped to the arm being discussed.
Time, token and cost cells are computed from the per-trial usage records. The Success column is different. The resource rows can be recomputed from data files; the denominators cannot all be. All five GPT-5.6 Luna sessions and all five Claude Fable 5.1 sessions are now on record, but three of the Luna sessions ended without a final report, through a timeout or an abort, and one Fable session has no report either. Those outcomes are recorded by the authors.
Every trial so far started from nothing. These three studies let the agent keep what it learned. Experience is ordinary files: measurements, procedures, corrections and working scripts, written at the end of one execution and read at the start of the next. Each execution still begins with a fresh agent context, so the only thing that carries over is what was written down.
Two comparisons on this page are two separate clips, not one composed video: ring cycle 1 beside ring cycle 4, and the same model with and without the transferred store. Each pair starts at the same task second and runs at the same speed, so the shorter run simply finishes first. Play them together; each still states its own trial and real elapsed time.
One arm takes a ridged ring off its base and puts it back, five times over, in a single continuous session with GPT-6 Astra at high thinking effort. Each cycle starts a new agent process that reads the previous checkpoint and updates the object poses from what it can see now. The first cycle has nothing to read: its first insertion stalls, the agent lifts the part safely, measures an offset of about 9 mm between the centre of the grasped ring and the tool, and corrects for it. Later cycles reuse that corrected alignment procedure and check it against current observations.
Cycle 1 of 5, with nothing in the experience store. The first insertion stalls; the agent measures a 9 mm grasp offset and retries.
The fourth cycle of the same continuous session, run by a fresh agent that reads what the earlier cycles wrote down. No retries.
| Cycle | Total runtime | Reasoning and programming | Tool execution | Experience writing | Retries | Calls |
|---|---|---|---|---|---|---|
| 1 | 989.7 | 702.9 | 167.4 | 119.4 | 1 | 37 / 37 |
| 2 | 522.9 | 282.1 | 126.7 | 114.1 | 0 | 32 / 33 |
| 3 | 489.9 | 245.0 | 127.3 | 117.6 | 0 | 32 / 33 |
| 4 | 484.4 | 247.7 | 124.6 | 112.2 | 0 | 32 / 33 |
| 5 | 461.7 | 268.3 | 125.0 | 68.5 | 0 | 32 / 33 |
Scroll the table sideways for the rest of the columns.
Total runtime falls from 989.7 to 461.7 seconds. The largest reduction follows the first cycle, after the agent establishes an alignment procedure and recovers from a failed insertion. Most of the savings come from reasoning and programming.
Task execution includes reasoning and programming and tool execution. Tool execution covers recorded time running programs, processing images and operating the robot. Overlapping activities count once, and tool time takes precedence when tools run during a model response. The remaining task time estimates reasoning and programming, including model response and tool startup delays. Experience writing covers the subsequent saving of updated experience. Components sum to total runtime up to rounding. Calls lists motion and gripper requests followed by image capture requests.
These are five dependent repetitions in one session, with no matched control in which every cycle starts from empty experience. They cannot separate the effect of the saved experience from anything else that changes between cycles.
| # | Reasoning and programming | Tool execution | Experience writing | Total |
|---|---|---|---|---|
| 1 | 702.9 | 167.4 | 119.4 | 989.7 |
| 2 | 282.1 | 126.7 | 114.1 | 522.9 |
| 3 | 245.0 | 127.3 | 117.6 | 489.9 |
| 4 | 247.7 | 124.6 | 112.2 | 484.4 |
| 5 | 268.3 | 125.0 | 68.5 | 461.7 |
All values in seconds. Components sum to total runtime up to rounding.
Source: Appendix Table 7, Execution breakdown over repeated tasks.
Five executions use GPT-6 Astra at high thinking effort with a single arm. Each starts with a fresh agent context and reads saved experience and programs from earlier executions. Total runtime decreases from 621.4 to 462.4 seconds. Reasoning and programming falls from 351.9 to 186.6 seconds, while tool execution rises slightly from 160.5 to 175.5 seconds.
Successful assembly is assessed by the agent from recorded images. Task execution includes the return to the observation posture, followed by experience writing.
Task execution includes reasoning and programming and tool execution. Tool execution covers recorded time running programs, processing images and operating the robot. Overlapping activities count once, and tool time takes precedence when tools run during a model response. The remaining task time estimates reasoning and programming, including model response and tool startup delays. Experience writing covers the subsequent saving of updated experience. Components sum to total runtime up to rounding. Calls lists motion and gripper requests followed by image capture requests.
| # | Reasoning and programming | Tool execution | Experience writing | Total |
|---|---|---|---|---|
| 1 | 351.9 | 160.5 | 108.9 | 621.4 |
| 2 | 281.7 | 186.3 | 100.1 | 568.1 |
| 3 | 237.1 | 173.8 | 96.6 | 507.5 |
| 4 | 221.0 | 174.1 | 96.6 | 491.8 |
| 5 | 186.6 | 175.5 | 100.3 | 462.4 |
All values in seconds. Components sum to total runtime up to rounding.
Source: Appendix Table 7, Execution breakdown over repeated tasks.
GPT-6 Astra ran the two pair assembly task five times and saved geometry, procedures, corrections and scripts. GPT-5.6 Terra, at high thinking effort, then received that collection with read access only, frozen, unchanged between trials. The comparison is Terra with nothing against Terra reading the Astra collection, with the same task references, the same interface and the same budgets on both sides.
| Condition | Success | Time (min) | Tokens (M) | Cost (USD) |
|---|---|---|---|---|
| GPT-5.6 Terra, empty experience | 1/5 | 19.0 | 6.28 | 1.98 |
| GPT-5.6 Terra, reading the Astra collection | 4/5 | 12.5 [10.1, 16.9] | 5.56 [4.81, 5.87] | 1.64 [1.56, 1.74] |
| GPT-6 Astra, the five runs that built the collection | 5/5 | 7.6 [6.5, 9.1] | 2.50 [2.24, 2.83] | 3.79 [3.45, 4.39] |
Scroll the table sideways for the rest of the columns.
Means over successful trials, with the observed minimum and maximum beneath, the same accounting as the model comparison above. The empty-experience figure is the one successful Terra trial from that table, so it has no spread.
The 5 of 5 in the third row is one of the four Success counts on this page that rest on the authors' own record. All five runs are on file with full time, token and cost accounting, but the operator verdict column of that batch record was left blank, so the count comes from the single decision of 11 September 2026, in which Yang Lin, the author who ran the trials, confirmed all four of these counts, rather than from the record.
Two pair assembly, GPT-5.6 Terra at high effort in Codex CLI, single arm, empty experience store: the only success in that row (1/5). 19.0 min real at 8x, with several settle-miss recoveries. Shown from task second 90 at the same speed as the clip beside it.
The same task and the same model, now reading a frozen experience store written by GPT-6 Astra and never written back to: 4/5 successful, 11.2 min real. Same 8x and the same task second 90 start, so the clip simply ends sooner.
With the collection in front of it, Terra succeeds in four trials instead of one, finishes faster, and uses fewer tokens. It does not reach the model that wrote the collection: Astra still finished in 7.6 minutes on average. Cost moves the other way, because Terra is the cheaper model per token.
| Condition | Success | Time (min), mean [min, max] | Tokens (M), mean [min, max] | Cost (USD), mean [min, max] | Per-trial completion time (min) |
|---|---|---|---|---|---|
| GPT-5.6 Terra, empty experience | 1/5 | 19.0 [19.0, 19.0] | 6.28 [6.28, 6.28] | 1.98 [1.98, 1.98] | 14.4 (failed), 19.0, 21.4 (failed), 22.1 (failed), 30.4 (failed) |
| GPT-5.6 Terra, reading a frozen GPT-6 Astra experience store | 4/5 | 12.5 [10.1, 16.9] | 5.56 [4.81, 5.87] | 1.64 [1.56, 1.74] | 10.1, 11.2, 11.7, 16.9, 19.6 (failed) |
| GPT-6 Astra, building the experience store | 5/5 | 7.6 [6.5, 9.1] | 2.50 [2.24, 2.83] | 3.79 [3.45, 4.39] | 6.5, 7.0, 7.3, 8.2, 9.1 |
Means, minima and maxima are over successful trials only. The third row is a different model, shown for reference.
Source: the pinned per-trial records of the two pair assembly experience transfer runs.
In the agent's own words
One failed assembly attempt preserved (cap30 failure): aligning TCP directly to receiver led to visible lateral offset and SETTLE_MISS at commanded z .028 (actual .0308). Fresh depth cap29 showed effective ring +.009 x from TCP; lifted safely .065, closed failure, started retry, corrected x then seated successfully. No hardware fault. Initial high downward observation IK candidates rejected without motion; successful observation is (.23,0,.15). Ordinary joint tolerance can allow 5–8 mm TCP error; final approaches use Cartesian. Repeated deltas accumulated x drift; use absolute measured targets. Ring moved/effective pose changed after release; remeasure. No force measurements.
The timelines of all 62 main result trials are reconstructed by matching robot session records with timestamped agent traces, covering 7,756 paired tool calls. The means below include successful and unsuccessful trials and cover the full recorded execution, including final reporting.
| Task | Trials | Total | Visual loop | Remaining response | Local tools | Action service | Check | Report |
|---|---|---|---|---|---|---|---|---|
| Four pair assembly | 10 | 48.5 | 25.5 | 9.9 | 0.3 | 11.6 | 0.5 | 0.8 |
| Pyramid | 10 | 22.3 | 8.7 | 3.4 | 0.1 | 8.8 | 0.6 | 0.7 |
| Two towers | 10 | 21.4 | 8.5 | 4.7 | 0.1 | 7.1 | 0.2 | 0.9 |
| Six block tower | 10 | 29.8 | 14.1 | 5.0 | 0.1 | 9.3 | 0.5 | 0.8 |
| Dice flipping | 10 | 38.6 | 16.2 | 7.7 | 0.2 | 12.9 | 0.8 | 0.8 |
| Targeted throwing, potato | 2 | 23.7 | 7.2 | 12.7 | 0.7 | 1.9 | 0.2 | 0.9 |
| Towel folding, sequential | 5 | 52.7 | 25.4 | 10.0 | 0.5 | 14.7 | 0.3 | 1.9 |
| Towel folding, simultaneous | 5 | 23.2 | 12.2 | 1.6 | 0.1 | 7.1 | 0.3 | 2.0 |
Scroll the table sideways for the rest of the columns.
Mean minutes per trial.
| Configuration | Trials | Visual loop | Remaining response | Local tools | Action service | Check | Report | Total |
|---|---|---|---|---|---|---|---|---|
| Four pair assembly | 10 | 25.5 | 9.9 | 0.3 | 11.6 | 0.5 | 0.8 | 48.5 |
| Block construction · pyramid | 10 | 8.7 | 3.4 | 0.1 | 8.8 | 0.6 | 0.7 | 22.3 |
| Block construction · two towers | 10 | 8.5 | 4.7 | 0.1 | 7.1 | 0.2 | 0.9 | 21.4 |
| Block construction · six block tower | 10 | 14.1 | 5.0 | 0.1 | 9.3 | 0.5 | 0.8 | 29.8 |
| Dice flipping | 10 | 16.2 | 7.7 | 0.2 | 12.9 | 0.8 | 0.8 | 38.6 |
| Targeted throwing · potato | 2 | 7.2 | 12.7 | 0.7 | 1.9 | 0.2 | 0.9 | 23.7 |
| Towel folding · sequential | 5 | 25.4 | 10.0 | 0.5 | 14.7 | 0.3 | 1.9 | 52.7 |
| Towel folding · simultaneous | 5 | 12.2 | 1.6 | 0.1 | 7.1 | 0.3 | 2.0 | 23.2 |
All values in minutes. Rows sum to the total.
Source: the execution time allocation records, rebuilt for all 62 main result trials.
The visual loop accounts for 25.5 of the 48.5 mean minutes in four pair assembly and 14.1 minutes in the six block tower. Throwing uses 1.9 minutes of action service and 12.7 minutes of remaining response, reflecting the analysis performed after the object lands.
Five of the 62 main result trials were unsuccessful, as were 11 of the 40 model comparison trials.
Three pairs went together. The hollow tube was knocked over and stood back up twice. On the final insertion the gripper was already partly open when the arm moved, the long pin left the jaws, and the agent could not locate it again.
Three pairs are assembled; on the last insertion the gripper opens too early, the long pin leaves the jaws and is never found.
An unsuccessful four pair assembly example. Three pairs are assembled; on the last insertion the gripper opens too early, the long pin leaves the jaws and is never found. 83:30-86:30 at 8x, then 86:30-89:00 at 8x. The agent stops and reports the failure itself.
The fourth cube went down out of alignment, and the agent regrasped and corrected it while the stack was still intact. The sixth level was beyond a straight-down reach, so it built the top two cubes on the table and carried them up as a pair. All six stood after release. The tower came down during the withdrawal that followed.
All six cubes stand, then the tower collapses as the arm withdraws and one cube lands out of reach. The collapse plays at 8x.
An unsuccessful six block tower example. 35.6 min. The agent builds the top two cubes as a pair on the table and transfers them; all six stand, then the tower collapses as the arm withdraws and one cube lands out of reach. Side view: 27:20-31:24 at 8x, then the collapse at 8x; 41 s and 63 s of the run are skipped between the three pieces.
The agent carried both short ends inward at the same time and staged the placements, which is what the demonstration asks for.
The concurrent inward carry works, but earlier corner slips and a curled long edge leave a loose triangular corner.
An unsuccessful simultaneous towel folding example. 27.6 min. The concurrent inward carry works, but earlier corner slips and a curled long edge leave a loose triangular corner. 8x. The agent called it an approximate match; the human review counted it a failure.
The circular pair went together. On the second pair the hexagonal ring slipped and knocked its post out of reach near the base of the arm. The agent tried to recover, could not, and reported partial success rather than success.
The round pair goes together; then the hex ring slips and pushes its post into the zone the arm cannot reach.
Two pair assembly example (1/5 for this model). GPT-5.6 Terra at high effort in Codex CLI, single arm. The round pair goes together; then the hex ring slips and pushes its post into the zone the arm cannot reach. 22.1 min to its own report, which calls it partial success, not task success. 8x.
Every trial starts a fresh agent session with the interface documentation, the task instructions, the reference media and an empty workspace. Solutions and logs from earlier trials are not supplied.
GPT-6 Astra at high thinking effort, inside Codex CLI, for every main result configuration.
Each trial begins from a documented physical reset and an initial scene drawn from a predefined set of placements. Model and experience comparisons use matched scenes with counterbalanced order.
Each task has a fixed wall-clock budget and a fixed budget for observation and action requests, shared across configurations. Recovery inside those budgets is allowed.
The agent reporting completion, the budget running out, or a reset or human intervention. A reset or intervention ends the autonomous trial. Physical failure, budget exhaustion, intervention and infrastructure interruption are recorded separately.
The physical outcome and the execution video, judged against criteria fixed before evaluation. The required assembly, support, orientation and fold relationships have to survive gripper release and arm withdrawal.
Measured from the delivery of the task and its references to the final physical verification. It includes reference processing, model calls, local computation, sensing, motion and recovery. Time is reported over successful trials only, so a configuration with no successful trial has no time.
Tokens are the sum of input and output usage across every model request in the trial, including reasoning and retries, reported in millions. Repeatedly submitted context is counted on each request, and cached input counts once inside that request's input total. Cost uses standard API rates with cached input priced as cached.
† Throwing time includes the trajectory analysis and verification that follow the throw. The throw itself completes an average of 13.9 minutes after the task is delivered, so roughly 40% of the reported 22.8 minutes comes after the object has landed.
| Configuration | Success | Time (min), mean [min, max] | Tokens (M), mean [min, max] | Cost (USD), mean [min, max] | Per-trial completion time (min) |
|---|---|---|---|---|---|
| Four pair assembly | 8/10 | 37.2 [16.0, 59.1] | 13.02 [3.96, 27.12] | 16.62 [5.55, 31.50] | 38.6, 53.0, 89.2 (failed), 47.2, 90.2 (failed), 16.3, 30.3, 16.0, 37.4, 59.1 |
| Block construction · pyramid | 10/10 | 21.6 [14.9, 32.2] | 9.24 [6.27, 12.94] | 11.69 [8.01, 15.28] | 16.8, 23.2, 26.9, 14.9, 32.2, 20.0, 17.0, 17.9, 24.1, 22.4 |
| Block construction · two towers | 10/10 | 20.6 [14.8, 29.2] | 7.94 [4.94, 14.39] | 9.92 [6.27, 16.85] | 21.0, 19.4, 14.8, 22.6, 29.2, 24.4, 17.6, 17.8, 21.9, 17.0 |
| Block construction · six block tower | 9/10 | 28.2 [11.6, 55.0] | 11.57 [4.50, 19.85] | 14.93 [6.50, 26.78] | 46.4, 35.6 (failed), 20.1, 17.3, 55.0, 25.4, 31.0, 29.0, 17.9, 11.6 |
| Dice flipping | 10/10 | 37.9 [22.6, 56.0] | 17.08 [10.16, 25.00] | 21.07 [12.57, 31.20] | 36.4, 56.0, 47.9, 51.4, 22.6, 37.3, 33.6, 32.1, 26.4, 34.9 |
| Targeted throwing · potato | 2/2 | 22.8 [22.2, 23.3] | 6.26 [6.25, 6.26] | 8.94 [8.71, 9.18] | 22.2, 23.3 |
| Towel folding · sequential | 5/5 | 50.8 [33.8, 63.3] | 18.12 [10.13, 25.26] | 24.14 [14.16, 30.90] | 61.6, 50.7, 44.6, 63.3, 33.8 |
| Towel folding · simultaneous | 3/5 | 21.4 [15.7, 31.2] | 5.06 [4.27, 5.68] | 7.46 [6.16, 8.67] | 14.0 (failed), 15.7, 27.6 (failed), 31.2, 17.4 |
Means, minima and maxima are over successful trials only.
Source: the per-trial completion time, token usage and cost table for all 62 main result trials.
The agent asks the robot for an overhead image, a wrist image with aligned depth, or the current joint and end effector state. Captures arrive with calibration and timestamps and are saved into the agent's own workspace.
The agent writes Python during the run to read those files: measure a part from depth, fit a surface, intersect a camera ray with a plane, triangulate a point seen from two views, or compose several robot calls into one step.
The agent submits a target: either joint angles or an end effector position and orientation, plus a separate gripper opening. The robot runtime turns the target into joint trajectories under workspace and motion limits, using Mink for inverse kinematics.
Each motion returns the measured state and the arrival error. The agent takes a new image when it wants one, and decides for itself whether the physical result matches its estimate.
A failed insertion can prompt a closer look, a new alignment estimate, or a different approach direction. The agent keeps going until it reports completion or ends the attempt inside the trial budget.
state, status, help, frames, move_ee, move_joints and gripper.The robot is an I2RT YAM arm with six revolute joints and a parallel gripper over a tabletop workspace. A wrist-mounted Intel RealSense D405 supplies aligned colour and depth at 640 by 360 pixels, and a fixed overhead Logitech BRIO supplies rectified colour at 1920 by 1080. Bimanual towel folding adds a second arm expressed in the first arm's coordinate frame. Targeted throwing uses a separate runtime with timed motion programs, so that the swing and the gripper release can be scheduled together.
Most trials end with a report written by the agent, and every trial has a separate review of the final state and the execution video by a person. Five of the 112 counted runs ended with no report at all, so on those there is nothing on the agent's side to compare. Where both exist they can differ. On this page a disagreement means the agent stated an outcome that the review contradicted: the agent called a trial successful and the review called it a failure, or the other way round. Trials whose report could not be classified automatically are counted apart, under a label of their own, because that is a limit of the parser and not a judgement by the agent.
Three pairs went together. The hollow tube was knocked over and stood back up twice. On the final insertion the gripper was already partly open when the arm moved, the long pin left the jaws, and the agent could not locate it again. It stopped and reported the failure rather than claiming the task was done.
Agent report: failure. Human review: failure. They agree.
The fourth cube went down out of alignment, and the agent regrasped and corrected it while the stack was still intact. The sixth level was beyond a straight-down reach, so it built the top two cubes on the table and carried them up as a pair. All six stood after release. The tower came down during the withdrawal that followed. The agent then measured where the cubes had landed, found one grey cube at about 0.70 m in the robot base frame against a 0.65 m radial grasp limit, and ended the attempt as unsuccessful with time and command budget still left.
Agent report: failure. Human review: failure. They agree.
This is why withdrawal counts as part of the task, and it is also where recovery stops: reobservation and regrasping fixed the earlier alignment error, but they could not reach a cube that had left the workspace.
The agent carried both short ends inward at the same time and staged the placements, which is what the demonstration asks for. Repeated corner slips earlier in the run, and a curled long-edge fold, left a loose triangular corner that is plain in the final overhead image.
Agent report: an approximate or partial match. Human review: failure. This is a difference of degree rather than a contradiction, and the trial is counted as a failure.
The circular pair went together. On the second pair the hexagonal ring slipped and knocked its post out of reach near the base of the arm. The agent tried to recover, could not, and reported partial success rather than success.
Agent report: partial success, not task success. Human review: failure.
The main fold order was attempted correctly, but the final flaps are skewed, unevenly overlapped, and not reasonably aligned. Success: no.
Towel folding, sequential. The review counted this example a success; the agent counted it a failure.
The page states two counts, how many trials carry an explicit disagreement between the agent's stated outcome and the human review, and how many reports the automatic parser could not classify. The second group is much the larger of the two, and it is not evidence of disagreement.
Across 65 recorded executions, 18 sessions retained 37 Python working files. These comprise 29 files for perception and geometry, 7 for composing robot calls, and 1 for drawing reference points into an image. The logs hold a further 135 shell events with inline Python across 48 sessions. One case can be followed all the way from a camera image to a physical motion.
The agent compared wrist camera views and depth measurements. Bright white surfaces sometimes gave unreliable depth readings.
An executed Python command converted selected pixels in captures 1 and 16 into viewing rays using their camera calibration. It computed the closest points on the two rays.
The hexagonal ring caught on the peg. Contact stops led to lifts and revised alignment using the visible ring hole and peg. The ring passed over the peg and settled near its foot after release.
The trial record reports success. The final photographs show all four assemblies after the gripper released the parts and the arm withdrew.
Executed 3D estimation code from the assembly run, with comments and line breaks added. The ray function uses the calibration for each capture to return the camera position and viewing direction.
# Camera positions p, q and viewing directions a, b
p,a=ray(1,450,175)
q,b=ray(16,293,189)
dot=lambda a,b:sum(x*y for x,y in zip(a,b))
# Geometry between the two viewing rays
d=[q[i]-p[i] for i in range(3)]
aa=dot(a,a); bb=dot(b,b); ab=dot(a,b)
ad=dot(a,d); bd=dot(b,d)
# Solve for the closest point on each ray
t=(ad*bb-ab*bd)/(aa*bb-ab*ab)
s=(ab*ad-aa*bd)/(aa*bb-ab*ab)
# Print the two estimated 3D positions as [x, y, z]
print([p[i]+t*a[i] for i in range(3)],
[q[i]+s*b[i] for i in range(3)])View the assembly run and its final photograph
What the programs contribute to performance is not established. Settling that would need comparisons that hold the interface capabilities and the inherited experience fixed.
@article{jia2026agentaspolicy,
title = {Agent as Policy for Robotic Manipulation},
author = {Jia, Mengzhao and Lin, Yang and Zhang, Xixin and Zhang, Zhihan and Liu, Xiaobai and Jiang, Meng},
journal = {arXiv preprint arXiv:2609.12541},
year = {2026}
}