Skip to the content Skip to the main results

Agent as Policy for Robotic Manipulation

A general purpose coding agent acts as a zero-shot robot policy. It observes the scene through robot cameras, writes executable programs, commands robot motions, and adapts its actions using physical feedback.

Mengzhao Jia1,* Yang Lin2,* Xixin Zhang3,* Zhihan Zhang1 Xiaobai Liu3 Meng Jiang1

1University of Notre Dame 2University of California San Diego 3San Diego State University

* Core contributors. Contact mjia2@nd.edu

arXivPDFHF DatasetBibTeX

AGP real world task examples

Bimanual towel folding

Follow a human demonstration using sequential or simultaneous folds.

Choose a variation

Task input

Video + images + text

Fold the long edge, then bring the short ends inward one at a time, following the demonstration.

Task instruction · condensed

Human demonstration · 1×
Before folding
Target fold
Top cameraWrist cameras8× playback
Each task in full, with its goal image and its part measurements ↗
Task reference image

Overview

Agent as Policy (AGP) lets a general purpose agent directly control a physical robot without training for individual tasks. The agent interprets visual observations, writes executable programs, commands motion, and adjusts its actions using physical feedback. We evaluate AGP on assembly from human videos, block construction from goal images, die reorientation, targeted throwing, and bimanual towel folding.

How AGP works

AGP gives the agent a task and a documented robot interface, and then leaves the runtime decisions to the agent.

How AGP runs a taskDiagram of AGP task preparation and execution. A separate preparation agent creates a reusable definition when a task type is first introduced. A launch program provides that definition and current instance details to the execution agent. Four numbered cards show a typical sequence of observing, writing or selecting and running programs, submitting action targets, and checking outcomes. The return arrow represents choosing the next observation, program or action using feedback. The agent chooses tool calls and observation timing and can combine operations in one tool call. Separate arrows show observation requests, returned images and state, action targets, and motion feedback. New images are acquired through observation requests. The tabletop interface has seven commands grouped into observation, interface information, and action. Programs process observations and compute action targets. The bridge checks requests against workspace and motion limits, and the robot provides cameras, trajectory execution and measured state. Targeted throwing uses a separate runtime of timed motion programs. A text description follows the diagram.How AGP runs a taskA general purpose coding agent executes tasks specified through video, images or language.It uses the task definition, reference materials, robot interface documentation and a writable workspace.TASK PREPARATIONonce for each new task typeA separate coding agent prepares a reusable task definition.A launch program loads it for each task instance.Goal, references, constraints and completion criteria.Interface rules, budgets and reporting requirements.saved definition and current instance detailsTHE EXECUTION AGENTA fresh session for every trial1ObserveAsk the robot for camera framesand its own state. OverheadRGB, wrist RGB with aligneddepth, joint angles and gripperpose, saved into the workspace.2Write, select andrun programsUse Python and availablelibraries to processobservations, compute targetsand compose robot calls. Reuseor revise saved programs.3Submit actiontargetsSend a target end effectorpose, joint angles, or gripperopening through the interface.The controller executesaccepted requests.4Check the outcomeInspect motion feedback. Requestfresh observations as needed.Compare the result with the taskgoal and choose the next step.use feedback to choose the next observation, program or actionTypical sequence. The agent chooses observation timing and can combine operations in one tool call.observationrequestsimages, depth,state, calibrationaction targetspose, joints, grippermotion feedbackstate, target errorsTHE INTERFACEseven commands for the tabletop setupPrograms process observations and compute action targets.ObservationframesstateInterface infostatushelpThe bridge checks requests against workspace and motion limits.Actionmove_eemove_jointsgripperTargeted throwing uses a separate runtime of timed motion programs.robot requestsobservations and motion feedbackTHE ROBOTsix-joint arm with a parallel gripper on a tabletop; a second arm for the two-arm taskCamerasA fixed overhead RGB camera and a wrist RGBcamera with aligned depth. Every capturecarries its calibration, a timestamp and therobot state.ControllerInverse kinematics and trajectory executionunder motion limits. Motor feedback trackstargets. Commands run in order per arm andconcurrently across arms.Robot stateJoint positions from motor feedback, theend-effector pose from forward kinematics,and the error against the commanded target.The cycle ends when the agent reports the task complete, or when the fixed time and request budget runs out.Physical success is evaluated afterwards using the final scene and recorded video, against criteria fixed before the evaluation.Success evaluation is not based on the agent’s own report.

Scroll the diagram sideways, or read it in words below.

The same diagram in words

The AGP loop, in words

When a task type is first introduced, a separate preparation agent creates a reusable task definition with the goal, references, constraints, completion criteria, interface rules, budgets and reporting requirements. A launch program supplies the saved definition and current instance details to the execution agent in a fresh session with interface documentation and a writable workspace. Tasks can be specified through video, images or language.

The four cards illustrate a typical sequence. The execution agent chooses its next tool call from the available history and workspace, determines observation timing, and can combine operations in one tool call.

  1. Observe. Request overhead RGB, wrist RGB with aligned depth, joint angles and gripper pose, saved into the workspace.
  2. Write, select and run programs. Use Python and available libraries to process observations, compute targets and compose robot calls. Reuse or revise saved programs.
  3. Submit action targets. Send a target end effector pose, joint angles or gripper opening through the interface. The controller executes accepted requests.
  4. Check the outcome. Inspect motion feedback. Request fresh observations as needed. Compare the result with the task goal and choose the next step.

The return arrow represents using feedback to choose the next observation, program or action. Separate arrows distinguish observation requests and their returned images, depth, state and calibration from action targets and their motion feedback.

The tabletop interface exposes seven commands. frames and state provide observations. status and help provide interface information. move_ee, move_joints and gripper request actions. The bridge checks requests against workspace and motion limits. Targeted throwing uses a separate runtime of timed motion programs.

The robot provides overhead and wrist cameras, a controller for inverse kinematics and trajectory execution, and state from motor feedback and forward kinematics. Captures include calibration, timestamps and robot state. Commands run in order per arm and concurrently across arms.

The cycle ends when the agent reports the task complete, or when the fixed time and request budget runs out. Physical success is evaluated afterwards using the final scene and recorded video, against criteria fixed before the evaluation. Success evaluation is not based on the agent's own report.

A real assembly example

During a four pair assembly run, the agent used wrist camera images to locate parts and align them for insertion. Bright white surfaces sometimes gave unreliable depth readings, so it compared views and ran a Python calculation using camera calibration.

Estimating a 3D position from two camera views

The same point is selected in two wrist camera images. Camera calibration converts each selection into a 3D viewing ray. This code finds the closest points on those rays and prints their x, y and z coordinates in the robot frame. Comments and line breaks have been added to the executed code for readability.

# Camera positions p, q and viewing directions a, b
p,a=ray(1,450,175)
q,b=ray(16,293,189)
dot=lambda a,b:sum(x*y for x,y in zip(a,b))

# Geometry between the two viewing rays
d=[q[i]-p[i] for i in range(3)]
aa=dot(a,a); bb=dot(b,b); ab=dot(a,b)
ad=dot(a,d); bd=dot(b,d)

# Solve for the closest point on each ray
t=(ad*bb-ab*bd)/(aa*bb-ab*ab)
s=(ab*ad-aa*bd)/(aa*bb-ab*ab)

# Print the two estimated 3D positions as [x, y, z]
print([p[i]+t*a[i] for i in range(3)],
      [q[i]+s*b[i] for i in range(3)])

During the hexagonal ring placement, the ring caught on the peg. The agent lifted it and revised the alignment using the visible ring hole and peg. The ring then passed over the peg and settled near its foot. The completed trial was recorded as successful, with all four assemblies standing after release.

View the assembly run and its final photograph

More details on the platform and agent setup

Main results across manipulation tasks

Eight task configurations, 62 trials, every trial in a fresh session.

62

real-robot trials

7 of 8

configurations at 80% or higher

20.6 to 50.8 min

and USD 7.46 to 24.14

All four figures are calculated from per-trial records.

Main results across manipulation tasks. Success is successful trials over evaluated trials. Time, tokens and cost are means over successful trials, with the observed minimum and maximum beneath.
TaskSuccess (n/N)Time (min)Tokens (M)Cost (USD)
Four pair assemblyLong horizon reasoningLong horizon VideoVideo8/10Four pair assembly, trial 1 of 10: successFour pair assembly, trial 2 of 10: successFour pair assembly, trial 3 of 10: failureFour pair assembly, trial 4 of 10: successFour pair assembly, trial 5 of 10: failureFour pair assembly, trial 6 of 10: successFour pair assembly, trial 7 of 10: successFour pair assembly, trial 8 of 10: successFour pair assembly, trial 9 of 10: successFour pair assembly, trial 10 of 10: success37.2[16.0, 59.1]13.02[3.96, 27.12]16.62[5.55, 31.50]
Block construction: pyramidLong horizon reasoningLong horizon ImageImage10/10Pyramid, trial 1 of 10: successPyramid, trial 2 of 10: successPyramid, trial 3 of 10: successPyramid, trial 4 of 10: successPyramid, trial 5 of 10: successPyramid, trial 6 of 10: successPyramid, trial 7 of 10: successPyramid, trial 8 of 10: successPyramid, trial 9 of 10: successPyramid, trial 10 of 10: success21.6[14.9, 32.2]9.24[6.27, 12.94]11.69[8.01, 15.28]
Block construction: two towersLong horizon reasoningLong horizon ImageImage10/10Two towers, trial 1 of 10: successTwo towers, trial 2 of 10: successTwo towers, trial 3 of 10: successTwo towers, trial 4 of 10: successTwo towers, trial 5 of 10: successTwo towers, trial 6 of 10: successTwo towers, trial 7 of 10: successTwo towers, trial 8 of 10: successTwo towers, trial 9 of 10: successTwo towers, trial 10 of 10: success20.6[14.8, 29.2]7.94[4.94, 14.39]9.92[6.27, 16.85]
Block construction: six block towerLong horizon reasoningLong horizon ImageImage9/10Six block tower, trial 1 of 10: successSix block tower, trial 2 of 10: failureSix block tower, trial 3 of 10: successSix block tower, trial 4 of 10: successSix block tower, trial 5 of 10: successSix block tower, trial 6 of 10: successSix block tower, trial 7 of 10: successSix block tower, trial 8 of 10: successSix block tower, trial 9 of 10: successSix block tower, trial 10 of 10: success28.2[11.6, 55.0]11.56[4.50, 19.85]14.93[6.50, 26.78]
Dice flippingAction diversityAction variety LanguageLanguage10/10Dice flipping, trial 1 of 10: successDice flipping, trial 2 of 10: successDice flipping, trial 3 of 10: successDice flipping, trial 4 of 10: successDice flipping, trial 5 of 10: successDice flipping, trial 6 of 10: successDice flipping, trial 7 of 10: successDice flipping, trial 8 of 10: successDice flipping, trial 9 of 10: successDice flipping, trial 10 of 10: success37.9[22.6, 56.0]17.08[10.16, 25.00]21.07[12.57, 31.20]
Targeted throwing: potato†Action diversityAction variety LanguageLanguage2/2Targeted throwing (potato), trial 1 of 2: successTargeted throwing (potato), trial 2 of 2: success22.8[22.2, 23.3]6.26[6.25, 6.26]8.94[8.71, 9.18]
Towel folding: sequentialObject diversityObject variety VideoVideo5/5Towel folding, sequential, trial 1 of 5: successTowel folding, sequential, trial 2 of 5: successTowel folding, sequential, trial 3 of 5: successTowel folding, sequential, trial 4 of 5: successTowel folding, sequential, trial 5 of 5: success50.8[33.8, 63.3]18.12[10.13, 25.26]24.14[14.16, 30.90]
Towel folding: simultaneousObject diversityObject variety VideoVideo3/5Towel folding, simultaneous, trial 1 of 5: failureTowel folding, simultaneous, trial 2 of 5: successTowel folding, simultaneous, trial 3 of 5: failureTowel folding, simultaneous, trial 4 of 5: successTowel folding, simultaneous, trial 5 of 5: success21.4[15.7, 31.2]5.05[4.27, 5.68]7.46[6.16, 8.67]

Scroll the table sideways for the rest of the columns.

All 62 main-result trials at a glance

How this table was made

How a trial was run and judged

One agent, one session

Every trial starts a fresh agent session with the interface documentation, the task instructions, the reference media and an empty workspace. Solutions and logs from earlier trials are not supplied.

Default configuration

GPT-6 Astra at high thinking effort, inside Codex CLI, for every main result configuration.

Fixed start

Each trial begins from a documented physical reset and an initial scene drawn from a predefined set of placements. Model and experience comparisons use matched scenes with counterbalanced order.

Budgets

Each task has a fixed wall-clock budget and a fixed budget for observation and action requests, shared across configurations. Recovery inside those budgets is allowed.

What ends a trial

The agent reporting completion, the budget running out, or a reset or human intervention. A reset or intervention ends the autonomous trial. Physical failure, budget exhaustion, intervention and infrastructure interruption are recorded separately.

What counts as success

The physical outcome and the execution video, judged against criteria fixed before evaluation. The required assembly, support, orientation and fold relationships have to survive gripper release and arm withdrawal.

Completion time

Measured from the delivery of the task and its references to the final physical verification. It includes reference processing, model calls, local computation, sensing, motion and recovery. Time is reported over successful trials only, so a configuration with no successful trial has no time.

Tokens and cost

Tokens are the sum of input and output usage across every model request in the trial, including reasoning and retries, reported in millions. Repeatedly submitted context is counted on each request, and cached input counts once inside that request's input total. Cost uses standard API rates with cached input priced as cached.

† Throwing time includes the trajectory analysis and verification that follow the throw. The throw itself completes an average of 13.9 minutes after the task is delivered, so roughly 40% of the reported 22.8 minutes comes after the object has landed.

What the capability column means

Long horizon reasoning (Long horizon)
the trial runs many steps and earlier progress has to survive later ones
Action diversity (Action variety)
the motion itself has to change, through grasp-dependent reorientation or a timed release
Object diversity (Object variety)
the object deforms, so the shape grasped is not the shape planned for

Reading the table

  • Success is the number of completely successful trials over the number of evaluated trials. Partial completion does not count.
  • Time, tokens and cost summarize successful trials within each configuration.
  • Taken together, the eight rows hold 57 successes in 62 trials.

Watch one run, second by second

Two complete two pair assembly trials, each with the video lined up against the agent's own record: what it said, which images it opened, which commands it ran, and what the robot answered. The overhead camera is the clock for both: every time in the log is a second of that recording, taken from the recorder's own frame log, and nothing here is aligned by hand. The overhead view is the only one that ships in this version.

How to read this
  • The strip under the video shows where the time goes: model generating, local tool runs, robot motion, camera captures, and the moments the agent opened an image.
  • Times in the log are seconds of the run. The second episode plays at 2x, so its video is half as long as the run it shows; the page converts before it seeks, and the elapsed clock on screen is always the run clock.
  • The still images below the video are full overhead frames, so the second arm, which another agent was driving at the same time on its own trial, is visible in them. The video is cropped to this agent's half of the table.
  • Clicking any entry in the log seeks the video to that moment. Deep links of the form t=123.4 point at an exact second.
  • The two agents expose different things. Codex CLI publishes short reasoning titles and occasional commentary. Claude Code publishes a one-line intent for each shell command and keeps its reasoning private. That is a difference in what the harness reports, not evidence about how much either model reasoned.
  • Text written by the agent is always marked as the agent's own words.

Codex CLI with GPT-6 Astra, high effort

Trial 5 of 5single armrecorded 2026-09-108:33 of recording
About this run

Two pair assembly, trial 5 of 5. GPT-6 Astra at high thinking effort, Codex CLI, single arm. Completion time 7.9 minutes; the overhead recording runs 8 minutes 33 seconds and is shown uncut at 1x, with a speed control. Completion time runs from the moment the task is handed over to the final physical check. The recording continues for about 40 seconds after that check. 150 logged events: 9 agent messages, 20 reasoning titles, 31 command runs, 25 image views over 37 distinct images, and 65 robot commands, 45 of which counted against the request budget.

Where the time went

Episode
514.0 s
Model generating
425.8 s
Robot motion
145.1 s
Tool runs
165.2 s
Arm moving while the model generated
80.4 s

The arm is moving for 145 seconds of the 8 minute 33 second recording. The model is generating for 426 seconds, 80 of which overlap with arm motion, because Codex CLI yields while a motion is running. Where the two episodes differ in that overlap, it is a harness scheduling difference — Codex CLI keeps generating while a motion command is in flight, Claude Code blocks on the tool result — not a property of either model.

What the log holds

Logged events
150
Robot commands
65
Counted against the budget
45
Camera captures
23
Images the agent opened
37

Codex CLI narrates as it goes but exposes only short reasoning titles.

Chapters

  1. 1Read the interface, study the demonstration0:00
  2. 2Survey the scene, measure the parts0:23
  3. 3Pick up the hex part and seat it on its post1:38
  4. 4Check the first pair, approach the round part3:23
  5. 5Pick up the round part and seat it on its cylinder4:55
  6. 6Withdraw, verify, report7:11

Claude Code with Claude Opus 5, high effort

Trial 3 of 5single armrecorded 2026-09-1015:18 of recording
About this run

The same task, trial 3 of 5. Claude Opus 5 at high thinking effort, Claude Code, single arm. Completion time 14.4 minutes; the overhead recording runs 15 minutes 18 seconds, nothing removed, played at 2x in 7 minutes 39 seconds. The clock on screen and every time in the log stay on the run. Completion time runs from the moment the task is handed over to the final physical check. The recording continues for about a minute after that check. 170 logged events: 62 command runs, 37 image views over 42 distinct images, 2 file reads, 2 file writes, and 64 robot commands, 53 of which counted.

Where the time went

Episode
919.0 s
Model generating
710.0 s
Robot motion
164.2 s
Tool runs
193.8 s
Arm moving while the model generated
0 s

The arm is moving for 164 seconds of the 15 minute 18 second recording, and none of that overlaps model generation, because the robot call runs inside the shell command and the agent waits for it. Where the two episodes differ in that overlap, it is a harness scheduling difference — Codex CLI keeps generating while a motion command is in flight, Claude Code blocks on the tool result — not a property of either model.

What the log holds

Logged events
170
Robot commands
64
Counted against the budget
53
Camera captures
24
Images the agent opened
42

Claude Code states its intent per command but hides its reasoning text, so this episode reads as commands rather than narration. The difference is what each harness publishes, not how much either model reasoned.

Chapters

  1. 1Read the interface, study the demonstration0:00
  2. 2Survey the scene, measure the parts0:32
  3. 3Pick up the hex part and seat it on its post5:56
  4. 4Check the first pair, approach the round part9:13
  5. 5Pick up the round part and seat it on its cylinder11:43
  6. 6Withdraw, verify, report13:16
About the published data

The published timeline carries only whitelisted fields. Account, rate-limit, cost and session identifiers, absolute paths, host and network addresses and embedded images are removed before anything is written to the site, and the thumbnails are regenerated from the saved camera frames.

Without JavaScript the section falls back to a poster strip and the agent's own words as video captions.

Reusing experience

Ring, cycle 1: 870 s

Cycle 1 of 5, with nothing in the experience store. The first insertion stalls; the agent measures a 9 mm grasp offset and retries.

Task
Ring disassembly and reassembly, cycle 1
Trial
1 of 5
Outcome
success after one retry
Model
GPT-6 Astra (high)
Harness
Codex CLI
Arm
single
Speed
8x
Real time
14.5 min real
Clip
109 s

Ring, cycle 4: 372 s

The fourth cycle of the same continuous session, run by a fresh agent that reads what the earlier cycles wrote down. No retries.

Task
Ring disassembly and reassembly, cycle 4
Trial
4 of 5
Outcome
success
Model
GPT-6 Astra (high)
Harness
Codex CLI
Arm
single
Speed
8x
Real time
6.2 min real
Clip
47 s

Handing that experience to a smaller model: GPT-5.6 Terra reached 1 of 5 with an empty store and 4 of 5 reading a frozen GPT-6 Astra store.

The three studies in full

Details and full results

See the links below for experimental details and results from all runs.

  • Failure cases

    Four recorded failures with video, agent reports and human evaluation.

  • All 112 counted trials

    All 112 slots: final photo, outcome, model, arm, time, tokens, cost.

  • Every number

    The model comparison, the experience studies, the time accounting, the protocol.

    • On the same task GPT-5.6 Luna reached 0 of 5 and Claude Fable 5.1 reached 3 of 5.

Limits

  • 20.6 to 50.8 minutes and USD 7.46 to 24.14 per success.
  • Limited trials per configuration due to the time and cost of real-robot evaluation.
  • Almost every clip is sped up; each states its speed.

The full list

Cite this work

@article{jia2026agentaspolicy,
  title   = {Agent as Policy for Robotic Manipulation},
  author  = {Jia, Mengzhao and Lin, Yang and Zhang, Xixin and Zhang, Zhihan and Liu, Xiaobai and Jiang, Meng},
  journal = {arXiv preprint arXiv:2609.12541},
  year    = {2026}
}

Credits

Assembly part geometry
Adapted from the AutoMate dataset. Tang et al., AutoMate: Specialist and Generalist Assembly Policies over Diverse Geometries, 2024, arXiv:2407.08028. The parts on this page are 3D prints of those meshes, scaled with their aspect ratios preserved.
Robot arm
I2RT YAM arm and the I2RT robot software.
Inverse kinematics
Mink, by Kevin Zakka: Python inverse kinematics based on MuJoCo.
Coding agent
Codex CLI, by OpenAI. Claude Code, by Anthropic.