All 112 counted trials
Browse final photographs, outcomes and resource usage for every evaluated trial. Success and failure are determined by human evaluation.
Explore detailed trial data in the HF Dataset.
62 main results
Four pair assemblytrial 1 of 10success- Time
- 38.6 min
- Tokens
- 12.84 M
- Cost
- USD 16.92
Four pair assemblytrial 2 of 10success- Time
- 53.0 min
- Tokens
- 17.14 M
- Cost
- USD 22.91
Four pair assemblytrial 3 of 10failure- Time
- 89.2 min
- Tokens
- 40.41 M
- Cost
- USD 48.39
Four pair assemblytrial 4 of 10success- Time
- 47.2 min
- Tokens
- 16.06 M
- Cost
- USD 19.94
Four pair assemblytrial 5 of 10failure- Time
- 90.2 min
- Tokens
- 29.36 M
- Cost
- USD 37.32
Four pair assemblytrial 6 of 10success- Time
- 16.3 min
- Tokens
- 4.62 M
- Cost
- USD 6.58
Four pair assemblytrial 7 of 10success- Time
- 30.3 min
- Tokens
- 10.46 M
- Cost
- USD 13.27
Four pair assemblytrial 8 of 10success- Time
- 16.0 min
- Tokens
- 3.96 M
- Cost
- USD 5.55
Four pair assemblytrial 9 of 10success- Time
- 37.4 min
- Tokens
- 11.98 M
- Cost
- USD 16.32
Four pair assemblytrial 10 of 10success- Time
- 59.1 min
- Tokens
- 27.12 M
- Cost
- USD 31.50
Pyramidtrial 1 of 10success- Time
- 16.8 min
- Tokens
- 6.36 M
- Cost
- USD 8.17
Pyramidtrial 2 of 10success- Time
- 23.2 min
- Tokens
- 8.66 M
- Cost
- USD 11.15
Pyramidtrial 3 of 10success- Time
- 26.9 min
- Tokens
- 11.12 M
- Cost
- USD 15.28
Pyramidtrial 4 of 10success- Time
- 14.9 min
- Tokens
- 6.27 M
- Cost
- USD 8.01
Pyramidtrial 5 of 10success- Time
- 32.2 min
- Tokens
- 12.94 M
- Cost
- USD 15.19
Pyramidtrial 6 of 10success- Time
- 20.0 min
- Tokens
- 11.15 M
- Cost
- USD 14.31
Pyramidtrial 7 of 10success- Time
- 17.0 min
- Tokens
- 6.85 M
- Cost
- USD 8.68
Pyramidtrial 8 of 10success- Time
- 17.9 min
- Tokens
- 9.25 M
- Cost
- USD 11.26
Pyramidtrial 9 of 10success- Time
- 24.1 min
- Tokens
- 9.95 M
- Cost
- USD 12.83
Pyramidtrial 10 of 10success- Time
- 22.4 min
- Tokens
- 9.86 M
- Cost
- USD 12.01
Two towerstrial 1 of 10success- Time
- 21.0 min
- Tokens
- 7.61 M
- Cost
- USD 9.57
Two towerstrial 2 of 10success- Time
- 19.4 min
- Tokens
- 7.80 M
- Cost
- USD 9.68
Two towerstrial 3 of 10success- Time
- 14.8 min
- Tokens
- 4.94 M
- Cost
- USD 6.27
Two towerstrial 4 of 10success- Time
- 22.6 min
- Tokens
- 6.91 M
- Cost
- USD 9.01
Two towerstrial 5 of 10success- Time
- 29.2 min
- Tokens
- 14.39 M
- Cost
- USD 16.85
Two towerstrial 6 of 10success- Time
- 24.4 min
- Tokens
- 9.58 M
- Cost
- USD 11.69
Two towerstrial 7 of 10success- Time
- 17.6 min
- Tokens
- 5.98 M
- Cost
- USD 8.11
Two towerstrial 8 of 10success- Time
- 17.8 min
- Tokens
- 7.80 M
- Cost
- USD 10.04
Two towerstrial 9 of 10success- Time
- 21.9 min
- Tokens
- 9.25 M
- Cost
- USD 11.37
Two towerstrial 10 of 10success- Time
- 17.0 min
- Tokens
- 5.12 M
- Cost
- USD 6.56
Six block towertrial 1 of 10success- Time
- 46.4 min
- Tokens
- 19.08 M
- Cost
- USD 25.29
Six block towertrial 2 of 10failure- Time
- 35.6 min
- Tokens
- 13.51 M
- Cost
- USD 17.02
Six block towertrial 3 of 10success- Time
- 20.1 min
- Tokens
- 8.44 M
- Cost
- USD 10.62
Six block towertrial 4 of 10success- Time
- 17.3 min
- Tokens
- 5.09 M
- Cost
- USD 6.69
Six block towertrial 5 of 10success- Time
- 55.0 min
- Tokens
- 19.85 M
- Cost
- USD 26.78
Six block towertrial 6 of 10success- Time
- 25.4 min
- Tokens
- 13.19 M
- Cost
- USD 15.72
Six block towertrial 7 of 10success- Time
- 31.0 min
- Tokens
- 11.30 M
- Cost
- USD 15.48
Six block towertrial 8 of 10success- Time
- 29.0 min
- Tokens
- 13.71 M
- Cost
- USD 16.55
Six block towertrial 9 of 10success- Time
- 17.9 min
- Tokens
- 8.94 M
- Cost
- USD 10.78
Six block towertrial 10 of 10success- Time
- 11.6 min
- Tokens
- 4.50 M
- Cost
- USD 6.50
Dice flippingtrial 1 of 10success- Time
- 36.4 min
- Tokens
- 10.16 M
- Cost
- USD 12.57
Dice flippingtrial 2 of 10success- Time
- 56.0 min
- Tokens
- 23.60 M
- Cost
- USD 28.76
Dice flippingtrial 3 of 10success- Time
- 47.9 min
- Tokens
- 20.19 M
- Cost
- USD 23.72
Dice flippingtrial 4 of 10success- Time
- 51.4 min
- Tokens
- 25.00 M
- Cost
- USD 31.20
Dice flippingtrial 5 of 10success- Time
- 22.6 min
- Tokens
- 13.35 M
- Cost
- USD 16.34
Dice flippingtrial 6 of 10success- Time
- 37.3 min
- Tokens
- 14.49 M
- Cost
- USD 18.82
Dice flippingtrial 7 of 10success- Time
- 33.6 min
- Tokens
- 16.46 M
- Cost
- USD 20.27
Dice flippingtrial 8 of 10success- Time
- 32.1 min
- Tokens
- 21.10 M
- Cost
- USD 25.70
Dice flippingtrial 9 of 10success- Time
- 26.4 min
- Tokens
- 11.78 M
- Cost
- USD 14.46
Dice flippingtrial 10 of 10success- Time
- 34.9 min
- Tokens
- 14.63 M
- Cost
- USD 18.83
Targeted throwing (potato)trial 1 of 2success- Time
- 22.2 min
- Tokens
- 6.25 M
- Cost
- USD 8.71
Targeted throwing (potato)trial 2 of 2success- Time
- 23.3 min
- Tokens
- 6.26 M
- Cost
- USD 9.18
Towel folding, sequentialtrial 1 of 5success- Time
- 61.6 min
- Tokens
- 17.58 M
- Cost
- USD 27.98
Towel folding, sequentialtrial 2 of 5success- Time
- 50.7 min
- Tokens
- 18.58 M
- Cost
- USD 23.53
Towel folding, sequentialtrial 3 of 5success- Time
- 44.6 min
- Tokens
- 19.06 M
- Cost
- USD 24.14
Towel folding, sequentialtrial 4 of 5success- Time
- 63.3 min
- Tokens
- 25.26 M
- Cost
- USD 30.90
Towel folding, sequentialtrial 5 of 5success- Time
- 33.8 min
- Tokens
- 10.13 M
- Cost
- USD 14.16
Towel folding, simultaneoustrial 1 of 5failure- Time
- 14.0 min
- Tokens
- 3.83 M
- Cost
- USD 5.86
Towel folding, simultaneoustrial 2 of 5success- Time
- 15.7 min
- Tokens
- 4.27 M
- Cost
- USD 6.16
Towel folding, simultaneoustrial 3 of 5failure- Time
- 27.6 min
- Tokens
- 8.93 M
- Cost
- USD 12.73
Towel folding, simultaneoustrial 4 of 5success- Time
- 31.2 min
- Tokens
- 5.68 M
- Cost
- USD 8.67
Towel folding, simultaneoustrial 5 of 5success- Time
- 17.4 min
- Tokens
- 5.21 M
- Cost
- USD 7.54
40 model comparison
GPT-6 Astra, low efforttrial 1 of 5success- Time
- 7.4 min
- Tokens
- 2.70 M
- Cost
- USD 3.77
GPT-6 Astra, low efforttrial 2 of 5success- Time
- 19.0 min
- Tokens
- 7.72 M
- Cost
- USD 9.56
GPT-6 Astra, low efforttrial 3 of 5success- Time
- 6.0 min
- Tokens
- 2.81 M
- Cost
- USD 3.63
GPT-6 Astra, low efforttrial 4 of 5success- Time
- 6.3 min
- Tokens
- 1.44 M
- Cost
- USD 2.15
GPT-6 Astra, low efforttrial 5 of 5success- Time
- 10.9 min
- Tokens
- 3.78 M
- Cost
- USD 4.82
GPT-6 Astra, medium efforttrial 1 of 5success- Time
- 6.3 min
- Tokens
- 1.84 M
- Cost
- USD 2.76
GPT-6 Astra, medium efforttrial 2 of 5success- Time
- 15.4 min
- Tokens
- 5.80 M
- Cost
- USD 7.48
GPT-6 Astra, medium efforttrial 3 of 5success- Time
- 8.8 min
- Tokens
- 2.51 M
- Cost
- USD 3.46
GPT-6 Astra, medium efforttrial 4 of 5success- Time
- 7.6 min
- Tokens
- 2.96 M
- Cost
- USD 3.86
GPT-6 Astra, medium efforttrial 5 of 5success- Time
- 7.6 min
- Tokens
- 2.01 M
- Cost
- USD 2.90
GPT-6 Astra, high efforttrial 1 of 5success- Time
- 8.9 min
- Tokens
- 4.02 M
- Cost
- USD 5.19
GPT-6 Astra, high efforttrial 2 of 5success- Time
- 10.6 min
- Tokens
- 3.14 M
- Cost
- USD 4.39
GPT-6 Astra, high efforttrial 3 of 5success- Time
- 9.0 min
- Tokens
- 2.58 M
- Cost
- USD 3.72
GPT-6 Astra, high efforttrial 4 of 5success- Time
- 9.7 min
- Tokens
- 2.81 M
- Cost
- USD 3.96
GPT-6 Astra, high efforttrial 5 of 5success- Time
- 7.9 min
- Tokens
- 3.94 M
- Cost
- USD 5.08
GPT-5.6 Soltrial 1 of 5success- Time
- 15.2 min
- Tokens
- 7.81 M
- Cost
- USD 4.35
GPT-5.6 Soltrial 2 of 5success- Time
- 15.5 min
- Tokens
- 7.83 M
- Cost
- USD 4.24
GPT-5.6 Soltrial 3 of 5success- Time
- 19.3 min
- Tokens
- 9.72 M
- Cost
- USD 5.30
GPT-5.6 Soltrial 4 of 5success- Time
- 12.3 min
- Tokens
- 6.16 M
- Cost
- USD 3.40
GPT-5.6 Soltrial 5 of 5success- Time
- 9.1 min
- Tokens
- 4.12 M
- Cost
- USD 2.40
GPT-5.6 Terratrial 1 of 5failure- Time
- 30.4 min
- Tokens
- 6.32 M
- Cost
- USD 2.04
GPT-5.6 Terratrial 2 of 5failure- Time
- 22.1 min
- Tokens
- 5.14 M
- Cost
- USD 1.86
GPT-5.6 Terratrial 3 of 5failure- Time
- 21.4 min
- Tokens
- 8.99 M
- Cost
- USD 2.62
GPT-5.6 Terratrial 4 of 5success- Time
- 19.0 min
- Tokens
- 6.28 M
- Cost
- USD 1.98
GPT-5.6 Terratrial 5 of 5failure- Time
- 14.4 min
- Tokens
- 2.70 M
- Cost
- USD 0.99
GPT-5.6 Lunatrial 1 of 5failure- Time
- not recorded
- Tokens
- not recorded
- Cost
- not recorded
GPT-5.6 Lunatrial 2 of 5failure- Time
- not recorded
- Tokens
- not recorded
- Cost
- not recorded
GPT-5.6 Lunatrial 3 of 5failure- Time
- not recorded
- Tokens
- not recorded
- Cost
- not recorded
GPT-5.6 Lunatrial 4 of 5failure- Time
- not recorded
- Tokens
- not recorded
- Cost
- not recorded
GPT-5.6 Lunatrial 5 of 5failure- Time
- not recorded
- Tokens
- not recorded
- Cost
- not recorded
Claude Opus 5trial 1 of 5success- Time
- 19.2 min
- Tokens
- 9.05 M
- Cost
- USD 7.90
Claude Opus 5trial 2 of 5success- Time
- 31.9 min
- Tokens
- 16.72 M
- Cost
- USD 13.03
Claude Opus 5trial 3 of 5success- Time
- 14.4 min
- Tokens
- 9.08 M
- Cost
- USD 6.94
Claude Opus 5trial 4 of 5success- Time
- 25.5 min
- Tokens
- 15.82 M
- Cost
- USD 11.83
Claude Opus 5trial 5 of 5success- Time
- 20.0 min
- Tokens
- 11.32 M
- Cost
- USD 9.04
Claude Fable 5.1trial 1 of 5success- Time
- 25.2 min
- Tokens
- 9.82 M
- Cost
- USD 11.20
Claude Fable 5.1trial 2 of 5success- Time
- 39.8 min
- Tokens
- 11.10 M
- Cost
- USD 14.36
Claude Fable 5.1trial 3 of 5success- Time
- 18.3 min
- Tokens
- 6.87 M
- Cost
- USD 8.33
Claude Fable 5.1trial 4 of 5failure- Time
- not recorded
- Tokens
- not recorded
- Cost
- not recorded
Claude Fable 5.1trial 5 of 5failure- Time
- not recorded
- Tokens
- not recorded
- Cost
- not recorded
10 experience transfer
GPT-6 Astra building the experience storetrial 1 of 5success- Time
- 9.1 min
- Tokens
- 2.83 M
- Cost
- USD 4.39
GPT-6 Astra building the experience storetrial 2 of 5success- Time
- 8.2 min
- Tokens
- 2.81 M
- Cost
- USD 4.08
GPT-6 Astra building the experience storetrial 3 of 5success- Time
- 7.3 min
- Tokens
- 2.25 M
- Cost
- USD 3.49
GPT-6 Astra building the experience storetrial 4 of 5success- Time
- 7.0 min
- Tokens
- 2.24 M
- Cost
- USD 3.45
GPT-6 Astra building the experience storetrial 5 of 5success- Time
- 6.5 min
- Tokens
- 2.36 M
- Cost
- USD 3.56
GPT-5.6 Terra reading the frozen storetrial 1 of 5success- Time
- 16.9 min
- Tokens
- 4.81 M
- Cost
- USD 1.56
GPT-5.6 Terra reading the frozen storetrial 2 of 5success- Time
- 10.1 min
- Tokens
- 5.82 M
- Cost
- USD 1.62
GPT-5.6 Terra reading the frozen storetrial 3 of 5failure- Time
- 19.6 min
- Tokens
- 7.28 M
- Cost
- USD 2.17
GPT-5.6 Terra reading the frozen storetrial 4 of 5success- Time
- 11.7 min
- Tokens
- 5.76 M
- Cost
- USD 1.74
GPT-5.6 Terra reading the frozen storetrial 5 of 5success- Time
- 11.2 min
- Tokens
- 5.87 M
- Cost
- USD 1.65
The same 112 slots as one table
| Slot | Configuration | Trial | Outcome | Model | Harness | Arm | Time (min) | Tokens (M) | Cost (USD) |
|---|---|---|---|---|---|---|---|---|---|
| assembly-fourpair-t01 | Four pair assembly | 1 of 10 | success | GPT-6 Astra | Codex CLI | single | 38.6 | 12.84 | 16.92 |
| assembly-fourpair-t02 | Four pair assembly | 2 of 10 | success | GPT-6 Astra | Codex CLI | single | 53.0 | 17.14 | 22.91 |
| assembly-fourpair-t03 | Four pair assembly | 3 of 10 | failure | GPT-6 Astra | Codex CLI | single | 89.2 | 40.41 | 48.39 |
| assembly-fourpair-t04 | Four pair assembly | 4 of 10 | success | GPT-6 Astra | Codex CLI | single | 47.2 | 16.06 | 19.94 |
| assembly-fourpair-t05 | Four pair assembly | 5 of 10 | failure | GPT-6 Astra | Codex CLI | single | 90.2 | 29.36 | 37.32 |
| assembly-fourpair-t06 | Four pair assembly | 6 of 10 | success | GPT-6 Astra | Codex CLI | single | 16.3 | 4.62 | 6.58 |
| assembly-fourpair-t07 | Four pair assembly | 7 of 10 | success | GPT-6 Astra | Codex CLI | single | 30.3 | 10.46 | 13.27 |
| assembly-fourpair-t08 | Four pair assembly | 8 of 10 | success | GPT-6 Astra | Codex CLI | single | 16.0 | 3.96 | 5.55 |
| assembly-fourpair-t09 | Four pair assembly | 9 of 10 | success | GPT-6 Astra | Codex CLI | single | 37.4 | 11.98 | 16.32 |
| assembly-fourpair-t10 | Four pair assembly | 10 of 10 | success | GPT-6 Astra | Codex CLI | single | 59.1 | 27.12 | 31.50 |
| blocks-pyramid-t01 | Pyramid | 1 of 10 | success | GPT-6 Astra | Codex CLI | single | 16.8 | 6.36 | 8.17 |
| blocks-pyramid-t02 | Pyramid | 2 of 10 | success | GPT-6 Astra | Codex CLI | single | 23.2 | 8.66 | 11.15 |
| blocks-pyramid-t03 | Pyramid | 3 of 10 | success | GPT-6 Astra | Codex CLI | single | 26.9 | 11.12 | 15.28 |
| blocks-pyramid-t04 | Pyramid | 4 of 10 | success | GPT-6 Astra | Codex CLI | single | 14.9 | 6.27 | 8.01 |
| blocks-pyramid-t05 | Pyramid | 5 of 10 | success | GPT-6 Astra | Codex CLI | single | 32.2 | 12.94 | 15.19 |
| blocks-pyramid-t06 | Pyramid | 6 of 10 | success | GPT-6 Astra | Codex CLI | single | 20.0 | 11.15 | 14.31 |
| blocks-pyramid-t07 | Pyramid | 7 of 10 | success | GPT-6 Astra | Codex CLI | single | 17.0 | 6.85 | 8.68 |
| blocks-pyramid-t08 | Pyramid | 8 of 10 | success | GPT-6 Astra | Codex CLI | single | 17.9 | 9.25 | 11.26 |
| blocks-pyramid-t09 | Pyramid | 9 of 10 | success | GPT-6 Astra | Codex CLI | single | 24.1 | 9.95 | 12.83 |
| blocks-pyramid-t10 | Pyramid | 10 of 10 | success | GPT-6 Astra | Codex CLI | single | 22.4 | 9.86 | 12.01 |
| blocks-twotowers-t01 | Two towers | 1 of 10 | success | GPT-6 Astra | Codex CLI | single | 21.0 | 7.61 | 9.57 |
| blocks-twotowers-t02 | Two towers | 2 of 10 | success | GPT-6 Astra | Codex CLI | single | 19.4 | 7.80 | 9.68 |
| blocks-twotowers-t03 | Two towers | 3 of 10 | success | GPT-6 Astra | Codex CLI | single | 14.8 | 4.94 | 6.27 |
| blocks-twotowers-t04 | Two towers | 4 of 10 | success | GPT-6 Astra | Codex CLI | single | 22.6 | 6.91 | 9.01 |
| blocks-twotowers-t05 | Two towers | 5 of 10 | success | GPT-6 Astra | Codex CLI | single | 29.2 | 14.39 | 16.85 |
| blocks-twotowers-t06 | Two towers | 6 of 10 | success | GPT-6 Astra | Codex CLI | single | 24.4 | 9.58 | 11.69 |
| blocks-twotowers-t07 | Two towers | 7 of 10 | success | GPT-6 Astra | Codex CLI | single | 17.6 | 5.98 | 8.11 |
| blocks-twotowers-t08 | Two towers | 8 of 10 | success | GPT-6 Astra | Codex CLI | single | 17.8 | 7.80 | 10.04 |
| blocks-twotowers-t09 | Two towers | 9 of 10 | success | GPT-6 Astra | Codex CLI | single | 21.9 | 9.25 | 11.37 |
| blocks-twotowers-t10 | Two towers | 10 of 10 | success | GPT-6 Astra | Codex CLI | single | 17.0 | 5.12 | 6.56 |
| blocks-sixblock-t01 | Six block tower | 1 of 10 | success | GPT-6 Astra | Codex CLI | single | 46.4 | 19.08 | 25.29 |
| blocks-sixblock-t02 | Six block tower | 2 of 10 | failure | GPT-6 Astra | Codex CLI | single | 35.6 | 13.51 | 17.02 |
| blocks-sixblock-t03 | Six block tower | 3 of 10 | success | GPT-6 Astra | Codex CLI | single | 20.1 | 8.44 | 10.62 |
| blocks-sixblock-t04 | Six block tower | 4 of 10 | success | GPT-6 Astra | Codex CLI | single | 17.3 | 5.09 | 6.69 |
| blocks-sixblock-t05 | Six block tower | 5 of 10 | success | GPT-6 Astra | Codex CLI | single | 55.0 | 19.85 | 26.78 |
| blocks-sixblock-t06 | Six block tower | 6 of 10 | success | GPT-6 Astra | Codex CLI | single | 25.4 | 13.19 | 15.72 |
| blocks-sixblock-t07 | Six block tower | 7 of 10 | success | GPT-6 Astra | Codex CLI | single | 31.0 | 11.30 | 15.48 |
| blocks-sixblock-t08 | Six block tower | 8 of 10 | success | GPT-6 Astra | Codex CLI | single | 29.0 | 13.71 | 16.55 |
| blocks-sixblock-t09 | Six block tower | 9 of 10 | success | GPT-6 Astra | Codex CLI | single | 17.9 | 8.94 | 10.78 |
| blocks-sixblock-t10 | Six block tower | 10 of 10 | success | GPT-6 Astra | Codex CLI | single | 11.6 | 4.50 | 6.50 |
| dice-flip-t01 | Dice flipping | 1 of 10 | success | GPT-6 Astra | Codex CLI | single | 36.4 | 10.16 | 12.57 |
| dice-flip-t02 | Dice flipping | 2 of 10 | success | GPT-6 Astra | Codex CLI | single | 56.0 | 23.60 | 28.76 |
| dice-flip-t03 | Dice flipping | 3 of 10 | success | GPT-6 Astra | Codex CLI | single | 47.9 | 20.19 | 23.72 |
| dice-flip-t04 | Dice flipping | 4 of 10 | success | GPT-6 Astra | Codex CLI | single | 51.4 | 25.00 | 31.20 |
| dice-flip-t05 | Dice flipping | 5 of 10 | success | GPT-6 Astra | Codex CLI | single | 22.6 | 13.35 | 16.34 |
| dice-flip-t06 | Dice flipping | 6 of 10 | success | GPT-6 Astra | Codex CLI | single | 37.3 | 14.49 | 18.82 |
| dice-flip-t07 | Dice flipping | 7 of 10 | success | GPT-6 Astra | Codex CLI | single | 33.6 | 16.46 | 20.27 |
| dice-flip-t08 | Dice flipping | 8 of 10 | success | GPT-6 Astra | Codex CLI | single | 32.1 | 21.10 | 25.70 |
| dice-flip-t09 | Dice flipping | 9 of 10 | success | GPT-6 Astra | Codex CLI | single | 26.4 | 11.78 | 14.46 |
| dice-flip-t10 | Dice flipping | 10 of 10 | success | GPT-6 Astra | Codex CLI | single | 34.9 | 14.63 | 18.83 |
| throw-potato-t01 | Targeted throwing (potato) | 1 of 2 | success | GPT-6 Astra | Codex CLI | single | 22.2 | 6.25 | 8.71 |
| throw-potato-t02 | Targeted throwing (potato) | 2 of 2 | success | GPT-6 Astra | Codex CLI | single | 23.3 | 6.26 | 9.18 |
| towel-sequential-t01 | Towel folding, sequential | 1 of 5 | success | GPT-6 Astra | Codex CLI | bimanual | 61.6 | 17.58 | 27.98 |
| towel-sequential-t02 | Towel folding, sequential | 2 of 5 | success | GPT-6 Astra | Codex CLI | bimanual | 50.7 | 18.58 | 23.53 |
| towel-sequential-t03 | Towel folding, sequential | 3 of 5 | success | GPT-6 Astra | Codex CLI | bimanual | 44.6 | 19.06 | 24.14 |
| towel-sequential-t04 | Towel folding, sequential | 4 of 5 | success | GPT-6 Astra | Codex CLI | bimanual | 63.3 | 25.26 | 30.90 |
| towel-sequential-t05 | Towel folding, sequential | 5 of 5 | success | GPT-6 Astra | Codex CLI | bimanual | 33.8 | 10.13 | 14.16 |
| towel-simultaneous-t01 | Towel folding, simultaneous | 1 of 5 | failure | GPT-6 Astra | Codex CLI | bimanual | 14.0 | 3.83 | 5.86 |
| towel-simultaneous-t02 | Towel folding, simultaneous | 2 of 5 | success | GPT-6 Astra | Codex CLI | bimanual | 15.7 | 4.27 | 6.16 |
| towel-simultaneous-t03 | Towel folding, simultaneous | 3 of 5 | failure | GPT-6 Astra | Codex CLI | bimanual | 27.6 | 8.93 | 12.73 |
| towel-simultaneous-t04 | Towel folding, simultaneous | 4 of 5 | success | GPT-6 Astra | Codex CLI | bimanual | 31.2 | 5.68 | 8.67 |
| towel-simultaneous-t05 | Towel folding, simultaneous | 5 of 5 | success | GPT-6 Astra | Codex CLI | bimanual | 17.4 | 5.21 | 7.54 |
| t2-astra_low-t01 | GPT-6 Astra, low effort | 1 of 5 | success | GPT-6 Astra | Codex CLI | single | 7.4 | 2.70 | 3.77 |
| t2-astra_low-t02 | GPT-6 Astra, low effort | 2 of 5 | success | GPT-6 Astra | Codex CLI | single | 19.0 | 7.72 | 9.56 |
| t2-astra_low-t03 | GPT-6 Astra, low effort | 3 of 5 | success | GPT-6 Astra | Codex CLI | single | 6.0 | 2.81 | 3.63 |
| t2-astra_low-t04 | GPT-6 Astra, low effort | 4 of 5 | success | GPT-6 Astra | Codex CLI | single | 6.3 | 1.44 | 2.15 |
| t2-astra_low-t05 | GPT-6 Astra, low effort | 5 of 5 | success | GPT-6 Astra | Codex CLI | single | 10.9 | 3.78 | 4.82 |
| t2-astra_medium-t01 | GPT-6 Astra, medium effort | 1 of 5 | success | GPT-6 Astra | Codex CLI | single | 6.3 | 1.84 | 2.76 |
| t2-astra_medium-t02 | GPT-6 Astra, medium effort | 2 of 5 | success | GPT-6 Astra | Codex CLI | single | 15.4 | 5.80 | 7.48 |
| t2-astra_medium-t03 | GPT-6 Astra, medium effort | 3 of 5 | success | GPT-6 Astra | Codex CLI | single | 8.8 | 2.51 | 3.46 |
| t2-astra_medium-t04 | GPT-6 Astra, medium effort | 4 of 5 | success | GPT-6 Astra | Codex CLI | single | 7.6 | 2.96 | 3.86 |
| t2-astra_medium-t05 | GPT-6 Astra, medium effort | 5 of 5 | success | GPT-6 Astra | Codex CLI | single | 7.6 | 2.01 | 2.90 |
| t2-astra_high-t01 | GPT-6 Astra, high effort | 1 of 5 | success | GPT-6 Astra | Codex CLI | single | 8.9 | 4.02 | 5.19 |
| t2-astra_high-t02 | GPT-6 Astra, high effort | 2 of 5 | success | GPT-6 Astra | Codex CLI | single | 10.6 | 3.14 | 4.39 |
| t2-astra_high-t03 | GPT-6 Astra, high effort | 3 of 5 | success | GPT-6 Astra | Codex CLI | single | 9.0 | 2.58 | 3.72 |
| t2-astra_high-t04 | GPT-6 Astra, high effort | 4 of 5 | success | GPT-6 Astra | Codex CLI | single | 9.7 | 2.81 | 3.96 |
| t2-astra_high-t05 | GPT-6 Astra, high effort | 5 of 5 | success | GPT-6 Astra | Codex CLI | single | 7.9 | 3.94 | 5.08 |
| t2-sol_high-t01 | GPT-5.6 Sol | 1 of 5 | success | GPT-5.6 Sol | Codex CLI | single | 15.2 | 7.81 | 4.35 |
| t2-sol_high-t02 | GPT-5.6 Sol | 2 of 5 | success | GPT-5.6 Sol | Codex CLI | single | 15.5 | 7.83 | 4.24 |
| t2-sol_high-t03 | GPT-5.6 Sol | 3 of 5 | success | GPT-5.6 Sol | Codex CLI | single | 19.3 | 9.72 | 5.30 |
| t2-sol_high-t04 | GPT-5.6 Sol | 4 of 5 | success | GPT-5.6 Sol | Codex CLI | single | 12.3 | 6.16 | 3.40 |
| t2-sol_high-t05 | GPT-5.6 Sol | 5 of 5 | success | GPT-5.6 Sol | Codex CLI | single | 9.1 | 4.12 | 2.40 |
| t2-terra_high-t01 | GPT-5.6 Terra | 1 of 5 | failure | GPT-5.6 Terra | Codex CLI | single | 30.4 | 6.32 | 2.04 |
| t2-terra_high-t02 | GPT-5.6 Terra | 2 of 5 | failure | GPT-5.6 Terra | Codex CLI | single | 22.1 | 5.14 | 1.86 |
| t2-terra_high-t03 | GPT-5.6 Terra | 3 of 5 | failure | GPT-5.6 Terra | Codex CLI | single | 21.4 | 8.99 | 2.62 |
| t2-terra_high-t04 | GPT-5.6 Terra | 4 of 5 | success | GPT-5.6 Terra | Codex CLI | single | 19.0 | 6.28 | 1.98 |
| t2-terra_high-t05 | GPT-5.6 Terra | 5 of 5 | failure | GPT-5.6 Terra | Codex CLI | single | 14.4 | 2.70 | 0.99 |
| t2-luna_high-t01 | GPT-5.6 Luna | 1 of 5 | failure | GPT-5.6 Luna | Codex CLI | single | not recorded | not recorded | not recorded |
| t2-luna_high-t02 | GPT-5.6 Luna | 2 of 5 | failure | GPT-5.6 Luna | Codex CLI | single | not recorded | not recorded | not recorded |
| t2-luna_high-t03 | GPT-5.6 Luna | 3 of 5 | failure | GPT-5.6 Luna | Codex CLI | single | not recorded | not recorded | not recorded |
| t2-luna_high-t04 | GPT-5.6 Luna | 4 of 5 | failure | GPT-5.6 Luna | Codex CLI | single | not recorded | not recorded | not recorded |
| t2-luna_high-t05 | GPT-5.6 Luna | 5 of 5 | failure | GPT-5.6 Luna | Codex CLI | single | not recorded | not recorded | not recorded |
| t2-opus5_high-t01 | Claude Opus 5 | 1 of 5 | success | Claude Opus 5 | Claude Code | single | 19.2 | 9.05 | 7.90 |
| t2-opus5_high-t02 | Claude Opus 5 | 2 of 5 | success | Claude Opus 5 | Claude Code | single | 31.9 | 16.72 | 13.03 |
| t2-opus5_high-t03 | Claude Opus 5 | 3 of 5 | success | Claude Opus 5 | Claude Code | single | 14.4 | 9.08 | 6.94 |
| t2-opus5_high-t04 | Claude Opus 5 | 4 of 5 | success | Claude Opus 5 | Claude Code | single | 25.5 | 15.82 | 11.83 |
| t2-opus5_high-t05 | Claude Opus 5 | 5 of 5 | success | Claude Opus 5 | Claude Code | single | 20.0 | 11.32 | 9.04 |
| t2-fable_high-t01 | Claude Fable 5.1 | 1 of 5 | success | Claude Fable 5.1 | Claude Code | single | 25.2 | 9.82 | 11.20 |
| t2-fable_high-t02 | Claude Fable 5.1 | 2 of 5 | success | Claude Fable 5.1 | Claude Code | single | 39.8 | 11.10 | 14.36 |
| t2-fable_high-t03 | Claude Fable 5.1 | 3 of 5 | success | Claude Fable 5.1 | Claude Code | single | 18.3 | 6.87 | 8.33 |
| t2-fable_high-t04 | Claude Fable 5.1 | 4 of 5 | failure | Claude Fable 5.1 | Claude Code | single | not recorded | not recorded | not recorded |
| t2-fable_high-t05 | Claude Fable 5.1 | 5 of 5 | failure | Claude Fable 5.1 | Claude Code | single | not recorded | not recorded | not recorded |
| transfer-astra_store-t01 | GPT-6 Astra building the experience store | 1 of 5 | success | GPT-6 Astra | Codex CLI | single | 9.1 | 2.83 | 4.39 |
| transfer-astra_store-t02 | GPT-6 Astra building the experience store | 2 of 5 | success | GPT-6 Astra | Codex CLI | single | 8.2 | 2.81 | 4.08 |
| transfer-astra_store-t03 | GPT-6 Astra building the experience store | 3 of 5 | success | GPT-6 Astra | Codex CLI | single | 7.3 | 2.25 | 3.49 |
| transfer-astra_store-t04 | GPT-6 Astra building the experience store | 4 of 5 | success | GPT-6 Astra | Codex CLI | single | 7.0 | 2.24 | 3.45 |
| transfer-astra_store-t05 | GPT-6 Astra building the experience store | 5 of 5 | success | GPT-6 Astra | Codex CLI | single | 6.5 | 2.36 | 3.56 |
| transfer-terra_exp-t01 | GPT-5.6 Terra reading the frozen store | 1 of 5 | success | GPT-5.6 Terra | Codex CLI | single | 16.9 | 4.81 | 1.56 |
| transfer-terra_exp-t02 | GPT-5.6 Terra reading the frozen store | 2 of 5 | success | GPT-5.6 Terra | Codex CLI | single | 10.1 | 5.82 | 1.62 |
| transfer-terra_exp-t03 | GPT-5.6 Terra reading the frozen store | 3 of 5 | failure | GPT-5.6 Terra | Codex CLI | single | 19.6 | 7.28 | 2.17 |
| transfer-terra_exp-t04 | GPT-5.6 Terra reading the frozen store | 4 of 5 | success | GPT-5.6 Terra | Codex CLI | single | 11.7 | 5.76 | 1.74 |
| transfer-terra_exp-t05 | GPT-5.6 Terra reading the frozen store | 5 of 5 | success | GPT-5.6 Terra | Codex CLI | single | 11.2 | 5.87 | 1.65 |
Scroll the table sideways for the rest of the columns.
Cite this work
@article{jia2026agentaspolicy,
title = {Agent as Policy for Robotic Manipulation},
author = {Jia, Mengzhao and Lin, Yang and Zhang, Xixin and Zhang, Zhihan and Liu, Xiaobai and Jiang, Meng},
journal = {arXiv preprint arXiv:2609.12541},
year = {2026}
}Credits
- Assembly part geometry
- Adapted from the AutoMate dataset. Tang et al., AutoMate: Specialist and Generalist Assembly Policies over Diverse Geometries, 2024, arXiv:2407.08028. The parts on this page are 3D prints of those meshes, scaled with their aspect ratios preserved.
- Robot arm
- I2RT YAM arm and the I2RT robot software.
- Inverse kinematics
- Mink, by Kevin Zakka: Python inverse kinematics based on MuJoCo.
- Coding agent
- Codex CLI, by OpenAI. Claude Code, by Anthropic.