Nine tasks in the RoboDojo simulator failed all five trials. SenseTime then raised the step ceiling to ten times each benchmark's official limit and let the model keep interacting. All nine passed five out of five. No weights were retrained. The report's own framing — that some failures were the evaluator pressing stop too early — is the right one, and it is a statement about evaluation design, not about a model's ceiling.
Across 12 simulated suites — tabletop arms, humanoids, wheeled arms, dexterous hands — and 8 physical robots, the study varied two things: access to training-set demonstrations before evaluation (in-context learning), and the step cap, lifted as high as ten times the official setting. With both, GPT-6-Astra reached 81.9% on RoboDojo, and saturated LIBERO and SIMPLE. The report also extrapolates each suite's curve toward an unlimited-step ceiling and argues that several leaderboards are now close to saturated for this model.
The five-trial flip
The clean 5/5 deserves statistical care. At n = 5, a five-out-of-five result is compatible with a true success rate as low as roughly 55% (the one-sided 95% lower bound). The defensible claim is therefore "no longer reliably failing", not "solved". The nine tasks were also selected because they had failed, which invites regression to the mean, and the earlier failures may themselves have been budget-limited rather than capability-limited. A larger trial count, or per-task confidence intervals, would settle which reading applies.
There is a second conflation: capability and budget sufficiency. A tenfold step allowance does not merely give the same task more attempts; it changes what the task is. Some official caps encode an assumption about acceptable latency, and a policy that needs ten times the steps has not become faster, only more persistent. The useful presentation is a success-versus-budget curve per task, with the knee visible, so a reader can see whether the gain comes from a few extra recovery attempts or from a qualitatively different regime. A single number at ten times the cap is one point on that curve.
The saturation argument is the report's strongest analytical move and its most fragile. If unlimited-step performance really plateaus near current scores, the leaderboard has stopped discriminating and the field needs harder tasks, not larger budgets. But extrapolation assumes a smooth, monotone curve. Longer horizons also introduce new failure modes — state drift, compounding perception error, and the completion-verification failures the report itself documents — any of which could bend the curve down instead of flat.
What counts as a fair protocol
In-context learning is where comparability gets tricky. Letting the model study demonstrations, write task notes, assemble reference actions and code tools before the first trial departs from the zero-shot protocol most leaderboard entries use. For deployment-oriented evaluation that is legitimate, arguably more realistic. It is not legitimate to place the resulting numbers beside zero-shot numbers without a label. The report is explicit that preparation happens once per task and is reused across trials, which also means the per-trial cost figures amortize a setup cost that a real deployment would pay per task.
Cost is reported in the authors' own conversion of about one dollar per million tokens: roughly 131 minutes and 19 dollars of inference to stack bowls, 263 minutes and 55 dollars to put a pen in a holder. Two adjustments matter. First, cost per success rather than per attempt: at a 20% success rate the effective price is five times the per-run figure. Second, wall-clock time is the binding constraint for anything time-sensitive; 263 minutes is orders of magnitude outside household tolerance regardless of price. Token accounting and billed inference can also diverge once retries, image tokens and caching are included, so these are estimates rather than invoices.
The harness experiments are the most transferable result. Pairing the slow planner with a fast policy (π0.5) in open loop — plan, execute, never check — cuts cost and damages performance. Closing the loop recovers performance without a cost advantage. The best reported combination, GPT-6-Astra with in-context learning and a closed-loop π0.5, reaches 54% on the harder 42-task RoboDojo set. That does not contradict 81.9%; it is a different suite and a different budget. But it makes the point sharply: verification, not planning, is where the marginal value sits in a two-model stack.
The hardware gap
On real hardware the picture narrows. The model drove all eight robots and handled tabletop grasping, but deformable materials, fine insertion and long-horizon navigation remained weak. Simulators omit the constraints that decide real deployments: collisions that stop a robot, motors that overheat, processes that continue while the model thinks. The report's egg example is the cleanest illustration — the model sees that the egg is done, decides to cut the heat, and by the time the decision becomes action the egg is burnt. That is a latency problem, and no amount of extra steps fixes it.
The reproducibility contribution is what makes the rest checkable: each benchmark is packaged as a Docker image behind a unified HTTP API, decoupling environment from agent and policy, so a result can be re-run end to end from a config file. Most embodied claims still cannot be. The accompanying 27B model also supplies a useful scale: 26.13 points and 20.83% success on RoboDojo, above 90 on LIBERO-Pro, and 26.0% to 46.5% on RoboCasa365 with 73% fewer tokens, at a stated one-fiftieth of the cost. The gap between 20.83% and 81.9% is roughly the share of the headline that comes from harness and budget rather than from weights.
What would settle the open questions is unglamorous: per-task success-versus-budget curves with confidence intervals; cost per success rather than per attempt; wall-clock time-to-completion distributions; separate accounting for in-context preparation; and an ablation that moves the step cap and the demonstrations independently, so their contributions are not read as one. Trial counts of at least 20 per task would make a ten-point difference meaningful.
The methodological instruction is the durable part. In embodied evaluation the harness sits inside the model under test: budget, retries, preparation and verification all change the measured object. Until those are reported alongside the score, a success rate describes a configuration, not a capability — and a benchmark that can be saturated by raising its own step limit is telling you where to look next.
Sources
SenseTime newsroom: GPT-6-Astra embodied evaluation and SenseNova-RoboRSI release
Technical report: Has GPT-6-Astra achieved embodied intelligence?