RobotGuide
Brain and cerebellum/Evaluation Truth

Foundation-model teardown · Evaluation Truth · 20 min

Evaluation Truth: between 90% and 15%

Simulation leaderboard #1 and a drop to 15% on three-axis OOD are facts from the same year. This page covers what each benchmark measures, the four definitions behind 'a 90% success rate', and how to design evaluation for your own robot.

20 minreal anchors, sourcedchecked 2026-11

What it really is, once the buzzwords are off

The waters of VLA evaluation run deeper than the models themselves. Without a definition, 'a 90% success rate' is empty words — 10 tasks at 30 trials each, or 1 task with the best 10 trials picked? The training scene, or after swapping objects, scenes, and instructions? The industry has simulation dual #1 and 15% on three-axis OOD coexisting — not a contradiction; the two rulers are simply not measuring the same thing.

The mainstream benchmark landscape: what each measures

BenchmarkWhat it measuresWho tops itLimitations
CALVINLong-horizon language-conditioned manipulation (tabletop)Early VLAs and imitation learningSimulated; small task pool
LIBEROFour task tiers (spatial/object/goal/long-horizon)OpenVLA-OFT and other open-source baselinesMostly tabletop pick-and-place
SimplerEnvSimulated recreations of Google robot tasksThe standard simulated evaluation for open-source VLAsLimited correlation with real robots
RoboCasa 365 / RoboDojoLarge-scale household tasksXiaomi Robotics-1 (dual #1 in 2026-07)Simulated; weak coverage of contact-rich tasks
Note a structural fact: not one benchmark in this table measures contact-rich tasks (pegs, snaps, flexible cables) — because simulation cannot produce reliable contact forces (see the Data Engine). So no leaderboard result is evidence of a model's ability to do assembly-type tasks.

Three-axis OOD: the multiplicative curse of generalization

The most valuable number in evaluation: new objects × new scenes × new instructions, all three axes shifted at once. Each axis drops a little alone, but stacking is multiplication — three 70% values stack into 34%, and sub-15% figures with all three axes open are not rare in industry measurements. Flip it around and this is the quantitative explanation of 'why demo videos can't be trusted': demo videos are recorded with all three axes aligned. Put two facts from the same year (2026) together: Xiaomi Robotics-1 took dual #1 on RoboCasa 365 and RoboDojo with 100,000 hours of real-robot data, while industry-wide success rates under three-axis OOD remain generally low. The two facts do not contradict — leaderboards measure in-distribution performance; delivery needs out-of-distribution reliability. Remember this sentence and you're immune to all VLA news.

How to design evaluation for your own robot

The minimum credible evaluation protocol

Split tasks into three tiers: pick-and-place (the acceptance floor), contact-rich (the real threshold), long-horizon multi-step (where gaps open) — at least 2 tasks per tier.

At least 20 trials per task: the minimum for a 95% confidence interval on success rate; '90% success' measured in 10 trials is statistically meaningless.

Two runs per tier: in-distribution (the training objects and scenes) + single-axis shift (one swapped object set) — single-axis numbers are far more useful than totals.

Log interruptions and recoveries: whether the policy self-recovers after getting stuck and how many human interventions were needed — production lines care about intervention rate, not success rate.

Four must-checks when reading someone else's evaluation

Trial floor: below 20 trials per task, read the 'success rate' as single-digit events.

Success criteria: 'object landed in the box' or 'the whole task sequence completed' — an order of magnitude apart.

Simulation or real robot: the conclusion that SimplerEnv scores have no reliable correlation with real-robot performance has been confirmed by multiple replication efforts.

Is OOD reported: evaluations reporting only in-distribution results should be filed under 'demo capability' by default, not under 'delivery capability' (this dichotomy comes from C8).

Sourcing: benchmark descriptions come from their papers and repos; three-axis OOD and 'limited sim-real correlation' summarize public replication work; Xiaomi's dual #1 is per the official 2026-07 release.

Previous · Deployment and InferenceDeployment and Inference: landing a 1–10Hz brain

All 6 teardown pages

01The model survey 02Taking a VLA Apart25 min03Data Engine: the real bottleneck of VLAs25 min04Post-training: from base model to your robot20 min05Deployment and Inference: landing a 1–10Hz brain20 min06Evaluation Truth: between 90% and 15%20 min