RobotGuide
Brain and cerebellum/Data Engine

Foundation-model teardown · Data Engine · 25 min

Data Engine: the real bottleneck of VLAs

Model architectures have converged; the gaps are all in data. Four data recipes — teleoperation, human video, simulation synthesis, cross-embodiment aggregation — differ in cost by three orders of magnitude, each solving one stretch and leaving one hole.

25 minreal anchors, sourcedchecked 2026-11

What it really is, once the buzzwords are off

In 2026 the architecture controversy around VLAs is shrinking (action heads converge on flow matching, backbones converge on off-the-shelf VLMs); the real arms race is entirely in data: whoever gets the 'right' manipulation data more cheaply gets a model that does one more class of jobs. This page puts the four data recipes side by side to do the math — their prices differ by three orders of magnitude, and none alone leads to a general robot.

The ledger first: four recipes compared

One table for the whole picture: each recipe's cost, what it provides, and what it inherently lacks. Each is unpacked below.
RecipeRepresentative projects / datasetsCost magnitudeWhat it gives the modelInherent flaw
Teleoperation collectionALOHA family, AgiBot World, Xiaomi's 100k hoursMost expensive: equipment + labor, minutes per trajectoryReal-robot action-vision pairing, highest qualitySlow, narrow distribution, human-operator bias
Human video learningDreamGen and EgoMimic routesCheap: existing video + a little retargetingMassive scene priors, hand-interaction priorsNo action labels; needs retargeting or implicit inference
Simulation synthesisMimicGen, RoboCasa, GR00T-BlueprintCheap: compute for data, mass-producible automaticallyPerfect annotations, unlimited variantssim gap: contact and materials are hardest to simulate
Cross-embodiment aggregationOpen X-Embodiment (22 embodiments, ~1M trajectories)Medium: multi-institution sharingEmbodiment diversity, task diversityHeterogeneity dilutes signal; uneven quality

Recipe one: teleoperation — expensive, but the foundation

Teleoperation data is the only source where action, vision, and force are all real and aligned. The engineering reality: one person collects dozens to a few hundred trajectories a day, and a single task past a thousand takes weeks. That dictates its use — not for training general capability, but for defining tasks and acceptance: a few dozen high-quality trajectories define 'what doing this task right looks like', and all other recipes' data align to that anchor. The largest teleoperation dataset of 2026 was built by Chinese teams: Zhiyuan's AgiBot World (million-scale real-robot trajectories, open-sourced) and Xiaomi Robotics-1's 100,000 hours of real-robot data (not fully open-sourced). This changed the industry's landscape — 'lots of data' used to be a Google-only advantage (the RT series' internal data), and now the open-source side has a foundation of the same magnitude for the first time. One hidden cost of teleoperation to watch: human operator bias (habitual approach from the same side, uniform speed) gets learned by the model as part of the task; collecting data with different operators is a cheap source of diversity.

Recipe two: human video — the cheapest prior, the hardest step

Internet and self-collected human manipulation video is nearly free; the problem is that video has no action labels — you can't read joint angles from it. Two technical routes turn it into usable data: retargeting (recover hand pose from video, then map kinematically onto the robot arm — the DreamGen family) and implicit learning (learn only 'how the scene will change' and let the model guess actions — the V-JEPA 2 family, covered in detail in the world model column). Its real role: instilling the prior of 'how things get manipulated', not teaching specific actions. Meta's V-JEPA 2 used 1M hours of video plus only 62 hours of real-robot data to reach 65–80% success on pick-and-place of new objects — the hardest evidence for this route: video priors push down the demand for real-robot data by two orders of magnitude. But it currently stops at pick-and-place-level tasks; contact-rich assembly remains far off.

Recipe three: simulation synthesis — mass production, with a ceiling

Simulation's lure is marginal cost near zero: MimicGen auto-generates tens of thousands of variants from a few dozen human demonstrations, RoboCasa scales home scenes to 365 tasks, and NVIDIA's GR00T-Blueprint turned 'simulation producing data' into a standard pipeline. It solves diversity and long-tail — collecting 10,000 object arrangements in the real world is unaffordable, while in simulation you just change the random seed. The ceiling is contact: high-precision contact of rigid objects (pegs, snaps, flexible cables) still has no reliable simulation solution — simulators' contact forces are either inaccurate or unstable, and contact-rich policies trained on them fail outright on real robots. That is why the sim-to-real column exists. Practical rule: use simulation freely for pick-and-place, scene navigation, and visual diversity; for contact-rich task data don't count on simulation — go back to teleoperation and real-robot fine-tuning.

Recipe four: cross-embodiment aggregation — trading diversity for quality

Open X-Embodiment aggregates 22 robot types and about 1M trajectories and proved one thing: the underlying regularities of 'how to grasp a cup' generalize across embodiments, and mixed training raises every embodiment's floor. The price is heterogeneity: 22 embodiments with different degrees of freedom, control frequencies, and fields of view — mixing directly dilutes the signal. Everyone's solution is 'shared semantic backbone, embodiment-specific action head', which is exactly another reason for the anatomy page's separated action-head design. When using someone else's dataset, ask three questions first: what is the success-rate definition (see Evaluation Truth), how much does their robot differ from yours, and does the data have force/tactile channels. The answer to the third is currently almost always 'no' — read on.

The hole common to all recipes: the hundred milliseconds after contact

Existing mainstream datasets almost all have only two channels: 'vision + action'. The pre-grasp phase can be learned from video, but what happens after the fingers contact the object — slip, deformation, grip tightness — no current recipe records it.

That's why gains like 56%→90% on cap twisting can only come from tactile data (LingBot-Touch from Ant Group used nearly 5,000 hours of tactile data). Tactile data collection (who collects it, how it's labeled) is the fifth row of the data-recipe table, still at the teleoperation stage in 2026 — roughly where visual data was in 2015.

Knowledge point covering this area: C9 Tactile Sensing and Dexterous-Hand Manipulation.

Practical advice for you

Starting order: lay a base with public cross-embodiment datasets → mass-produce variants of your task in simulation → teleoperate 50–200 trajectories to define the task and acceptance → fine-tune on the real robot.

Budget criterion: if your task can be simulated (non-contact-rich), simulation can take 80%+ of the share; for contact-rich tasks, teleoperation gets at least half the budget.

Always reserve 10–20% of the budget for 'out-of-task diversity' (different lighting, distractors, arrangements) — the cheapest axis to buy in three-axis OOD evaluation (see Evaluation Truth).

Sourcing: dataset sizes and cost magnitudes come from the Open X-Embodiment paper, the AgiBot World open-source page, Xiaomi's official releases, and the DreamGen paper; costs are industry-standard relative magnitudes, not precise quotes.

Previous · VLA AnatomyTaking a VLA Apart Next · Post-trainingPost-training: from base model to your robot

All 6 teardown pages

01The model survey 02Taking a VLA Apart25 min03Data Engine: the real bottleneck of VLAs25 min04Post-training: from base model to your robot20 min05Deployment and Inference: landing a 1–10Hz brain20 min06Evaluation Truth: between 90% and 15%20 min