Perception & intelligence · topic C8 · 40 hours
VLA & embodied foundation models
The industry's hottest direction, and also its noisiest. You must be able to tell «demonstrated capability» from «deliverable capability».
What to learn
- The VLA input-output paradigm: vision + language → actions (or action chunks)
- Its relation to the layered architecture: the VLA usually only makes high-level decisions; the low level remains a control loop
- The real constraints of inference latency and frequency: how a few-Hz decision layer cooperates with 1 kHz control
- The data flywheel: how real-robot data flows back, and how increments are evaluated
- The input paradigm is not only vision+language: tactile tokens — a unified tactile representation output as a third modality — can add only ~0.15% parameters (see C9 and /wm/open/)
The lesson · plain words
Everything before this page was «specialist models»: detection trained for detection, grasping trained for grasping. VLA (vision-language-action) changes the approach: pack seeing images, hearing instructions, and producing actions into one large model, and let it «emit actions» the way a language model emits words.
Right now it sits at the stage of «stunning demos, productization on the way». Your job is to understand its structure — neither deify it nor ignore it.
Structure: everything is a token
Watch the animation: the camera frame is cut into vision tokens, the instruction into text tokens, and both go into the large model; like writing a sentence, it autoregressively emits a string of action chunks (say, the incremental poses of «move the end effector left for 8 steps»), and the low-level controller executes them chunk by chunk.
The only difference from GPT is the «vocabulary»: a language model's words are text, a VLA's vocabulary adds image patches and action chunks. Understand one, and you understand the other's structure.
Interactive figure · drag it and see
Why instructions can command actions
The secret is pretraining: from internet text and image-text pairs, the large model has already learned the spatial and semantic meaning of concepts like «left», «red», and «push it over»; fine-tuning needs only tens of thousands of «instruction + demonstrated action» pairs to wire those concepts onto concrete actions.
That is why small datasets already produce results — the concepts are not taught, they arrive with the model. This is also why open-source VLAs (OpenVLA, π0 reproductions) replicate on a standard tabletop yet drop accuracy on an unseen table: generalization lives at the «concept» layer, not the «geometry» layer.
Limitations: an honest list
Frequency: one action-chunk inference takes tens of milliseconds — 200 Hz reaction is out of reach, so the low level must remain a real-time controller. Precision: language-level generalization ≠ millimeter-level assembly; peg-in-hole work is still safer with specialist models + calibration. Safety: the model may misread an instruction («put it in the box» understood as «knock the box over»), so software limits and e-stop must backstop it.
Cost: real-robot data is expensive and one fine-tune is not cheap; edge deployment needs quantization (G10). Do not treat the VLA as an all-purpose gripper — treat it as a «task-level translator».
Where the engineering opportunities are
The three most realistic things right now: ① data-collection pipelines (teleop stations, demonstration management, cloud return) — the roles every team is short of; ② evaluation sets — a VLA can claim anything, but benchmarks decide, and whoever defines the eval holds the discourse; ③ downstream fine-tuning and deployment optimization (quantization, distillation down to a size that fits on the robot).
Learning path: run OpenVLA/π0 inference demos → LoRA-fine-tune with 20 teleop episodes of a new task → probe its failure modes (change the lighting / background / tabletop and retry). Walk these three steps and your understanding already beats the eighty percent who only read papers.
Three questions — do you really get it?
1. The biggest structural difference between a VLA and GPT?
- AThe vocabulary adds vision patches and action chunks; the model body is the same kind
- BA larger parameter count
- CNo pretraining needed
Show answer
A. Both are autoregressive transformers; the difference is the token types. Concept ability comes from pretraining, action ability from fine-tuning.
2. Why do tens of thousands of episodes suffice to teach a VLA a new task?
- ASpatial and semantic concepts come from pretraining; fine-tuning only learns the «concept → action» mapping
- BThe model is big enough to figure it out automatically
- CIt actually takes millions of episodes
Show answer
A. «Left» and «red» were already learned in pretraining; fine-tuning merely wires those concepts onto this robot's action space.
3. What a VLA cannot yet do at this stage?
- AUnderstand Chinese-language instructions
- BMillisecond-level closed-loop control and millimeter-level assembly
- CGeneralize to tabletop objects it has never seen
Show answer
B. Inference latency and precision both fall short of servo level. Low-level real-time control and high-precision assembly remain the traditional stack's job.
Keep going deeper
Links without a marker stay on this site; the rest point to the Chinese edition until its translation batch lands.
How it fails in the field
- Assuming a VLA can directly output joint torques, ignoring the control layer and safety constraints
- Treating demo videos as proof of capability without asking about success rate, working conditions, or failure recovery
You have learned enough when
The Chinese edition carries the full topic page for VLA & embodied foundation models, including the complete outline, depth requirements per career track, and further derivations: robotguide.dev/k/C8/.