RobotGuide

Perception & intelligence · topic C8 · 40 hours

VLA & embodied foundation models

The industry's hottest direction, and also its noisiest. You must be able to tell «demonstrated capability» from «deliverable capability».

Specialization stage · Hardcore
Algorithm · Proficient System software · Aware

Tools
Tools

What to learn

  • The VLA input-output paradigm: vision + language → actions (or action chunks)
  • Its relation to the layered architecture: the VLA usually only makes high-level decisions; the low level remains a control loop
  • The real constraints of inference latency and frequency: how a few-Hz decision layer cooperates with 1 kHz control
  • The data flywheel: how real-robot data flows back, and how increments are evaluated
  • The input paradigm is not only vision+language: tactile tokens — a unified tactile representation output as a third modality — can add only ~0.15% parameters (see C9 and /wm/open/)

The lesson · plain words

Everything before this page was «specialist models»: detection trained for detection, grasping trained for grasping. VLA (vision-language-action) changes the approach: pack seeing images, hearing instructions, and producing actions into one large model, and let it «emit actions» the way a language model emits words.

Right now it sits at the stage of «stunning demos, productization on the way». Your job is to understand its structure — neither deify it nor ignore it.

Structure: everything is a token

Watch the animation: the camera frame is cut into vision tokens, the instruction into text tokens, and both go into the large model; like writing a sentence, it autoregressively emits a string of action chunks (say, the incremental poses of «move the end effector left for 8 steps»), and the low-level controller executes them chunk by chunk.

The only difference from GPT is the «vocabulary»: a language model's words are text, a VLA's vocabulary adds image patches and action chunks. Understand one, and you understand the other's structure.

Interactive figure · drag it and see

What this figure shows: the VLA pipeline — a camera frame is cut into vision tokens, the instruction «push the red block to the left» into text tokens, and both feed an embodied foundation model (VLM + action head); it streams out an action-chunk sequence (Δpose × 8 steps, one whole chunk per prediction) into the low-level controller while the arm executes continuously. Takeaway: seeing, saying, and acting are squeezed into one token stream — all autoregressive within a single sequence.

Why instructions can command actions

The secret is pretraining: from internet text and image-text pairs, the large model has already learned the spatial and semantic meaning of concepts like «left», «red», and «push it over»; fine-tuning needs only tens of thousands of «instruction + demonstrated action» pairs to wire those concepts onto concrete actions.

That is why small datasets already produce results — the concepts are not taught, they arrive with the model. This is also why open-source VLAs (OpenVLA, π0 reproductions) replicate on a standard tabletop yet drop accuracy on an unseen table: generalization lives at the «concept» layer, not the «geometry» layer.

Limitations: an honest list

Frequency: one action-chunk inference takes tens of milliseconds — 200 Hz reaction is out of reach, so the low level must remain a real-time controller. Precision: language-level generalization ≠ millimeter-level assembly; peg-in-hole work is still safer with specialist models + calibration. Safety: the model may misread an instruction («put it in the box» understood as «knock the box over»), so software limits and e-stop must backstop it.

Cost: real-robot data is expensive and one fine-tune is not cheap; edge deployment needs quantization (G10). Do not treat the VLA as an all-purpose gripper — treat it as a «task-level translator».

Where the engineering opportunities are

The three most realistic things right now: ① data-collection pipelines (teleop stations, demonstration management, cloud return) — the roles every team is short of; ② evaluation sets — a VLA can claim anything, but benchmarks decide, and whoever defines the eval holds the discourse; ③ downstream fine-tuning and deployment optimization (quantization, distillation down to a size that fits on the robot).

Learning path: run OpenVLA/π0 inference demos → LoRA-fine-tune with 20 teleop episodes of a new task → probe its failure modes (change the lighting / background / tabletop and retry). Walk these three steps and your understanding already beats the eighty percent who only read papers.

Three questions — do you really get it?

1. The biggest structural difference between a VLA and GPT?

  • AThe vocabulary adds vision patches and action chunks; the model body is the same kind
  • BA larger parameter count
  • CNo pretraining needed
Show answer

A. Both are autoregressive transformers; the difference is the token types. Concept ability comes from pretraining, action ability from fine-tuning.

2. Why do tens of thousands of episodes suffice to teach a VLA a new task?

  • ASpatial and semantic concepts come from pretraining; fine-tuning only learns the «concept → action» mapping
  • BThe model is big enough to figure it out automatically
  • CIt actually takes millions of episodes
Show answer

A. «Left» and «red» were already learned in pretraining; fine-tuning merely wires those concepts onto this robot's action space.

3. What a VLA cannot yet do at this stage?

  • AUnderstand Chinese-language instructions
  • BMillisecond-level closed-loop control and millimeter-level assembly
  • CGeneralize to tabletop objects it has never seen
Show answer

B. Inference latency and precision both fall short of servo level. Low-level real-time control and high-precision assembly remain the traditional stack's job.

Keep going deeper

Links without a marker stay on this site; the rest point to the Chinese edition until its translation batch lands.

How it fails in the field

  • Assuming a VLA can directly output joint torques, ignoring the control layer and safety constraints
  • Treating demo videos as proof of capability without asking about success rate, working conditions, or failure recovery

You have learned enough when

You can state clearly where a VLA sits in a layered architecture, propose a «decisions at 5 Hz + control at 1 kHz» interface scheme, and read the experimental setup of a VLA paper.

The Chinese edition carries the full topic page for VLA & embodied foundation models, including the complete outline, depth requirements per career track, and further derivations: robotguide.dev/k/C8/.