RobotGuide
Brain and cerebellum/Post-training

Foundation-model teardown · Post-training · 20 min

Post-training: from base model to your robot

The base model 'can move'; your robot needs to 'do your job'. How much data, what to fine-tune, how not to wash out what it already knew — one decision table says it all.

20 minreal anchors, sourcedchecked 2026-11

What it really is, once the buzzwords are off

The cost of fine-tuning is determined by one variable: how far your task is from the base's pre-training distribution. New workspace, new camera angle, new objects — a few dozen trajectories suffice; new task type (pick-and-place to assembly) — thousands to start; new embodiment — that's not fine-tuning, that's retraining. The viral claims of 'adapting to a new robot with 200 samples' all belong to tier one; applying them to tier three is 2026's most common failure.

Find your tier first: which case is yours

What you changedData magnitudeTraining methodSource of the real figures
Only scene/lighting/objects/camera pose50–200 trajectoriesLoRA fine-tuning near the action headGemini Robotics 2 official figures: <200 samples, hours, to adapt to a new embodiment
Same embodiment, new task type (pick-and-place → peg insertion)Thousands of trajectoriesFull-parameter or large LoRA; action head must be trainedMulti-task fine-tuning experiments in the π0 and OpenVLA papers
New embodiment form (arm → humanoid/dual-arm/mobile manipulator)Pre-training magnitudeRedo cross-embodiment pre-trainingThe entire motivation of the Open X-Embodiment / GR00T route
Note that Gemini's '<200 samples' loses its premise when widely quoted — that figure assumes your task is already within the semantic and skill base and only the embodiment is being adapted. Fine-tune an open-source base (OpenVLA, π0 openpi) on a task type it has never seen and 50 trajectories produce something that can't do the job — the most common expectation mismatch.

The four-step process

Step one: pick a base. As of 2026-11 the real candidates are four: openpi (π0/π0.5, Apache-2.0, flow-matching action head), LeRobot (the Hugging Face ecosystem, with the ACT/Diffusion Policy/π0/SmolVLA full set), OpenVLA-OFT (discrete + FAST, smoothest with the LM toolchain), and GR00T N1.7 (humanoid; mind the license — the repo model card and blog disagree on commercial terms, check the LICENSE file yourself before use). Closed-source APIs (Gemini Robotics 2 already in beta, the π-family API on the waitlist) are another option beyond tier-two tasks, but you get no weights — don't expect deep customization. Step two: collect data. Match the volume to the table above; quality beats quantity — 20 clean trajectories beat 100 contaminated ones (teleoperation bias and definition issues: see the Data Engine). Step three: fine-tune. Tier one: LoRA (rank 16–64, mainly the action head and the last few layers) is enough; tier two: full-parameter or larger rank, and the action head must be trained. Step four: evaluate. Design the protocol per Evaluation Truth; don't report just one overall success rate.

Language instructions: real need or decoration

The first superstition to cut when fine-tuning is 'my robot must understand natural language'. The status quo, unpacked: the real value of instruction following is in task switching and error correction ('use the left hand', 'the one on the right'), not in task definition (parameter definitions are more stable than language on production lines). If your instruction texts are highly templated, what the model learns is template retrieval — it fails on unseen phrasing. Keeping phrasing diversity during fine-tuning (3–5 phrasings per task) preserves generalization better than piling up task count.

Three fine-tuning pitfalls

Catastrophic forgetting: fine-tuning a new task washes out old skills — the mechanism that erases pre-trained representations during fine-tuning is the same mathematical pathology as in the world-model line. Mitigation: mix 10–25% old-task data for replay; experimental evidence in World-Model Experiment Pack · Experiment B (retention 1.00→0.00 without replay, fully preserved with it).

High simulation scores, real-robot collapse: running SimplerEnv-style simulated evaluations right after fine-tuning is easy, but simulation rankings have no reliable correlation with real-robot delivery (see Evaluation Truth); small-sample real-robot evaluation cannot be skipped.

Success-rate definition gaming: reporting only easy subtask success rates and not interruptions and recoveries is the most common trick in model cards. Before reproducing someone's numbers, check how they counted.

The first step you can do today

Single-GPU (24G VRAM) start: install SmolVLA with LeRobot or π0 with openpi, LoRA fine-tune on a public dataset (an AgiBot World or DROID subset) — no self-collected data needed at any point.

Once it runs, switch to your own 50 teleoperated trajectories and compare before/after fine-tuning — one complete run-through beats reading ten deep dives.

The discipline standard for command-level tutorials is in the hands-on tutorial layer (the Gazebo and RL pieces are live; the VLA fine-tuning tutorial is written to the same standard).

Sourcing: data-volume tiers are a summary of public figures (the Gemini Robotics 2 official blog, OpenVLA/π0 papers, GR00T license files); the tier-one <200 samples is embodiment adaptation, not new-task learning.

Previous · Data EngineData Engine: the real bottleneck of VLAs Next · Deployment and InferenceDeployment and Inference: landing a 1–10Hz brain

All 6 teardown pages

01The model survey 02Taking a VLA Apart25 min03Data Engine: the real bottleneck of VLAs25 min04Post-training: from base model to your robot20 min05Deployment and Inference: landing a 1–10Hz brain20 min06Evaluation Truth: between 90% and 15%20 min