Tools & simulation · topic G9 · 35 hours
Training frameworks & data pipelines
Between «a demo that runs» and «a policy that trains» lies a pipeline: environments, parallelism, logging, checkpoints, data formats. If the pipeline is broken, more compute just burns electricity.
What to learn
- The parts of PyTorch that are enough: autograd, device, DataLoader, mixed precision, fixed random seeds
- RL library division of labor: Stable-Baselines3 (single-machine entry point, clean implementations), rsl_rl (Isaac Lab's default PPO), SKRL, CleanRL (for reading code, not running experiments)
- Parallelism: vectorized environments (SubprocVecEnv), GPU parallelism (Isaac / MJX), seeding and sync policies; converting between worker counts and environment steps
- Experiment logging: what wandb / tensorboard should record — return, success rate, episode length, gradient norm, entropy; logging the return only will mislead you
- Data formats: LeRobotDataset (the community mainstream), HDF5 (the ALOHA/ACT family), RLDS/TFDS, and rosbag → training-set conversion
- The imitation-learning pipeline: teleoperated collection → cleaning (drop the failures and idle segments) → train ACT / Diffusion Policy → deploy on the real robot
The lesson · plain words
The training framework and data pipeline are «all the logistics of model learning»: how data enters the GPU, how multiple cards parallelize, how experiments get tracked. When training is slow or results are hard to reproduce, nine times out of ten the problem lives in this layer.
GPU utilization: the health metric of the data pipeline
Watch the animation: with single-threaded data feeding, the GPU spends 75% of its time waiting for food; raise the DataLoader workers to 3–4 and utilization surges past 90%; beyond that the marginal gain drops to zero (the GPU is full). When training is slow, this is the first metric to look at.
The usual bottleneck ranking: disk reads (switch to NVMe / preload into memory), decoding (JPEG decoding is a CPU hog — switch to a more efficient format), preprocessing on the main thread (move it into workers), batch size too small.
Interactive figure · drag it and see
Distributed training: data parallelism first
Multi-GPU training defaults to data parallelism: each card holds one model copy, each eats different data, gradients sync and average. DDP (PyTorch) is nearly zero-config. Robot RL's parallelism is different: environment parallelism (the thousands of sims from G5) stacked on top of data parallelism.
Mixed precision (FP16/BF16) is acceleration for free: half the memory, double the speed, with GradScaler guarding against numerical underflow. Training without mixed precision today is money wasted.
Experiment tracking and reproducibility
The experiment-tracking trio: a config snapshot (every hyperparameter from before training), the code version (git commit), metric curves (loss/lr/success rate). The tool is your choice (wandb/tensorboard/csv); the discipline is mandatory — «which parameter set produced that great result last week» is a disaster every team has lived through.
The reproducibility floor: fix the seeds (data order/initialization/simulation), save checkpoints (every N steps), record the environment (library versions). Robot-RL-specific: record the simulator version too — one MuJoCo upgrade can leave the policy dead.
How deep to go
Diagnose one «slow training» (GPU utilization → locate the bottleneck → fix it); set up two-GPU DDP training; establish your team's experiment-tracking discipline. With these done, the training framework is transparent to you.
Three questions — do you really get it?
1. Training is slow and GPU utilization is 30% — what do you check first?
- AThe learning rate
- BThe batch size
- CThe data pipeline bottleneck (IO/decoding/preprocessing)
Show answer
C. The GPU is starving. Add workers, switch to NVMe, move preprocessing into the workers — the three standard moves.
2. The default parallelism for multi-GPU training?
- AModel parallelism
- BData parallelism (DDP)
- CPipeline parallelism
Show answer
B. If the model fits on one card, use DDP. Model parallelism is a choice only oversized models need.
3. The robot-RL-specific requirement for reproducible experiments?
- ATurning off mixed precision
- BUsing a larger learning rate
- CRecording the simulator version
Show answer
C. MuJoCo/Isaac version changes alter physics behavior. The environment is part of the experiment configuration.
Keep going deeper
Links without a marker stay on this site; the rest point to the Chinese edition until its translation batch lands.
How it fails in the field
- Watching only the mean return: the policy learns «stand still and never lose points» — success rate 0 while the curve looks beautiful
- Train and validation sets sharing the same random seeds and the same trajectory: metrics absurdly good, everything collapses on the real robot
- 80% of the dataset is failed demonstrations (the human missed too), and the model very faithfully learns to fail
You have learned enough when
The Chinese edition carries the full topic page for Training frameworks & data pipelines, including the complete outline, depth requirements per career track, and further derivations: robotguide.dev/k/G9/.