Perception & intelligence · topic C6 · 60 hours
Reinforcement learning
Quadruped and humanoid gaits, dexterous manipulation — more and more of it is trained with RL. It is also the classic case of «spectacular in demos, brutal in deployment».
What to learn
- MDP, reward, policy gradient, the PPO pipeline end to end (run it once yourself)
- Simulation environments and parallel sampling: why RL needs thousands of parallel environments
- Domain randomization: which parameters to randomize, and what happens when you overdo it
- Evaluation: a pretty training curve does not mean the real robot works
The lesson · plain words
Everything before this page was «humans write the rules». Reinforcement learning flips it: give only a reward signal and let the robot figure out what to do. It can learn policies no human writes (a quadruped's skittering gait, an arm's flick-throw) — at the price of expensive training and hard-to-explain behavior.
Understanding RL does not require deriving formulas; it requires two concepts: value (how many points is this situation worth) and policy (which action to take in each situation).
Value iteration: the score propagates itself
Look at the corridor below: goal +1, pit −1, everything else 0. Initially no cell knows its worth; repeatedly back up «which next step pays more» and the score propagates back from the goal — after 30 rounds the whole corridor's value map has grown.
This is the minimal kernel of RL: the Bellman backup — a state's value equals its immediate reward plus a discount times the next state's value. Every RL algorithm is a variation on that one sentence.
Interactive figure · drag it and see
Exploration and exploitation: the trial ratio
Always picking the current best (exploitation) locks you onto the first decent-looking option; pure random flailing (exploration) never learns anything. ε-greedy: 90% follow the policy, 10% random, decaying as training proceeds.
The real difficulty of robot RL is data, not algorithms: 100,000 tries in simulation are affordable; on the real machine you break one robot an hour. Hence the mainstream: «train in simulation, fine-tune on hardware» (sim2real, see F2).
Reward design: the real hard part
Skew the reward a little and the robot games it: reward «forward motion» and it learns to rub against a wall in place; reward «object grabbed» and it learns to shove the box off the table so the detector false-positives. The reward is the robot's law, and loopholes will be exploited.
Engineering practice: sparse task reward (arrived / grabbed) + dense shaping terms (distance shrinking, pose alignment) + anti-cheating audits (humans watching replays regularly). Shape too hard and the robot «farms points» instead of finishing the task.
The relationship with classical control
RL does not replace MPC/WBC — it is another layer: the high-level policy issues commands while the low level stays classical (B9). Pure end-to-end RL has real successes (quadruped locomotion), but safe fallbacks and explainability remain engineering weaknesses.
What to learn: run one simulation training (the Isaac/MuJoCo environments of G5/G6), understand what PPO's clip guards against (one oversized update wrecking the policy), and be able to explain where the sim2real gap comes from.
Three questions — do you really get it?
1. The core idea of the Bellman backup is?
- ARewards must be random to work
- BThe more complex the policy the better
- CA state's value = immediate reward + discount × next state's value
Show answer
C. Value propagates back from the goal. Iterate that sentence to convergence and you get the value map and the policy.
2. In ε-greedy, ε decays during training in order to?
- AExplore more early, exploit more late
- BSave compute
- CRaise the sampling rate
Show answer
A. Early on, exploration discovers good actions; later, the learned policy gets locked in and refined.
3. Rewarding «distance to the goal», the robot stands still and farms points — this is called?
- AVanishing gradients
- BOverfitting
- CReward hacking (shaped reward exploited)
Show answer
C. When shaping terms diverge from the true goal, the agent optimizes the proxy metric instead of the task. The audit method: watch the replays.
Keep going deeper
Links without a marker stay on this site; the rest point to the Chinese edition until its translation batch lands.
How it fails in the field
- A «looks reasonable» term sneaks into the reward and the policy exploits it (spinning in place for points)
- Trained only on one terrain type, collapses on a new floor material — then blamed on hardware
You have learned enough when
The Chinese edition carries the full topic page for Reinforcement learning, including the complete outline, depth requirements per career track, and further derivations: robotguide.dev/k/C6/.