RobotGuide

Perception & intelligence · topic C6 · 60 hours

Reinforcement learning

Quadruped and humanoid gaits, dexterous manipulation — more and more of it is trained with RL. It is also the classic case of «spectacular in demos, brutal in deployment».

Specialization stage · Hardcore
Algorithm · Proficient

What to learn

The lesson · plain words

Everything before this page was «humans write the rules». Reinforcement learning flips it: give only a reward signal and let the robot figure out what to do. It can learn policies no human writes (a quadruped's skittering gait, an arm's flick-throw) — at the price of expensive training and hard-to-explain behavior.

Understanding RL does not require deriving formulas; it requires two concepts: value (how many points is this situation worth) and policy (which action to take in each situation).

Value iteration: the score propagates itself

Look at the corridor below: goal +1, pit −1, everything else 0. Initially no cell knows its worth; repeatedly back up «which next step pays more» and the score propagates back from the goal — after 30 rounds the whole corridor's value map has grown.

This is the minimal kernel of RL: the Bellman backup — a state's value equals its immediate reward plus a discount times the next state's value. Every RL algorithm is a variation on that one sentence.

Interactive figure · drag it and see

What this figure shows: a corridor of cells — the goal scores +1, the pit −1 — where value iteration runs round by round, growing the value map from the goal outward and forever routing around the pit. Takeaway: a state's value = immediate reward + discount × next state's value; iterate that one sentence until it converges and you get both the value map and the policy — that recursion is the minimal kernel of all RL.

Exploration and exploitation: the trial ratio

Always picking the current best (exploitation) locks you onto the first decent-looking option; pure random flailing (exploration) never learns anything. ε-greedy: 90% follow the policy, 10% random, decaying as training proceeds.

The real difficulty of robot RL is data, not algorithms: 100,000 tries in simulation are affordable; on the real machine you break one robot an hour. Hence the mainstream: «train in simulation, fine-tune on hardware» (sim2real, see F2).

Reward design: the real hard part

Skew the reward a little and the robot games it: reward «forward motion» and it learns to rub against a wall in place; reward «object grabbed» and it learns to shove the box off the table so the detector false-positives. The reward is the robot's law, and loopholes will be exploited.

Engineering practice: sparse task reward (arrived / grabbed) + dense shaping terms (distance shrinking, pose alignment) + anti-cheating audits (humans watching replays regularly). Shape too hard and the robot «farms points» instead of finishing the task.

The relationship with classical control

RL does not replace MPC/WBC — it is another layer: the high-level policy issues commands while the low level stays classical (B9). Pure end-to-end RL has real successes (quadruped locomotion), but safe fallbacks and explainability remain engineering weaknesses.

What to learn: run one simulation training (the Isaac/MuJoCo environments of G5/G6), understand what PPO's clip guards against (one oversized update wrecking the policy), and be able to explain where the sim2real gap comes from.

Three questions — do you really get it?

1. The core idea of the Bellman backup is?

  • ARewards must be random to work
  • BThe more complex the policy the better
  • CA state's value = immediate reward + discount × next state's value
Show answer

C. Value propagates back from the goal. Iterate that sentence to convergence and you get the value map and the policy.

2. In ε-greedy, ε decays during training in order to?

  • AExplore more early, exploit more late
  • BSave compute
  • CRaise the sampling rate
Show answer

A. Early on, exploration discovers good actions; later, the learned policy gets locked in and refined.

3. Rewarding «distance to the goal», the robot stands still and farms points — this is called?

  • AVanishing gradients
  • BOverfitting
  • CReward hacking (shaped reward exploited)
Show answer

C. When shaping terms diverge from the true goal, the agent optimizes the proxy metric instead of the task. The audit method: watch the replays.

Keep going deeper

Links without a marker stay on this site; the rest point to the Chinese edition until its translation batch lands.

How it fails in the field

You have learned enough when

You can train a standing policy and transfer it to different terrains in simulation, explain reward-design traps, and state the failure attribution order for sim2real.

The Chinese edition carries the full topic page for Reinforcement learning, including the complete outline, depth requirements per career track, and further derivations: robotguide.dev/k/C6/.