RobotGuide

Foundations

Derive the formulas until you can write them yourself.

A full derivation path for reinforcement learning: from the four math tools to PPO and SAC, every article walks claim → derivation → hand-workable example → 20-line numpy check → 4 self-test questions. No RL framework needed — every number on these pages was actually computed.

How to read: look at the one-line conclusion and the «why this section exists» first. If they make sense, jump straight to the worked example, verify it by hand, then go back to the derivation. All example numbers were really run — the code is copy-paste executable and needs nothing but numpy.

Play first, derive later

Every formula in this series is animated somewhere: rotation matrices, Bayesian counting, gradient descent on a hill — all in the 15 interactive labs and on the lesson pages. Understand the phenomenon before the proof.

Every article has the same shape

One-line conclusion → step-by-step derivation (each step names the math tool it uses) → a hand-workable example → a 20-line numpy check → 4 self-test questions

Why the example must be worked by hand

Between «I followed the derivation» and «I can write it myself» there is exactly one hand computation. All example numbers were really run — if yours do not match the page, some sign or boundary is off.

Groundwork: math and modeling

If these three are not solid, everything after is memorized formulas

01 The Only Four Math Tools You NeedLinearity of expectation, the tower property of conditional expectation, the score function, importance sampling. Every derivation in the eleven articles that follow is a recombination of these four 40 min6 derivation steps4 self-test 02 MDP and Return: Writing 'What To Do' as Something ComputableStates, actions, transitions, rewards, discount. Once these are pinned down the problem has no ambiguity left - all algorithms are solving the same problem 45 min6 derivation steps4 self-test 03 The Bellman Equation: Why V Is a Fixed Point and Why Iteration Must ConvergeFrom the closed-form solution of a linear system, to the convergence guarantee of a contraction mapping, to the one spot of nonlinearity that max introduces - and the systematic overestimation it causes 60 min6 derivation steps4 self-test

Value functions: from tables to function approximation

Where «guaranteed convergence» breaks down

04 Model-Free: The Bias-Variance Ledger of MC vs TDReal return as the target or a bootstrap as the target - this one choice decides the shape of every algorithm that follows. Includes a reproducible measured comparison 60 min6 derivation steps4 self-test 05 Function approximation: the three ingredients that take you from 'guaranteed convergence' to 'blowing up'Semi-gradient, the deadly triad, the distribution decides the answer, target networks. This piece explains why deep RL training curves often don't 'converge slowly' but 'explode' 55 min6 derivation steps4 self-test

Policy gradients: optimizing the policy directly

REINFORCE → GAE → PPO, one continuous derivation

06 Policy gradient: deriving REINFORCE line by lineThe whole derivation uses one single trick, but three times. By the end you will know why 'no environment model needed' is not a coincidence but a necessity 70 min6 derivation steps4 self-test 07 Actor–critic and GAE: turning the variance knob to λThe advantage function is the only necessity of the policy gradient. GAE gives a continuous knob: λ=0 degenerates to one-step TD, λ=1 degenerates to MC 60 min6 derivation steps4 self-test 08 PPO: what does clip actually clipFrom the importance sampling ratio, to TRPO's KL constraint, to that min-clip piecewise function. With a gradient table for the four cases 70 min6 derivation steps4 self-test

Continuous control and field practice

Maximum entropy, hindsight relabeling, and how to diagnose non-convergence

09 SAC: entropy, reparameterization, double Q - three parts, each solving one problemThe max-entropy objective keeps the policy stochastic, reparameterization lets gradients pass through sampling, double Q pushes the positive bias of max back down. Remove any one of them and it doesn't work 75 min7 derivation steps4 self-test 10 HER: why goal relabeling is mathematically legitimateChange the goal without changing the dynamics - that is the entire argument. Plus the quantitative condition almost nobody mentions: 'the tolerance must exceed the per-step displacement' 50 min6 derivation steps4 self-test 11 How to debug non-convergence: map symptoms back to formulasEleven common curve shapes mapped to the specific term in the formula. Plus three pre-training health checks and two controlled experiments you must run 45 min8 derivation steps4 self-test

Relationship to the lesson pages

The 74 lessons tell you what to learn and how deep; this series tells you why the tools in track «algo» actually work. Recommended order: meet the phenomenon on a lesson or lab page first, then read the matching derivation — at that point every symbol has a phenomenon attached to it.