RobotGuide
Foundations/The Only Four Math Tools You Need

Foundations 01 / 11 · Tools

The Only Four Math Tools You Need

Linearity of expectation, the tower property of conditional expectation, the score function, importance sampling. Every derivation in the eleven articles that follow is a recombination of these four

40 minPrerequisite6 derivation steps4 self-test questions

One-line conclusion

Reinforcement learning (RL) derivations use no 'advanced' math. Only four tools get used over and over, and the most critical one is called the score function - it turns 'taking a derivative with respect to a probability distribution' into 'taking a derivative with respect to a sampled point'.

This article exists to kill the fear early. You will see plenty of expressions full of ∇ and E later on, but each one uses exactly one of the four rules below, and I will flag which one. If you are short on time, you can still follow everything by reading only sections 3 and 4.

Symbol table

SymbolMeaning
E_{x~p}[f(x)]The average of f over x drawn from distribution p - the limit of a 'sample average'
∇_θThe gradient with respect to parameters θ, a vector with the same dimension as θ
p_θ(a|s)The policy: the probability of choosing action a in state s, determined by θ
H(p)Entropy, a measure of how uncertain a distribution is; maximal for the uniform distribution
KL(p||q)The difference between two distributions: non-negative, asymmetric, and 0 when p=q

Derivation

  1. Expectation is linear, and it does not require independence
  2. Tower property: take the expectation within each branch, then the expectation over branches
  3. The score function: take a log, then a derivative, and the probability becomes a sample
  4. The expectation of the score is zero (the root of why a baseline adds no bias)
  5. Importance sampling: estimate a new target from old data, at the price of variance
  6. Entropy and KL: all of SAC runs on these two

1. Expectation is linear, and it does not require independence

This one is so basic that most textbooks skip it, yet it is the entire basis for 'splitting the total return into per-step pieces'. Many people implicitly assume you need independence to split it; you do not.

E[aX + bY] = a·E[X] + b·E[Y]

Why this holdsAs long as it is a sum over the same set of samples, coefficients can be pulled out freely. Later, the Bellman equation manages to write 'an infinite sum' as 'one term plus the expectation of the rest' on the strength of this rule alone.

2. Tower property: take the expectation within each branch, then the expectation over branches

This is the engine of the Bellman equation. It says: if you want the average of X, you can first group by the value of Y, compute the average within each group, then weight those averages by the probabilities of Y.

E[X] = E_Y[ E[X | Y] ]

Why this holdsTreat 'the future' as Y and 'the current step' as the outer expectation, and you have the Bellman equation. The most commonly overlooked step in this whole series is right here: this split is valid only because the distribution of Y does not depend on the thing you are averaging.

3. The score function: take a log, then a derivative, and the probability becomes a sample

This is the only formula in the whole article you need to memorize. The left side is 'a gradient with respect to a probability'; the right side is 'the gradient divided by the probability'. It comes straight from the chain rule, with no approximation anywhere.

∇_θ log p_θ(x) = ∇_θ p_θ(x) / p_θ(x)

Why this holdsThis quantity is called the score. Its usefulness: once you multiply it by p and sum, p cancels, and 'an integral over a distribution' becomes 'an average over samples' - you never need to know what the distribution looks like, only that you can sample from it.

4. The expectation of the score is zero (the root of why a baseline adds no bias)

Multiply both sides of rule 3 by p and sum over x, and you get a beautiful result. This result is the entire basis for 'subtracting a baseline changes no expectation, only variance'.

Ex~p_θ[∇log p_θ(x)] = ∑_x ∇p_θ(x) = ∇∑_x p_θ(x) = ∇1 = 0

Why this holdsNote the third step: probabilities always sum to 1, 1 is a constant, and its gradient is 0. So anything that 'depends only on the state, not the action' multiplied by the score still has expectation 0 - that is the legitimacy proof for the baseline.

5. Importance sampling: estimate a new target from old data, at the price of variance

PPO, the DQN replay buffer, and offline RL are all built on this rule. If you want to estimate an expectation under distribution p using samples drawn from distribution q, multiply each sample by a ratio.

Ex~p[f(x)] = Ex~q[ (p(x)/q(x))·f(x) ]

Why this holdsThe prerequisite is coverage: q must be non-zero wherever p is non-zero. The price is variance: samples with a large ratio carry enormous weight, and a handful of them can dominate the whole estimate. PPO's clip exists to restrict the range of this ratio; see article 8.

6. Entropy and KL: all of SAC runs on these two

Entropy measures 'how uncertain'; KL measures 'how far apart two distributions are'. They are not optional decoration - entropy appears as an explicit term inside the SAC objective.

H(p) = -∑_a p(a) log p(a), KL(p||q) = ∑_a p(a) log(p(a)/q(a))

Why this holdsWith two actions at 0.5 each, H=0.693 (that is ln 2); the more lopsided the two action probabilities, the lower the entropy. KL is not a distance: KL(p||q) ≠ KL(q||p), and this will bite you in PPO's early-stopping criterion.

Work it by hand

A two-action policy with θ=(0.2, -0.1); probabilities come from a softmax. Compute the scores ∇log π(a1) and ∇log π(a2), and verify that their weighted sum is 0.

  1. p1 = e^0.2/(e^0.2+e^-0.1) = 1.2214/2.1262 = 0.5744, p2 = 0.4256
  2. score(a1) = (1,0) - p = (0.4256, -0.4256)
  3. score(a2) = (0,1) - p = (-0.5744, 0.5744)
  4. Weighted sum = 0.5744×(0.4256,-0.4256) + 0.4256×(-0.5744,0.5744) = (0.2445-0.2445, -0.2445+0.2445) = (0, 0)
Answer

The two scores are (0.4256, -0.4256) and (-0.5744, 0.5744), and the weighted sum is exactly (0,0). This is the hand-computed version of 'the score has zero expectation' for the two-action case - no sampling needed; plug it in and it cancels.

Run it

Verify 'the expectation of the score is zero' with 200000 samples. If this check fails, every policy gradient implementation afterward will be wrong, and it will not raise an error.

python · numpy only
import numpy as np
rng = np.random.default_rng(0)
th = np.array([0.2, -0.1])
p = np.exp(th); p = p / p.sum()
N = 200000
a = rng.choice(2, size=N, p=p)
score = np.eye(2)[a] - p        # onehot(a) - p is the score
print('E[score] =', score.mean(0), ' std err =', score.std(0)/np.sqrt(N))
print('theory = [0, 0]; passes if the mean falls within ±3 std errs')

You pass when you see this

The output looks like [0.0014 -0.0014] with std err 0.0011, a ratio of about 1.3 - within noise. If your implementation writes 1-p (using 1-p for every dimension), the second dimension will drift steadily to about -0.42, and you can spot the bug at a glance.

Self-test · 4 questions

Answer first, then expand. Each question traces back to a specific derivation section.

01Why does subtracting a baseline that depends only on the state not make the policy gradient biased?

Because the expectation of the score is 0: E[∇log π(a|s)·b(s)] = b(s)·∑_a ∇π(a|s) = b(s)·∇1 = 0. The multiplied-in term vanishes as a whole, leaving the original gradient.

02Under what conditions does the importance sampling ratio p/q destroy the estimate?

When q puts almost no samples where p has mass (insufficient coverage), the ratio explodes and the variance diverges. In practice this shows up as 'training suddenly jumping' or 'loss going inf'.

03What is the maximum entropy? Why does SAC set the target entropy to minus the action dimension?

With n actions the maximum entropy is log n. SAC uses continuous actions and sets the target entropy to -dim(A), which amounts to 'unit-Gaussian uncertainty per dimension' - a constant independent of the action scale.

04Does the tower property E[X]=E[E[X|Y]] require X and Y to be independent?

No. It only requires that the distribution of Y not depend on the quantity you condition on in the inner expectation. In the Bellman equation Y is 'the next state' and the inner part is 'everything from the next state onward', so you can split directly.

Keep reading

Reinforcement learning (knowledge page, 60h)Hands-on tutorial: run it once, then come back for the formulasNext: MDP and return
NextMDP and Return: Writing 'What To Do' as Something Computable

The whole series · 11 articles

01The Only Four Math Tools You Need40 min02MDP and Return: Writing 'What To Do' as Something Computable45 min03The Bellman Equation: Why V Is a Fixed Point and Why Iteration Must Converge60 min04Model-Free: The Bias-Variance Ledger of MC vs TD60 min05Function approximation: the three ingredients that take you from 'guaranteed convergence' to 'blowing up'55 min06Policy gradient: deriving REINFORCE line by line70 min07Actor–critic and GAE: turning the variance knob to λ60 min08PPO: what does clip actually clip70 min09SAC: entropy, reparameterization, double Q - three parts, each solving one problem75 min10HER: why goal relabeling is mathematically legitimate50 min11How to debug non-convergence: map symptoms back to formulas45 min