RobotGuide
Foundations/Policy gradient: deriving REINFORCE line by line

Foundations 06 / 11 · policy gradient

Policy gradient: deriving REINFORCE line by line

The whole derivation uses one single trick, but three times. By the end you will know why 'no environment model needed' is not a coincidence but a necessity

70 minAdvanced6 derivation steps4 self-test questions

One-line conclusion

The policy gradient does exactly one thing: raise the log-probability of good actions, lower that of bad ones. And the standard for 'good' is the return after that step. Nowhere in the whole derivation do you need the environment's transition probabilities - that is the fundamental reason it works on real hardware.

This is the watershed of deep RL. The previous four pieces were all 'learn a value first, then pick actions by it'; from this piece on, the parameters live directly on the policy. PPO, SAC, and A3C are all variants of this piece, differing only in 'what replaces G_t' and 'how the step size is limited'.

Symbol table

SymbolMeaning
τa trajectory (s0,a0,r1,s1,a1,...), which is random
p_θ(τ)the probability of this trajectory under policy π_θ
J(θ)the objective: expected return, the bigger the better
∇_θ log π_θ(a|s)the score function, the one from piece 1 section 3
b(s)a baseline: depends on state only, not on the action

Derivation

  1. The objective: expected return, written as an integral
  2. Gradient into the integral, then turn p into p·∇log p
  3. Factorizing trajectory probability: the environment term vanishes
  4. Get the final form, and swap the return for 'what comes after'
  5. The baseline: subtract any state-only quantity and the expectation is unchanged
  6. Why log is mandatory, and why you cannot differentiate π directly

1. The objective: expected return, written as an integral

Write the objective as 'the returns of all trajectories weighted by probability'. This step is only a definition, but it turns the problem into 'differentiating a parameter-dependent probability distribution'.

J(θ) = Eτ~p_θ[ R(τ) ] = ∫ p_θ(τ) R(τ) dτ

Why this holdsKey observation: R(τ) does not depend on θ. The reward comes from the environment; changing your policy does not change the environment's rules. This 'does not depend on' is the premise that lets the gradient move inside the integral in the next step.

2. Gradient into the integral, then turn p into p·∇log p

The gradient only acts on p_θ (because R contains no θ). Then use the score identity from piece 1 section 3 to replace ∇p with p·∇log p.

∇J = ∫ ∇p_θ(τ)·R(τ) dτ = ∫ p_θ(τ)·∇log p_θ(τ)·R(τ) dτ = E[ ∇log p_θ(τ)·R(τ) ]

Why this holdsThis step is the core transformation of the whole piece. Its meaning: the left side is 'differentiating a distribution'; the right side is 'differentiating sampled trajectories and averaging' - the right side can be estimated directly from samples. The feasibility of all model-free RL rests on this step.

3. Factorizing trajectory probability: the environment term vanishes

A trajectory's probability = initial state distribution × per-step policy probability × per-step transition probability. After taking log it becomes a sum of three terms; take the gradient and only the middle term survives.

p_θ(τ) = ρ(s_0) ∏_t π_θ(a_t|s_t) P(st+1|s_t,a_t)

Why this holdsAfter log: ∇log p_θ(τ) = ∑_t ∇log π_θ(a_t|s_t), because ρ and P contain no θ and their gradient is 0. This is the entire reason 'policy gradients need no environment model', and the single most memorable line of this piece.

4. Get the final form, and swap the return for 'what comes after'

Substitute item 3 into item 2 to get the per-step sum form. Then one simplification: rewards before t are unrelated to a_t (an action cannot affect the past), so they can be dropped.

∇J = E[ ∑_t ∇log π_θ(a_t|s_t)·G_t ], where G_t = ∑k≥t γk-t rk+1

Why this holdsDropping 'past rewards' is not just saving computation: it significantly reduces variance. Intuition: within one trajectory, how good a_0 is should not be influenced by r_0 (a reward received before the action was taken). Replacing G_t with Q^π(s_t,a_t) is equivalent in expectation, but G_t can be sampled directly and Q cannot.

5. The baseline: subtract any state-only quantity and the expectation is unchanged

G_t is usually positive (e.g. a small reward every step), which raises the probability of every action, only by different amounts. Subtract an average so that 'better than average' and 'worse than average' separate.

E[ ∇log π(a|s)·b(s) ] = b(s)∑_a ∇π(a|s) = b(s)·∇1 = 0

Why this holdsSo b(s) can be anything: a constant, V(s), a batch's average return. The most common choice is V(s), at which point G_t - V(s) becomes the advantage A (piece 7). Measured (three-armed bandit, 4000 samples): the gradient standard deviation drops from 0.362 to 0.137, the variance falls to 14% of its original value, and the gradient mean barely moves (0.171 → 0.168).

6. Why log is mandatory, and why you cannot differentiate π directly

Differentiating π directly gives ∇π, which is not in the form of an expectation - you cannot estimate it from samples (you do not know which observable quantity corresponds to 'the gradient of a probability').

∇π = π·∇log π

Why this holdsThis one identity turns 'gradient of a probability' into 'probability × a quantity computable on samples'. With it, the gradient becomes 'compute a number on each sampled action, then average'. Without it, the policy gradient would remain theory-only. It is also why real implementations are always log_prob(a).backward(), never differentiating the probability itself.

Work it by hand

Three-armed bandit, initial θ=(0,0,0) so all three actions are equiprobable. Action a1 is taken once and yields reward 1; the other two actions are not taken (rewards unknown). Write the θ direction after one update with REINFORCE without a baseline and with a baseline (b=0.5), and explain the difference between the two.

  1. equiprobable → p=(1/3,1/3,1/3); when a1 is taken, score = onehot(a1) - p = (2/3, -1/3, -1/3)
  2. without baseline: gradient estimate = (2/3, -1/3, -1/3) × 1.0
  3. with baseline: gradient estimate = (2/3, -1/3, -1/3) × (1.0 - 0.5) = 0.5 times the one above
  4. the two directions are exactly identical, because 'a global rescale' does not change direction - the baseline's effect shows up across many samples, not in a single one
Answer

For a single update the two differ only in length (0.5 times), same direction. The baseline's value lies in the variance of the expectation: without one, if this trajectory's rewards are all positive (very common), every action's probability gets pushed up, differing only in relative magnitude; with one, below-average actions get explicitly pushed down. The measured variance ratio of 14% comes from 4000 samples; seeing no difference in a single draw is normal.

Run it

30 lines, directly testing 'a baseline reduces variance'. Flip the use_base switch to compare; no gym install needed.

python · numpy only
import numpy as np
rng = np.random.default_rng(1)
R = np.array([1.0, 0.5, 0.0])      # true expected rewards of the three actions
def grads(use_base, trials=4000):
    out = []
    for _ in range(trials):
        th = np.zeros(3); p = np.exp(th)/np.exp(th).sum()
        a = rng.choice(3, p=p)
        b = R.mean() if use_base else 0.0
        out.append((np.eye(3)[a] - p) * (R[a] - b))
    return np.array(out)
for use in (False, True):
    g = grads(use)
    print('baseline' if use else 'no base ', 'std =', np.round(g.std(0), 3), ' mean =', np.round(g.mean(0), 3))

You pass when you see this

Without a baseline the standard deviations are about [0.362 0.273 0.136]; with one about [0.137 0.136 0.136]. Both means are close to [0.17, 0, -0.17] (how much better that action is than average) - the variance drops about 7-fold while the mean is unchanged; that is the entire effect of a baseline.

Self-test · 4 questions

Answer first, then expand. Each question traces back to a specific derivation section.

01In the policy gradient derivation, which step uses the fact that 'the reward does not depend on θ'?

The first step: ∇ only acts on p_θ, with R treated as a constant. If the reward depended on the policy in reverse (e.g. a penalty on action magnitude where the magnitude is determined by θ), this step would fail and an extra term would be needed.

02Why does the environment's transition probability P vanish completely from the final expression?

Because log turns products into sums, and P contains no θ so its gradient is 0. Hence policy gradients are 'model-free' - you can compute the gradient without knowing the robot's dynamics.

03Why is REINFORCE almost unusable on real robot tasks?

Variance. G_t is the return of an entire trajectory: high-dimensional, noisy, and it must wait for the episode to end. On real hardware an episode can take tens of seconds, and sample efficiency is nowhere near enough, so you use a critic to replace G_t with an advantage estimate.

04What happens if you set the baseline to a very large positive number?

The expectation and direction are unchanged, but the signal-to-noise ratio worsens: all advantages become negative with similar magnitudes, and the variance of the gradient estimate actually rises. A baseline should track V(s); bigger is not better.

Keep reading

Previous: function approximationNext: actor–critic and GAEHands-on: getting it running
PreviousFunction approximation: the three ingredients that take you from 'guaranteed convergence' to 'blowing up' NextActor–critic and GAE: turning the variance knob to λ

The whole series · 11 articles

01The Only Four Math Tools You Need40 min02MDP and Return: Writing 'What To Do' as Something Computable45 min03The Bellman Equation: Why V Is a Fixed Point and Why Iteration Must Converge60 min04Model-Free: The Bias-Variance Ledger of MC vs TD60 min05Function approximation: the three ingredients that take you from 'guaranteed convergence' to 'blowing up'55 min06Policy gradient: deriving REINFORCE line by line70 min07Actor–critic and GAE: turning the variance knob to λ60 min08PPO: what does clip actually clip70 min09SAC: entropy, reparameterization, double Q - three parts, each solving one problem75 min10HER: why goal relabeling is mathematically legitimate50 min11How to debug non-convergence: map symptoms back to formulas45 min