RobotGuide
Foundations/Actor–critic and GAE: turning the variance knob to λ

Foundations 07 / 11 · policy gradient

Actor–critic and GAE: turning the variance knob to λ

The advantage function is the only necessity of the policy gradient. GAE gives a continuous knob: λ=0 degenerates to one-step TD, λ=1 degenerates to MC

60 minAdvanced6 derivation steps4 self-test questions

One-line conclusion

The essence of actor–critic is not 'two networks' but 'using a learned V to build a low-variance advantage estimate'. GAE hands 'how far to trust bootstrapping' to a number between 0 and 1, and λ=0.95 is the default validated over and over in practice.

In a PPO implementation, the single line that computes the advantage decides success or failure. This piece derives that line and gives you a way to automatically check whether the implementation is right (at λ=1 it must equal the MC returns bit for bit).

Symbol table

SymbolMeaning
A^π(s,a)the advantage: how much better this action is than 'the policy average'
δ_tthe TD error r + γV(s') - V(s)
λthe decay of GAE, controlling how long to trust bootstrapping
γλthe compound discount that actually works, determining how fast the weights decay

Derivation

  1. The advantage: swapping 'absolute value' for 'relative value'
  2. The one-step TD error is an unbiased estimate of the advantage (provided V is accurate)
  3. n-step advantages and GAE's geometric weighting
  4. Recursive implementation: one backward sweep, O(T)
  5. The two extremes of λ (letting measured numbers speak)
  6. Compute one by hand to confirm you really got it

1. The advantage: swapping 'absolute value' for 'relative value'

Q tells you how much an action is worth, but it includes 'how good this state itself is'. In a good state every action's Q is large and the gradients get pushed up together - exactly the disease the baseline fixes.

A^π(s,a) = Q^π(s,a) - V^π(s)

Why this holdsProperty: ∑_a π(a|s)A^π(s,a) = 0 (the advantage averages to zero under the policy). So 'good actions' and 'bad actions' separate naturally in the gradient, with no extra centering needed.

2. The one-step TD error is an unbiased estimate of the advantage (provided V is accurate)

This is the pivot on which actor–critic stands. Expand Q one step with the Bellman equation and you will find δ is exactly A.

E[δ_t | s_t,a_t] = r + γE[V(st+1)] - V(s_t) = Q^π(s_t,a_t) - V^π(s_t) = A^π(s_t,a_t)

Why this holdsSo δ can serve as the advantage directly - A2C is exactly δ. But it uses only 'one step of real reward', so its bias is large. Use a few more steps of real reward and a little less estimation, and the bias shrinks. That spectrum is GAE.

3. n-step advantages and GAE's geometric weighting

Carry the n-step returns of piece 4 over to advantages and you get n-step advantages. GAE does not pick a single n; it averages all n-step advantages with geometric weights.

AGAE_t = ∑k=0∞ (γλ)^k δt+k

Why this holdsThe closer λ is to 1, the slower the weights decay and the more MC-like it is; at λ=0 only δ_t remains. Note the decay rate is γλ and not λ - γ is already inside δ, and the two discounts stacked together give the effective horizon.

4. Recursive implementation: one backward sweep, O(T)

The formula above is an infinite sum and slow to compute directly. Written recursively, one loop from back to front suffices - the standard form in every implementation.

A_t = δ_t + γλ·At+1 (A_T = 0)

Why this holdsThe easiest thing to get wrong is the 'episode boundary': at the end of a trajectory A must restart from 0, not continue from the previous one. Mixing them lets advantages leak across episodes - the symptom is that early training looks fine and later it degrades more and more.

5. The two extremes of λ (letting measured numbers speak)

Given δ=(0.78, 0.39, -0.30, 0) and γ=0.9, compute the results for λ=0.95 and λ=1.0.

GAE(0.95) = (0.894, 0.134, -0.300); GAE(1.0) = (0.888, 0.120, -0.300)

Why this holdsThe λ=1.0 result equals 'MC return minus V' bit for bit, residual 2.2e-16 (floating-point precision). This property is the best unit test: run your GAE implementation at λ=1 and compare against hand-computed (G_t - V_t); a mismatch means the boundary handling is wrong. λ=0.95 is smoother (the second value 0.134 sits between δ and MC), which is exactly why it is the default.

6. Compute one by hand to confirm you really got it

Take δ=(1, 0.5, -0.2), with γ=0.9 and λ=0.95, and compute A_0.

γλ = 0.855, (γλ)² = 0.731; A_0 = 1 + 0.855×0.5 + 0.731×(-0.2) = 1.281

Why this holdsThe three terms are the contributions of 'this step', 'the next step', and 'the step after'. Note the second term's weight 0.855 is already 15% smaller than the first - GAE naturally weights the near term, and that is how it cuts variance.

Work it by hand

Two trajectories in the same batch: trajectory A rewards [1,1,1], trajectory B rewards [0,0,0], and V is estimated as 0.5 throughout. Compute the advantages with the λ=0 GAE (i.e. δ) and explain what error 'mixing across trajectories' causes.

  1. trajectory A: δ = 1+0.9×0.5-0.5 = 0.95 (all three steps alike); trajectory B: δ = 0+0.9×0.5-0.5 = -0.05
  2. without resetting the boundary, A's last step attaches onto B's first step, and δ_A3 becomes 1 + 0.9×V(B0) - V(A3), where V(B0) is a state estimate from another trajectory
  3. consequence: A's advantage is contaminated by B's states, the gradient direction is wrong, and no error is raised
Answer

The correct fix is to zero A at episode termination and at the same time set next_value to 0 (the terminal state's V is defined as 0). The typical symptom of this bug is 'success rate climbs but never gets high, and the variance is large' - half the samples have wrong advantages, just not wrong enough to be obvious.

Run it

15 lines of GAE with a built-in assertion. That assertion (λ=1 equals MC) is the only reliable way to judge whether the implementation is right.

python · numpy only
import numpy as np
def gae(rew, val, gamma=0.99, lam=0.95, last_val=0.0):
    T = len(rew)
    adv = np.zeros(T)
    gae_run = 0.0
    for t in reversed(range(T)):
        nv = val[t+1] if t+1 < T else last_val
        delta = rew[t] + gamma * nv - val[t]
        gae_run = delta + gamma * lam * gae_run
        adv[t] = gae_run
    return adv
rw = np.array([1.0, 0.5, -0.2, 0.0]); vs = np.array([0.4, 0.2, 0.1, 0.0, 0.0])
a95 = gae(rw, vs, 0.9, 0.95); a100 = gae(rw, vs, 0.9, 1.0)
mc = np.zeros(4); gt = 0.0
for t in reversed(range(4)):
    gt = rw[t] + 0.9 * gt; mc[t] = gt
print('GAE(0.95) =', np.round(a95, 4))
print('GAE(1.0)  =', np.round(a100, 4), ' MC-V =', np.round(mc - vs[:-1], 4))
assert np.abs(a100 - (mc - vs[:-1])).max() < 1e-9, 'boundary handling is wrong'
print('assertion passed: lambda=1 matches the MC returns exactly')

You pass when you see this

It prints GAE(0.95)=[0.894 0.134 -0.3 0], GAE(1.0)=[0.888 0.12 -0.3 0], and the assertion passes. If the assertion fails, 99% of the time the last state used V[T] instead of last_val=0.

Self-test · 4 questions

Answer first, then expand. Each question traces back to a specific derivation section.

01Why is the advantage function's expectation under the policy 0? What is this property good for?

Because ∑π(a|s)Q(s,a) = V(s), subtracting V makes the weighted sum zero. It keeps the gradient uncontaminated by 'how good this state itself is', and it centerizes the advantages naturally, making the numbers more stable.

02Why is it γλ and not λ in GAE?

δ already contains one γ (the discount of the future value); multiplying by λ gives the geometric weights (γλ)^k. If you write λ and drop γ, λ=1 no longer equals the MC returns - the assertion catches it directly.

03Why is λ=0.95 the default?

It sits at a measured sweet spot between bias and variance that works for most continuous control tasks: much smoother than λ=0, yet unlike λ=1 it does not swallow all the long-horizon noise. When the task horizon is very long you can raise it to 0.98.

04What happens if the critic learns badly? Will the actor be unable to learn at all?

The advantage estimate becomes biased (δ's bias comes straight from V's error), but usually the actor can still learn, because δ keeps a real-reward term. What is truly fatal is a wrong critic scale (e.g. outputting 1e3), which explodes the gradient. Hence implementations often normalize the advantage.

Keep reading

Previous: policy gradientNext: PPO's clippingHands-on: what these parameters look like in SB3
PreviousPolicy gradient: deriving REINFORCE line by line NextPPO: what does clip actually clip

The whole series · 11 articles

01The Only Four Math Tools You Need40 min02MDP and Return: Writing 'What To Do' as Something Computable45 min03The Bellman Equation: Why V Is a Fixed Point and Why Iteration Must Converge60 min04Model-Free: The Bias-Variance Ledger of MC vs TD60 min05Function approximation: the three ingredients that take you from 'guaranteed convergence' to 'blowing up'55 min06Policy gradient: deriving REINFORCE line by line70 min07Actor–critic and GAE: turning the variance knob to λ60 min08PPO: what does clip actually clip70 min09SAC: entropy, reparameterization, double Q - three parts, each solving one problem75 min10HER: why goal relabeling is mathematically legitimate50 min11How to debug non-convergence: map symptoms back to formulas45 min