RobotGuide
Foundations/PPO: what does clip actually clip

Foundations 08 / 11 · policy gradient

PPO: what does clip actually clip

From the importance sampling ratio, to TRPO's KL constraint, to that min-clip piecewise function. With a gradient table for the four cases

70 minAdvanced6 derivation steps4 self-test questions

One-line conclusion

PPO's clip is a piecewise function, not a slogan about 'limiting update magnitude'. It zeroes the gradient in two situations: the action is much better than under the old policy (already enough, stop pushing) and much worse (already avoided, stop suppressing). Understand these two and the hyperparameter ε loses its mystique.

PPO is the most used algorithm in industry, and the one most likely to be 'used like a library call'. The goal of this piece is that when your loss curve looks abnormal, you can immediately say which branch of the clip is at fault.

Symbol table

SymbolMeaning
r_t(θ)the importance sampling ratio π_θ(a|s)/π_old(a|s); r=1 means unchanged
A_tthe advantage, computed in piece 7
εthe clip range, default 0.2
L^CPIthe unconstrained importance sampling objective r·A

Derivation

  1. Why a ratio is needed: the data was sampled by the old policy
  2. The unconstrained objective, and why it must blow up
  3. TRPO: a hard constraint buys a theoretical guarantee
  4. PPO's replacement: turning the constraint into a piecewise function
  5. The four cases: which two have zero gradient
  6. Three terms of the full loss that are often ignored

1. Why a ratio is needed: the data was sampled by the old policy

Every on-policy update makes the earlier data 'stale'. To reuse it for several rounds, you must fold the old data onto the new policy with importance sampling.

r_t(θ) = π_θ(a_t|s_t) / π_θold(a_t|s_t)

Why this holdsr is computable (the ratio of two policies' probabilities for the same action) and needs no environment. That is the basis for PPO updating multiple epochs on the same batch of data. The ratio is computed per action, not as one global scalar.

2. The unconstrained objective, and why it must blow up

The most direct objective multiplies the return by the ratio. Mathematically correct, but optimization will walk into regions the data never covered.

LCPI(θ) = E_t[ r_t(θ)·A_t ]

Why this holdsThe blow-up mechanism: r has no upper bound. When an action's advantage is overestimated, the optimizer pushes its probability toward 1, r becomes 10, 100, and the objective rises along - but the states those probabilities correspond to were never sampled under the new policy, so the estimate is entirely untrustworthy. The next step is the policy collapsing onto a single action.

3. TRPO: a hard constraint buys a theoretical guarantee

Since the problem is 'going too far', limit how far you go. TRPO uses the average KL divergence as a trust region and maximizes the objective inside it.

max_θ E[ r_t(θ)A_t ] s.t. E[ KL(π_old || π_θ) ] ≤ δ

Why this holdsIt has a monotonic improvement guarantee, but requires Fisher matrix-vector products + conjugate gradients: complex to implement, expensive per step. In engineering, 'guarantees' are not that valuable; simplicity is.

4. PPO's replacement: turning the constraint into a piecewise function

Instead of adding a constraint, change the objective to 'the worse of the two'. That one change yields a lower bound that needs no second-order optimization.

LCLIP = E_t[ min( r_t A_t , clip(r_t, 1-ε, 1+ε)·A_t ) ]

Why this holdsWhy min: it guarantees L^CLIP ≤ r·A, i.e. 'never overestimate the gain of this update'. If you use clip directly (without min), then when A<0 and r is small, clip shrinks the penalty - effectively encouraging 'getting worse'. This is the trap many people step on when implementing PPO themselves.

5. The four cases: which two have zero gradient

Combine the sign of A with whether r leaves the range: only four cases. Measured (ε=0.2, A=±2):

A>0, r=0.5 → L=1.0 (gradient follows A=2); A>0, r=1.5 → L=2.4 (gradient 0)

Why this holdsFull results: for A=2, r=0.5/0.9/1.0/1.15 all give gradient 2, and r=1.5 gives gradient 0; for A=-2, r=1.15/1.5 still give gradient -2 (keep punishing), and r=0.5 gives gradient 0. The pattern: a good action pushed too far → stop; a bad action suppressed too far → stop; all other cases update as usual. So ε essentially means 'how large a probability change is allowed', and 0.2 means the probability ratio may change by at most ±20%.

6. Three terms of the full loss that are often ignored

The real implementation is three parts summed. Everyone knows the first two; the third often gets dropped or double-counted.

L = -LCLIP + c1·MSE(V, R) - c2·H(π), commonly c1=0.5, c2=0.01

Why this holdsThree details: (1) the advantage must be normalized (subtract the mean, divide by the standard deviation), otherwise the gradient scale is unstable on small batches; (2) the value loss is sometimes clipped too (compared against the old value, it must not change too much), preventing a collapsing critic from dragging the actor down; (3) an entropy coefficient too small makes the policy settle prematurely, too large prevents learning a deterministic policy - 0.01 is the common value for continuous control, and 0.01~0.05 for discrete-action tasks.

Work it by hand

With ε=0.2, what is L^CLIP for A=2 and r=1.3? Is the gradient with respect to θ positive, negative, or zero? Then compute the case A=-2, r=1.3.

  1. A=2, r=1.3: the unclipped term = 2.6, the clipped term = 2×1.2 = 2.4, min picks 2.4 → and because r already exceeds the upper bound, this branch is constant, so the gradient is 0
  2. A=-2, r=1.3: the unclipped term = -2.6, the clipped term = -2×1.2 = -2.4, min picks -2.6 → the unclipped term wins, the gradient is computed with A=-2 (the probability is still being pushed down)
Answer

The former: 2.4, gradient 0; the latter: -2.6, gradient -2 (this action keeps being pushed down). At the same spot r>1 (the probability grew), for a good action it means 'enough, stop', for a bad action 'it grew, push harder' - that is the asymmetry of min, and the reason PPO is more stable than naive truncation.

Run it

20 lines verifying the table above. This code needs no torch - pure numpy is enough to make the four cases and the gradient signs clear.

python · numpy only
import numpy as np
EPS = 0.2
def lclip(A, r):
    return min(r * A, np.clip(r, 1-EPS, 1+EPS) * A)
for A in (2.0, -2.0):
    for r in (0.5, 0.9, 1.0, 1.15, 1.5):
        # numerical gradient: differentiate with respect to r to see whether this branch is still active
        h = 1e-6
        g = (lclip(A, r+h) - lclip(A, r-h)) / (2*h)
        print('A=%+.0f r=%.2f L=%+.3f dL/dr=%+.3f %s' % (A, r, lclip(A, r), g, 'gradient clipped' if abs(g) < 1e-9 else ''))

You pass when you see this

The row A=+2 with r=1.5 and the row A=-2 with r=0.5 will show 'gradient clipped'. The remaining rows have gradients of +2 or -2. If those two rows are not 0 in your implementation, check whether min became max, or the clip bounds are swapped.

Self-test · 4 questions

Answer first, then expand. Each question traces back to a specific derivation section.

01Is PPO's clip the same thing as gradient clipping (grad clip)?

No. PPO's clip acts on the importance sampling ratio, changing the shape of the objective function; grad clip acts on the gradient norm, an optimizer-level safeguard. Both are often on at the same time, but they solve different problems.

02Why does PPO update multiple epochs, and doesn't that break the on-policy assumption?

Multiple epochs improve sample efficiency, at the real cost of drifting off-policy. Clipping exists precisely to limit that drift. Hence epochs are usually 3~10; beyond that clip fires constantly and updates nearly stall (the policy loss stops moving).

03The policy loss becomes 0 and stops changing - what could be the cause?

Two options: (1) the advantages are all normalized to nearly 0 (the critic collapsed or the rewards are all identical); (2) a large share of samples triggers clip and the gradient is clipped away. How to check: print the clip fraction; normally it should be 0.1~0.3, and persistently >0.6 means learning too hard or the data is too stale.

04What exactly does ε=0.2 constrain?

It keeps the new/old probability ratio of the same action inside [0.8, 1.2]. Because the ratio is computed per (s,a), what is actually constrained is 'each action's probability changes by at most about 20%', not the policy's overall KL.

Keep reading

Previous: actor–critic and GAENext: SACHands-on: running SB3 PPO once
PreviousActor–critic and GAE: turning the variance knob to λ NextSAC: entropy, reparameterization, double Q - three parts, each solving one problem

The whole series · 11 articles

01The Only Four Math Tools You Need40 min02MDP and Return: Writing 'What To Do' as Something Computable45 min03The Bellman Equation: Why V Is a Fixed Point and Why Iteration Must Converge60 min04Model-Free: The Bias-Variance Ledger of MC vs TD60 min05Function approximation: the three ingredients that take you from 'guaranteed convergence' to 'blowing up'55 min06Policy gradient: deriving REINFORCE line by line70 min07Actor–critic and GAE: turning the variance knob to λ60 min08PPO: what does clip actually clip70 min09SAC: entropy, reparameterization, double Q - three parts, each solving one problem75 min10HER: why goal relabeling is mathematically legitimate50 min11How to debug non-convergence: map symptoms back to formulas45 min