RobotGuide
Foundations/SAC: entropy, reparameterization, double Q - three parts, each solving one problem

Foundations 09 / 11 · Policy Gradient

SAC: entropy, reparameterization, double Q - three parts, each solving one problem

The max-entropy objective keeps the policy stochastic, reparameterization lets gradients pass through sampling, double Q pushes the positive bias of max back down. Remove any one of them and it doesn't work

75 minAdvanced7 derivation steps4 self-test questions

One-line conclusion

SAC's objective is not 'maximize return' but 'maximize return plus entropy'. This single change turns 'one deterministic optimum' into 'a crowd of equally good actions' - exploration, robustness, and multimodal policies are all byproducts of it, not additional mechanisms bolted on.

On continuous control tasks SAC is usually an order of magnitude more sample-efficient than PPO, at the cost of four networks and a pile of mutually dependent terms. This article explains where each term comes from, so you know what happens when you turn each knob.

Symbol table

SymbolMeaning
αtemperature, controls the relative weight of the entropy term
H(π(·|s))the entropy of the policy at state s
ξstandard Gaussian noise, used for reparameterization
tanhsquashes an unbounded Gaussian into (-1,1); must be paired with the log-det correction
H_targettarget entropy, set to -dim(A) for continuous control

Derivation

  1. The max-entropy objective: add an 'uncertainty' term to the return
  2. The soft value function: stuff entropy into Q
  3. The soft Bellman equation: the target carries the current policy's log π
  4. Reparameterization: let gradients pass through the sampled point
  5. tanh's log-det correction (skip it and you get 'fake high entropy')
  6. Automatic α tuning: turn the most sensitive hyperparameter into an adaptive one
  7. Double Q: push the positive bias of max back down

1. The max-entropy objective: add an 'uncertainty' term to the return

Ordinary RL only asks 'which action has the highest return'; max-entropy RL asks 'among actions whose returns are close to the highest, which policy is the least picky'.

J(π) = ∑_t E[ r(s_t,a_t) + α·H(π(·|s_t)) ]

Why this holdsAs α→0 it degenerates back to ordinary RL (the policy determinizes quickly); as α→∞ it becomes random flailing. The benefit of the middle ground: multiple equally good actions all retain probability mass, so the policy doesn't lock onto one suboptimal action because of a tiny estimation error. This is why SAC is more noise-tolerant on real hardware.

2. The soft value function: stuff entropy into Q

When defining soft V, subtract the entropy term directly. That way Q's target naturally carries entropy, and no exploration bonus needs to be added elsewhere.

V(s) = Ea~π[ Q(s,a) - α log π(a|s) ]

Why this holdsSince H = -E[log π], 'adding entropy' equals 'subtracting log π'. Note that log π is negative (probability less than 1), so -α log π is positive - the entropy term is awarding points to 'uncertain actions'.

3. The soft Bellman equation: the target carries the current policy's log π

Substituting V back into Q's Bellman equation gives the soft version. It differs from DQN in exactly one place: the target has one extra term, -α log π(a'|s').

Q(s,a) = r + γ·Es'[ Q(s',a') - α log π(a'|s') ]

Why this holdsImplementation point: a' is sampled from the current policy at s' (not argmax), and log π is also computed with the current policy. So the target moves with the policy - which is why SAC must use a target network (polyak 0.005) to stabilize it.

4. Reparameterization: let gradients pass through the sampled point

The policy outputs μ and σ, and the action is obtained via a = μ + σ·ξ. Moving the randomness onto a ξ that is independent of the parameters turns sampling into a deterministic function, and gradients can backpropagate directly.

a = tanh( μ_θ(s) + σ_θ(s) ⊙ ξ ), ξ ~ N(0, I)

Why this holdsDDPG sidesteps this problem with a deterministic policy (no sampling, differentiate a directly), at the cost of exploration relying on externally added noise. SAC uses reparameterization, so exploration is part of the policy itself - that is its advantage.

5. tanh's log-det correction (skip it and you get 'fake high entropy')

A change of variables must be multiplied by the Jacobian determinant. tanh maps u to a, so the probability density must be divided by |da/du|.

log π(a|s) = log μ(u|s) - ∑_i log( 1 - tanh²(u_i) )

Why this holdsThe consequence of omitting this term is sneaky: log π gets computed too large (a positive number is subtracted one time too few), the policy looks 'very high entropy', the automatic α tuning then pushes the temperature down to nearly 0, exploration disappears, and the final symptom is 'the policy determinizes halfway through training and the success rate plateaus'. This is the single most common bug in hand-rolled SAC implementations, bar none.

6. Automatic α tuning: turn the most sensitive hyperparameter into an adaptive one

Hand-tuning α is hard (it is extremely sensitive to the reward scale). SAC simply does gradient descent on α, with the goal of making the policy entropy equal a preset value.

J(α) = E[ -α( log π(a|s) + H_target ) ], H_target = -dim(A)

Why this holdsThe logic: entropy below target → the parenthesis is positive → α gets pushed up → the entropy term's weight rises → the policy becomes more stochastic. Taking H_target=-dim(A) means 'every action dimension keeps an entropy close to that of a unit Gaussian', independent of the action scale; if the action dimension is 6, the target entropy is -6

7. Double Q: push the positive bias of max back down

Article 3 derived the overestimation of max (measured +1.54 with σ=1 and 10 actions). SAC uses two independent critics and takes the smaller one for the target.

y = r + γ( min(Q1'(s',a'), Q2'(s',a')) - α log π(a'|s') )

Why this holdsWhy taking the min works: the noise of the two estimates is independent, so taking the min amounts to 'it only counts as overestimated if both overestimate', and that probability drops sharply. The cost is a possible slight negative bias, but in engineering that is far better than a positive bias (a positive bias gets amplified by positive feedback).

Work it by hand

Action dimension 2, the policy gives μ=0, σ=0.5, and ξ=1 is sampled (both dimensions take 1). Reward r=0.5, γ=0.99, the two critics give Q1=0.8 and Q2=0.6 at the next state, α=0.2. Find a, log π(a|s), and the soft target y.

  1. u = μ + σ·ξ = 0 + 0.5×1 = 0.5; a = tanh(0.5) = 0.4621
  2. log μ(u) = -0.5·[ (0.5-0)²/0.25 + 2·ln0.5 + ln(2π) ] = -0.7258
  3. correction = -log(1 - tanh²(0.5)) = -log(1-0.2135) = 0.2402, so log π(a) = -0.7258 + 0.2402 = -0.4856
  4. min(Q1,Q2) = 0.6; y = 0.5 + 0.99×(0.6 - 0.2×(-0.4856)) = 0.5 + 0.99×0.6971 = 1.1901
Answer

a=0.4621, log π=-0.4856, y=1.1901. Note the sign in the last expression: -α log π = +0.0971, because log π is negative. This 'subtracting a negative equals adding' spot is where sign errors are most frequent - writing +α log π makes the target systematically too low, and the policy grows ever more conservative.

Run it

25 lines, computing every step above and verifying how much log π is off when 'the log-det correction is skipped'.

python · numpy only
import numpy as np
mu, log_std, xi = 0.0, np.log(0.5), 1.0
u = mu + np.exp(log_std) * xi
a = np.tanh(u)
log_mu = -0.5 * ((u - mu)**2 / np.exp(2*log_std) + 2*log_std + np.log(2*np.pi))
corr = np.log(1 - np.tanh(u)**2 + 1e-7)
log_pi = log_mu - corr
print('u=%.4f a=%.4f log_mu=%.4f correction=%.4f log_pi=%.4f' % (u, a, log_mu, -corr, log_pi))
print('if the correction is skipped, log_pi would be computed as %.4f (off by %.4f, entropy overestimated by %.4f)' % (log_mu, log_pi-log_mu, log_mu-log_pi))
r, g, q1, q2, alpha = 0.5, 0.99, 0.8, 0.6, 0.2
y = r + g * (min(q1, q2) - alpha * log_pi)
print('soft target y = %.4f' % y)
print('sign check: -alpha*log_pi = %+.4f (should be positive)' % (-alpha*log_pi))

You pass when you see this

Output: u=0.5000 a=0.4621 log_mu=-0.7258 correction=0.2402 log_pi=-0.4856, y=1.1901, and the last line shows +0.0971. If log_pi is not negative enough relative to log_mu (say the gap is only 0.05), the tanh derivative term is wrong.

Self-test · 4 questions

Answer first, then expand. Each question traces back to a specific derivation section.

01Why doesn't SAC need exploration noise like ε-greedy?

Because the max-entropy objective itself requires the policy to stay uncertain, and automatic α tuning maintains the target entropy. Exploration is part of the optimization objective, not a heuristic bolted on.

02Under what conditions does automatic α tuning fail?

① When the reward scale is changed (say rewards are multiplied by 100), α gets pushed extremely large early on and learning is very slow; ② when the action dimension is high, the target entropy -dim(A) may be too large and the policy stays too stochastic forever. The former is solved by reward normalization, the latter by manually lowering H_target.

03SAC's critic is only used to compute the target - is argmax still needed?

No. That is the key difference from DQN: the actor differentiates Q directly (through reparameterization), with no max over the whole action space. So SAC is naturally suited to continuous actions and doesn't need ε-greedy.

04Mid-training the actor loss suddenly becomes a large negative number and the policy entropy explodes. What could be the cause?

Q values exploding (the critic broke) → the actor chases the inflated Q. Check the reward scale, whether γ for the target is too large, and whether double Q is used. An entropy explosion is usually the consequence, not the cause, of Q magnitudes running out of control.

Keep reading

Previous article: PPONext article: HERHands-on: actually training SB3 SAC
PreviousPPO: what does clip actually clip NextHER: why goal relabeling is mathematically legitimate

The whole series · 11 articles

01The Only Four Math Tools You Need40 min02MDP and Return: Writing 'What To Do' as Something Computable45 min03The Bellman Equation: Why V Is a Fixed Point and Why Iteration Must Converge60 min04Model-Free: The Bias-Variance Ledger of MC vs TD60 min05Function approximation: the three ingredients that take you from 'guaranteed convergence' to 'blowing up'55 min06Policy gradient: deriving REINFORCE line by line70 min07Actor–critic and GAE: turning the variance knob to λ60 min08PPO: what does clip actually clip70 min09SAC: entropy, reparameterization, double Q - three parts, each solving one problem75 min10HER: why goal relabeling is mathematically legitimate50 min11How to debug non-convergence: map symptoms back to formulas45 min