RobotGuide
Foundations/HER: why goal relabeling is mathematically legitimate

Foundations 10 / 11 · Sparse Reward

HER: why goal relabeling is mathematically legitimate

Change the goal without changing the dynamics - that is the entire argument. Plus the quantitative condition almost nobody mentions: 'the tolerance must exceed the per-step displacement'

50 minAdvanced6 derivation steps4 self-test questions

One-line conclusion

Under sparse reward, Q is identically 0 and so is the gradient - not 'learning slowly' but 'zero signal'. HER's fix is to replace the goal of failed trajectories with the place they actually reached. This is mathematically legitimate because the goal changes no transition probability - only the reward.

Tasks like grasping, assembly, and door opening are almost all sparse-reward, and HER is the switch that turns them from 'untrainable' into 'trainable'. But the reasons it fails are very specific; without reading this article it's hard to tune it right.

Symbol table

SymbolMeaning
gthe goal (desired goal), an extra input to the policy
φ(s)the achieved goal extracted from the state, e.g. gripper coordinates
εthe tolerance for the success check
r(s,a,g)the goal-conditioned reward, usually 1[||φ(s')-g||≤ε]

Derivation

  1. The mathematical difficulty of sparse reward: the signal is zero, not weak
  2. The goal-conditioned MDP: put the goal into the observation
  3. The legitimacy proof (three lines)
  4. The future strategy and a quantitative condition almost nobody mentions
  5. The j=i degeneracy: why you can't use relabeling exclusively
  6. Three disciplines of implementation

1. The mathematical difficulty of sparse reward: the signal is zero, not weak

The reward is an indicator function that is 0 or 1. If a trajectory never got near the goal, every reward is 0, and so every regression target for Q is 0.

r(s,a,g) = 1[ ||φ(s') - g|| ≤ ε ], ≡ 0 when unsuccessful

Why this holdsConsequence: Q converges to identically 0, and the actor's gradient, ∇Q times 0, is also 0 - there is no 'faint gradient' anywhere in the process to climb. So the symptom is loss dropping to 0 quickly while the success rate doesn't budge - that is a different thing from 'learning slowly'; don't just add training steps there.

2. The goal-conditioned MDP: put the goal into the observation

Make g part of the policy input: π(a|s,g). The same transition is a failure under g1 and a success under g2.

M_g = (S, A, P, R_g, γ, G), π(a|s,g)

Why this holdsWatch the subscripts on P and R_g: the transition probability carries no g, the reward does. That one-subscript difference is the entire source of HER's legitimacy - it only permits you to swap the subscript on R.

3. The legitimacy proof (three lines)

What relabeling does: take an already-sampled transition (s,a,s') and change its goal from g to g', recomputing the reward for the new goal. What must be shown is that this triple is still a legal sample from the same MDP.

P(s'|s,a) is independent of g ⟹ swapping g does not change the transition ⟹ only r needs recomputing, and the sample stays legal

Why this holdsConversely, if relabeling also changes s' (e.g. 'assume it reached the goal'), you have fabricated a transition that doesn't exist in the environment - that is wrong, and it makes Q learn nonexistent dynamics. This is the sole criterion for judging whether a HER implementation is correct: did it touch s'.

4. The future strategy and a quantitative condition almost nobody mentions

The future strategy takes an achieved goal from a later step of the same trajectory as the new goal. The intuition: 'I got there later, so treat that place as the goal'. But the success rate depends on the ratio of tolerance to step size.

requires ε ≳ the order of magnitude of the per-step displacement

Why this holdsMeasured: a 4-step trajectory with adjacent-step displacement 0.42. With ε=0.05 or 0.15, the future strategy (j>i) produces 0 positive pairs; with ε=0.35, 1 pair; with ε=0.5, 2 pairs. The implication is direct: if the tolerance is smaller than a single step, 'later steps' can never land inside 'the current step's tolerance ball', and the future strategy degenerates to relying only on j=i. So 'HER doesn't work' is often not an algorithm error - it's ε being an order of magnitude too small.

5. The j=i degeneracy: why you can't use relabeling exclusively

When j=i is allowed, the new goal is the current step's own achieved goal, the distance is 0, and success is guaranteed.

r'(i,i) = 1[ ||φ(s_i) - φ(s_i)|| ≤ ε ] ≡ 1

Why this holdsMeasured: under the original goal, among 4 steps there is only 1 positive sample; after relabeling, out of 10 pairs there are 4 positives, all from j=i. This is equivalent to a self-supervised 'reaching means success' signal that gives Q a nonzero starting point, but using it exclusively teaches the policy to 'stay put and don't move' - because every state's goal is itself. That's why SB3 uses her_ratio=0.8, keeping 20% true goals so the 'true goal' branch always has gradient. More importantly: even with ε tuned right, the future strategy can only rescue the steps 'close to success'; long-range exploration still needs other means.

6. Three disciplines of implementation

Getting the order wrong doesn't raise an error - it just fails silently. Write these three into your code comments directly.

① swap the goal → ② recompute the reward → ③ store back to the buffer

Why this holdsThe most common bug is 'store r first, then swap g': the reward is still computed for the old goal, Q's target and conditioning mismatch, and training turns into noise. Second: reward recomputation must use the same φ and the same ε (don't change the tolerance mid-training). Third: true-goal and relabeled samples must be mixed; the ratio is a hyperparameter, not a constant.

Work it by hand

A 4-step trajectory with achieved goals (0.2,0.1), (0.5,0.4), (0.8,0.9), (0.97,0.98) in order, true goal (1.0,1.0), ε=0.05. List the original rewards and the reward matrix for future relabeling (j≥i), and count the positive-sample rate.

  1. Original: only step 4 is 0.042 < 0.05 from the goal, rewards [0,0,0,1], 1/4 positives
  2. Rows i and columns j≥i of the relabeled matrix: i=0 → j=0:1 (distance 0), j=1..3:0; i=1 → j=1:1, rest 0; i=2 → j=2:1; i=3 → j=3:1
  3. In total, out of 10 pairs there are 4 positives, all landing on the j=i diagonal
Answer

Relabeling turns 1 positive sample into 4, but every increment comes from 'goal = itself'. That's not worthless (it gives Q a nonzero starting point), but it shows this trajectory's future strategy (j>i) produced not a single positive sample - because each step displaces 0.42, far more than the tolerance 0.05. In a real task, either raise ε to the same order of magnitude as the per-step displacement, or raise the control frequency to shrink the per-step displacement.

Run it

25 lines, counting the positive-sample rate and sweeping the tolerance. You can paste this code straight into your replay buffer as a self-check.

python · numpy only
import numpy as np
ach = np.array([[0.2,0.1],[0.5,0.4],[0.8,0.9],[0.97,0.98]])
goal = np.array([1.0,1.0])
def rew(a, g, eps):
    return float(np.linalg.norm(a - g) <= eps)
eps = 0.05
print('original rewards:', [rew(a, goal, eps) for a in ach])
for i in range(4):
    row = [(j, rew(ach[i], ach[j], eps)) for j in range(i, 4)]
    print('t=%d relabeled:' % i, row)
print('--- tolerance sensitivity (future means j>i) ---')
for e in (0.05, 0.15, 0.35, 0.5):
    n_gt = sum(1 for i in range(4) for j in range(i+1,4) if rew(ach[i], ach[j], e))
    n_ge = sum(1 for i in range(4) for j in range(i,4) if rew(ach[i], ach[j], e))
    print('eps=%.2f  j>i:%d pairs   j>=i:%d pairs' % (e, n_gt, n_ge))
print('adjacent-step displacement =', round(float(np.linalg.norm(ach[1]-ach[0])), 2))

You pass when you see this

Original rewards [0,0,0,1]; with tolerance 0.05/0.15, j>i gives 0 pairs in both cases, 1 pair at 0.35, 2 pairs at 0.5; adjacent displacement 0.42. This comparison is the criterion of 'tolerance vs step size': when ε is clearly below 0.42, the future strategy is effectively switched off.

Self-test · 4 questions

Answer first, then expand. Each question traces back to a specific derivation section.

01HER must recompute the reward when relabeling - why can't the old reward be reused?

Because the reward is a goal-conditioned function r(s,a,g). Swapping g without recomputing means training 'the new goal's Q' with 'the old goal's reward'; conditioning and label mismatch, and what gets learned is noise.

02Can relabeling also change next_state to 'assume it reached the goal'?

No. That fabricates a transition that doesn't exist in the environment, and Q learns wrong dynamics (believing some action can teleport to the goal). Legitimacy comes only from 'P is independent of g'; the moment you touch P, it's illegal.

03What goes wrong if ε is set too large?

Many irrelevant samples are judged successful, Q is +1 everywhere, and the policy learns 'wiggle a bit and it counts as success'. Typical symptom: the success-rate metric looks great but the real robot grasps nothing. Rule of thumb: take ε at 1~2 times the per-step displacement, then maybe one notch higher.

04What is the relationship between HER and reward shaping?

HER can be seen as 'goal conditioning + automatic shaping': it uses hindsight to create dense goals, but every goal was genuinely reached, so unlike hand-crafted shaping it introduces no wrong preferences. The cost: it only helps when you are 'close to success'.

Keep reading

Previous article: SACNext article: how to debug non-convergenceHands-on: how to turn on HER in SB3
PreviousSAC: entropy, reparameterization, double Q - three parts, each solving one problem NextHow to debug non-convergence: map symptoms back to formulas

The whole series · 11 articles

01The Only Four Math Tools You Need40 min02MDP and Return: Writing 'What To Do' as Something Computable45 min03The Bellman Equation: Why V Is a Fixed Point and Why Iteration Must Converge60 min04Model-Free: The Bias-Variance Ledger of MC vs TD60 min05Function approximation: the three ingredients that take you from 'guaranteed convergence' to 'blowing up'55 min06Policy gradient: deriving REINFORCE line by line70 min07Actor–critic and GAE: turning the variance knob to λ60 min08PPO: what does clip actually clip70 min09SAC: entropy, reparameterization, double Q - three parts, each solving one problem75 min10HER: why goal relabeling is mathematically legitimate50 min11How to debug non-convergence: map symptoms back to formulas45 min