RobotGuide
Foundations/How to debug non-convergence: map symptoms back to formulas

Foundations 11 / 11 · Diagnostics

How to debug non-convergence: map symptoms back to formulas

Eleven common curve shapes mapped to the specific term in the formula. Plus three pre-training health checks and two controlled experiments you must run

45 minCapstone8 derivation steps4 self-test questions

One-line conclusion

Training non-convergence is not mysticism. Every curve shape corresponds to one specific term in a formula - map the symptom to the term and you can localize it, no need to gamble on grid search. This article is the index to the previous ten; each entry says which article's derivation to go back to.

The previous ten articles were 'why'; this one is 'what to do'. On real hardware 90% of your time goes here, and most people's approach is to change the learning rate and rerun - that's replacing diagnosis with random search.

Symbol table

SymbolMeaning
health checkthree checks that can all be done before training starts
controlled experimentchange one variable and see whether the difference exceeds seed noise
seed noisethe metric variation from rerunning the same code with a different random seed, typically 5~20 points

Derivation

  1. Three health checks before training (skip them and none of the later curves are trustworthy)
  2. Jagged curves, loss bouncing up and down
  3. Q rising monotonically without falling back, success rate flat
  4. Success rate stuck at 0, loss dropping to 0 fast
  5. Policy entropy collapsing to zero fast, actions determinizing
  6. Good training metrics, poor evaluation (the generalization gap)
  7. Two controlled experiments you must run
  8. A quick-reference table (matching symptoms to article numbers)

1. Three health checks before training (skip them and none of the later curves are trustworthy)

All three finish in a few minutes and eliminate half the possible causes of failure in advance.

① can 1 batch be overfit to loss≈0 ② Q value range ③ policy entropy

Why this holds① Train repeatedly on 1 batch for 200 steps; loss must drop to near 0; failing that → model capacity, learning rate, or label construction is wrong, unrelated to the algorithm. ② Q should stay between 0 and R_max/(1-γ) (article 2). Reaching 1e4 → wrong reward scale or target not synced. ③ Entropy should decline slowly, not hit 0 in one step (α from article 9).

2. Jagged curves, loss bouncing up and down

This is a gradient signal-to-noise problem, not an algorithm problem. Three common sources, ordered by probability.

jaggedness ↑ → batch too small / learning rate too large / advantage not normalized

Why this holdsFirst check advantage normalization (article 7): the advantage scale can differ several-fold across batches, and skipping normalization means the learning rate fluctuates randomly. Then look at the batch: with PPO's minibatch below 64, jaggedness is almost guaranteed. The learning rate comes last.

3. Q rising monotonically without falling back, success rate flat

Eight out of nine cases are a reward-scale or target problem. Because the only source of Q's positive feedback is bootstrapping.

Q ↑↑ → reward given every step / γ too close to 1 / target network not synced

Why this holdsHow to tell: print the mean Q and the mean target for one batch. If both rise together and far exceed R_max/(1-γ) → wrong reward scale. If the target exceeds Q by a fixed amount → the reward sign or an extra term (e.g. the entropy term's sign is wrong, see article 9's worked example).

4. Success rate stuck at 0, loss dropping to 0 fast

This is the classic signature of 'zero signal', not 'learning slowly'. Adding training steps does nothing.

loss→0 and reward identically 0 → sparse reward with no effective HER / tolerance too small / actions clipped into an ineffective region

Why this holdsCheck in order: is the reward given only at termination (yes → correct); is HER on and is ε larger than the per-step displacement (article 10 - the most commonly botched spot); are actions clipped to [-1,1] while the environment expects a different range (symptom: action saturation, trajectories repeating exactly).

5. Policy entropy collapsing to zero fast, actions determinizing

Determinization itself isn't the problem; premature determinization is. It means 'exploration stopped before a solution was found'.

entropy ↓↓ → α too small / entropy coefficient too small / log π computed wrong (missing the tanh correction fakes high entropy)

Why this holdsThe pit from article 9, section 5: missing the log-det correction makes log π too large, the automatic α tuning mistakenly believes 'it's already random enough', and pushes α to 0 - how to check: print |log π|; it should be on the order of the action dimension (e.g. 6 dims → around 6); clearly too small means the correction is missing.

6. Good training metrics, poor evaluation (the generalization gap)

This is a distribution mismatch, not overfitting. The word 'overfitting' is frequently misused in RL.

good training poor evaluation → observation normalization statistics inconsistent / not enough domain randomization / different initial state distribution at evaluation

Why this holdsThe sneakiest one: training normalizes observations with running mean/std, but at evaluation you forget to load the same statistics (or use fresh running statistics). Symptom: the policy's outputs at evaluation are complete nonsense and the code raises no error. Second most common: evaluation uses fixed initial states while training uses random ones.

7. Two controlled experiments you must run

They answer two different questions; missing either one leads to a misdiagnosis.

① turn off randomization and see if it can still learn ② fix the seed and repeat 3 times to see variance

Why this holds① separates 'task too hard' from 'algorithm wrong': if it still can't learn with domain randomization off → the algorithm/reward is broken; stop tweaking the randomization magnitude. ② separates 'real improvement' from 'seed luck': RL's seed variance is often 10~20 points, larger than your change's effect. An improvement only counts if all three seeds improve.

8. A quick-reference table (matching symptoms to article numbers)

Below is the index to the first ten articles of this series. When debugging, read the symptom first, then go back to that article's derivation.

Q overestimation → article 3; high variance → articles 4/7; policy doesn't move → article 6; updates too violent → article 8; exploration vanishes → article 9; zero signal → article 10

Why this holdsOne final piece of experience: change one variable, run three seeds, record the baseline. The most expensive thing in RL experiments isn't compute - it's 'you thought you changed something but didn't' and 'you changed two variables and don't know whose credit it is'.

Work it by hand

A grasping task: Q values sit stable around 0.3, loss drops fast from 0.8 to 0.001 and the success rate is 0, and inspection shows the reward is 'distance under 5cm gives 1, otherwise 0', HER is on, ε=0.01, mean per-step displacement 4cm. Diagnose and give the fix.

  1. loss drops to 0 fast and Q is constant → classic zero signal, not learning slowly (article 10, section 1)
  2. ε=0.01 is far smaller than the per-step displacement 0.04 → future relabeling produces almost no positive samples (article 10, section 4)
  3. Only the j=i diagonal samples remain usable, and they teach 'stay put' → the policy doesn't move, success rate 0
Answer

Fix: raise ε to 0.04~0.08 (1~2 times the per-step displacement), confirm her_ratio is around 0.8 to keep true goals, and rerun with three seeds. Secondary fix: raise the control frequency to shrink the per-step displacement - more fundamental than widening the tolerance, but it lengthens the episode.

Run it

A 20-line 'pre-training health check' script skeleton. It contains no training logic; it just pins down what you should be printing - this step saves hours later.

python · numpy only
import numpy as np

def health_check(q_batch, rew_batch, logp_batch, gamma, r_max, act_dim):
    q = np.asarray(q_batch); r = np.asarray(rew_batch)
    bound = r_max / (1 - gamma)
    print('Q mean %.3f  min %.3f  max %.3f' % (q.mean(), q.min(), q.max()))
    print('theoretical upper bound R_max/(1-gamma) = %.2f' % bound)
    if q.max() > 3 * bound:
        print('  [FAIL] Q far exceeds the bound: check whether reward is given every step / whether target is synced')
    print('nonzero reward ratio %.3f' % (r != 0).mean())
    if (r != 0).mean() < 0.01:
        print('  [WARN] rewards extremely sparse: confirm HER is on and eps exceeds the per-step displacement')
    lp = np.asarray(logp_batch)
    print('log pi mean %.3f  expected magnitude about -%d' % (lp.mean(), act_dim))
    if abs(lp.mean()) < 0.3 * act_dim:
        print('  [FAIL] log pi too small: possibly missing tanh log-det correction')
    print('done')

health_check([0.3, 0.31, 0.29], [0, 0, 0], [-0.02, -0.03, -0.01], 0.99, 1.0, 2)

You pass when you see this

The three output lines will trigger exactly the 'rewards extremely sparse' and 'log pi too small' warnings - precisely the health-check result for the task in the worked example above. Getting this function to run on your own project beats running ten training jobs.

Self-test · 4 questions

Answer first, then expand. Each question traces back to a specific derivation section.

01Why does the 'overfit 1 batch' check alone eliminate half the failure causes?

Because it separates 'can it learn' from 'should it learn'. If even 3 samples can't be fit, the problem must be in the model, loss, or data pipeline - not the algorithm, hyperparameters, or environment.

02Training with three seeds gives 40%, 55%, 35%. You make one change and get 50%. Does that count as an improvement?

No. The difference is inside the seed-noise range (±10 points). Use at least three seeds and look at the mean and the minimum, or directly increase the number of evaluation episodes to reduce noise.

03Why is 'good on training, poor on evaluation' usually not overfitting in RL?

Because RL samples are not i.i.d., and the case where model capacity is far smaller than the data volume is rare. The more common causes are inconsistent observation distributions, normalization statistics, or initial state distributions between training and evaluation.

04What is knowing Q's theoretical upper bound good for?

It is the only hard indicator that can tell in real time during training whether 'the critic is already wrong'. Curves can't show it, but once this number crosses the bound by 3x, it must be a reward-scale or target problem - no need to look further.

Keep reading

Previous article: HERBack to the first article: four mathematical toolsDrills: 24 debugging problems
PreviousHER: why goal relabeling is mathematically legitimate

The whole series · 11 articles

01The Only Four Math Tools You Need40 min02MDP and Return: Writing 'What To Do' as Something Computable45 min03The Bellman Equation: Why V Is a Fixed Point and Why Iteration Must Converge60 min04Model-Free: The Bias-Variance Ledger of MC vs TD60 min05Function approximation: the three ingredients that take you from 'guaranteed convergence' to 'blowing up'55 min06Policy gradient: deriving REINFORCE line by line70 min07Actor–critic and GAE: turning the variance knob to λ60 min08PPO: what does clip actually clip70 min09SAC: entropy, reparameterization, double Q - three parts, each solving one problem75 min10HER: why goal relabeling is mathematically legitimate50 min11How to debug non-convergence: map symptoms back to formulas45 min