RobotGuide
Frontier/Direction 1

Frontier · Direction 1 · 18 min

Continual Reinforcement Learning: Learn the New Without Forgetting the Old

When an agent learns task A and then task B, the shared parameters get dragged away by the new data and A's capability collapses — catastrophic forgetting. How to avoid forgetting on the world model track.

18 min3 paper anchors4 math touchpoints3 pitfalls

What it really is, once the buzzwords are off

The parameter-efficient fine-tuning the original post mentions is not the mainstream solution. There are two real solutions: shallow online models refit from scratch on current data each time, so forgetting does not exist by mathematical construction; or mixture-of-experts style task conditioning plus generative replay, letting the old model "dream" replays of old data.

The original post

The original post claims: «Build incrementally updatable world model architectures, combining parameter-efficient fine-tuning and knowledge reuse techniques to solve catastrophic forgetting across scenarios.»

Post vs reality

"Parameter-efficient fine-tuning" is boilerplate imported from NLP — methods like LoRA and Adapter are secondary tools in the world model continual learning literature. The ICML 2025 Spotlight answer is to pull the rug out: don't make deep world models carry continual learning; instead use a shallow Follow-The-Leader online model plus MPC planning. The model refits from scratch on current data at every step, so old-task capability is never stored in parameters and there is simply nothing to forget — the paper also proves a regret bound of O(√(K²D·log T)).

To keep the expressiveness of deep models, the mainstream is another route: task-conditioned mixture dynamics (e.g., Gaussian mixtures learning task-specific latent dynamics) plus generative experience replay — letting the old world model "dream" samples following the distribution of old data, using the dreamed samples to keep representations from being dragged away. The Life-Long World Model from ECML PKDD 2025 beat the direct combination of "continual learning tricks × visual RL" on DMC and Meta-World.

There is also a frequently ignored problem: beyond forgetting, networks also stiffen — Loss of Plasticity (CoLLAs 2023) points out that networks stop learning in continual deep RL. Forgetting and plasticity are the two ends of one lever; there is no joint optimum.

Real paper anchors

Continual RL by Planning with Online World ModelsarXiv 2507.09177 · ICML 2025 Spotlight

Shallow FTL online model + MPC, immune to forgetting by construction, with a regret bound; ships with the Continual Bench environment — before it, this field didn't even have a unified benchmark.

Continual Visual RL with A Life-Long World ModelECML PKDD 2025 · Shanghai Jiao Tong University

Gaussian-mixture task-conditioned latent dynamics + generative experience replay + exploration-conservative value estimation, continual learning for visual continuous control.

Loss of Plasticity in Continual Deep RLCoLLAs 2023

Background problem: in continual learning the network loses plasticity — a second failure mode besides forgetting.

Math you need

Concept hereWhere it landsWhere to learn it
The mechanism of forgettingThree elements of function approximation: parameter sharing, data distribution, target drift/fund/approx/ →
Why online refitting can keep upConvergence and fixed points/math/fixed/ →
How to define old-task retentionTime average vs. ensemble average/math/markov/ →
The accounting of value functions in non-stationary environmentsBias and variance of MC and TD/fund/tabular/ →

A first move you can actually run

How

No need to touch deep models. Build a 5×5 grid world with tasks A→B presented sequentially: run Q-learning with a shared two-layer MLP once and measure how much A's success rate drops after learning B; then mix 25% of A's old samples into B's training stream and run a second time. Doable in a day, and you get hands-on contact with the forgetting-plasticity lever.

Result on recordAlready run on-site: expB_forgetting.py. When goal cells are adjacent, naive sequential training has forgetting of 1.00, the replay version has forgetting of 0.00, and the independent tabular control group is naturally forget-free — three conditions forming a complete contrast.

Pitfalls

Keep reading

Full numbers for Experiment BThe mathematical roots of forgettingThe rigorous version of time averaging
Next · Direction 2Open-Environment Adaptation: Pre-train a Worldview First, Patch Tasks Online Later

All 6 directions

1Continual Reinforcement Learning: Learn the New Without Forgetting the Old18 min2Open-Environment Adaptation: Pre-train a Worldview First, Patch Tasks Online Later20 min3Uncertainty Quantification: Let the Model Know What It Doesn't Know16 min4Efficient Inference: Planning Pays Compute for Return18 min5Hierarchical Control: Macro-actions Save 480× the Imagined Compute18 min6Sim-to-Real: Pixel Gap Is the Appearance, Physics Gap Is the Essence18 min