RobotGuide
Frontier/Starter experiments

Frontier · experiments · about 30 min to read

Four minimal experiments, really run

Entry-level actions accompanying four of the directions, all actually run to numbers: pure numpy (no PyTorch), fixed seeds, one experiment in a few dozen lines, about 7 minutes of total runtime. Toy scale; the numbers exist only to make the mechanisms visible, not for comparison with papers. Each experiment's code is a standalone script, reproducible given an interpreter path.

Run through these four experiments and you will have touched by hand the core mechanisms of directions 1, 3, 4, and 5 of the six. The entry experiments for directions 2 (open environment) and 6 (sim2real) presuppose much larger frameworks; the site provides alternative paths.

Experiment A (Direction 3): Ensemble Uncertainty — and a Reversal Finding

Setup — 2D smooth nonlinear system + σ=0.05 Gaussian noise; 5 independent MLPs (same structure, different seeds) trained on the same data; on 200 fixed test inputs, measure how the inter-member standard deviation of predictions (disagreement) changes with data size.

n_traindisagreementmean-prediction test MSEtheoretical noise floor σ
250.05310.01520.05
1000.01610.00330.05
4000.01000.00260.05
16000.01000.00240.05
160000.00800.00240.05

What the numbers say

The reversal finding: the common claim is that "disagreement converges to the noise floor"; the actual run shows it converges to 0.008 — far below σ=0.05. Reason: the five MSE-trained members are all fitting the same conditional mean E[y|x], so of course they agree with each other. Evidence: the MSE of the ensemble mean converges to 0.0024 ≈ σ²=0.0025 (the conditional mean is fitted well), yet the members show almost no disagreement. MSE ensembles cannot see aleatoric noise at all.

The added contrast: give the network a probabilistic head (outputting μ and log σ², with a Gaussian NLL loss); at n=400 the mean predicted σ is 0.0477 (truth 0.05), range 0.040-0.063. The probabilistic head explicitly supplies the aleatoric noise, complementing the ensemble disagreement — this is exactly where PETS's two words "probabilistic ensemble" come from.

Experiment B (Direction 1): Catastrophic Forgetting — from 1.00 to 0.00 by Moving One Goal Cell

Setup — 5×5 grid world, task A goal (0,4), task B goal (0,3) — adjacent cells, with highly conflicting optimal policies. Shared MLP Q-function (position onehot + goal onehot, hidden layer 16), DQN-style training, 300 episodes per task.

conditionA after training AB after training BA after training Bforgetting
naive sequential training (buffer discarded)1.001.000.001.00
replay (mixing 25% old A samples)1.001.001.000.00
tabular control group (no shared parameters)1.001.001.000.00

What the numbers say

Move the goal cell by one cell and the old task's success rate goes 1.00→0.00; keeping 25% of the old samples in the new task's training stream preserves it completely. The root of forgetting is the parameters of shared function approximation being dragged away by new data, not the RL algorithm itself — the tabular control group proves that no sharing means no forgetting.

Paper-level methods are all scaled-up versions of the same principle: experience replay (store old samples), generative replay (the old world model dreams up samples), task conditioning, or no sharing at all (shallow online refitting).

Experiment C (Direction 4): CEM Search Budget — the Return Curve and the Collapse Point

Setup — Pendulum swing-up (θ=0 hanging, θ=π inverted, torque limit 2.0 smaller than the gravity term, so pumping is required). The world model = an MLP learned from 50K random interactions (single-step RMSE: Δθ 0.058 rad, ω 0.45 rad/s); CEM plans entirely inside the model's imagination, and results are always accepted on the ground-truth dynamics. Flat CEM horizon: 30 steps.

candidates C × iterations Isuccess ratemean returnimagined forward passes per step
16 × 10.001.1314,400
16 × 30.001.6643,200
64 × 30.003.60172,800
256 × 30.003.66691,200
256 × 60.054.341,382,400

What the numbers say

Return rises monotonically with budget (1.13→4.34), but success rate appears only at the maximum budget tier — what "efficient inference" earns is its position on the budget-return curve, and the collapse point exists definitively. TD-MPC2's shortcut planning and FOCUS's sparsification both earn a shift of this entire curve to the left.

Debugging trap 1: the ±π jump of angle wrap is mathematically unlearnable — predicting θ' directly gives a model RMSE of 0.51 rad; predicting continuous Δθ and then wrapping works fine. The discontinuities of angles and contact in real-robot world models are the isomorphic problem.

Debugging trap 2: without receding-horizon warm start, success rate is 0 at every budget — elite averaging averages out-of-phase pump-swing candidates (swing left first, then right) into zero torque. Warm start is a necessity, not an optimization.

Experiment D (Direction 5): Macro-action Hierarchy — the Same Return with 480× Less Imagined Compute

Setup — The same pendulum, the same world model. The high-level CEM plans only over 5 macro-actions (constant torques -2/-1/0/1/2, each lasting 5 steps), horizon of 6 macros = 30 steps; the search space is compressed from 30-dimensional continuous to 6-dimensional discrete.

candidates C × iterations Isuccess ratemean returnimagined forward passes per step
8 × 10.002.20240
8 × 30.002.56720
16 × 30.053.211,440
32 × 30.154.122,880

What the numbers say

The core contrast: macro-action 32×3 achieves return 4.12 with 2,880 imagined forward passes ≈ flat 256×6's 4.34 with 1,382,400 passes — about 480× less imagined compute for the same return. Moreover flat CEM's success rate is still 0.00 at 240× the budget (256×3, 691,200 passes) while macro-actions achieve 0.15: temporal abstraction buys not only compute but also the difficulty of the search space itself.

The model-bias lesson screening alongside: no experiment's success rate rises (≤0.15), and the root cause is not insufficient search — the model still has 0.058 rad single-step Δθ error, and a 30-step imagined rollout diverges by compounding; the perfect plan in imagination goes off the rails on the ground-truth dynamics. Model bias cashes out directly as success or failure — this is why Direction 3 (quantify it) and Direction 6 (fix it) exist.

What the four experiments say together

The combined ledger

Experiment A: the model should "know what it doesn't know" — but MSE ensembles see only epistemic; aleatoric requires a probabilistic head.

Experiment B: the pathology of continual learning = shared parameters dragged away — one dose of replay cures it (toy version); paper-level solutions are the same principle.

Experiment C: planning pays compute for return — the budget-return curve has a collapse point; efficient inference = shifting the curve left.

Experiment D: temporal abstraction is the sharpest instrument for shifting the curve left — a 480× compute difference.

Put together: deploying world models = learn accurately (A) + remember (B) + afford the search (C) + search cleverly (D) — corresponding exactly to directions 3, 1, 4, 5 of the six; directions 2 and 6 are their data-side prerequisites.

Keep reading

The six directions Direction 3: uncertainty quantification Direction 1: continual reinforcement learning