RobotGuide
Frontier/Direction 5

Frontier · Direction 5 · 18 min

Hierarchical Control: Macro-actions Save 480× the Imagined Compute

On long-horizon tasks, let a slow and abstract high level emit subgoals/macro-actions and a fast and concrete low level emit raw actions, with the two layers aligned in latent space. The origin is Sutton 1999's Options framework; the world model version = the high level plans macro-action sequences in imagination.

18 min4 paper anchors4 math touchpoints3 pitfalls

What it really is, once the buzzwords are off

The direction is real and is the current mainstream structure, but the "co-training" the original post mentions is precisely the hardest part — much of the literature chooses to freeze one layer and train only the other, because jointly training both layers is extremely unstable. The original post stated the hardest step as the default option.

The original post

The original post claims: «Build co-training mechanisms between high-level planners and low-level controllers to precisely decompose long-horizon complex tasks and execute them step by step.»

Post vs reality

The clearest entry-level example is Shepherd-Dreamer: an upper DreamerV3 world model plus CEM plans over 12 discrete macro-actions, the lower layer is pure geometric primitives (no learning), each macro-action lasts 15 low-level steps. A herding task with hundreds of steps of horizon and sparse reward converges in 2.5 hours on an RTX 4090, with the architecture diagram, training configuration, and CEM pseudocode all public.

HWM in 2026 settled this layer's theoretical account: long-horizon planning has two fundamental difficulties — prediction error compounds over the rollout, and the search tree explodes exponentially with the horizon. Its solution is not to learn a hierarchical policy but to do hierarchical planning: two world models share the same latent space, the high level predicts long-jump transitions under macro-actions (an action encoder compresses low-level action sequences into latent vectors), and subgoals convert directly between the two layers. This fixes the single-layer planner's weakness of "greedy on tasks that reward greed, failing on tasks that require temporarily moving away from the goal".

Industry-side isomorphs: V-JEPA 2-AC's "AC emits actions, world model emits imagination", and Cosmos-Reason's "reasoning layer emits decisions, execution layer emits actions" are both generalized hierarchy. The origin is Options (Sutton 1999) and feudal networks.

Real paper anchors

Shepherd-Dreamer2025 · open source

DreamerV3 + CEM planning over macro-actions, lower layer of fixed primitives; the clearest open-source entry example, converging in 2.5 hours on an RTX 4090.

Hierarchical Planning with Latent World Models (HWM)arXiv 2604.03208 · 2026

Dual-timescale world model + learned macro-action encoding; not learning a hierarchical policy but doing hierarchical planning, with subgoals converting directly between layers.

Training Agents inside of Scalable World ModelsHafner · arXiv 2509.24527 · 2025

The official roadmap of the Dreamer line; hierarchy is an explicit direction.

Options frameworkSutton, Precup & Singh · 1999

The origin of temporal abstraction: the trio of macro-actions, internal policies, and termination conditions.

Math you need

Concept hereWhere it landsWhere to learn it
What macro-actions changeThe MDP's temporal granularity coarsens: semi-Markov decision processes/math/mdp/ →
How to settle accounts when the two layers each own a horizonThe two kinds of dynamic programming iteration/math/dp/ →
The training mechanism of the low-level policyActor-Critic and GAE/fund/ac/ →
How the low level learns when the high level gives no rewardGoal relabeling with HER/fund/her/ →

A first move you can actually run

How

Replicate Shepherd-Dreamer's structure (macro-action primitives can be swapped for your own): in MiniGrid, set up "high level may only emit 4 subgoals + low level executes 10 steps", and compare the flat policy's success rate and sample efficiency on a 50-step task. Watch closely whether the high level converges on its own after the lower layer is frozen, and what happens when both layers are released at once.

Result on recordAlready run on-site: expCD_plan.py, Experiment D. 5 macros (constant torques -2/-1/0/1/2, each lasting 5 steps) × high-level horizon of 6 macros: macro-action CEM reaches return 4.12 with 2,880 imagined forward passes, while flat CEM needs 1,382,400 passes for 4.34.

Pitfalls

Keep reading

Experiment D: the full table of the 480× gapThe mathematical language of temporal abstractionThe other side of the same coin: efficient inference
Previous · Direction 4Efficient Inference: Planning Pays Compute for Return Next · Direction 6Sim-to-Real: Pixel Gap Is the Appearance, Physics Gap Is the Essence

All 6 directions

1Continual Reinforcement Learning: Learn the New Without Forgetting the Old18 min2Open-Environment Adaptation: Pre-train a Worldview First, Patch Tasks Online Later20 min3Uncertainty Quantification: Let the Model Know What It Doesn't Know16 min4Efficient Inference: Planning Pays Compute for Return18 min5Hierarchical Control: Macro-actions Save 480× the Imagined Compute18 min6Sim-to-Real: Pixel Gap Is the Appearance, Physics Gap Is the Essence18 min