RobotGuide
Hands-on/Train a grasping policy from scratch: MuJoCo + SAC + HER

Hands-on 3 / 4

Train a grasping policy from scratch: MuJoCo + SAC + HER

First get the arm to reach the target, then learn to pick it up. Training runs with a single command, about 5 hours

5–6 h10 stepsKnows Python and has installed pip packages; RL concepts can be zero

Math: Derivatives and probability (Math bridge T2/T4)Coding: Knows Python and pipHardware: A normal laptop works (slow); GPU is faster

What you will have in hand when done

The same code produces two results: Reach success rate above 95% and PickAndPlace (grasping) success rate above 60%; you can explain what each hyperparameter changes.

The hard part of robot RL is not the algorithm; it is how to set these four things: environment, action space, reward, curriculum. Nobody walks you through these steps, and online tutorials jump straight to formulas. This tutorial does them in order, every step has a verifiable result — if it doesn't work, some step is wrong. No guessing.

Before you start

You needVersion / notes
Python 3.10+Recommend isolating with venv
mujoco + gymnasiumPhysics engine and API
gymnasium-roboticsFetch task suite
stable-baselines3SAC + HER implementation
C6 Reinforcement LearningG6 MuJoCo and training environments

Contents

  1. Install the environment: four lines, no more
  2. Inspect the environment first: no training yet, just make it move
  3. Understand the task: why HER is required
  4. First training run: Reach, 40 minutes to 95%
  5. Read the curves: four shapes of failure
  6. Second run: reshape the reward to keep it 'gentle'
  7. Third run: the grasping task, from 0 to 60%
  8. Replay: prove with code that it really learned
  9. Swap in your own model: plug in an MJCF
  10. Three things to add before going to the real robot
01

Install the environment: four lines, no more

Do not start by installing the full Isaac or CUDA stack. This task produces results on CPU in 40 minutes; installing the GPU stack only adds failure points.

terminal / code
python3 -m venv ~/rl && source ~/rl/bin/activate
pip install --upgrade pip
pip install "mujoco>=3.1" "gymnasium>=1.0" gymnasium-robotics \
  "stable-baselines3>=2.3" tensorboard
python -c "import mujoco, gymnasium, gymnasium_robotics, stable_baselines3 as sb3; print(mujoco.__version__, sb3.__version__)"

You pass when you see this

Prints version numbers, no ImportError.

Stuck?

SymptomMost likely causeWhat to do
Importing `gymnasium_robotics` reports missing `mujoco`Install order or version conflictInstall mujoco first, then gymnasium-robotics; use Python 3.10/3.11
Fetch environment raises `ModuleNotFoundError`New versions require explicit registrationAdd `gym.register_envs(gymnasium_robotics)`; see next step
02

Inspect the environment first: no training yet, just make it move

The first step of any RL project is to run the environment and watch it. Run 200 steps with random actions and observe: what the action space is, which keys the observation has, how reward is given. If you don't understand these, every tuning attempt is blind.

terminal / code
cat > ~/check_env.py <<'EOF'
import gymnasium as gym, gymnasium_robotics, numpy as np

gym.register_envs(gymnasium_robotics)
env = gym.make("FetchReach-v3", render_mode="human")
obs, info = env.reset(seed=0)
print("obs keys:", list(obs.keys()))
print("obs dim:", {k: v.shape for k, v in obs.items()})
print("act dim:", env.action_space.shape, env.action_space.low, env.action_space.high)

for i in range(200):
    obs, r, term, trunc, info = env.step(env.action_space.sample())
    if i % 50 == 0:
        print(i, "reward", round(r, 3), "is_success", info.get("is_success"))
env.close()
EOF
python ~/check_env.py

You pass when you see this

A window pops up and the arm moves randomly; prints the three keys observation/achieved_goal/desired_goal, action dim 4, range ±1.

Stuck?

SymptomMost likely causeWhat to do
No window pops upNo display or GL libraries missingUse `export MUJOCO_GL=egl` or run on a machine with a desktop; you can drop render_mode during training
obs is an array, not a dictUsed an old environment name other than FetchReach-v3Use the `-v3` environment name, and you must call register_envs
03

Understand the task: why HER is required

The reward in grasping tasks is sparse: 1 only at the instant of touching the target, 0 otherwise. Random exploration never hits it, and plain SAC would never learn it. HER (Hindsight Experience Replay) rewrites 'failed experience' into 'achieved a different goal' experience — the standard solution for robot force-control tasks.

terminal / code
# No code to type here, just read this (matches SB3's HerReplayBuffer)
# A collected trajectory: goal G, actually reached G'
# Original reward: 0 (did not reach G)
# HER rewrite: swap the goal to G', and this trajectory becomes a 'success' sample with reward 1
# So every ten thousand failed samples turn into tens of thousands of success samples

You pass when you see this

You can state in your own words: sparse reward + HER = relabeling failures as successes, so the gradient has something to learn from.

Stuck?

SymptomMost likely causeWhat to do
Success rate still 0 after hundreds of epochsNot using HER, or n_sampled_goal too smallIn SB3 you must explicitly pass replay_buffer_class=HerReplayBuffer
04

First training run: Reach, 40 minutes to 95%

Train the simplest task, Reach, first: it only requires the end effector to reach the target point. It converges fast and validates the whole pipeline (environment, HER, logging, saving). If this step fails, do not move on.

terminal / code
cat > ~/train_reach.py <<'EOF'
import gymnasium as gym, gymnasium_robotics
from stable_baselines3 import SAC
from stable_baselines3.her import HerReplayBuffer
from stable_baselines3.common.callbacks import EvalCallback

gym.register_envs(gymnasium_robotics)
env = gym.make("FetchReach-v3", max_episode_steps=50)
eval_env = gym.make("FetchReach-v3", max_episode_steps=50)

model = SAC(
    "MultiInputPolicy", env,
    replay_buffer_class=HerReplayBuffer,
    replay_buffer_kwargs=dict(n_sampled_goal=4, goal_selection_strategy="future"),
    learning_rate=1e-3, buffer_size=1_000_000,
    batch_size=256, gamma=0.95, tau=0.05,
    train_freq=1, gradient_steps=-1,
    verbose=1, tensorboard_log="./tb",
)
model.learn(total_timesteps=200_000,
            callback=EvalCallback(eval_env, n_eval_episodes=20,
                                  eval_freq=10_000, best_model_save_path="./best"))
model.save("sac_reach")
EOF
python ~/train_reach.py
tensorboard --logdir ./tb --port 6006

You pass when you see this

In TensorBoard, eval/success_rate climbs from 0 to above 0.9; the console rollout/ep_rew_mean rises clearly.

Stuck?

SymptomMost likely causeWhat to do
Loss stays large and success rate doesn't risegamma left at the default 0.99 but the episode is only 50 stepsFor grasping tasks use gamma 0.95 to shorten credit assignment
Training slow, 20k steps per hourCPU threads fighting for coresUse `OMP_NUM_THREADS=1 python train_reach.py`; with a single environment it is actually faster
HER reports `not enough transitions`buffer too small or batch too largebuffer_size at least 1e5; with n_sampled_goal=4 the buffer needs 1e4+ to start
05

Read the curves: four shapes of failure

The most expensive part of training is 'not knowing what went wrong'. These four shapes map to four causes; recognize the shape first, then change parameters.

terminal / code
# Shape 1: success rate 0 and ep_len pinned at 50 (full run) -> not enough exploration or reward too sparse
# Shape 2: success rate rises to 0.5 then falls back to 0.1 -> learning rate too large / buffer flushed by new data
# Shape 3: success rate 0.9 but eval only 0.3 -> overfit to the training initial distribution
# Shape 4: loss normal but action saturates at ±1 -> improper action scaling

You pass when you see this

You can point at your own TensorBoard plot, identify which shape it is, and name which parameter to change.

Stuck?

SymptomMost likely causeWhat to do
Curve oscillates wildly up and downbatch_size too small + learning rate too largebatch 256, lr 1e-3 is a stable starting point; don't tune network size first
06

Second run: reshape the reward to keep it 'gentle'

Reach trains a 'lunging' policy: it gets there but jerks. In engineering you must add an action-smoothness penalty, otherwise motor current will trip protection on the real robot. Use a Wrapper to modify the reward, not the environment itself.

terminal / code
cat > ~/wrappers.py <<'EOF'
import gymnasium as gym

class SmoothReward(gym.Wrapper):
    """Add an action-change penalty on top of the raw reward"""
    def __init__(self, env, coef=0.05):
        super().__init__(env)
        self.coef = coef
        self.prev = None

    def reset(self, **kw):
        self.prev = None
        return self.env.reset(**kw)

    def step(self, action):
        obs, r, term, trunc, info = self.env.step(action)
        if self.prev is not None:
            r -= self.coef * float(((action - self.prev) ** 2).sum())
        self.prev = action.copy()
        return obs, r, term, trunc, info
EOF
# Wrap one layer when training: env = SmoothReward(gym.make("FetchReach-v3"), coef=0.05)

You pass when you see this

At the same 200k steps the success rate stays above 0.9, but the recorded actions are visibly smoother (compare with a play script).

Stuck?

SymptomMost likely causeWhat to do
Success rate drops to 0.5coef too large, drowning out the main rewardStart coef at 0.01, double it upward, never exceed 0.1
07

Third run: the grasping task, from 0 to 60%

Switch to FetchPickAndPlace. It requires 'reach -> grasp -> lift -> move to the goal', four sub-skills inside one trajectory. Training directly easily gets stuck at 0; use a curriculum: widen the goal range first, then tighten it.

terminal / code
cat > ~/train_pick.py <<'EOF'
import gymnasium as gym, gymnasium_robotics, numpy as np
from stable_baselines3 import SAC
from stable_baselines3.her import HerReplayBuffer
from stable_baselines3.common.callbacks import EvalCallback

gym.register_envs(gymnasium_robotics)

def make():
    return gym.make("FetchPickAndPlace-v3", max_episode_steps=50)

env, eval_env = make(), make()
model = SAC(
    "MultiInputPolicy", env,
    replay_buffer_class=HerReplayBuffer,
    replay_buffer_kwargs=dict(n_sampled_goal=4, goal_selection_strategy="future"),
    learning_rate=1e-3, buffer_size=1_000_000, batch_size=256,
    gamma=0.95, tau=0.05, train_freq=1, gradient_steps=1,
    policy_kwargs=dict(net_arch=[256, 256, 256]),
    verbose=1, tensorboard_log="./tb_pick",
)
model.learn(total_timesteps=1_500_000,
            callback=EvalCallback(eval_env, n_eval_episodes=20,
                                  eval_freq=25_000, best_model_save_path="./best_pick"))
model.save("sac_pick")
EOF
OMP_NUM_THREADS=1 python ~/train_pick.py

You pass when you see this

After 1.5M steps, eval/success_rate stabilizes above 0.6 (about 8–12 hours on CPU; faster with a GPU, but SAC is fine on CPU too).

Stuck?

SymptomMost likely causeWhat to do
Stays 0 for the first 500k stepsNot enough exploration: action noise too smallSAC defaults to ent_coef='auto'; if it stays 0, temporarily raise target_entropy, or warm up on Reach first and initialize from those weights
Stuck around 0.3 success rateNetwork too small or HER sampling strategy too narrowRaise n_sampled_goal to 8, use future for goal_selection_strategy, widen net_arch to [256,256,256]
08

Replay: prove with code that it really learned

After training you must be able to reproduce it. Load the best model, run 100 episodes, compute the success rate, and record a video. Don't trust the numbers in the training log.

terminal / code
cat > ~/eval_pick.py <<'EOF'
import gymnasium as gym, gymnasium_robotics, numpy as np
from stable_baselines3 import SAC

gym.register_envs(gymnasium_robotics)
env = gym.make("FetchPickAndPlace-v3", max_episode_steps=50, render_mode="rgb_array")
model = SAC.load("./best_pick/best_model", env=env)

succ, frames = 0, []
for ep in range(100):
    obs, _ = env.reset(seed=1000 + ep)
    done = False
    while not done:
        act, _ = model.predict(obs, deterministic=True)
        obs, r, term, trunc, info = env.step(act)
        done = term or trunc
        if ep == 0:
            frames.append(env.render())
    succ += int(info.get("is_success", 0))
print("success rate:", succ / 100)

import imageio
imageio.mimsave("pick.gif", frames, fps=30)
EOF
python ~/eval_pick.py

You pass when you see this

The printed success rate is close to the eval number in the training log (within ±0.1), and pick.gif shows the full pick-and-place motion.

Stuck?

SymptomMost likely causeWhat to do
Eval says 0.6 but the video shows failuresDifferent seed distribution or deterministic behavior differencesCompare stochastic/deterministic success rates on the same batch of seeds; a large gap means the policy is on the boundary
The gif is too largeToo many framesKeep every 2nd frame, or record only the last 50 frames
09

Swap in your own model: plug in an MJCF

Fetch is someone else's model; running it doesn't mean you can do real work. When switching to your own MJCF only three things change: action dims, observation assembly, and how the goal is defined. Keep the reward as 'negative distance + 1 on success'.

terminal / code
cat > ~/my_env.py <<'EOF'
import gymnasium as gym, mujoco, numpy as np

class MyArmEnv(gym.Env):
    def __init__(self, xml="arm6.xml"):
        self.model = mujoco.MjModel.from_xml_path(xml)
        self.data = mujoco.MjData(self.model)
        self.n = self.model.nu                    # action dim = number of actuators
        self.goal = np.zeros(3)
        self.observation_space = gym.spaces.Dict({
            "observation": gym.spaces.Box(-np.inf, np.inf, (self.n * 2 + 3,), np.float32),
            "achieved_goal": gym.spaces.Box(-np.inf, np.inf, (3,), np.float32),
            "desired_goal": gym.spaces.Box(-np.inf, np.inf, (3,), np.float32),
        })
        self.action_space = gym.spaces.Box(-1, 1, (self.n,), np.float32)
        self.site = self.model.site("tool0").id

    def _obs(self):
        q = self.data.qpos[:self.n]; dq = self.data.qvel[:self.n]
        p = self.data.site_xpos[self.site]
        return {"observation": np.concatenate([q, dq, p]).astype(np.float32),
                "achieved_goal": p.astype(np.float32),
                "desired_goal": self.goal.astype(np.float32)}

    def reset(self, seed=None, options=None):
        super().reset(seed=seed)
        mujoco.mj_resetData(self.model, self.data)
        self.goal = self.np_random.uniform(-0.1, 0.1, 3) + np.array([0.4, 0.0, 0.3])
        for _ in range(50):
            mujoco.mj_step(self.model, self.data)
        return self._obs(), {}

    def step(self, action):
        self.data.ctrl[:] = action * 0.5          # action scaling: ±1 -> ±0.5 rad
        for _ in range(20):                        # run 20 physics steps per RL step
            mujoco.mj_step(self.model, self.data)
        obs = self._obs()
        d = np.linalg.norm(obs["achieved_goal"] - self.goal)
        r = -float(d) + (1.0 if d < 0.03 else 0.0)
        return obs, r, False, d < 0.02, {"is_success": d < 0.03}
EOF
# The XML must contain <site name="tool0"/> and a matching number of actuators

You pass when you see this

Your own environment can be trained by SB3 with the same code (MultiInputPolicy + HER), and after reset the end-effector position is within a reasonable range.

Stuck?

SymptomMost likely causeWhat to do
SB3 reports a `Dict space` mismatchKey names must be observation/achieved_goal/desired_goalRename them; don't invent your own keys
Training doesn't convergeNo action scaling; ±1 applied directly as torque/positionScale to the physical range first, then tune coef; print qpos once in the environment to check for joint-limit hits
10

Three things to add before going to the real robot

A policy trained in simulation usually loses more than half its success rate on the real robot. These three items are ordered by cost-effectiveness; do them and see how much gap remains.

terminal / code
# 1) Domain randomization: mass ±20%, friction 0.5–1.5x, actuator gain ±30%, observation delay 0–20 ms
# 2) Observation alignment: real joint angles have encoder noise and zero offset; add the same noise in simulation
# 3) Action clipping and filtering: add a first-order low-pass before the policy output; cutoff from your control frequency (20–30 Hz at 1 kHz)

You pass when you see this

After adding randomization and retraining, sim success rate drops 10–20%, but the drop on the real robot shrinks markedly — that is the right direction.

Stuck?

SymptomMost likely causeWhat to do
Sim success rate collapses after randomizationRanges too wideRandomize one item at a time, ordered by sensitivity; friction and delay first

Acceptance checklist

Without looking at notes or chat history, complete each item below on your own — only then is the tutorial really done.

Keep learning

C6 Reinforcement LearningF2 Sim2Real and system identificationNext: GPU parallel training

Other tutorials

From zero to your first Python script: terminal lesson one

For anyone who has never installed an environment or opened a terminal, this is the first step up. No Linux, no GPU, nothing to buy throughout — just an ordinary laptop.

1–2 h

Build a six-axis arm in Gazebo from scratch and make it obey you

URDF → gz-sim → gz_ros2_control → MoveIt 2, one chain walked end to end, about 6 hours

6–8 h

Isaac Lab parallel training: 4096 environments at once

From environment setup to exporting a deployable policy, about 4 hours (excluding training time)

4 h + training