RobotGuide

Tools & simulation · topic G5 · 40 hours

Isaac Sim / Isaac Lab

GPU parallelism compresses «training one policy» from weeks to hours. This is the mainstream sim2real route today — and the layer where the GPUs are most expensive. The barrier is not algorithms; it is scene formats and parallel configuration.

Specialization stage · Hardcore
Algorithm · Proficient System software · Hands-on

Tools
Tools

What to learn

  • USD is the scene format: stage / prim / xform / material; the URDF relationship goes through an importer, and after conversion you must check joint axes and mass
  • PhysX 5 and articulations: drive modes (position / velocity / effort), solver iterations, joint friction and armature
  • Isaac Lab (formerly Orbit) has two styles: Manager-based (the configuration five-pack: scene / observation / action / reward / termination) and Direct (you write the step yourself)
  • Parallelism: num_envs, cloner, the VRAM-to-FPS trade — double the environments and step rate roughly halves; compute VRAM before you scale
  • Domain randomization: mass, friction, PD gains, observation delay, external pushes, visual materials and lighting; the ranges must cover the real robot's uncertainty — wider is not better
  • Sensors and synthetic data: RTX lidar, fisheye cameras, semantic/instance segmentation, depth; annotation is this toolchain's most underrated use
  • ROS 2 bridge: publishing topics, clock synchronization, frame naming, and the pitfalls of «sim time vs wall time»

The lesson · plain words

Isaac Lab (Isaac Sim) is the flagship of GPU-parallel simulation: one GPU runs thousands of robot environments at once, speeding up RL training a hundredfold. Its core methodology is domain randomization — «don't chase an accurate sim; chase a robust policy».

GPU parallelism: how the training paradigm changed

Traditional simulation runs one environment at a time; learning one skill takes days. Isaac pushes thousands of environments into a single kernel call via a GPU-ized physics engine (PhysX) — a quadruped gait trains in hours, and the paradigm is changed for good.

The price: contact accuracy gives way to throughput (F1's trade-off). Fine contact tasks (insertion) are actually worse on Isaac than MuJoCo — pick the tool by task type first.

Domain randomization: the art of the inverted U

Randomize parameters during training (friction ±, mass ±, latency ±, noise ±) and the policy is forced to become insensitive to them — the real robot's «one value» must fall inside the random range. Drag the magnitude in the figure: too small overfits the sim, too large and the policy gives up; there is a peak in between.

Tuning the randomization magnitude is core hands-on work for deploying RL: layer by parameter importance (less randomization for critical ones, more for minor ones), annealed by training progress (small randomness early to learn structure, large randomness later to build robustness).

Interactive figure · drag it and see

What this figure shows: sliders for friction ±, mass ±, latency ±, and sensor noise ±, a randomization readout, and a curve of real-robot success that rises and falls in an inverted U as the randomization magnitude changes. Takeaway: too little randomization overfits the sim, too much makes the policy timid — tuning is hunting the peak.

The pipeline from training to deployment

Isaac Lab training (RL, thousands of parallel environments) → export the policy (ONNX/TensorRT, G10) → real-robot deployment + low-rate fine-tuning. The whole chain's pitfalls concentrate in two places: is the observation space actually available on the real robot (never use sim-only perfect state), and can the real robot keep up with the control frequency.

The engineering checklist: an observation table (sim value ↔ real sensor), an action table (command ↔ actuator), a frequency table (training Hz ↔ deployment Hz). Align all three tables, and sim2real becomes negotiable.

How deep to go

Run a built-in Isaac Lab task end to end and modify its randomization parameters; explain the GPU-parallelism vs MuJoCo-accuracy trade-off; list the sim2real checklist. In RL robotics interviews, these three are mandatory questions.

Three questions — do you really get it?

1. Isaac Lab's biggest advantage over MuJoCo is?

  • AGPU-parallel thousands of environments: a hundredfold RL training throughput
  • BContact accuracy
  • CIt is free
Show answer

A. GPU-izing PhysX is a paradigm-level change. The cost is conceding contact accuracy (the F1/G6 trade-off).

2. The consequence of «too much» domain randomization is?

  • AMore stable training
  • BVRAM explodes
  • CThe policy learns conservatively; real-robot success actually drops
Show answer

C. The inverted U: under extreme randomization the policy dares only to stand still. Layering by importance + annealing is the correct answer.

3. The most commonly overlooked item on a sim2real checklist is?

  • ANetwork architecture
  • BRandom seeds
  • CWhether the observation space is obtainable from real sensors
Show answer

C. Use a sim-only perfect state (ground-truth pose) that the real robot cannot measure, and the policy fails immediately.

Keep going deeper

Links without a marker stay on this site; the rest point to the Chinese edition until its translation batch lands.

How it fails in the field

  • Training a week with camera rendering on; turning rendering off reveals a 20× speedup — disable vision during training, add it back at the end
  • num_envs pushed to the VRAM edge: no error, but the step rate sinks to 3 Hz, and it looks like «training is not converging»
  • Randomization ranges wider than the real robot's actual error: the sim learns an «immune to everything» conservative policy that is slow and shaky on hardware

You have learned enough when

Get a built-in task running: the training curve converges, play mode shows the policy walking/standing; produce a measured num_envs-vs-FPS table; name the three domain randomization items that affect sim2real most.

The Chinese edition carries the full topic page for Isaac Sim / Isaac Lab, including the complete outline, depth requirements per career track, and further derivations: robotguide.dev/k/G5/.