RobotGuide

Perception & intelligence · topic C5 · 30 hours

Multi-sensor fusion

Every single sensor has a hard flaw. Fusion is not «averaging makes it accurate» — it is «cover each other's blind spots and cross-check each other».

Specialization stage · Hardcore
Algorithm · Proficient System software · Hands-on Embedded & controls · Hands-on

What to learn

The lesson · plain words

Every single sensor has a weak spot: the IMU is fast but drifts, the camera is accurate but slow, GPS is stable but dead indoors, wheel odometry is cheap but slips. Fusion makes them cover for each other — not a naive average, but weighting by «who to trust right now».

Two words you must keep apart: complementary filter (cheap, easy to tune, the engineering default) and the Kalman filter (optimal, needs modeling — covered in B7).

IMU and camera: born companions

The IMU runs at hundreds of hertz with near-zero latency, but integration drifts — in 10 seconds it wanders a visible angle. The camera returns frames at 2–30 Hz with tens of milliseconds latency, but never drifts. They complement exactly: trust the IMU at high frequency, the camera at low frequency.

Drag α below: a complementary filter has exactly one parameter. α = 0.95 means each step takes 95% from the IMU increment and pulls 5% back toward the camera. Watch how far the IMU-alone line drifts after 10 s and how steady the fused line stays.

Interactive figure · drag it and see

What this figure shows: the IMU's drifting estimate (gray) against the camera-anchored fused estimate (orange), with a slider for the pull-back time constant τ. Takeaway: high frequencies trust the IMU, low frequencies pull back to the camera — smaller τ tracks the camera harder and drifts less, while a too-large τ lets the IMU drift walk in; the fused error RMS ends up far below the IMU running alone.

Kalman-style fusion: turning «whom to trust» into math

B7 covered the single-sensor version. Multi-sensor fusion just performs the update step twice: the IMU arrives → predict; the camera arrives → correct. Whoever arrives gets a correction; nobody arrives → coast on the model.

The key engineering decisions: how to choose the state vector (position and velocity, or attitude error too), how to write the observation equation (does the camera give poses or pixels), and how to align timestamps across sensors (B1's old problem).

Time and space: the two thresholds of fusion

Time sync: two streams 20 ms apart are 2 cm of position error at 1 m/s — the trap from B1 is the number-one accident source in fusion. Spatial alignment: a 1° error in the camera–IMU extrinsic amplifies with speed in every turn.

Acceptance method: hold still 10 minutes and watch for drift; drive a straight line and look for systematic offset; do figure-eights and check corner tracking. All three pass before the fusion counts as usable.

Engineering default configurations

Low-cost attitude: complementary filter (start with α 0.95–0.99). Navigation grade: EKF/ESKF (error-state Kalman) with multiple sensors. Learning-based: the Microsoft-paper school that fuses directly in a network. Get the complementary filter running first, then the EKF — do not jump to the summit in one step.

The failure list: garbage Q/R (B7), uncalibrated extrinsics, timestamps by arrival time instead of sample time, and reusing a stale observation after a dropped frame. Those four account for nine of ten fusion bugs.

Three questions — do you really get it?

1. In a complementary filter, what does α = 0.98 mean?

  • AA 98 Hz sampling rate
  • BA 98% probability of using the camera
  • CEach step takes 98% from the IMU increment, 2% from the camera's absolute value
Show answer

C. α sets the trust ratio between the two sources. Larger α tracks harder and drifts faster.

2. In IMU+camera fusion, what does a 20 ms timestamp gap directly cause?

  • ABlurry images
  • BIMU overheating
  • CAbout 2 cm of position error at 1 m/s — easily misdiagnosed as a calibration error
Show answer

C. Time gap × speed = position gap. This error has a constant direction and is most easily blamed on the extrinsics.

3. When camera frames drop out, the EKF should?

  • AReuse the last observation once more
  • BReset the filter
  • CPredict only, and update when the next frame arrives
Show answer

C. Reusing an old observation injects a fake measurement. The correct move: run the predict step only, let uncertainty inflate naturally, and wait for a real observation.

Keep going deeper

Links without a marker stay on this site; the rest point to the Chinese edition until its translation batch lands.

How it fails in the field

You have learned enough when

You can design a time-sync scheme and state its accuracy, explain how extrinsic error contaminates the fused result, and write a failure criterion for one sensor.

The Chinese edition carries the full topic page for Multi-sensor fusion, including the complete outline, depth requirements per career track, and further derivations: robotguide.dev/k/C5/.