Perception & intelligence · topic C5 · 30 hours
Multi-sensor fusion
Every single sensor has a hard flaw. Fusion is not «averaging makes it accurate» — it is «cover each other's blind spots and cross-check each other».
What to learn
- Time sync: hard sync (PTP, hardware trigger) versus soft sync (interpolation) and their accuracy gap
- The calibration chain: camera–IMU and lidar–camera extrinsics, and how their errors propagate into the final pose
- Fusion levels: data-level, feature-level, decision-level — which scenes fit which
- Failure detection and graceful degradation: how to notice one sensor is lying
The lesson · plain words
Every single sensor has a weak spot: the IMU is fast but drifts, the camera is accurate but slow, GPS is stable but dead indoors, wheel odometry is cheap but slips. Fusion makes them cover for each other — not a naive average, but weighting by «who to trust right now».
Two words you must keep apart: complementary filter (cheap, easy to tune, the engineering default) and the Kalman filter (optimal, needs modeling — covered in B7).
IMU and camera: born companions
The IMU runs at hundreds of hertz with near-zero latency, but integration drifts — in 10 seconds it wanders a visible angle. The camera returns frames at 2–30 Hz with tens of milliseconds latency, but never drifts. They complement exactly: trust the IMU at high frequency, the camera at low frequency.
Drag α below: a complementary filter has exactly one parameter. α = 0.95 means each step takes 95% from the IMU increment and pulls 5% back toward the camera. Watch how far the IMU-alone line drifts after 10 s and how steady the fused line stays.
Interactive figure · drag it and see
Kalman-style fusion: turning «whom to trust» into math
B7 covered the single-sensor version. Multi-sensor fusion just performs the update step twice: the IMU arrives → predict; the camera arrives → correct. Whoever arrives gets a correction; nobody arrives → coast on the model.
The key engineering decisions: how to choose the state vector (position and velocity, or attitude error too), how to write the observation equation (does the camera give poses or pixels), and how to align timestamps across sensors (B1's old problem).
Time and space: the two thresholds of fusion
Time sync: two streams 20 ms apart are 2 cm of position error at 1 m/s — the trap from B1 is the number-one accident source in fusion. Spatial alignment: a 1° error in the camera–IMU extrinsic amplifies with speed in every turn.
Acceptance method: hold still 10 minutes and watch for drift; drive a straight line and look for systematic offset; do figure-eights and check corner tracking. All three pass before the fusion counts as usable.
Engineering default configurations
Low-cost attitude: complementary filter (start with α 0.95–0.99). Navigation grade: EKF/ESKF (error-state Kalman) with multiple sensors. Learning-based: the Microsoft-paper school that fuses directly in a network. Get the complementary filter running first, then the EKF — do not jump to the summit in one step.
The failure list: garbage Q/R (B7), uncalibrated extrinsics, timestamps by arrival time instead of sample time, and reusing a stale observation after a dropped frame. Those four account for nine of ten fusion bugs.
Three questions — do you really get it?
1. In a complementary filter, what does α = 0.98 mean?
- AA 98 Hz sampling rate
- BA 98% probability of using the camera
- CEach step takes 98% from the IMU increment, 2% from the camera's absolute value
Show answer
C. α sets the trust ratio between the two sources. Larger α tracks harder and drifts faster.
2. In IMU+camera fusion, what does a 20 ms timestamp gap directly cause?
- ABlurry images
- BIMU overheating
- CAbout 2 cm of position error at 1 m/s — easily misdiagnosed as a calibration error
Show answer
C. Time gap × speed = position gap. This error has a constant direction and is most easily blamed on the extrinsics.
3. When camera frames drop out, the EKF should?
- AReuse the last observation once more
- BReset the filter
- CPredict only, and update when the next frame arrives
Show answer
C. Reusing an old observation injects a fake measurement. The correct move: run the predict step only, let uncertainty inflate naturally, and wait for a real observation.
Keep going deeper
Links without a marker stay on this site; the rest point to the Chinese edition until its translation batch lands.
How it fails in the field
- Each sensor runs its own thread with unsynchronized timestamps — that is not fusion at all
- No fallback logic after fusion: one sensor dies and the whole vehicle loses control
You have learned enough when
The Chinese edition carries the full topic page for Multi-sensor fusion, including the complete outline, depth requirements per career track, and further derivations: robotguide.dev/k/C5/.