Perception & intelligence · topic C1 · 20 hours
Camera models & calibration
The first gate of vision is not the model — it is calibration. When calibration is off, every downstream pose error starts at those 5 mm and amplifies from there.
What to learn
- What the pinhole model, intrinsics, extrinsics, and distortion coefficients each are, and how the error shows up
- The Zhang calibration procedure, and what actually matters in the reprojection error (smaller is not always better)
- The two hand-eye calibration setups: eye-in-hand (camera on the wrist) and eye-to-hand (camera fixed off the arm)
- The failure modes of the three depth-camera routes (structured light / ToF / stereo)
The lesson · plain words
A camera is not an eye; it is a machine that converts meters into pixels. The conversion rule is a single equation: image height = focal length × object size ÷ distance. Calibration means measuring the constants hidden in that equation — focal length, optical center, distortion.
Without calibration, every size you measure is in «pixel units»; after calibration, pixels can be translated back into meters — grasping, positioning, and measurement all depend on this one step.
The pinhole model: one equation governs all projection
Simplify the camera into a pinhole: light passes through the optical center and lands on the image plane, flipped up-down and left-right. Farther objects project smaller; longer focal length narrows the field of view. The core intuition is one sentence: pixels = focal length × meters ÷ distance.
Drag the object point and the focal length below: watch the projected point move with depth and focal length. The f (focal length) and c (optical-center offset) that calibration must solve are the two gears of this conversion machine.
Interactive figure · drag it and see
Distortion: where the lens is imperfect
A real lens is not a pinhole: wide angles bend straight lines into curves (barrel distortion), telephoto does the opposite (pincushion distortion). The distortion coefficients are a few numbers describing how much the edges get bent.
During calibration you photograph a checkerboard: however the grid's straight lines bend in the photo is exactly what the distortion coefficients encode. OpenCV gets you started with one line, findChessboardCorners, but you must judge the reprojection error yourself — under 0.5 px is acceptable, above 1 px reshoot.
The engineering discipline of calibration
The checkerboard must reach all four corners of the frame (distortion is strongest in the corners), poses must be varied (shooting head-on cannot solve for all the parameters), and lighting must be even. Calibrate at a fixed focal length — zoom once and everything before it is wasted.
The biggest traps: intrinsics shift slightly while focusing, a lens change means recalibration, and stacking the distortion model too high overfits. The reprojection error is the only acceptance metric.
Beyond calibration: hand-eye calibration
With the camera mounted on the arm's flange, one step remains: the fixed relation between the camera frame and the flange frame (hand-eye calibration, AX = XB). Pose the arm a few ways while photographing the same calibration board, and this fixed transform falls out.
The symptom of bad hand-eye calibration is distinctive: accurate when static, drifting the moment the workcell pose changes. The error hides in the fixed «camera-to-flange» relation and gets amplified by pose.
Three questions — do you really get it?
1. What does camera intrinsics calibration mainly solve for?
- AThe camera's position in the world frame
- BThe image's exposure parameters
- CFocal length, optical center, and distortion coefficients
Show answer
C. Intrinsics describe the camera's own conversion rule; the camera's position in the world belongs to extrinsics (plus hand-eye calibration).
2. The same object point moves to twice the distance — what happens to its image height?
- AIt doubles
- BUnchanged
- CIt halves
Show answer
C. Image height = f·size/distance; doubling the denominator halves the height. This is also why distant objects pile up in the frame.
3. Your calibration reports a reprojection error of 1.5 px. What should you do?
- AUse it as is — the error is not big
- BReshoot and recalibrate; normal is <0.5 px
- CDouble the checkerboard area
Show answer
B. The reprojection error is the only hard acceptance metric. Above 1 px, corner detection or pose diversity is the problem.
Keep going deeper
Links without a marker stay on this site; the rest point to the Chinese edition until its translation batch lands.
How it fails in the field
- Shooting only a small patch at the center of the frame with the calibration board: edge distortion stays unconstrained, and grasp error at the edges blows up to centimeters
- Swapping the lens without recalibrating — every bit of the resulting error gets blamed on the algorithm
You have learned enough when
The Chinese edition carries the full topic page for Camera models & calibration, including the complete outline, depth requirements per career track, and further derivations: robotguide.dev/k/C1/.