RobotGuide

Perception & intelligence · topic C1 · 20 hours

Camera models & calibration

The first gate of vision is not the model — it is calibration. When calibration is off, every downstream pose error starts at those 5 mm and amplifies from there.

Core stage · Beginner
Algorithm · Proficient System software · Aware Embedded & controls · Aware

Tools
Tools

What to learn

  • What the pinhole model, intrinsics, extrinsics, and distortion coefficients each are, and how the error shows up
  • The Zhang calibration procedure, and what actually matters in the reprojection error (smaller is not always better)
  • The two hand-eye calibration setups: eye-in-hand (camera on the wrist) and eye-to-hand (camera fixed off the arm)
  • The failure modes of the three depth-camera routes (structured light / ToF / stereo)

The lesson · plain words

A camera is not an eye; it is a machine that converts meters into pixels. The conversion rule is a single equation: image height = focal length × object size ÷ distance. Calibration means measuring the constants hidden in that equation — focal length, optical center, distortion.

Without calibration, every size you measure is in «pixel units»; after calibration, pixels can be translated back into meters — grasping, positioning, and measurement all depend on this one step.

The pinhole model: one equation governs all projection

Simplify the camera into a pinhole: light passes through the optical center and lands on the image plane, flipped up-down and left-right. Farther objects project smaller; longer focal length narrows the field of view. The core intuition is one sentence: pixels = focal length × meters ÷ distance.

Drag the object point and the focal length below: watch the projected point move with depth and focal length. The f (focal length) and c (optical-center offset) that calibration must solve are the two gears of this conversion machine.

Interactive figure · drag it and see

What this figure shows: a pinhole camera with the optical center and the image plane, plus an object point you can drag sideways to change depth z; the image forms below the axis (inverted), the readout shows image height = f·y/z in px, and sliders adjust object depth z (m) and focal length f (px) together with the field of view. Takeaway: intrinsic calibration solves exactly f and the optical-center offset — the two gears that translate pixels back into meters.

Distortion: where the lens is imperfect

A real lens is not a pinhole: wide angles bend straight lines into curves (barrel distortion), telephoto does the opposite (pincushion distortion). The distortion coefficients are a few numbers describing how much the edges get bent.

During calibration you photograph a checkerboard: however the grid's straight lines bend in the photo is exactly what the distortion coefficients encode. OpenCV gets you started with one line, findChessboardCorners, but you must judge the reprojection error yourself — under 0.5 px is acceptable, above 1 px reshoot.

The engineering discipline of calibration

The checkerboard must reach all four corners of the frame (distortion is strongest in the corners), poses must be varied (shooting head-on cannot solve for all the parameters), and lighting must be even. Calibrate at a fixed focal length — zoom once and everything before it is wasted.

The biggest traps: intrinsics shift slightly while focusing, a lens change means recalibration, and stacking the distortion model too high overfits. The reprojection error is the only acceptance metric.

Beyond calibration: hand-eye calibration

With the camera mounted on the arm's flange, one step remains: the fixed relation between the camera frame and the flange frame (hand-eye calibration, AX = XB). Pose the arm a few ways while photographing the same calibration board, and this fixed transform falls out.

The symptom of bad hand-eye calibration is distinctive: accurate when static, drifting the moment the workcell pose changes. The error hides in the fixed «camera-to-flange» relation and gets amplified by pose.

Three questions — do you really get it?

1. What does camera intrinsics calibration mainly solve for?

  • AThe camera's position in the world frame
  • BThe image's exposure parameters
  • CFocal length, optical center, and distortion coefficients
Show answer

C. Intrinsics describe the camera's own conversion rule; the camera's position in the world belongs to extrinsics (plus hand-eye calibration).

2. The same object point moves to twice the distance — what happens to its image height?

  • AIt doubles
  • BUnchanged
  • CIt halves
Show answer

C. Image height = f·size/distance; doubling the denominator halves the height. This is also why distant objects pile up in the frame.

3. Your calibration reports a reprojection error of 1.5 px. What should you do?

  • AUse it as is — the error is not big
  • BReshoot and recalibrate; normal is <0.5 px
  • CDouble the checkerboard area
Show answer

B. The reprojection error is the only hard acceptance metric. Above 1 px, corner detection or pose diversity is the problem.

Keep going deeper

Links without a marker stay on this site; the rest point to the Chinese edition until its translation batch lands.

How it fails in the field

  • Shooting only a small patch at the center of the frame with the calibration board: edge distortion stays unconstrained, and grasp error at the edges blows up to centimeters
  • Swapping the lens without recalibrating — every bit of the resulting error gets blamed on the algorithm

You have learned enough when

You can independently complete a camera intrinsics + hand-eye calibration and report the reprojection error, explain why the calibration board must cover the whole field of view, and describe why depth cameras fail on reflective and transparent objects.

The Chinese edition carries the full topic page for Camera models & calibration, including the complete outline, depth requirements per career track, and further derivations: robotguide.dev/k/C1/.