Perception & intelligence · topic C3 · 45 hours
2D / 3D vision
Detection, segmentation, pose estimation, point-cloud processing — the daily work of perception roles. Grasping roles almost always test pose estimation.
What to learn
- The mainstream detection and segmentation models and when to use which (speed-first or accuracy-first)
- Point-cloud processing: filtering, registration (ICP and its variants), segmentation
- The two routes to 6D pose estimation: feature-based (e.g. PPF) and learning-based
- Visual servoing: image-based IBVS versus position-based PBVS
The lesson · plain words
2D vision answers «what is in the image, and in which box»; 3D vision answers «which point in space is it at, and which way is it facing». Grasping needs the latter — a robot arm does not understand pixels, only meters.
Between 2D and 3D, the missing number is called depth. This page covers how depth is obtained and how pose is estimated.
Stereo: the pixel offset between two eyes
Two cameras, left and right, look at the same point; on the image plane it is displaced horizontally by d pixels, and depth z = focal length × baseline ÷ d. Drag the depth and the baseline below: the nearer the object, the larger the disparity; the farther, the smaller — at range, depth starts to be guesswork.
This is stereo's Achilles heel: the longer the baseline (the distance between the two cameras), the more accurately near objects are measured. A stereo rig with a 12 cm baseline already shows centimeter-plus depth error at 3 m.
Interactive figure · drag it and see
Depth sensors: the trade-offs of active schemes
Structured light (speckle): accurate up close (mm-level), fails at range and under strong light. ToF: works at range, but multiple units interfere with each other. Monocular depth networks: cheap, but scale is guessed (absolute depth is unconstrained). LiDAR: true 3D but sparse and expensive.
Selection logic in one sentence: indoor short-range, structured light; outdoor navigation, LiDAR; tight budget, monocular network + motion (SfM) to recover scale. There is no all-round player, only a match to the working conditions.
Pose estimation: from a box to six numbers
Grasping wants the object's 3 translations + 3 rotations in the camera frame. Two routes: model-based (detect keypoints in pixels, then solve the pose with PnP) and end-to-end (the network regresses the 6D pose directly).
PnP's intuition: knowing the pixel positions of at least 4 points on the object, plus those points' true positions in the object frame, lets you back-solve the camera's pose relative to the object. It is the mirror image of B2's inverse-kinematics idea inside vision.
Validation: judge by «did it grasp»
3D vision metrics cannot stop at geometric errors like ADD-0.02d; verify closed-loop: command grasps at the estimated poses and tally the grasp success rate. A 2° pose error passes on a dataset but drops parts on a stack of workpieces.
Common pitfalls: depth map and RGB misaligned (wrong extrinsics), flying depth pixels at object edges, missing depth on transparent/reflective objects. The engineering fix for such objects: change the wavelength band, or add occlusion and reshoot.
Three questions — do you really get it?
1. In stereo depth, as the object point gets farther, the disparity will...
- AGet larger
- BStay the same
- CGet smaller, and depth accuracy degrades
Show answer
C. z = f·b/d: as d shrinks, z grows; the pixel quantization error of d is amplified multiplicatively into depth error at range.
2. Given the pixel coordinates of 4 points on an object and their true 3D coordinates, the method that solves the camera pose is called?
- AICP
- BSLAM
- CPnP
Show answer
C. PnP solves the «n-point projection» equations for the camera pose; ICP registers point clouds, SLAM builds maps and localizes.
3. The final acceptance metric for a grasping task should be?
- AThe pose estimation's ADD error
- BThe mAP of detection boxes
- CThe success rate of grasps commanded at the estimated poses
Show answer
C. Geometric metrics are only proxies. Grasping successfully is the true metric of the grasping system.
Keep going deeper
Links without a marker stay on this site; the rest point to the Chinese edition until its translation batch lands.
How it fails in the field
- Pointing a depth camera straight at reflective or transparent objects — the point cloud comes back full of holes — and then blaming the algorithm
- Reporting pose estimation only as a success rate, never an error distribution, mixing mm-level and cm-level results in one conversation
You have learned enough when
The Chinese edition carries the full topic page for 2D / 3D vision, including the complete outline, depth requirements per career track, and further derivations: robotguide.dev/k/C3/.