Engineering practice · topic F4 · 20 hours
Data collection & labeling
In the era of embodied AI, data is an asset. Whether you collect it correctly decides whether the model is any use.
What to learn
- Collection specs: time synchronization, camera calibration, metadata (who, what task, what conditions)
- Annotation quality assessment: consistency checks and sampling-review rates
- Dataset versioning and reproducibility (seeds, splits, statistics)
- The value of failure samples and how to collect them
- Tactile data collection specs: metadata for the two streams — a human wearing a tactile glove, and teleoperating a dexterous hand — must be kept distinct (sensor layout, calibration version); the two streams can be mixed for unified-representation training but must be separated for action-model training (see C9)
The lesson · plain words
Data is the food of perception models, and annotation quality sets the ceiling. This page covers the three keys of annotation: the IoU ruler, QA discipline, and «which data is worth labeling».
IoU: the ruler of annotation quality
IoU = intersection / union; 1.0 is a perfect overlap. Drag the prediction box below to get a feel: IoU>0.9 counts as passing, 0.5–0.9 teaches the model crooked boundaries, <0.5 is poison — the model learns your mistakes.
Detection's mAP is just the statistic of «counting a hit by IoU threshold»: [email protected] means IoU>0.5 counts as correct. Understand IoU and you understand every detection metric.
Interactive figure · drag it and see
Three annotation types and QA discipline
2D boxes (detection) are cheapest; segmentation masks (pixel-level) cost three times more; 3D pose annotation is the most expensive (usually generated by registering an object model). The cost gap drives data design: if a box solves it, do not reach for segmentation.
QA discipline: human-review a 5–10% sample, plus a consistency test (have two people annotate the same batch; if their IoU comes back below 0.8, the annotation guide is not written clearly enough). The annotation guide itself is a key document: edge cases (occlusion, truncation) must be written as rules.
Which data is worth labeling
Priority order: ① online failure cases (label exactly where the model errs — highest value); ② boundary samples (lighting changes, occlusion, new objects); ③ routine data (stop once diversity is sufficient; piling up volume has diminishing returns).
The robotics-specific account: teleoperation demonstration data (C7) goes through the same pipeline — a demo's «action annotation» is the pose stream the arm itself records, generated automatically at zero annotation cost. That is the underlying reason VLA data is exploding.
How deep to go
You can compute IoU and design a QA sampling plan; write a one-page annotation guide (edge cases included); and judge «label in-house, outsource, or use auto-labeling». Thinking in terms of the data pipeline is worth more than tool fluency.
Three questions — do you really get it?
1. What is the formula for IoU?
- AIntersection / ground-truth area
- BUnion area / intersection area
- CIntersection area / union area
Show answer
C. IoU is symmetric and unbiased; 1 is perfect, 0 is completely wrong. It is the common foundation of detection and segmentation metrics.
2. Which annotated data is worth the most?
- ARoutine data — the more the better
- BNice-looking photos
- COnline failure cases and boundary samples
Show answer
C. Label where the model errs. Past sufficient diversity, routine data's marginal return drops fast.
3. Two people independently annotate the same batch and their IoU is only 0.6. What does that mean?
- AThe annotation guide is unclear — the rules are ambiguous
- BThe model is too weak
- CThere is too much data
Show answer
A. A consistency test measures whether «the guide is executable». Fix the guide before annotating, or the rework doubles.
Keep going deeper
Links without a marker stay on this site; the rest point to the Chinese edition until its translation batch lands.
How it fails in the field
- Data with no metadata: three months later it cannot be reproduced and you do not know which conditions are missing
- Collecting only success samples; the policy has no recovery ability
You have learned enough when
The Chinese edition carries the full topic page for Data collection & labeling, including the complete outline, depth requirements per career track, and further derivations: robotguide.dev/k/F4/.