RobotGuide

Engineering practice · topic F4 · 20 hours

Data collection & labeling

In the era of embodied AI, data is an asset. Whether you collect it correctly decides whether the model is any use.

Engineering stage · Beginner
Algorithm · Proficient System software · Hands-on Embedded & controls · Aware

Tools
Tools

What to learn

  • Collection specs: time synchronization, camera calibration, metadata (who, what task, what conditions)
  • Annotation quality assessment: consistency checks and sampling-review rates
  • Dataset versioning and reproducibility (seeds, splits, statistics)
  • The value of failure samples and how to collect them
  • Tactile data collection specs: metadata for the two streams — a human wearing a tactile glove, and teleoperating a dexterous hand — must be kept distinct (sensor layout, calibration version); the two streams can be mixed for unified-representation training but must be separated for action-model training (see C9)

The lesson · plain words

Data is the food of perception models, and annotation quality sets the ceiling. This page covers the three keys of annotation: the IoU ruler, QA discipline, and «which data is worth labeling».

IoU: the ruler of annotation quality

IoU = intersection / union; 1.0 is a perfect overlap. Drag the prediction box below to get a feel: IoU>0.9 counts as passing, 0.5–0.9 teaches the model crooked boundaries, <0.5 is poison — the model learns your mistakes.

Detection's mAP is just the statistic of «counting a hit by IoU threshold»: [email protected] means IoU>0.5 counts as correct. Understand IoU and you understand every detection metric.

Interactive figure · drag it and see

What this figure shows: a ground-truth box and a draggable prediction box over one object, with a live IoU readout — aligned it reads pass, a visible offset warns that training will learn crooked boxes, and severe misalignment marks the frame as poison. Takeaway: every detection and segmentation metric (mAP and friends) starts from IoU, and QA-sampling 5% of annotations for human review is the floor discipline of a data pipeline.

Three annotation types and QA discipline

2D boxes (detection) are cheapest; segmentation masks (pixel-level) cost three times more; 3D pose annotation is the most expensive (usually generated by registering an object model). The cost gap drives data design: if a box solves it, do not reach for segmentation.

QA discipline: human-review a 5–10% sample, plus a consistency test (have two people annotate the same batch; if their IoU comes back below 0.8, the annotation guide is not written clearly enough). The annotation guide itself is a key document: edge cases (occlusion, truncation) must be written as rules.

Which data is worth labeling

Priority order: ① online failure cases (label exactly where the model errs — highest value); ② boundary samples (lighting changes, occlusion, new objects); ③ routine data (stop once diversity is sufficient; piling up volume has diminishing returns).

The robotics-specific account: teleoperation demonstration data (C7) goes through the same pipeline — a demo's «action annotation» is the pose stream the arm itself records, generated automatically at zero annotation cost. That is the underlying reason VLA data is exploding.

How deep to go

You can compute IoU and design a QA sampling plan; write a one-page annotation guide (edge cases included); and judge «label in-house, outsource, or use auto-labeling». Thinking in terms of the data pipeline is worth more than tool fluency.

Three questions — do you really get it?

1. What is the formula for IoU?

  • AIntersection / ground-truth area
  • BUnion area / intersection area
  • CIntersection area / union area
Show answer

C. IoU is symmetric and unbiased; 1 is perfect, 0 is completely wrong. It is the common foundation of detection and segmentation metrics.

2. Which annotated data is worth the most?

  • ARoutine data — the more the better
  • BNice-looking photos
  • COnline failure cases and boundary samples
Show answer

C. Label where the model errs. Past sufficient diversity, routine data's marginal return drops fast.

3. Two people independently annotate the same batch and their IoU is only 0.6. What does that mean?

  • AThe annotation guide is unclear — the rules are ambiguous
  • BThe model is too weak
  • CThere is too much data
Show answer

A. A consistency test measures whether «the guide is executable». Fix the guide before annotating, or the rework doubles.

Keep going deeper

Links without a marker stay on this site; the rest point to the Chinese edition until its translation batch lands.

How it fails in the field

  • Data with no metadata: three months later it cannot be reproduced and you do not know which conditions are missing
  • Collecting only success samples; the policy has no recovery ability

You have learned enough when

You can write a collection spec and produce a small dataset, and you can state its coverage and its gaps.

The Chinese edition carries the full topic page for Data collection & labeling, including the complete outline, depth requirements per career track, and further derivations: robotguide.dev/k/F4/.