Perception & intelligence · topic C2 · 50 hours
Deep learning foundations
Perception, pose estimation, VLA — all of it stands on deep learning. But a robotics job does not want «package tuner»; it wants someone who can judge whether a model may run on the machine.
What to learn
- Backprop, loss functions, optimizers: know what each hyperparameter actually changed
- The structural difference between CNNs and Transformers, and why vision foundation models all moved to Transformers
- Train/validation/test splits, and how to recognize overfitting and data leakage
- Deployment side: quantization, distillation, ONNX/TensorRT basics and their accuracy cost
The lesson · plain words
The smallest part of deep learning is the convolution: a small square matrix (a kernel) slides over the image and does one multiply-add per position. Every «look at the picture» network — detection, segmentation, grasp pose — is a thousand-layer stack of this one step.
Entry level does not require training networks from scratch; being able to explain what happens between «one image in, one list of numbers out» already counts as understanding.
Convolution: one kernel, sweeping the image
Switch kernels below: the edge kernel traces the outline of the bright blob, the blur kernel flattens everything, the sharpen kernel adds contrast back. Same image, different kernel, wildly different output — a convolution layer learns precisely these kernel numbers.
Wherever the window lands, the multiply-add happens there; sweeping the whole image yields a new image (a feature map). A network = hundreds of kernels sweeping at once + nonlinearity + downsampling, translating «pixels» into «semantics» layer by layer.
Interactive figure · drag it and see
Training: the loss function leads the network
Give the network labeled images (input → correct answer); it guesses, measures the gap (loss), and nudges every kernel number along the gradient (A3's gradient descent). Repeat tens of thousands of times and kernels evolve from random noise into edge/texture/part detectors.
The validation set is the referee: training loss still falling while validation loss rises means overfitting — the network is memorizing answers instead of learning rules. Early stopping and data augmentation exist to fight it.
Three things specific to robots
① Small data: real-robot data is expensive; simulation and domain randomization (F2/G5) make up the volume. ② Latency budget: a 50 Hz control loop leaves the network a dozen or so milliseconds, hence quantization, pruning, distillation (G10). ③ Failure cost: one wrong detection can crash the machine, hence confidence scores, multi-model voting, and fallbacks on anomalies.
Evaluate always on «never-seen data»: new lighting, new occlusion, new object arrangements. A lab result of 99% dropping to 60% in the field usually means the eval set and the field are not the same distribution.
How deep to go
Be able to run inference and fine-tune pretrained models (not train ResNet from zero); read the architecture diagram of a detection network; explain what receptive field, downsampling, and BN each do. Hand-writing backprop is not required — but you must be able to say why gradients vanish.
Interview favorites: convolution vs fully-connected, why pooling, how mAP is computed, the three tools against overfitting. This page's text is the skeleton of those answers.
Three questions — do you really get it?
1. As a convolution kernel sweeps the image, how is each output pixel produced?
- AThe average of the pixels in the window
- BThe maximum value in the window
- CThe kernel multiplied element-wise with the window, then summed
Show answer
C. Multiply-add is the body of convolution; «take the max» is pooling's rule, not convolution.
2. Training loss still falling, validation loss starting to rise — what does that mean?
- ALearning rate too small
- BOverfitting — early-stop or add data augmentation
- CNetwork too small
Show answer
B. The network has started memorizing the training set. Diverging train/validation curves are the standard overfitting signal.
3. Which of these is usually NOT a worry when deploying robot perception?
- AInference latency
- BWhether the training set is public
- CCompute and power
Show answer
B. Latency and power are hard constraints (a 50 Hz loop); training-set publicity does not affect the running system.
Keep going deeper
Links without a marker stay on this site; the rest point to the Chinese edition until its translation batch lands.
How it fails in the field
- Frames of one video clip split into both train and test sets — metrics inflate, then collapse on the real machine
- Watching mAP only, never checking inference time or VRAM — the model never reaches 10 Hz on the Jetson
You have learned enough when
The Chinese edition carries the full topic page for Deep learning foundations, including the complete outline, depth requirements per career track, and further derivations: robotguide.dev/k/C2/.