RobotGuide

Perception & intelligence · topic C2 · 50 hours

Deep learning foundations

Perception, pose estimation, VLA — all of it stands on deep learning. But a robotics job does not want «package tuner»; it wants someone who can judge whether a model may run on the machine.

Specialization stage · Intermediate
Algorithm · Proficient

What to learn

The lesson · plain words

The smallest part of deep learning is the convolution: a small square matrix (a kernel) slides over the image and does one multiply-add per position. Every «look at the picture» network — detection, segmentation, grasp pose — is a thousand-layer stack of this one step.

Entry level does not require training networks from scratch; being able to explain what happens between «one image in, one list of numbers out» already counts as understanding.

Convolution: one kernel, sweeping the image

Switch kernels below: the edge kernel traces the outline of the bright blob, the blur kernel flattens everything, the sharpen kernel adds contrast back. Same image, different kernel, wildly different output — a convolution layer learns precisely these kernel numbers.

Wherever the window lands, the multiply-add happens there; sweeping the whole image yields a new image (a feature map). A network = hundreds of kernels sweeping at once + nonlinearity + downsampling, translating «pixels» into «semantics» layer by layer.

Interactive figure · drag it and see

What this figure shows: a 6×6 input image with a 3×3 kernel sliding over it, producing a 4×4 output — buttons switch between an edge kernel, a blur kernel, and a sharpen kernel. Takeaway: the image never changes; only the kernel's numbers do, and the outputs differ completely — a conv layer learns exactly those numbers, which is how pixels become edges, textures, and parts.

Training: the loss function leads the network

Give the network labeled images (input → correct answer); it guesses, measures the gap (loss), and nudges every kernel number along the gradient (A3's gradient descent). Repeat tens of thousands of times and kernels evolve from random noise into edge/texture/part detectors.

The validation set is the referee: training loss still falling while validation loss rises means overfitting — the network is memorizing answers instead of learning rules. Early stopping and data augmentation exist to fight it.

Three things specific to robots

① Small data: real-robot data is expensive; simulation and domain randomization (F2/G5) make up the volume. ② Latency budget: a 50 Hz control loop leaves the network a dozen or so milliseconds, hence quantization, pruning, distillation (G10). ③ Failure cost: one wrong detection can crash the machine, hence confidence scores, multi-model voting, and fallbacks on anomalies.

Evaluate always on «never-seen data»: new lighting, new occlusion, new object arrangements. A lab result of 99% dropping to 60% in the field usually means the eval set and the field are not the same distribution.

How deep to go

Be able to run inference and fine-tune pretrained models (not train ResNet from zero); read the architecture diagram of a detection network; explain what receptive field, downsampling, and BN each do. Hand-writing backprop is not required — but you must be able to say why gradients vanish.

Interview favorites: convolution vs fully-connected, why pooling, how mAP is computed, the three tools against overfitting. This page's text is the skeleton of those answers.

Three questions — do you really get it?

1. As a convolution kernel sweeps the image, how is each output pixel produced?

  • AThe average of the pixels in the window
  • BThe maximum value in the window
  • CThe kernel multiplied element-wise with the window, then summed
Show answer

C. Multiply-add is the body of convolution; «take the max» is pooling's rule, not convolution.

2. Training loss still falling, validation loss starting to rise — what does that mean?

  • ALearning rate too small
  • BOverfitting — early-stop or add data augmentation
  • CNetwork too small
Show answer

B. The network has started memorizing the training set. Diverging train/validation curves are the standard overfitting signal.

3. Which of these is usually NOT a worry when deploying robot perception?

  • AInference latency
  • BWhether the training set is public
  • CCompute and power
Show answer

B. Latency and power are hard constraints (a 50 Hz loop); training-set publicity does not affect the running system.

Keep going deeper

Links without a marker stay on this site; the rest point to the Chinese edition until its translation batch lands.

How it fails in the field

You have learned enough when

You can train a small model independently and explain every hyperparameter, diagnose overfitting and data leakage, and state what quantization costs in accuracy.

The Chinese edition carries the full topic page for Deep learning foundations, including the complete outline, depth requirements per career track, and further derivations: robotguide.dev/k/C2/.