RobotGuide

Systems & software · topic D10 · 30 hours

Profiling & debugging

The same symptom can come from six different layers. Whether you can localize by layer decides whether you «fixed it» or merely «made it stop showing up for now».

Engineering stage · Hardcore
Algorithm · Aware System software · Proficient Embedded & controls · Proficient

Tools
Tools

What to learn

  • Localization paths for three problem classes: CPU usage, context switching, memory growth
  • Basic use of perf / ftrace / eBPF and which metrics to read
  • Latency instrumentation: a GPIO toggle + an oscilloscope is the only trustworthy end-to-end measurement
  • A symptom-to-layer troubleshooting order (bus first, then scheduling, then algorithm)

The lesson · plain words

The first law of performance work: measure first, optimize second. The hotspot your intuition points at is wrong nine times out of ten. This page covers how to measure, how to read a flame graph, and how Amdahl's law pours cold water on your plans.

The flame graph: wide blocks are the hotspots

A flame graph draws call stacks as stacked rectangles: the horizontal axis is share of time, and the wider a block, the more it deserves optimization. Watch the figure: the widest block is cudaMemcpy (30% of the total) — optimize it and everything visibly speeds up; optimizing the small blocks next to it is wasted effort.

After clicking «optimize the hotspot»: the PCIe copy drops from 30% to 3%, yet total time falls only from 100 ms to 71 ms — nowhere near a 97 ms improvement. That is Amdahl's law: the overall speedup is locked down by the part you did not touch.

Interactive figure · drag it and see

What this figure shows: a flame graph of stacked call-stack rectangles — main, processFrame, ros spin, logging, with conv2d (GPU upload), resize, nms, and a wide cudaMemcpy block; a before/after toggle shows total ms and the headroom at 30 Hz, with the 30% PCIe copy cut down to 3%. Takeaway: the widest block is the hotspot — measure first, optimize second, never guess; when only that 30% block shrinks, Amdahl caps the overall win at about 1.4×.

The toolbox: pick by scenario

CPU hotspots: perf record + the FlameGraph scripts (first choice on Linux). Latency distributions: cyclictest (kernel), hand-rolled timestamp histograms (application). Memory: valgrind is slow but exact, heaptrack is fast. IO/syscalls: strace -c for a summary.

ROS 2 specifics: ros2 trace records DDS-layer events and can answer «where along the route did this message get stuck». Measuring real-time loops needs your own instrumentation via rdtsc/clock registers — ordinary timing functions themselves carry microsecond-level overhead.

The optimization priority list

The usual hotspot leaderboard: ① copies that should not happen (images/point clouds passed by value, serialization) ② blocking waits (synchronous I/O, lock contention) ③ per-element loops left unvectorized ④ frequent small allocations ⑤ logs written synchronously to disk.

Each entry has an off-the-shelf fix (zero-copy / asynchronization / vectorization / preallocation / async logging). Match the flame graph first, then act — this list is not a checklist to grind through; it is a menu for finding the entry your figure points at.

Accepting optimizations and preventing regressions

Every optimization needs a before/after comparison: same test data, same metrics (p50/p99 latency, frame rate). An optimization whose numbers are not archived counts as not done — two weeks later nobody remembers why the code was changed that way.

Regression prevention lives in CI benchmarks: every commit runs a standard workload, and latency over the threshold triggers an alarm. Performance rots quietly (dependency upgrades, data growth); without a gatekeeper, six months later you are back to square one.

Three questions — do you really get it?

1. What does Amdahl's law tell us?

  • AThe overall speedup is capped by the unoptimized part: a hotspot with a 30% share cut to zero still caps you at 1.4×
  • BOptimizing a hotspot always speeds things up linearly
  • CThe more parallelism the better
Show answer

A. A 30% part sped up 10× → 1/(0.7+0.03)≈1.37. Expectation management is part of optimization.

2. In a flame graph, what should you optimize first?

  • AThe topmost blocks
  • BThe block with the longest name
  • CThe horizontally widest blocks
Show answer

C. Width = share of time. Wide blocks are the real global bottleneck.

3. What mechanism stops performance from quietly regressing?

  • ACode review
  • BCI benchmarks with latency alarms
  • CMore comments
Show answer

B. Performance regression is gradual; only continuous benchmarking can intercept it.

Keep going deeper

Links without a marker stay on this site; the rest point to the Chinese edition until its translation batch lands.

How it fails in the field

  • Measuring end-to-end latency with software timestamps, baking the measurement overhead itself into the result
  • Diving straight into code changes with no bisection — after touching three places you cannot tell which one did the work

You have learned enough when

You can capture an end-to-end latency curve with an oscilloscope, locate one hotspot function with perf, and write a «symptom → hypothesis → verification» troubleshooting log.

The Chinese edition carries the full topic page for Profiling & debugging, including the complete outline, depth requirements per career track, and further derivations: robotguide.dev/k/D10/.