Systems & software · topic D4 · 35 hours
Real-time systems & PREEMPT_RT
The most valuable piece of insight in robot software: real-time is not 'fast', it is 'deterministic'. Very few people can produce jitter data.
What to learn
- Measuring determinism: jitter, worst-case latency, WCET, and why the average is meaningless
- Three key PREEMPT_RT mechanisms: sleeping spinlocks, priority inheritance, interrupt threading
- The four-piece tuning kit: isolcpus/nohz_full/rcu_nocbs isolation, SCHED_FIFO, mlockall, disabling deep C-states and frequency scaling
- Measurement method: three cyclictest comparison groups (stock kernel / PREEMPT_RT / with isolation), stressed with stress-ng
The lesson · plain words
Ordinary Linux chases 'fast on average'; real-time systems chase 'guaranteed even in the worst case'. A control loop beats at 1 ms and cannot miss a single beat — that is why PREEMPT_RT exists.
Real-time is not 'fast', it is 'deterministic'. This page builds your intuition for the 'latency tail'.
The latency tail: jitter is the killer
Look at the histogram: a stock kernel's average latency may be only tens of microseconds, but once load rises the tail drags to hundreds of microseconds or even milliseconds. A control loop needs 99.99% of latencies below the deadline — the average is meaningless; the tail is what decides survival.
Where the tail comes from: long non-preemptible stretches in the kernel (interrupt handling, memory-management locks), page faults, and all the moments the kernel queues your thread together with thousands of ordinary tasks.
Interactive figure · drag it and see
What the PREEMPT_RT patch changes
It turns almost all 'non-preemptible sections' in the kernel into preemptible ones: spinlocks become sleeping mutexes, interrupts are threaded, priority inheritance is added. The cost is slightly lower throughput (5–15%), traded for a bounded worst-case latency.
Current status: since mainline kernel 6.12 most of the RT patch has been merged; installing an RT kernel on a Raspberry Pi or industrial PC is now routine. The de facto standard at robot companies: control-related processes run on an RT kernel with real-time scheduling policy and reserved cores.
Four things to make user space real-time
① mlockall to lock memory (prevent page faults); ② SCHED_FIFO with tiered priorities (sampling > control > communication); ③ preallocate the hot path (D1); ④ pin cores + isolcpus to fence kernel daemons into other cores. Only after all four does user space cash in the RT kernel's guarantees.
Acceptance: run cyclictest for the latency distribution and look at max and the 99.9 percentile, never the average. Load testing must be thorough (stress-ng hammering CPU+IO while the control loop runs).
When you don't need RT
Perception, planning, and communication do not need it — they are oblivious to 100 ms of jitter. Don't RT-ize the whole machine; fence only the hard real-time parts (servo loops, safety I/O) into RT cores and leave the rest on plain Linux.
The criterion in one line: only what causes physical consequences when it misses a deadline (crashing, exploding, shaking) deserves a real-time budget. This also answers 'why ROS 2 is not a real-time system'.
Three questions — do you really get it?
1. The goal of a real-time system is?
- AFaster average response
- BHigher CPU utilization
- CBounded, deterministic worst-case latency
Show answer
C. The essence of real-time is determinism. A system fast on average but occasionally stuck for 5 ms is a disaster in a control loop.
2. Real-time performance should be evaluated by looking at?
- AAverage latency
- BThe 99.9 percentile and the maximum latency
- CThroughput
Show answer
B. The tail is what kills the control loop. cyclictest's max and percentiles are the metrics.
3. Which of the following does NOT need real-time guarantees?
- AA 1 kHz servo loop
- BVisual detection and planning
- CSafety I/O sampling
Show answer
B. Perception and planning are oblivious to tens of milliseconds of jitter; fencing them into RT only wastes the budget. Reserve hard real-time for the stages with physical consequences.
Keep going deeper
Links without a marker stay on this site; the rest point to the Chinese edition until its translation batch lands.
How it fails in the field
- Swapping the kernel without doing isolation, measuring jitter similar to the stock kernel, then drawing the wrong conclusion that 'the patch is useless'
- Reporting averages without P99 or max, so the real problem never gets localized
You have learned enough when
The Chinese edition carries the full topic page for Real-time systems & PREEMPT_RT, including the complete outline, depth requirements per career track, and further derivations: robotguide.dev/k/D4/.