RobotGuide

Systems & software · topic D4 · 35 hours

Real-time systems & PREEMPT_RT

The most valuable piece of insight in robot software: real-time is not 'fast', it is 'deterministic'. Very few people can produce jitter data.

Engineering stage · Hardcore
Algorithm · Aware System software · Proficient Embedded & controls · Proficient

Tools
Tools

What to learn

  • Measuring determinism: jitter, worst-case latency, WCET, and why the average is meaningless
  • Three key PREEMPT_RT mechanisms: sleeping spinlocks, priority inheritance, interrupt threading
  • The four-piece tuning kit: isolcpus/nohz_full/rcu_nocbs isolation, SCHED_FIFO, mlockall, disabling deep C-states and frequency scaling
  • Measurement method: three cyclictest comparison groups (stock kernel / PREEMPT_RT / with isolation), stressed with stress-ng

The lesson · plain words

Ordinary Linux chases 'fast on average'; real-time systems chase 'guaranteed even in the worst case'. A control loop beats at 1 ms and cannot miss a single beat — that is why PREEMPT_RT exists.

Real-time is not 'fast', it is 'deterministic'. This page builds your intuition for the 'latency tail'.

The latency tail: jitter is the killer

Look at the histogram: a stock kernel's average latency may be only tens of microseconds, but once load rises the tail drags to hundreds of microseconds or even milliseconds. A control loop needs 99.99% of latencies below the deadline — the average is meaningless; the tail is what decides survival.

Where the tail comes from: long non-preemptible stretches in the kernel (interrupt handling, memory-management locks), page faults, and all the moments the kernel queues your thread together with thousands of ordinary tasks.

Interactive figure · drag it and see

What this figure shows: a latency histogram under system load (compiling / big copies) where the stock kernel's tail drags out to hundreds of microseconds while the RT kernel keeps 99.9% under a tight bound, with a 1 ms deadline line marked. Takeaway: PREEMPT_RT patches out the non-preemptible kernel sections — that is where the tail went; in a 1 ms control beat, one 500 μs jitter spike is half a beat lost.

What the PREEMPT_RT patch changes

It turns almost all 'non-preemptible sections' in the kernel into preemptible ones: spinlocks become sleeping mutexes, interrupts are threaded, priority inheritance is added. The cost is slightly lower throughput (5–15%), traded for a bounded worst-case latency.

Current status: since mainline kernel 6.12 most of the RT patch has been merged; installing an RT kernel on a Raspberry Pi or industrial PC is now routine. The de facto standard at robot companies: control-related processes run on an RT kernel with real-time scheduling policy and reserved cores.

Four things to make user space real-time

① mlockall to lock memory (prevent page faults); ② SCHED_FIFO with tiered priorities (sampling > control > communication); ③ preallocate the hot path (D1); ④ pin cores + isolcpus to fence kernel daemons into other cores. Only after all four does user space cash in the RT kernel's guarantees.

Acceptance: run cyclictest for the latency distribution and look at max and the 99.9 percentile, never the average. Load testing must be thorough (stress-ng hammering CPU+IO while the control loop runs).

When you don't need RT

Perception, planning, and communication do not need it — they are oblivious to 100 ms of jitter. Don't RT-ize the whole machine; fence only the hard real-time parts (servo loops, safety I/O) into RT cores and leave the rest on plain Linux.

The criterion in one line: only what causes physical consequences when it misses a deadline (crashing, exploding, shaking) deserves a real-time budget. This also answers 'why ROS 2 is not a real-time system'.

Three questions — do you really get it?

1. The goal of a real-time system is?

  • AFaster average response
  • BHigher CPU utilization
  • CBounded, deterministic worst-case latency
Show answer

C. The essence of real-time is determinism. A system fast on average but occasionally stuck for 5 ms is a disaster in a control loop.

2. Real-time performance should be evaluated by looking at?

  • AAverage latency
  • BThe 99.9 percentile and the maximum latency
  • CThroughput
Show answer

B. The tail is what kills the control loop. cyclictest's max and percentiles are the metrics.

3. Which of the following does NOT need real-time guarantees?

  • AA 1 kHz servo loop
  • BVisual detection and planning
  • CSafety I/O sampling
Show answer

B. Perception and planning are oblivious to tens of milliseconds of jitter; fencing them into RT only wastes the budget. Reserve hard real-time for the stages with physical consequences.

Keep going deeper

Links without a marker stay on this site; the rest point to the Chinese edition until its translation batch lands.

How it fails in the field

  • Swapping the kernel without doing isolation, measuring jitter similar to the stock kernel, then drawing the wrong conclusion that 'the patch is useless'
  • Reporting averages without P99 or max, so the real problem never gets localized

You have learned enough when

You can produce three sets of cyclictest data on an ARM board, explain the reasons for the differences between each pair, and judge whether 'this machine can sustain 1 kHz'.

The Chinese edition carries the full topic page for Real-time systems & PREEMPT_RT, including the complete outline, depth requirements per career track, and further derivations: robotguide.dev/k/D4/.