RobotGuide

Systems & software · topic D3 · 50 hours

Linux systems programming

The robot's host computer is a Linux box. Scheduling, memory, I/O, interrupts — none of it can be bypassed.

Core stage · Intermediate
Algorithm · Aware System software · Proficient Embedded & controls · Hands-on

Tools
Tools

What to learn

  • Processes/threads/scheduling policies, priorities and CPU affinity
  • Virtual memory, page errors, and how page faults relate to mlock
  • I/O models: blocking, non-blocking, epoll; device files and ioctl
  • Diagnostic tools: what top, perf, strace, ftrace, and dmesg each reveal

The lesson · plain words

Nine out of ten robots run Linux. The divide between an application-layer engineer and a system-layer engineer is whether you understand the 'stuff underneath': processes, file descriptors, scheduling.

This page covers the three most valuable things: processes and threads, I/O models, and why your program 'gets stuck'.

Blocking I/O: why programs stall

Four camera streams plus one lidar, each blocking-read: four threads each stuck on their own, and the waiting time piles up serially. Look at the Gantt chart on the left — orange waiting occupies the vast majority, and green real computation is pitifully small.

Switch to the epoll mode: one thread's waiting serves all connections, processing only when data is ready. Waits overlap, handling is compact, and the same hardware multiplies throughput several times over. This is the principle behind nginx/redis, and behind ROS 2's executor event loop.

Interactive figure · drag it and see

What this figure shows: a Gantt-style comparison of four threads doing blocking read() — orange = stuck waiting, green = actually computing, waits piling up serially — versus a single thread + epoll where one epoll_wait serves all connections and the waits overlap. Takeaway: event-driven I/O concentrates the waiting so the CPU only works when data is ready; waits overlap, handling is tight, and the same hardware multiplies its throughput.

Processes, threads, and scheduling

Processes have independent address spaces (a crash does not take neighbors down); threads share memory (fast but they can trample each other). The robot norm: critical processes run isolated, threads inside a process, shared data through lock-free queues or mutexes.

The scheduler decides 'who uses the CPU next'. Ordinary programs do not care; real-time programs must hold the scheduler's leash: SCHED_FIFO/RR to give critical threads priority, cgroups to fence off cores (isolcpus), taskset to pin cores. These are the foundations of D4's real-time work.

The system toolbox you must master

ps/top/htop to see who is eating CPU; strace to see which syscall a program is stuck in; lsof for file and port occupancy; journalctl for service logs; cgroup/systemd for resources and services. One hour each, useful for life.

The filesystem deserves understanding too: frantic logging wears out SD cards (the number-one Raspberry Pi death); /tmp is a RAM disk; on embedded boards ext4 needs its journaling options configured. Storage is the most fragile component on a robot.

How deep to go

You can write an epoll TCP server (within 50 lines); explain why a high %wa in top means waiting on I/O; and take a program from 'stuck' down to 'stuck in which syscall'.

Interview favorites: process vs thread (address space and scheduling unit), the differences among select/poll/epoll (polling vs event callbacks), and how zombie processes arise and get reaped.

Three questions — do you really get it?

1. The core advantage of single-threaded epoll over thread-per-connection blocking reads?

  • AShorter code
  • BNo kernel needed
  • COverlapping waits: one wait serves all connections, fewer context switches
Show answer

C. Event-driven gathers all the 'waiting' together; the CPU only works when data is ready.

2. A very high %wa (iowait) in top means?

  • AThe CPU is waiting on disk/network I/O — the bottleneck is storage
  • BNot enough CPU power
  • CA memory leak
Show answer

A. iowait means 'wants to work but the data has not arrived'. Optimize the I/O (better storage, async), not the CPU.

3. To keep a critical real-time thread from being squeezed out by ordinary processes, use?

  • AThe SCHED_FIFO real-time scheduling policy plus core isolation
  • BA lower nice value
  • CA few more threads
Show answer

A. Real-time policies let the thread run first when ready; pinning plus isolation physically separates the interference sources.

Keep going deeper

Links without a marker stay on this site; the rest point to the Chinese edition until its translation batch lands.

How it fails in the field

  • When things break, only knowing how to reboot — never reading dmesg or perf, so the same fault keeps recurring
  • A process killed by the OOM killer mistaken for a program crash, because nobody looked at the kernel log

You have learned enough when

You can root-cause an 'occasional 200 ms stall' and produce a chain of evidence, and explain what problem mlockall solves.

The Chinese edition carries the full topic page for Linux systems programming, including the complete outline, depth requirements per career track, and further derivations: robotguide.dev/k/D3/.