RobotGuide

Tools & simulation · topic G10 · 30 hours

Deployment runtimes & edge inference

Training happens on an A100; deployment happens on an Orin. Between them sit quantization, operators, memory, and the latency budget. This layer decides whether a demo can become a product — it is also the line between «an algorithm engineer» and «an algorithm engineer who can ship».

Engineering stage · Hardcore
Algorithm · Proficient System software · Proficient Embedded & controls · Hands-on

Tools
Tools

What to learn

  • Export: torch.onnx.export's opset and dynamic axes; if the ONNX export fails, no TensorRT engine can be built
  • TensorRT: engines are bound to hardware/driver, serialized and then loaded; FP16 is basically lossless, INT8 needs a calibration set and per-layer sensitivity analysis
  • Inference integration: C++ TensorRT or ONNX Runtime, with GPU memory zero-copied into ROS messages (avoid the D2H→H2D double transfer)
  • Edge platforms: Jetson Orin power modes (15W/30W/MAXN) and clock locking, x86 + discrete GPU, the compute limits of the RK3588
  • The end-to-end latency budget: capture exposure + transfer + preprocessing + inference + postprocessing + communication — measure every segment, not just inference
  • Engineering practice: model versioning, canary rollout, rollback, field diagnosability (log the per-segment timing)

The lesson · plain words

The deployment runtime is where a trained model gets stuffed into the real machine: TensorRT/ONNX Runtime/quantization/scheduling. The 99% accuracy from training becomes, on hardware, purely a budget question: how many milliseconds per frame, how many watts, how much memory.

Quantization: the exchange rate between precision and speed

Watch the animation: the weight distribution gets «staircased» — INT8 has only 256 steps, error ~0.4%; INT4 is coarser but faster. The default setting for robot inference is INT8: accuracy loss usually <1%, speed 2–3×.

The two ways to quantize: PTQ (quantize directly after training — fast but may lose accuracy) and QAT (simulate quantization during training — better accuracy, more work). The deployment chain: export ONNX → TensorRT/edge inference engines consume the ONNX → output an INT8 engine file.

Interactive figure · drag it and see

What this figure shows: a quantization-bits selector (FP32 / INT8 / INT4) staircasing a weight distribution — the readout lists the number of steps, the relative quantization error, and inference time in ms per frame; fewer bits mean coarser steps, bigger error, and faster frames. Takeaway: robot inference defaults to INT8 (~0.4% error, 2–3× faster); fall back to FP16 or move to QAT if accuracy loss exceeds the budget.

The inference budget: how to do the latency math

Inside a 50 Hz control loop only a few milliseconds are left for perception — the inference budget is a hard constraint. The ledger has three columns: model compute (FLOPs), memory bandwidth (memory access is often the bottleneck, as E8 noted), and preprocessing/postprocessing (the CPU-side chunk that is routinely ignored).

Measurement always wins: benchmark the same model on the Orin in FP32 and INT8, and the numbers directly decide the selection. Do not trust the advertised TOPS — as D8 covered, effective compute comes discounted.

Scheduling: how inference and control coexist

Perception inference (tens of ms) and real-time control (1 kHz) fight over the CPU: push perception onto the big cores to run asynchronously, pin control to the small cores (D8's layering); lower the inference process's priority, give the control process SCHED_FIFO (D4). GPU sharing conflicts: keep the control loop off the GPU as much as possible and let the vision chain own it exclusively.

Memory management: preallocate the inference engine's workspace; move images into the GPU zero-copy (DMA buf); disable swap (D4's mlockall). With these three done, edge inference latency jitter can be squeezed under 5%.

How deep to go

Walk the full chain «ONNX export → TensorRT INT8 → benchmark»; run latency-distribution tests on the inference process (p50/p99); design a scheduling scheme where perception and control coexist. These three are the dividing line between «can train models» and «can put a model on a robot».

Three questions — do you really get it?

1. The default quantization level for robot inference?

  • AFP16
  • BINT2
  • CINT8 (2–3× speed, <1% accuracy loss)
Show answer

C. INT8 is the sweet spot of accuracy vs speed. If accuracy loss exceeds budget, fall back to FP16 or move up to QAT.

2. The hidden bulk of edge-inference latency is often?

  • AMatrix multiplication
  • BModel download
  • CCPU-side preprocessing/postprocessing
Show answer

C. Decoding, resize, and NMS all run on the CPU and often take half the latency. Quantization only saved the GPU side.

3. The key to perception inference and real-time control coexisting?

  • APin both to the big cores
  • BLayered scheduling: control gets real-time priority and pinned cores, perception runs asynchronously at lower priority
  • CDisable perception
Show answer

B. D8's fast/slow split plus D4's scheduling policy. Give the GPU exclusively to vision; keep control off the GPU.

Keep going deeper

Links without a marker stay on this site; the rest point to the Chinese edition until its translation batch lands.

How it fails in the field

  • Measured 5 ms on the server; on the Orin it becomes 25 ms — the power mode is 15W and the clocks are not locked
  • After INT8 quantization accuracy drops 40%; no per-layer analysis, quantization abandoned outright — in fact only the first layer was sensitive
  • Rebuilding the engine on every startup; at the live demo you wait 40 seconds just for startup

You have learned enough when

Export, quantize, and run a policy model on the target hardware at the set frequency, and produce an end-to-end latency breakdown table (ms and share per segment); you can locate which layer loses accuracy under quantization.

The Chinese edition carries the full topic page for Deployment runtimes & edge inference, including the complete outline, depth requirements per career track, and further derivations: robotguide.dev/k/G10/.