RobotGuide

Perception & intelligence · topic C9 · 30 hours

Tactile sensing & dexterous manipulation

Vision cannot see the fingertip slipping or whether the cap is loose — the margin in fine manipulation lives in the hundred milliseconds after contact. The only mech-heavy topic in group C: without contact mechanics, tactile data is just a string of meaningless numbers.

Core stage · Intermediate
Algorithm · Hands-on System software · Aware Embedded & controls · Aware Mechanical · Hands-on

What to learn

The lesson · plain words

However good vision is, it has a physical blind spot: what happens after contact. Is the object slipping, how hard is it pressed, is it snagged — only the «skin» knows. Tactile sensing gives the machine a skin; dexterous hands give that skin fingers.

This page covers three things: how touch is measured, what makes dexterous hands hard, and why touch is the last puzzle piece of manipulation.

The tactile array: a patch of skin that reports pressure

The mainstream design is a small high-density array, mostly piezoresistive, capacitive, or magnetic. Each little cell (taxel) reports one pressure value — an 8×8 array is 64 numbers. Watch the heatmap below: drag the contact, change the contact shape, see how the pressure blob moves.

The green ring is the force centroid — judging «am I gripping straight, am I about to slip» is exactly what a dexterous hand reads off it. In industry, the truly common setup is not this «HD skin» but a fingertip-sized six-axis force sensor plus joint-current estimation — adequate and nearly indestructible.

Interactive figure · drag it and see

What this figure shows: an 8×8 taxel array rendered as a pressure heatmap — drag the contact around and switch between flat, edge, and point contact shapes; the green ring marks the force centroid. Takeaway: each cell reports one pressure value (64 numbers for 8×8), and the centroid's offset plus the tangential/normal force ratio is the classic slip-prediction signal a dexterous hand relies on.

Dexterous hands: hard in mechanics, harder in control

The human hand has 27 degrees of freedom; lab dexterous hands usually 12–24. Mechanically, every added DOF brings another drive + transmission + sensor set — a tendon-driven finger must be strong, nimble, and light at once.

Control is crueler still: coordinating 12 DOFs is not 12 independent PIDs, it is the finger edition of B9's whole-body control. Recent breakthroughs come from RL trained in simulation (C6/G5) — training directly on hardware breaks too many robots.

What tasks does touch unlock

Insertion-class: with millimeter-level visual error, the final 1 mm is «felt in» via fingertip slip detection. Compliant objects: wiring, folding, snap-fastening — force-feedback loops beat visual loops by a wide margin. Dynamic grasping: catch-and-throw, quick workpiece swaps — all riding on 1 kHz force feedback.

The criterion is plain: the larger the share of «after contact» actions in a task, the more irreplaceable touch becomes. Pure palletizing is fine with vision; assembly and compliant manipulation cannot take a step without it.

A realistic path in

Do not start with a dexterous hand (expensive and fragile). Buy a cheap fingertip force sensor, mount it on a gripper, and use B8's impedance control for a «press 5 N without slipping» experiment — that single loop touches every element of tactile closed-loop control.

Level up to visuotactile sensors like DIGIT (a camera watching the fingertip deform): a few hundred units of currency, rich community material — the best learning toy for tactile perception. The real industrial opportunities sit in «tactile datasets» and «VLA fine-tuning with touch» — both new ground, few people there.

Three questions — do you really get it?

1. The most common industrial «touch» is actually?

  • AAn 8×8 piezoresistive array
  • BA temperature sensor
  • CA six-axis force/torque sensor plus joint-current estimation
Show answer

C. Force sensors and current estimation are cheap, rugged, and adequate. The HD skin remains mostly a lab device.

2. What is the force centroid used for in grasp control?

  • AComputing the object's color
  • BJudging whether the grip is centered and whether slip is starting
  • CMeasuring ambient temperature
Show answer

B. Centroid offset plus the tangential/normal force ratio is the classic slip-prediction signal.

3. Which task depends on touch the most?

  • ABox palletizing
  • BPath planning
  • CThe final 1 mm of a peg-in-hole insertion
Show answer

C. Insertion finishes inside the vision blind zone, felt out via fingertip slip and force feedback. Palletizing mostly needs vision plus a torque cap.

Keep going deeper

Links without a marker stay on this site; the rest point to the Chinese edition until its translation batch lands.

How it fails in the field

You have learned enough when

You can state the physical quantity and failure mode of each sensing principle, explain why the unified tactile representation is architected that way, and design a tactile ablation with a predicted ordering of the three conditions.

The Chinese edition carries the full topic page for Tactile sensing & dexterous manipulation, including the complete outline, depth requirements per career track, and further derivations: robotguide.dev/k/C9/.