Perception & intelligence · topic C9 · 30 hours
Tactile sensing & dexterous manipulation
Vision cannot see the fingertip slipping or whether the cap is loose — the margin in fine manipulation lives in the hundred milliseconds after contact. The only mech-heavy topic in group C: without contact mechanics, tactile data is just a string of meaningless numbers.
What to learn
- Three sensing principles: piezoresistive (cheap, hysteresis), capacitive (dense arrays, crosstalk), magnetic (occlusion- and noise-resistant) — each one's transfer function and failure mode
- Two visuotactile routes: internal imaging to reconstruct contact geometry (normals → depth), and marker tracking to estimate the tangential field
- The «raw-signal trap» of touch: swap the sensor and you swap the language — the data is void. Why a unified representation encodes by sensing principle first, then maps into a shared fingertip space
- Data collection: why glove teleoperation and teleoperated dexterous hands can co-train (it depends on decoupling representation from hardware)
- Plugging into action models: tactile tokens as part of the policy observation, and how the ~0.15% parameter increment is accounted
- Evaluation: task success must be paired with tactile ablations (no-touch / raw-signal / unified-representation) to carry any information
The lesson · plain words
However good vision is, it has a physical blind spot: what happens after contact. Is the object slipping, how hard is it pressed, is it snagged — only the «skin» knows. Tactile sensing gives the machine a skin; dexterous hands give that skin fingers.
This page covers three things: how touch is measured, what makes dexterous hands hard, and why touch is the last puzzle piece of manipulation.
The tactile array: a patch of skin that reports pressure
The mainstream design is a small high-density array, mostly piezoresistive, capacitive, or magnetic. Each little cell (taxel) reports one pressure value — an 8×8 array is 64 numbers. Watch the heatmap below: drag the contact, change the contact shape, see how the pressure blob moves.
The green ring is the force centroid — judging «am I gripping straight, am I about to slip» is exactly what a dexterous hand reads off it. In industry, the truly common setup is not this «HD skin» but a fingertip-sized six-axis force sensor plus joint-current estimation — adequate and nearly indestructible.
Interactive figure · drag it and see
Dexterous hands: hard in mechanics, harder in control
The human hand has 27 degrees of freedom; lab dexterous hands usually 12–24. Mechanically, every added DOF brings another drive + transmission + sensor set — a tendon-driven finger must be strong, nimble, and light at once.
Control is crueler still: coordinating 12 DOFs is not 12 independent PIDs, it is the finger edition of B9's whole-body control. Recent breakthroughs come from RL trained in simulation (C6/G5) — training directly on hardware breaks too many robots.
What tasks does touch unlock
Insertion-class: with millimeter-level visual error, the final 1 mm is «felt in» via fingertip slip detection. Compliant objects: wiring, folding, snap-fastening — force-feedback loops beat visual loops by a wide margin. Dynamic grasping: catch-and-throw, quick workpiece swaps — all riding on 1 kHz force feedback.
The criterion is plain: the larger the share of «after contact» actions in a task, the more irreplaceable touch becomes. Pure palletizing is fine with vision; assembly and compliant manipulation cannot take a step without it.
A realistic path in
Do not start with a dexterous hand (expensive and fragile). Buy a cheap fingertip force sensor, mount it on a gripper, and use B8's impedance control for a «press 5 N without slipping» experiment — that single loop touches every element of tactile closed-loop control.
Level up to visuotactile sensors like DIGIT (a camera watching the fingertip deform): a few hundred units of currency, rich community material — the best learning toy for tactile perception. The real industrial opportunities sit in «tactile datasets» and «VLA fine-tuning with touch» — both new ground, few people there.
Three questions — do you really get it?
1. The most common industrial «touch» is actually?
- AAn 8×8 piezoresistive array
- BA temperature sensor
- CA six-axis force/torque sensor plus joint-current estimation
Show answer
C. Force sensors and current estimation are cheap, rugged, and adequate. The HD skin remains mostly a lab device.
2. What is the force centroid used for in grasp control?
- AComputing the object's color
- BJudging whether the grip is centered and whether slip is starting
- CMeasuring ambient temperature
Show answer
B. Centroid offset plus the tangential/normal force ratio is the classic slip-prediction signal.
3. Which task depends on touch the most?
- ABox palletizing
- BPath planning
- CThe final 1 mm of a peg-in-hole insertion
Show answer
C. Insertion finishes inside the vision blind zone, felt out via fingertip slip and force feedback. Palletizing mostly needs vision plus a torque cap.
Keep going deeper
Links without a marker stay on this site; the rest point to the Chinese edition until its translation batch lands.
How it fails in the field
- Treating touch as «a second camera» and stuffing it into the visual backbone, ignoring the kHz-vs-30Hz space-time scale gap
- Claiming the representation is universal after validating on a single hardware rig — cross-hardware reuse needs the «sensor model absent from the pretraining set» control experiment
- Reporting success rates without the ablation protocol — you cannot tell whether the gain came from touch or just from extra parameters
You have learned enough when
The Chinese edition carries the full topic page for Tactile sensing & dexterous manipulation, including the complete outline, depth requirements per career track, and further derivations: robotguide.dev/k/C9/.