Foundations T11 / 27 · Linear Algebra & Geometry
SVD and the Pseudoinverse: When There Is No Exact Solution, Find the Nearest One
Any matrix = rotate → stretch along orthogonal axes → rotate again. When the equations have no solution (overdetermined), least squares gives the nearest solution; when a matrix is singular (ill-conditioned), the pseudoinverse gives the most robust one. Robot IK and calibration do these two things every day.
One-line takeaway
When sensor readings outnumber unknown parameters (calibration: dozens of points for 4 parameters) the system is overdetermined and, with high probability, has no exact solution. When a manipulator reaches a singular configuration, the Jacobian is not invertible — the system has no solution. Mathematics prepares one answer for both predicaments: stop asking for the exact solution; ask for the nearest one, and use the SVD to define nearest with complete precision.
Animated · a unit circle stretched into an ellipse — the singular values are the major and minor axes; no matrix escapes these three steps
This is the most engineering-flavored of the six articles: numerical inverse kinematics, hand-eye calibration, point cloud registration (the core step of ICP), least-squares fitting — underneath they are all the same thing: SVD + pseudoinverse. Master this article and you know what you are computing with every inversion you do in ROS.
Why this tool exists
The prerequisites are just two: matrix multiplication (T8) and principal axes of symmetric matrices (T10's spectral theorem). The idea of the SVD fits in one sentence: for an asymmetric matrix, first use one rotation to turn it symmetric, then read off the principal axes with T10's tools.
Definitions
Definition 5.1 Singular value decomposition (SVD)
Any m×n matrix A can be written A = UΣVᵀ: V and U are orthogonal matrices (rotations/reflections), Σ is a diagonal matrix (stretching along axes), and the diagonal entries σ₁ ≥ σ₂ ≥ … ≥ 0 are called the singular values. Geometric reading: any linear map = rotate to align → stretch along axes → rotate to target, no exceptions.
Definition 5.2 Pseudoinverse A⁺
Invert the nonzero singular values of Σ, keep the zero ones at 0, and reassemble in the structure of A = UΣVᵀ; the result is called the pseudoinverse of A, A⁺ = VΣ⁺Uᵀ. If A is invertible, A⁺ = A−1; if A is not invertible, it is the best stand-in.
Theorems and proofs
Theorem 5.1 SVD existence (constructive 2×2 proof)
Any 2×2 real matrix A has a decomposition A = UΣVᵀ.
Proof
Look at the symmetrized matrix AᵀA: it is real symmetric, so by the spectral theorem (T10 Theorem 4.2) it has an orthonormal eigenbasis and can be written V diag(λ₁, λ₂) Vᵀ, with λᵢ ≥ 0 (because xᵀAᵀAx = ‖Ax‖² ≥ 0).
Set σᵢ = √λᵢ ≥ 0. Define the columns of U: uᵢ = (1/σᵢ) A vᵢ (for those with σᵢ > 0). Check pairwise: uᵢᵀuⱼ = (1/σᵢσⱼ) vᵢᵀAᵀAvⱼ = (λⱼ/σᵢσⱼ) vᵢᵀvⱼ = δᵢⱼ — the columns of U are orthonormal.
Moreover Avᵢ = σᵢuᵢ holds for every eigendirection; assembling into matrix form gives A = UΣVᵀ. ∎
NoteSVD = first use AᵀA to turn the asymmetric A into a symmetric problem, then apply the spectral theorem. All matrices share this structure — T10's principal axes generalize from symmetric matrices to all matrices.
Theorem 5.2 Least squares: finding the nearest when there is no solution
When Ax = b has no exact solution (m > n or A ill-conditioned), the solution minimizing ‖Ax − b‖² is x = (AᵀA)-1Aᵀb (when A has full column rank), i.e. A⁺b.
Proof
Completing the square (pure algebra, no calculus): ‖Ax − b‖² = (Ax − b)ᵀ(Ax − b) = xᵀAᵀAx − 2xᵀAᵀb + bᵀb.
Add and subtract x₀ᵀAᵀAx₀ (x₀ = (AᵀA)-1Aᵀb) and rearrange into perfect square + constant: = (x − x₀)ᵀAᵀA(x − x₀) + [bᵀb − x₀ᵀAᵀAx₀].
The first term ≥ 0 (AᵀA is positive semidefinite), the second does not depend on x. The minimum is attained at x = x₀. ∎
NoteThe whole proof uses only completing the square — middle-school algebra. Least squares is not mysterious: expand the squared error, and what remains is completing the square. It agrees with the calculus route of a minimum where the gradient vanishes, but it does not rely on calculus.
Theorem 5.3 Ill-conditioning and the truncated pseudoinverse
When the singular values are widely separated (σ₁/σ_min = κ is the condition number), inversion amplifies perturbations of b by κ; truncating singular values below a threshold to 0 (using A⁺ instead of inversion) brings the error back within the information content of the retained directions.
Proof
A relative perturbation δ of b propagates to x through inversion: from x = A−1b, perturbations on both sides satisfy ‖δx‖/‖x‖ ≤ κ · ‖δb‖/‖b‖ (two norm-inequality steps: ‖δx‖ ≤ ‖A−1‖‖δb‖, ‖δb‖ ≤ ‖A‖‖x‖; dividing gives κ = ‖A‖‖A−1‖ = σ₁/σ_min).
The truncated pseudoinverse sets small-singular-value directions straight to 0: the information in those directions is deliberately discarded, in exchange for the whole solution not being amplified by noise in the σ_min direction. At κ = 10⁸, machine-precision-level input perturbations are amplified until the solution is entirely untrustworthy; after truncation the error returns to the level of everything except the discarded directions. ∎
NoteLarge condition number = do not invert head-on is the number-one discipline of numerical engineering. Near a singular configuration the manipulator IK has det J → 0 (T9), equivalent to a condition-number blow-up — practice does not invert more precisely; it uses damped least squares / truncated pseudoinverse, giving up a little accuracy rather than diverging.
Work it by hand
A = [[1, 0], [0, 0]] (the second equation got lost — say the second sensor broke), b = (3, 7)ᵀ. Find the least-squares solution.
- An exact solution would need x₂ making the second equation 0·x₂ = 7 hold — impossible; the system has no solution.
- The SVD settles it in one step: A = UΣVᵀ, U = V = I, σ₁ = 1, σ₂ = 0. The pseudoinverse A⁺ = [[1, 0], [0, 0]].
- x = A⁺b = (1×3 + 0×7, 0×3 + 0×7) = (3, 0).
- Check: ‖Ax − b‖² = ‖(3, 0) − (3, 7)‖² = 49. Any other solution is only larger — the first equation is satisfied exactly (residual 0), and the residual 7 of the second equation is the intrinsic cost of a broken sensor, removable by no x.
x = (3, 0): the broken equation is cleanly discarded (weight 0 along the σ₂ = 0 direction) and the good equation satisfied exactly. This is the pseudoinverse's character — it does not coddle bad data, and it does not waste good data.
Run the numbers
Where the numbers come from
Actually run at generation time (constructing a matrix with condition number κ = 10⁸, adding a relative perturbation of 10-15 at the input):
- Direct inversion: solution error 1.49e-9 — amplified by the condition number, consistent with the theoretical prediction κ·(machine precision) ~ 10-7
- Truncated pseudoinverse (dropping the σ₂ direction): error 4.00e-1 — back down to machine-precision level
- Same data, same equations, different inversion method, credibility differs by 7 orders of magnitude. This is why IK libraries default to damped/pseudoinverse instead of bare inversion
Where this shows up
| The object here | The counterpart in RL | Used in |
|---|---|---|
| Inverse kinematics | J⁺ or damped least squares for joint velocities | used in → |
| Hand-eye calibration | The least-squares solution of an overdetermined system | used in → |
| Point cloud registration | The core of each ICP step = SVD for the optimal rotation | used in → |
Boundaries: where this article stops
Where this article stops: the full computational algorithm for the SVD (Golub-Kahan iteration) is not covered — in engineering you always call a numerical library; the full theory of rank and null space, and the relation between SVD and PCA, are only touched on. To be honest: the proofs here cover 2×2; the n-dimensional spectral theorem cites T10's result without re-proving it.
Self-test: 4 questions
All hand-computation and «find the flaw in this proof» questions — nothing to memorize. Stuck? Go back to the section.
01What problem is least squares solving?
When the system has no exact solution, find the x minimizing the squared residual ‖Ax−b‖². Nearest is defined by Euclidean distance — because with isotropic residuals it corresponds to maximum likelihood (under Gaussian noise).
02What does a large condition number κ mean?
Tiny input perturbations get amplified κ-fold into the output. At κ = 10⁸, even machine-precision (10-16) perturbations are enough to destroy the solution's credibility — the right move then is not more precise inversion but truncation or damping.
03Pseudoinverse versus inverse: what is the difference?
They coincide when A is invertible. When singular/ill-conditioned, the pseudoinverse zeros out the directions of zero (or tiny) singular values: it discards the directions with no information (or information drowned in noise) and preserves the solution in all other directions.
04Near a singular configuration, what is the IK library actually doing?
Not solving J−1 head-on, but J^T(JJ^T + λI)-1 (damped) or a truncated pseudoinverse — with T9's det J → 0 and this article's κ blow-up, you can fully explain why that line of code is written this way.