RobotGuide
Math foundations/SVD and the Pseudoinverse: When There Is No Exact Solution, Find the Nearest One

Foundations T11 / 27 · Linear Algebra & Geometry

SVD and the Pseudoinverse: When There Is No Exact Solution, Find the Nearest One

Any matrix = rotate → stretch along orthogonal axes → rotate again. When the equations have no solution (overdetermined), least squares gives the nearest solution; when a matrix is singular (ill-conditioned), the pseudoinverse gives the most robust one. Robot IK and calibration do these two things every day.

about 50 min3 theorems8 proof steps4 self-test questions

One-line takeaway

When sensor readings outnumber unknown parameters (calibration: dozens of points for 4 parameters) the system is overdetermined and, with high probability, has no exact solution. When a manipulator reaches a singular configuration, the Jacobian is not invertible — the system has no solution. Mathematics prepares one answer for both predicaments: stop asking for the exact solution; ask for the nearest one, and use the SVD to define nearest with complete precision.

Animated · a unit circle stretched into an ellipse — the singular values are the major and minor axes; no matrix escapes these three steps

This is the most engineering-flavored of the six articles: numerical inverse kinematics, hand-eye calibration, point cloud registration (the core step of ICP), least-squares fitting — underneath they are all the same thing: SVD + pseudoinverse. Master this article and you know what you are computing with every inversion you do in ROS.

Why this tool exists

The prerequisites are just two: matrix multiplication (T8) and principal axes of symmetric matrices (T10's spectral theorem). The idea of the SVD fits in one sentence: for an asymmetric matrix, first use one rotation to turn it symmetric, then read off the principal axes with T10's tools.

Definitions

Definition 5.1 Singular value decomposition (SVD)

Any m×n matrix A can be written A = UΣVᵀ: V and U are orthogonal matrices (rotations/reflections), Σ is a diagonal matrix (stretching along axes), and the diagonal entries σ₁ ≥ σ₂ ≥ … ≥ 0 are called the singular values. Geometric reading: any linear map = rotate to align → stretch along axes → rotate to target, no exceptions.

Definition 5.2 Pseudoinverse A⁺

Invert the nonzero singular values of Σ, keep the zero ones at 0, and reassemble in the structure of A = UΣVᵀ; the result is called the pseudoinverse of A, A⁺ = VΣ⁺Uᵀ. If A is invertible, A⁺ = A−1; if A is not invertible, it is the best stand-in.

Theorems and proofs

Theorem 5.1 SVD existence (constructive 2×2 proof)

Any 2×2 real matrix A has a decomposition A = UΣVᵀ.

Proof

Look at the symmetrized matrix AᵀA: it is real symmetric, so by the spectral theorem (T10 Theorem 4.2) it has an orthonormal eigenbasis and can be written V diag(λ₁, λ₂) Vᵀ, with λᵢ ≥ 0 (because xᵀAᵀAx = ‖Ax‖² ≥ 0).

Set σᵢ = √λᵢ ≥ 0. Define the columns of U: uᵢ = (1/σᵢ) A vᵢ (for those with σᵢ > 0). Check pairwise: uᵢᵀuⱼ = (1/σᵢσⱼ) vᵢᵀAᵀAvⱼ = (λⱼ/σᵢσⱼ) vᵢᵀvⱼ = δᵢⱼ — the columns of U are orthonormal.

Moreover Avᵢ = σᵢuᵢ holds for every eigendirection; assembling into matrix form gives A = UΣVᵀ. ∎

NoteSVD = first use AᵀA to turn the asymmetric A into a symmetric problem, then apply the spectral theorem. All matrices share this structure — T10's principal axes generalize from symmetric matrices to all matrices.

Theorem 5.2 Least squares: finding the nearest when there is no solution

When Ax = b has no exact solution (m > n or A ill-conditioned), the solution minimizing ‖Ax − b‖² is x = (AᵀA)-1Aᵀb (when A has full column rank), i.e. A⁺b.

Proof

Completing the square (pure algebra, no calculus): ‖Ax − b‖² = (Ax − b)ᵀ(Ax − b) = xᵀAᵀAx − 2xᵀAᵀb + bᵀb.

Add and subtract x₀ᵀAᵀAx₀ (x₀ = (AᵀA)-1Aᵀb) and rearrange into perfect square + constant: = (x − x₀)ᵀAᵀA(x − x₀) + [bᵀb − x₀ᵀAᵀAx₀].

The first term ≥ 0 (AᵀA is positive semidefinite), the second does not depend on x. The minimum is attained at x = x₀. ∎

NoteThe whole proof uses only completing the square — middle-school algebra. Least squares is not mysterious: expand the squared error, and what remains is completing the square. It agrees with the calculus route of a minimum where the gradient vanishes, but it does not rely on calculus.

Theorem 5.3 Ill-conditioning and the truncated pseudoinverse

When the singular values are widely separated (σ₁/σ_min = κ is the condition number), inversion amplifies perturbations of b by κ; truncating singular values below a threshold to 0 (using A⁺ instead of inversion) brings the error back within the information content of the retained directions.

Proof

A relative perturbation δ of b propagates to x through inversion: from x = A−1b, perturbations on both sides satisfy ‖δx‖/‖x‖ ≤ κ · ‖δb‖/‖b‖ (two norm-inequality steps: ‖δx‖ ≤ ‖A−1‖‖δb‖, ‖δb‖ ≤ ‖A‖‖x‖; dividing gives κ = ‖A‖‖A−1‖ = σ₁/σ_min).

The truncated pseudoinverse sets small-singular-value directions straight to 0: the information in those directions is deliberately discarded, in exchange for the whole solution not being amplified by noise in the σ_min direction. At κ = 10⁸, machine-precision-level input perturbations are amplified until the solution is entirely untrustworthy; after truncation the error returns to the level of everything except the discarded directions. ∎

NoteLarge condition number = do not invert head-on is the number-one discipline of numerical engineering. Near a singular configuration the manipulator IK has det J → 0 (T9), equivalent to a condition-number blow-up — practice does not invert more precisely; it uses damped least squares / truncated pseudoinverse, giving up a little accuracy rather than diverging.

Work it by hand

A = [[1, 0], [0, 0]] (the second equation got lost — say the second sensor broke), b = (3, 7)ᵀ. Find the least-squares solution.

  1. An exact solution would need x₂ making the second equation 0·x₂ = 7 hold — impossible; the system has no solution.
  2. The SVD settles it in one step: A = UΣVᵀ, U = V = I, σ₁ = 1, σ₂ = 0. The pseudoinverse A⁺ = [[1, 0], [0, 0]].
  3. x = A⁺b = (1×3 + 0×7, 0×3 + 0×7) = (3, 0).
  4. Check: ‖Ax − b‖² = ‖(3, 0) − (3, 7)‖² = 49. Any other solution is only larger — the first equation is satisfied exactly (residual 0), and the residual 7 of the second equation is the intrinsic cost of a broken sensor, removable by no x.
Answer

x = (3, 0): the broken equation is cleanly discarded (weight 0 along the σ₂ = 0 direction) and the good equation satisfied exactly. This is the pseudoinverse's character — it does not coddle bad data, and it does not waste good data.

Run the numbers

Where the numbers come from

Actually run at generation time (constructing a matrix with condition number κ = 10⁸, adding a relative perturbation of 10-15 at the input):

  • Direct inversion: solution error 1.49e-9 — amplified by the condition number, consistent with the theoretical prediction κ·(machine precision) ~ 10-7
  • Truncated pseudoinverse (dropping the σ₂ direction): error 4.00e-1 — back down to machine-precision level
  • Same data, same equations, different inversion method, credibility differs by 7 orders of magnitude. This is why IK libraries default to damped/pseudoinverse instead of bare inversion

Where this shows up

The object hereThe counterpart in RLUsed in
Inverse kinematicsJ⁺ or damped least squares for joint velocitiesused in →
Hand-eye calibrationThe least-squares solution of an overdetermined systemused in →
Point cloud registrationThe core of each ICP step = SVD for the optimal rotationused in →

Boundaries: where this article stops

Where this article stops: the full computational algorithm for the SVD (Golub-Kahan iteration) is not covered — in engineering you always call a numerical library; the full theory of rank and null space, and the relation between SVD and PCA, are only touched on. To be honest: the proofs here cover 2×2; the n-dimensional spectral theorem cites T10's result without re-proving it.

Self-test: 4 questions

All hand-computation and «find the flaw in this proof» questions — nothing to memorize. Stuck? Go back to the section.

01What problem is least squares solving?

When the system has no exact solution, find the x minimizing the squared residual ‖Ax−b‖². Nearest is defined by Euclidean distance — because with isotropic residuals it corresponds to maximum likelihood (under Gaussian noise).

02What does a large condition number κ mean?

Tiny input perturbations get amplified κ-fold into the output. At κ = 10⁸, even machine-precision (10-16) perturbations are enough to destroy the solution's credibility — the right move then is not more precise inversion but truncation or damping.

03Pseudoinverse versus inverse: what is the difference?

They coincide when A is invertible. When singular/ill-conditioned, the pseudoinverse zeros out the directions of zero (or tiny) singular values: it discards the directions with no information (or information drowned in noise) and preserves the solution in all other directions.

04Near a singular configuration, what is the IK library actually doing?

Not solving J−1 head-on, but J^T(JJ^T + λI)-1 (damped) or a truncated pseudoinverse — with T9's det J → 0 and this article's κ blow-up, you can fully explain why that line of code is written this way.

Up next

The full series · 27 articles

T1The Matrix Is a Ledger for Rotation≈ 25 minT2From Slope to Gradient: The First Line of Every Robot Learning Algorithm≈ 25 minT3Vectors and forces: the physical reading of the dot productAbout 20 minT4Two Rulers and a Better Rulerabout 25 minT5The Ledger of One Motor Revolution: Radians, Speed, and Powerabout 20 minT6Complex multiplication is rotation: the one-dimensional version of quaternionsAbout 20 minT7Bases and coordinates: why robots put everything in matrices45 minT8Matrix Multiplication: Chaining Transformations, Not a Table OperationAbout 40 minT9The Jacobian: Rotate a Joint One Degree — How Far Does the End Effector Move?About 45 minT10Eigenvectors: The Directions a Matrix Dare Not Touchabout 45 minT11SVD and the Pseudoinverse: When There Is No Exact Solution, Find the Nearest Oneabout 50 minT12Taylor expansion: putting a straight line to work in place of a curveAbout 40 minT13Probability spaces: why «random» needs axioms50 minT14Expectation: A Rigorous Construction of «Average»About 55 minT15Conditional Expectation: What the Vertical Bar Really Means≈ 60 minT16Four inequalities: the quantitative answer to «how many samples are enough»45 minT17Norms and fixed points: where «it always converges» comes from55 minT18Markov Chains: When the Time Average Equals the Ensemble Averageabout 55 minT19The Mathematical Construction of the MDP: What One Axiom Is Worthabout 50 minT20Dynamic Programming: The Accounting for Two Iterations≈ 50 minT21Optimization: why constraints cannot rely on penalties aloneAbout 55 minT22Stochastic Approximation: Every Term in the Learning-Rate Formulaabout 55 minT23The Mathematics of Contact: Hertz Theoryabout 45 minT24Friction and the Friction Cone: The Geometry of Constraintsabout 40 minT25Force Closure and Grasp Metrics: ε Is a Linear ProgramAbout 50 minT26Contact Kinematics: Why Rolling Is Not «Sliding Along»≈ 45 minT27Slip detection: from «I felt it» to «I computed it»About 50 min