RobotGuide
Math foundations/From Slope to Gradient: The First Line of Every Robot Learning Algorithm

Foundations T2 / 27 · From High School

From Slope to Gradient: The First Line of Every Robot Learning Algorithm

You already know that the derivative of f(x) = x² is 2x. Generalize the «one-dimensional slope» to many dimensions and you get the most frequent symbol in robot learning algorithms: ∇.

≈ 25 min1 theorems3 proof steps3 self-test questions

One-line takeaway

The gradient is the vector packing «one derivative per variable». Walking against it decreases the function fastest — not a mnemonic, but a one-line corollary of the cosine formula.

Animated · drop the ball on a slope and drag the learning rate: too small crawls, too big overshoots, about 0.25 is just right

Every learning algorithm in /fund/ (policy gradient, TD, PPO) has updates shaped like θ ← θ + α∇J. Understand this lesson and the most intimidating part of those formulas is dismantled first.

Why this tool exists

Tuning a robot's parameters: change one parameter — does performance improve or worsen, and by how much? That is just differentiation.

Once there are many parameters, «which way to adjust» becomes a direction question. The answer is called the gradient.

How far each step goes (the learning rate) directly decides whether training works at all: too slow won't get there in time, too large overshoots.

Definitions

Partial derivatives and the gradient

The gradient of f(x, y) at (a, b) is written ∇f = (∂f/∂x, ∂f/∂y): differentiate with respect to x holding y constant, and with respect to y holding x constant, then stack the two results into a vector.

Why define it this way«Partial derivative» is not a new operation — it is the differentiation you already know, applied to one variable at a time. For example, f = x² + 2y² gives ∂f/∂x = 2x and ∂f/∂y = 4y.

Theorems and proofs

Theorem (direction of steepest descent)

The rate of change along a unit direction u is ∇f·u = |∇f|·cosφ (φ is the angle between u and the gradient). At φ=180° (the anti-gradient direction) the descent is fastest.

Proof

Let g(t) = f(a + t·cosφ, b + t·sinφ), the function value after walking t steps along direction u.

g'(0) = ∂f/∂x·cosφ + ∂f/∂y·sinφ (cosφ and sinφ are constants when differentiating in t) — which is exactly ∇f·u.

By the geometric definition of the dot product, ∇f·u = |∇f||u|cosφ = |∇f|cosφ, so φ=0 is the maximum and φ=π the minimum (−|∇f|).

NoteThe legitimacy of «gradient descent» is exactly this line. Machine learning's θ ← θ − η∇J translates to: walk a distance η along the anti-gradient.

Work it by hand

f(x, y) = x² + 2y². Starting at (2, 4) with learning rate η=0.1, how much does f drop in one step?

  1. ∇f(2, 4) = (2x, 4y) = (4, 16).
  2. New point = (2 − 0.1×4, 4 − 0.1×16) = (1.6, 2.4).
  3. f was 4 + 32 = 36; now it is 1.6² + 2×2.4² = 2.56 + 11.52 = 14.08.
Answer

One step from 36 to 14.08 — a 61% drop. Right direction, steady steps, and it works.

Run the numbers

Where the numbers come from

Same f, same starting point (2,4), three learning rates actually run (the generator computes for real; stop criterion f<0.000001):

  • η=0.1: reached in 35 steps (stable but slow).
  • η=0.25: reached in 11 steps (the y direction lands in one step — the fastest).
  • η=0.49: reached in 212 steps (the y direction oscillates back and forth — slow to converge).
  • η=0.6: the y direction diverges; after 10 steps f has grown from 36 to 2.68e+4 — the mathematical body of «the learning rate was too large and training blew up».

Where this shows up

The object hereThe counterpart in RLUsed in
∇fThe policy gradient ∇J(θ)used in →
Learning rate ηLearning rate α / step sizeused in →
Direction of steepest descentThe policy-update direction of gradient ascentused in →

Boundaries: where this article stops

This lesson stops at first-order methods for smooth functions. Why η=0.25 lands the y direction in one step (the optimal step size for a 1D quadratic is 1/(2λ)), why oscillation happens, and the pits of nonconvex functions — see /math/sa/ and /fund/debug/.

Self-test: 3 questions

All hand-computation and «find the flaw in this proof» questions — nothing to memorize. Stuck? Go back to the section.

01What is the gradient of f(x,y)=x²+2y² at (1, 0)?

(2, 0) — the y term is 4y, which is 0 at y=0. The gradient points along +x.

02Use the |1−ηλ| accounting to explain: why does η=0.6 diverge in the y direction?

Each step is y ← (1−4η)y; at η=0.6 the coefficient is −1.4, |−1.4|>1, so it walks ever farther. The convergence condition |1−4η|<1 is η<0.5.

03What is the relation between the directions of two consecutive gradient-descent steps?

On a quadratic they are nearly perpendicular — the previous step «used up» the gradient direction, and the new gradient is mostly the perpendicular component. Hand-compute f=x² from any point with the optimal step size and you will see it (one step to the target, gradient becomes 0).

Up next

The full series · 27 articles

T1The Matrix Is a Ledger for Rotation≈ 25 minT2From Slope to Gradient: The First Line of Every Robot Learning Algorithm≈ 25 minT3Vectors and forces: the physical reading of the dot productAbout 20 minT4Two Rulers and a Better Rulerabout 25 minT5The Ledger of One Motor Revolution: Radians, Speed, and Powerabout 20 minT6Complex multiplication is rotation: the one-dimensional version of quaternionsAbout 20 minT7Bases and coordinates: why robots put everything in matrices45 minT8Matrix Multiplication: Chaining Transformations, Not a Table OperationAbout 40 minT9The Jacobian: Rotate a Joint One Degree — How Far Does the End Effector Move?About 45 minT10Eigenvectors: The Directions a Matrix Dare Not Touchabout 45 minT11SVD and the Pseudoinverse: When There Is No Exact Solution, Find the Nearest Oneabout 50 minT12Taylor expansion: putting a straight line to work in place of a curveAbout 40 minT13Probability spaces: why «random» needs axioms50 minT14Expectation: A Rigorous Construction of «Average»About 55 minT15Conditional Expectation: What the Vertical Bar Really Means≈ 60 minT16Four inequalities: the quantitative answer to «how many samples are enough»45 minT17Norms and fixed points: where «it always converges» comes from55 minT18Markov Chains: When the Time Average Equals the Ensemble Averageabout 55 minT19The Mathematical Construction of the MDP: What One Axiom Is Worthabout 50 minT20Dynamic Programming: The Accounting for Two Iterations≈ 50 minT21Optimization: why constraints cannot rely on penalties aloneAbout 55 minT22Stochastic Approximation: Every Term in the Learning-Rate Formulaabout 55 minT23The Mathematics of Contact: Hertz Theoryabout 45 minT24Friction and the Friction Cone: The Geometry of Constraintsabout 40 minT25Force Closure and Grasp Metrics: ε Is a Linear ProgramAbout 50 minT26Contact Kinematics: Why Rolling Is Not «Sliding Along»≈ 45 minT27Slip detection: from «I felt it» to «I computed it»About 50 min