Foundations T2 / 27 · From High School
From Slope to Gradient: The First Line of Every Robot Learning Algorithm
You already know that the derivative of f(x) = x² is 2x. Generalize the «one-dimensional slope» to many dimensions and you get the most frequent symbol in robot learning algorithms: ∇.
One-line takeaway
The gradient is the vector packing «one derivative per variable». Walking against it decreases the function fastest — not a mnemonic, but a one-line corollary of the cosine formula.
Animated · drop the ball on a slope and drag the learning rate: too small crawls, too big overshoots, about 0.25 is just right
Every learning algorithm in /fund/ (policy gradient, TD, PPO) has updates shaped like θ ← θ + α∇J. Understand this lesson and the most intimidating part of those formulas is dismantled first.
Why this tool exists
Tuning a robot's parameters: change one parameter — does performance improve or worsen, and by how much? That is just differentiation.
Once there are many parameters, «which way to adjust» becomes a direction question. The answer is called the gradient.
How far each step goes (the learning rate) directly decides whether training works at all: too slow won't get there in time, too large overshoots.
Definitions
Partial derivatives and the gradient
The gradient of f(x, y) at (a, b) is written ∇f = (∂f/∂x, ∂f/∂y): differentiate with respect to x holding y constant, and with respect to y holding x constant, then stack the two results into a vector.
Why define it this way«Partial derivative» is not a new operation — it is the differentiation you already know, applied to one variable at a time. For example, f = x² + 2y² gives ∂f/∂x = 2x and ∂f/∂y = 4y.
Theorems and proofs
Theorem (direction of steepest descent)
The rate of change along a unit direction u is ∇f·u = |∇f|·cosφ (φ is the angle between u and the gradient). At φ=180° (the anti-gradient direction) the descent is fastest.
Proof
Let g(t) = f(a + t·cosφ, b + t·sinφ), the function value after walking t steps along direction u.
g'(0) = ∂f/∂x·cosφ + ∂f/∂y·sinφ (cosφ and sinφ are constants when differentiating in t) — which is exactly ∇f·u.
By the geometric definition of the dot product, ∇f·u = |∇f||u|cosφ = |∇f|cosφ, so φ=0 is the maximum and φ=π the minimum (−|∇f|).
NoteThe legitimacy of «gradient descent» is exactly this line. Machine learning's θ ← θ − η∇J translates to: walk a distance η along the anti-gradient.
Work it by hand
f(x, y) = x² + 2y². Starting at (2, 4) with learning rate η=0.1, how much does f drop in one step?
- ∇f(2, 4) = (2x, 4y) = (4, 16).
- New point = (2 − 0.1×4, 4 − 0.1×16) = (1.6, 2.4).
- f was 4 + 32 = 36; now it is 1.6² + 2×2.4² = 2.56 + 11.52 = 14.08.
One step from 36 to 14.08 — a 61% drop. Right direction, steady steps, and it works.
Run the numbers
Where the numbers come from
Same f, same starting point (2,4), three learning rates actually run (the generator computes for real; stop criterion f<0.000001):
- η=0.1: reached in 35 steps (stable but slow).
- η=0.25: reached in 11 steps (the y direction lands in one step — the fastest).
- η=0.49: reached in 212 steps (the y direction oscillates back and forth — slow to converge).
- η=0.6: the y direction diverges; after 10 steps f has grown from 36 to 2.68e+4 — the mathematical body of «the learning rate was too large and training blew up».
Where this shows up
| The object here | The counterpart in RL | Used in |
|---|---|---|
| ∇f | The policy gradient ∇J(θ) | used in → |
| Learning rate η | Learning rate α / step size | used in → |
| Direction of steepest descent | The policy-update direction of gradient ascent | used in → |
Boundaries: where this article stops
This lesson stops at first-order methods for smooth functions. Why η=0.25 lands the y direction in one step (the optimal step size for a 1D quadratic is 1/(2λ)), why oscillation happens, and the pits of nonconvex functions — see /math/sa/ and /fund/debug/.
Self-test: 3 questions
All hand-computation and «find the flaw in this proof» questions — nothing to memorize. Stuck? Go back to the section.
01What is the gradient of f(x,y)=x²+2y² at (1, 0)?
(2, 0) — the y term is 4y, which is 0 at y=0. The gradient points along +x.
02Use the |1−ηλ| accounting to explain: why does η=0.6 diverge in the y direction?
Each step is y ← (1−4η)y; at η=0.6 the coefficient is −1.4, |−1.4|>1, so it walks ever farther. The convergence condition |1−4η|<1 is η<0.5.
03What is the relation between the directions of two consecutive gradient-descent steps?
On a quadratic they are nearly perpendicular — the previous step «used up» the gradient direction, and the new gradient is mostly the perpendicular component. Hand-compute f=x² from any point with the optimal step size and you will see it (one step to the target, gradient becomes 0).