Gradient Descent

Gradient descent is how most machine learning models "learn": start with a bad guess, measure how wrong it is, and nudge the parameters in whichever direction reduces the error the fastest. Repeat a few dozen times and the guess stops being bad.

The setup

We're fitting a line, y = m·x + b, to a scatter of noisy points. The error for one point is how far the line misses it; the total error (the loss) is the average of the squared misses across every point:

loss(m, b) = (1/n) · Σ (m·xᵢ + b − yᵢ)²

Squaring the miss does two things: it makes big misses count more than small ones, and it turns the loss into a smooth bowl shape with a single lowest point — which is exactly what makes "always step downhill" work.

The step

At each step, we compute the slope of that bowl with respect to m and b (the gradient), then move a little in the opposite direction — downhill:

m ← m − learning_rate · ∂loss/∂m
b ← b − learning_rate · ∂loss/∂b

The learning rate controls how big each step is. Too small and it crawls; too large and it can overshoot the bottom of the bowl and bounce around, or even diverge. The animation uses a fixed rate small enough to converge smoothly.

Press play and watch the line rotate and slide into place over about 80 steps, or step through it one update at a time.