In the gradient descent update rule:

w ← w - learning_rate × gradient

the learning_rate (often written η or lr) is a small positive number that controls how big each step is. In the simplest form of gradient descent it stays constant for the whole training run, and you pick it before training starts.

Choosing the right learning rate is one of the most important (and annoying) practical decisions in machine learning.

Too small: slow but safe

weight loss the bottom (the goal) start each loop is one tiny step
A small learning rate takes many tiny steps, drawn here as a red coil winding down the loss curve, one loop per step. It is safe and steady, but reaching the bottom can take a great many iterations.

If the learning rate is very small, each step moves only a tiny distance down the loss curve. Training will eventually converge, but it can take thousands of iterations to get there. In practice, for large models, this means hours or days of extra computation.

Too large: oscillation and divergence

If the learning rate is too large, each step overshoots the minimum. Instead of landing near the bottom of the curve, you land on the other side, and the next step overshoots back. The weight oscillates back and forth around the minimum, never actually reaching it. With an extreme learning rate, the loss can actually increase with each step (divergence).

Interactive demo: compare learning rates

The animation below runs gradient descent on the same dataset (y = 2x + 1) with three different learning rates. Watch how each one traces a different path on the MSE loss curve.

Three simultaneous runs: LR = 0.005 (slow) · LR = 0.05 (good) · LR = 0.125 (oscillating)

What to look for

In the slow panel (LR = 0.005), each step is tiny. The trajectory creeps along the bottom of the curve. It will eventually converge but takes many iterations.

In the good panel (LR = 0.05), steps are larger. The trajectory moves efficiently towards the minimum in far fewer iterations.

In the oscillating panel (LR = 0.125), steps overshoot the minimum. The trajectory bounces back and forth: weight = 0 overshoots to weight ≈ 4, then back, then back again. The weight never settles.

Adaptive learning rates

In practice, manually picking a fixed learning rate is hard. Modern optimisers like Adam, RMSProp, and AdaGrad adapt the learning rate automatically: they start with a larger step and shrink it as they approach the minimum, or they use different rates for different parameters. But even with these methods, you still need to provide an initial learning rate to start from.

Next: Overfitting and Generalisation: what "good enough" really means, why a model can ace its training data and still fail in the real world, and how held-back data keeps it honest.