Gradient Descent

We know that we need to find the right values for weight and bias. We know how to measure how wrong we are (the loss). Now the question is: how do we use the loss to find better values?

The answer is gradient descent. And here is something worth saying plainly: when people talk about "training a model", this is what they mean. Training is nothing more mysterious than the process of finding good values for the weights and biases.

The training loop

Gradient descent turns training into a simple loop. The steps below are numbered to match the diagram:

1. Set the initial weight and bias. We have already met this step: we simply start with some randomly chosen values. The line will fit badly at first, and that is fine.
2. Evaluate the model. Measure how wrong the current line is by computing its loss score, the MAE or MSE from the last part.
3. Is the model good enough? If the loss is small enough, stop. If not, carry on.
4. Tweak the weight and bias. Nudge them in a direction that lowers the loss, then go back to step 2. (Working out that direction is what the rest of this article is about.)
5. Success. Once the model is good enough, training is done.

Steps 2 to 4 are the part that repeats. Each pass through them is one iteration. (You will also meet the word epoch in ML libraries: an epoch is one full pass through the entire training dataset. In our tiny examples every iteration uses all the data, so the two happen to be the same thing here, but on big datasets a single epoch is made up of many iterations.) Think of it like training for a marathon: you don't wake up one morning and run 26 miles. You practice every day, get a little better, and gradually build towards your goal.

One label on the diagram deserves a note before we move on: the arrow that loops back is marked backpropagation. That is the name of the technique used to work out which way to tweak each weight and bias when a model has many layers of them. For our single-line model the tweak is simple enough to compute directly, as you are about to see, and we will come back to what "propagating backwards" means when we get to networks of neurons at the end of the series.

How do we know which direction to tweak?

This is the clever part. The loss curve is shaped like a U (a bowl): the loss is high out on the sides and lowest at the single best weight and bias, right at the bottom. Gradient descent wants to reach that bottom. But how does the algorithm know which direction is "downhill" from where it currently sits?

The answer is the gradient, the slope of the loss curve at the current point.

To turn "which way is downhill" into an actual number, we use a partial derivative of the loss with respect to the weight, written ∂MSE/∂w. The word partial means we wiggle only the weight and hold the bias still, then ask: if we nudge the weight up by a tiny amount, does the loss rise or fall, and how fast? That is exactly the slope of the loss curve in the weight direction, and its sign points us the way that would lower the loss.

Here is the formula for MSE:

∂MSE/∂w = (2/n) × Σ (y_pred - y_actual) × x

The key is what that sum is doing. The partial derivative takes account of every individual data point and tells us how each one is affected by changing the weight. For each point it takes the error (y_pred - y_actual), multiplies it by that point's feature x, and averages over all n points. Points that are badly fitted, or that sit at a large x, pull harder on the result. The single number that comes out is the overall slope: which way, and how steeply, the loss changes as we change the weight.

That one number is all we need to decide the move:

• If it is positive, nudging the weight up would raise the loss, so we go the other way and decrease the weight.
• If it is negative, nudging the weight up would lower the loss, so we increase the weight.
• If it is near zero, the curve is flat, we are at the bottom, and there is nowhere better to go.

So we always step in the direction opposite to the gradient. The actual amount we add to or subtract from each value is that gradient scaled down by a small learning rate:

w_new = w - learning_rate × ∂MSE/∂w
b_new = b - learning_rate × ∂MSE/∂b

A worked step

That formula is a lot to take in, so let us run it on real numbers. We will use the y = 2x + 1 data from the last part and start from a poor guess of weight = 1 (with the bias fixed at 1). For each point we work out the prediction 1·x + 1, its error, and the error times x:

Now we add up that last column and apply the 2/n factor (here n = 4):

Σ (error × x) = -1 - 4 - 9 - 16 = -30
∂MSE/∂w      = (2/4) × (-30)   = -15

The gradient is -15. It is negative, so increasing the weight would lower the loss, and we should move the weight up. The size of the move is the gradient scaled by the learning rate, say 0.05:

Δw    = - learning_rate × ∂MSE/∂w = - 0.05 × (-15) = +0.75
w_new = 1 + 0.75 = 1.75

One step has carried the weight from 1 to 1.75, well on its way to the true value of 2, and the loss drops. Feed the new weight back into the same formula, repeat, and gradient descent walks the weight down to the bottom of the U. (The bias is updated the same way each step, using its own partial derivative.)

The learning rate

Look again at the small number the gradient gets multiplied by in the update: the learning rate. It is what turns a raw gradient into an actual step.

The gradient on its own can be large. In the worked step above it was -15, and moving the weight by a full 15 would fling it far past the bottom of the curve. So we do not step by the whole gradient. We keep its direction (its sign) but shrink its size, multiplying it by a small learning rate such as 0.01 or 0.05:

step  = learning_rate × gradient
w_new = w - step

Multiplying by the learning rate scales the gradient down to a safe, controlled amount, and subtracting that step nudges the weight the right way, downhill towards lower loss. In the worked step, 0.05 × -15 = -0.75, and 1 - (-0.75) moved the weight up from 1 to 1.75.

The value of the learning rate is a balancing act: too large and the steps overshoot the bottom and bounce around, too small and training crawls. Choosing it well matters enough to be the subject of the whole next part.

What about the bias?

Until now we quietly held the bias fixed and only moved the weight, which is what let us draw the loss as a simple 2D curve. In a real model the bias is a free parameter too, and it gets its own partial derivative:

∂MSE/∂b = (2/n) × Σ (y_pred - y_actual)

It is almost the same as the weight’s gradient, only with no × x on the end. That makes sense: nudging the bias shifts every prediction up or down by the same amount, whatever x is, so each point’s error counts equally.

Once both the weight and the bias can move, the loss is no longer a curve over a single number. It becomes a surface over the whole weight-and-bias plane: a bowl, U-shaped in every direction, with one lowest point that marks the best weight and bias together.

Gradient descent now works in two dimensions at once. At each step it computes both partial derivatives, ∂MSE/∂w and ∂MSE/∂b, and nudges the weight and the bias together, rolling the guess a little further down the bowl towards that lowest point.

When to stop

Training stops when one of three conditions is met:

Convergence: the happy ending. The loss is no longer improving: either the loss itself is near zero, or the parameters are barely changing between iterations, or the gradient has flattened out near zero. We have found the bottom of the curve.

Early stopping: something is wrong. The loss keeps falling on the data the model is trained on, but when we check it against a small stash of data we deliberately held back, its predictions there start getting worse. That means the model has begun memorising its training data instead of learning the underlying pattern, so we stop early and rethink. (This idea, called overfitting, and the held-back data used to catch it, get a whole part of their own later in the series.)

Maximum iterations: a safety net. Set a hard limit to prevent infinite loops and also useful for benchmarking (comparing two models at the same number of training steps).

Weight and bias together, step by step

Let us watch the full loop run. Starting from a blank slate of weight = 0 and bias = 0 on the same y = 2x + 1 data, at every step we compute both partial derivatives and nudge each value by the learning rate (0.05). Here are the first few iterations:

Read the first row like this: both gradients are large and negative, so each update pushes its value up. The weight becomes 0 - 0.05 × (-35) = 1.75 and the bias becomes 0 - 0.05 × (-12) = 0.60. Those are the numbers on the next row, where we work out the gradients again and repeat.

The loss collapses from 41 to under 0.05 in just two steps. The weight snaps close to 2 almost at once, while the bias climbs towards 1 more gradually. Left to keep running, gradient descent settles both at the values that make the loss as small as it can be. That is the whole algorithm: find the slope for each parameter, take one small step downhill, and repeat.

Interactive demo: gradient descent in action

The animation below shows gradient descent running on a simple dataset (y = 2x + 1, 4 data points). The left panel shows the predicted line chasing the true line. The right panel shows the gradient descent trajectory on the MSE loss curve.

Press Start to begin. Watch how each step moves the weight slightly closer to the true value of 2.

Then experiment with the learning rate box above the buttons. Set it to 0.01 and run: the weight creeps towards 2 in tiny, steady steps, taking many iterations to get there. Now set it to 0.13 and run again: the steps are now so big that the weight shoots past the minimum and bounces back and forth across it, taking far longer to settle (a little higher still and it would fly off and diverge). Getting this number right is the whole subject of the next part.

What to notice

• The red dot on the loss curve moves step by step towards the minimum at weight = 2.
• The tangent line (dashed red) shows the gradient at the current point, steep far from the minimum, nearly flat when close.
• The predicted line (red) in the left panel converges towards the true line (black dashed).
• Try changing the learning rate. A smaller value takes more steps but is stable. A larger value can overshoot. See the next part for details.

Next: The Learning Rate: why the size of each step matters so much, and how too large or too small a learning rate can wreck or stall training.