We have all heard the dismissal: "LLMs are just predicting the next token." Technically true, but it's a bit like saying a conversation is just saying the next word. There is a lot going on under the hood, and the ideas in this series: weights, biases, loss functions, gradient descent. These are the same core building blocks whether you are looking at the simplest possible model or at ChatGPT. That is why some of the most prominent voices in AI argue that the ability to predict should not be underestimated.

Tweet by Richard Ngo: LLMs are just doing next-token prediction without any understanding is by now so clearly false it is no longer worth debating.
The debate over whether "just predicting the next token" is a meaningful dismissal. It isn't.

"It is the ability to make predictions about the future that is the crux of intelligence." Jeff Hawkins, On Intelligence

Prediction is everywhere

Humans have always been prediction engines. Our ancestors predicted weather patterns to hunt and farm. We predicted planetary motion to reach space. The ability to make accurate predictions about the future has been one of the defining advantages of human intelligence.

It is no surprise, then, that many academic fields have their own name for prediction: forecasting in meteorology, modeling in economics, projection in finance, extrapolation in statistics. And in machine learning, for reasons we are about to see, we use the term regression.

Early human standing on a cliff in a storm, looking at lightning, illustrating our ancestral drive to predict weather
Predicting weather patterns was a survival skill long before machine learning existed.

Galton and the origin of the word

The word "regression" in machine learning traces back to a Victorian-era polymath named Sir Francis Galton (1822–1911), a half-cousin of Charles Darwin. In 1886 he published a paper titled Regression Towards Mediocrity in Hereditary Stature.

Portrait of Sir Francis Galton
Sir Francis Galton (1822–1911)

Galton was studying the relationship between parents' heights and their children's heights. He focused specifically on families where the parents were extreme outliers, very tall or very short, and found something striking:

  • Tall parents tended to have tall children, but not quite as tall as their parents.
  • Short parents tended to have short children, but not quite as short as their parents.

In both cases the children's height regressed towards the mean, it pulled back towards average, rather than tracking the extreme of the parent. Galton called this regression to the mean, or regression towards mediocrity.

Galton's original 1886 chart: Rate of Regression in Hereditary Stature, showing mid-parent height vs child height with the regression line
Galton's original 1886 chart. The first regression plot ever published.

When he plotted the data, the heights fell roughly along a straight line. He called it the regression line, because the heights of each new generation tended to regress towards it.

From Galton's paper to the linear model

Before going further, two words you will see throughout this series. The thing we use to make a prediction is called the feature (here, the parents' height). The thing we are trying to predict is called the target (the child's height). A dataset is just a collection of feature values with their known target values.

Galton also worked out a rough formula to predict a child's height from their parents' mid-height (the average of the two parents):

child_height ≈ 0.66 × parent_mid_height + 23

Notice the shape of this formula. It looks like:

target = weight × feature + bias

This is the fundamental equation of linear regression. weight (here 0.66) controls the slope of the line which is how steeply the target changes as the feature changes. bias (here 23) controls the intercept or where the line intersects the Y axis when the feature is zero.

These two values, weight and bias, are the two tuning knobs of any linear model. Given any weight and bias, you can reproduce any straight line. And once you have them, you can plug in a feature value you have never seen before to get a prediction.

That is exactly what machine learning does: it finds the values for weight and bias that best fit the training data, and then uses those values to predict targets for new inputs.

Try it: Galton's prediction machine

You can use Galton's 140-year-old formula right now. Slide the parents' mid-height below and the model predicts the child's adult height. Notice the regression-to-the-mean effect Galton discovered: very tall parents are predicted to have children shorter than themselves, and very short parents to have children taller than themselves, the prediction always pulls back towards the average.

predicted child height = 0.66 × 68.0 + 23 = 67.9 inches

Slide all the way to the right: parents with a mid-height of 76 inches get a predicted child of about 73 inches, tall, but below the dashed "same as parents" line. Slide to the far left: 60-inch parents get a predicted child of about 63 inches, above the dashed line. The two lines cross at roughly 68 inches, the population average, and every prediction is pulled towards it. That pull is regression to the mean, and the red line's two numbers, 0.66 and 23, are all the model needs to make a prediction for parents it has never seen.

What "linear" means

A linear relationship between a feature and a target means that as the feature increases, the target changes in a steady, predictable, straight-line way.

Some everyday examples:

  • Average study hours and exam score: more hours, steadily higher scores (roughly).
  • Years of experience and salary: more experience, steadily higher pay (roughly).
  • Square footage and house price: bigger house, proportionally higher price (roughly).

The data will not lie perfectly on a line, there is always noise, but the underlying trend is linear enough that a straight line is a useful model.

Next: Weights, Bias, and the Linear Model where we make the weight and bias interactive so you can see exactly what they do to a line.