Lines Through the Data

Every test so far has compared groups: pupils with the guide against pupils without, at two schools, then at School C, then at twelve. The other shape a question takes is how does one thing change with another: does revising for longer buy a higher mark, does the guide help more on the hard topics than the easy ones, does its effect grow with how often a pupil opened it. Those questions are answered with lines, and the machinery of lines, correlation and regression, turns out to contain everything from the earlier parts as special cases. It is also the machinery the twelve-school trial used to cross-check its permutation tests, so by the end of this part you will be able to read the one line in its report that has not yet been explained.

Correlation

The twenty-four pupils have a second number each: the minutes of revision each of them logged in the fortnight before the exam, which Part 11 used for its ratio of medians. Pupils with the guide revised for longer, around 640 minutes, because the guide sets practice questions. Pupils without revised for around 266 minutes. Plot minutes against mark and ask the correlation question: when one is high, is the other?

Pearson's r answers it with a number between −1 and +1: +1 means the points lie exactly on a rising line, −1 exactly on a falling one, 0 means no straight-line relationship at all. Across all twenty-four pupils, r = 0.34. Minutes and marks rise together, moderately. Spearman's rho is the same idea on ranks, so it does not care whether the relationship is a straight line, only whether it is consistently upward, and it is the one to use with skewed variables like the tutoring bills of Part 2. Here rho = 0.30, close to r because nothing is very skewed.

Now look inside each condition. Among the twelve pupils with the guide, the correlation between minutes and mark is 0.00. Among the twelve without, also 0.00. Revising for longer buys nothing within a condition. The correlation across all twenty-four is entirely the guide: it raises the mark and it raises the minutes, and a line through the whole cloud connects the two clusters without saying anything about what happens inside them.

This is what "correlation is not causation" means in practice. It is rarely that the correlation is spurious. It is that a third thing, here the condition, moves both variables, and the line faithfully reports their joint motion. The twelve-school trial met the reverse case and reported it: the schools whose pupils watched most of the deluxe guide's video solutions were not the schools that gained most from the deluxe guide, so the trial declined to credit its apparent benefit to the videos. A correlation that is absent where the mechanism says it should be present is evidence against the mechanism, which is a legitimate and underused kind of finding.

Two more cautions. Pearson's r is as sensitive to a single outlier as the mean is, and one unusual pupil can create or destroy a correlation on their own, which the demo below lets you do by hand. And r says nothing about the size of the effect: a correlation of 0.9 can describe a slope of half a mark per thousand minutes of revision if the noise is tiny. For the size, you need the line.

The line, and how to read it

Linear regression fits the line that comes closest to the points, closest meaning that the squared vertical distances add up to the least, and reports two numbers: the intercept, where the line crosses the vertical axis, and the slope, how much the outcome changes per unit of the input. On the twenty-four pupils the line is mark = 54.5 + 0.029 × minutes, so every extra 100 minutes of revision goes with 2.9 more marks. The slope is the effect size, in the outcome's units per unit of input, and like every effect size it needs an interval, which the bootstrap of Part 11 gives by resampling the pairs.

The square of the correlation, r², is the share of the outcome's variation that the line accounts for. Here it is 0.11: minutes explain 11 per cent of why marks differ, and 89 per cent is other things. In a write-up, a slope with a tight interval and an r² of 0.1 means "a real but small influence among many". A slope with an r² of 0.9 means the input nearly determines the outcome.

Regression is every comparison so far

Here is the trick that makes regression the general method. Give the guide a number, 1 for with and 0 for without, and fit mark = a + b × guide. The slope b is the gap between the two means, 11.9, and the intercept is the mean without the guide. The t-test of Part 12 is a regression with one yes-or-no input.

Now add the school as a second input, 1 for School B and 0 for School A: mark = a + b × guide + c × school. The fit gives b = +11.9, c = −28.1, and the interval on b is +5.4 to +18.5. That interval measures the same thing as Part 6's stratified bootstrap, +6.2 to +17.7, arrived at by a different road. Adding the school as a term does exactly what shuffling within the school did: it takes the 28-mark difference between the schools out of the noise, so that the guide's 12 marks can be seen against 15 marks of pupil-to-pupil spread instead of 30 marks of school spread. This is called adjusting for or controlling for the school, and it is what a write-up means when a results table says "adjusted for school and topic".

Four names, one idea. Each one says: compare like with like, and let the design remove the noise it was built to remove.

Mixed effects: letting each school have its own baseline

With two schools, adding the school as a term is fine. The twelve-school trial had twelve, and it treated them differently. Instead of estimating twelve separate baselines as fixed numbers, a mixed-effects model treats the schools as draws from a population of possible schools, each with its own baseline pulled from a shared distribution, and estimates the guide's effect as something that holds across that population. The "mixed" is that the guide's effect is a fixed effect, the same for everyone, and each school's baseline is a random effect, one draw per school.

On the twenty-four pupils, the mixed model gives the guide +11.9 with a standard error of 3.2 (the standard error is how far the estimate would typically wander if the study were repeated with fresh pupils), the same as the plain regression, because with two schools there is little difference. The estimated spread of school baselines is 19.7 marks, which is the school gap seen from the mixed model's side. With twelve schools and seven topics, both as random effects, the mixed model does the stratifying automatically and gives an honest standard error for a guide effect that is meant to generalise to schools and topics not in the trial.

That is the cross-check the twelve-school trial reported. Every one of its six predictions was tested by stratified permutation, and then re-estimated by a completely different method, a mixed-effects regression, and the two sets of estimates matched to two decimal places. When a resampling method that assumes nothing and a model-based method that assumes a lot agree, the agreement says the assumptions were not doing the work, which is the strongest kind of robustness check a study can offer.

Interactions: why P5 was so wide

The last thing regression adds is the ability to ask whether an effect depends on something. Does the guide help pupils resitting the exam more than pupils sitting it for the first time? That is P5, and in regression language it is an interaction: mark = a + b × guide + c × resit + d × (guide × resit). The coefficient d is the guide's gap among resit pupils minus its gap among first-timers, the difference of differences that Part 8 mentioned, and P5's prediction was that d is about zero.

Interactions are expensive. On made-up data with the same 12-mark gap for both kinds of pupil and 12 pupils per cell, the guide's main effect has a standard error of 7.2 and the interaction has a standard error of 10.2, because d is built from two subtractions and every subtraction adds noise. Halving the standard error of an interaction takes four times the pupils. That is the arithmetic behind P5's interval of −13.7 to +12.7, built from the 144 resit pupils and their first-time classmates, and it is the general warning: any claim of the form "the effect is bigger for X than for Y" needs several times the data of the claim "there is an effect".

Try it: the line

Twelve draggable points, a fitted line, and the correlation and slope updated live. Drag one point far away and watch r and the slope follow it.

Press Add an unusual pupil: one pupil who revised for nearly a thousand minutes and scored 30 turns a flat line into a falling one and r from 0 to −0.5. Then drag the dot up to a mark of 100 and the line rises instead. Twelve ordinary pupils and one strange one, and the one decides the story. Spearman's rho would move much less, because it only sees that the new point is last in one ranking and first in the other.

In Python

The table of pupils gets a minutes column, and the rest is SciPy for the correlations and the line, and statsmodels for the regression, the mixed model, and the interaction, using the formula language you will see in published analysis scripts: mark ~ guide + school means "mark as a function of guide and school".

import numpy as np, pandas as pd
from scipy import stats
import statsmodels.formula.api as smf

# The twenty-four pupils as a table, with the minutes each one spent revising in the fortnight
pupils = pd.DataFrame({
    "mark":    [88, 91, 79, 84, 95, 86, 62, 58, 71, 49, 66, 55,   78, 84, 70, 75, 89, 62, 46, 53, 40, 58, 49, 37],
    "minutes": [530, 810, 575, 590, 615, 720, 540, 690, 655, 700, 612, 640,   300, 230, 290, 250, 265, 270, 240, 255, 275, 262, 310, 245],
    "guide":   [1] * 12 + [0] * 12,
    "school":  (["A"] * 6 + ["B"] * 6) * 2,
})

# 1. Correlation: do pupils who revise for longer get higher marks?
r, p = stats.pearsonr(pupils.minutes, pupils.mark)
rho, _ = stats.spearmanr(pupils.minutes, pupils.mark)
print(f"all 24 pupils: Pearson r = {r:.2f} (p = {p:.3f}), Spearman rho = {rho:.2f}")
for g, grp in pupils.groupby("guide"):
    print(f"  within {'with' if g else 'without'} the guide: r = {stats.pearsonr(grp.minutes, grp.mark)[0]:+.2f}")

# 2. A line through the data, and how to read its slope
fit = stats.linregress(pupils.minutes, pupils.mark)
print(f"mark = {fit.intercept:.1f} + {fit.slope:.3f} x minutes, so +100 minutes goes with {100 * fit.slope:+.1f} marks; r^2 = {fit.rvalue**2:.2f}")

# 3. Regression as the general comparison: the guide's gap, adjusted for school
ols = smf.ols("mark ~ guide + school", data=pupils).fit()
print(f"OLS: guide {ols.params['guide']:+.1f} (SE {ols.bse['guide']:.1f}, 95% {ols.conf_int().loc['guide', 0]:+.1f} to {ols.conf_int().loc['guide', 1]:+.1f}), "
      f"School B {ols.params['school[T.B]']:+.1f}")

# 4. The same thing as a mixed-effects model: school as a random intercept, the twelve-school trial's cross-check
mixed = smf.mixedlm("mark ~ guide", data=pupils, groups=pupils["school"]).fit()
print(f"mixed: guide {mixed.params['guide']:+.1f} (SE {mixed.bse['guide']:.1f}), spread of school baselines {np.sqrt(mixed.cov_re.iloc[0, 0]):.1f}")

# 5. P5 as an interaction: does the guide's gap differ between first-time and resit pupils?
rng = np.random.default_rng(20260703)
p5 = pd.DataFrame({"sitting": np.repeat(["first", "resit"], 24), "guide": np.tile(np.repeat([0, 1], 12), 2)})
p5["mark"] = 60 + 12 * p5.guide - 8 * (p5.sitting == "resit") + rng.normal(0, 15, 48)   # same 12-mark gap for both kinds of pupil
inter = smf.ols("mark ~ guide * sitting", data=p5).fit()
print(f"interaction (guide x resit): {inter.params['guide:sitting[T.resit]']:+.1f}, SE {inter.bse['guide:sitting[T.resit]']:.1f}; "
      f"the main guide effect's SE on the same data: {inter.bse['guide']:.1f}")

which prints:

all 24 pupils: Pearson r = 0.34 (p = 0.106), Spearman rho = 0.30
  within without the guide: r = -0.00
  within with the guide: r = +0.00
mark = 54.5 + 0.029 x minutes, so +100 minutes goes with +2.9 marks; r^2 = 0.11
OLS: guide +11.9 (SE 3.2, 95% +5.4 to +18.5), School B -28.1
mixed: guide +11.9 (SE 3.2), spread of school baselines 19.7
interaction (guide x resit): +3.5, SE 10.2; the main guide effect's SE on the same data: 7.2

Where this leaves us

A correlation is a number for how consistently two things move together, and the most common reason they do is a third thing moving both, which is why the twenty-four pupils show a correlation between minutes of revision and marks that vanishes inside each condition. A regression line puts a size on the relationship, and regression with a yes-or-no input is the t-test, with a second input it is the stratified analysis, with the schools as random draws it is the mixed model the twelve-school trial used as its cross-check, and with a product term it is the interaction that made P5 so hard to pin down. One method, and every comparison in the series is a special case of it. The next part takes all of this to the place most readers of this series will actually use it: a league table.

Next: Reading a League Table: a league-table figure is a rate with an interval, best of k sittings and the estimator that gets it right, why one sitting is not the pupil, comparing two exam papers on the same pupils with McNemar's test and the paired bootstrap, and what twenty league tables do to the word "top".