One Number Is Not a Finding

Here is a sentence from the kind of write-up most of us produce the first time we test anything. A maths teacher tries a new revision guide with some of her pupils in the fortnight before the end-of-year exam, and reports: "The pupils who had the guide scored 74 out of 100 on average. The pupils who did not scored 62. The guide works."

Every number in that sentence is real, and the conclusion might even be right. But the headteacher, who is being asked to buy the guide for every pupil in the school, will not accept it as it stands, and the questions she will ask are the subject of this series. How many pupils? How far apart do two pupils land when they had exactly the same preparation? If you shuffled which pupils had the guide, how often would a gap of 12 appear by accident? How sure are you of the 12, and would you have written it up the same way if it had come out as 2?

I will use the teacher's study for the whole series, because it has everything a study needs and nothing that requires special knowledge. It is made up, but it is made up carefully, and it grew the way real studies grow. In its first year it was twenty-four pupils at two schools, which is small enough to shuffle by hand. In its second year a third school joined with thirty-six more. In its third year twelve schools ran it together, 480 pupils, with six predictions written down before anyone sat the exam, a test for each, a correction for having tested six things, and a table at the end that says "supported" beside three of the predictions and "not supported" beside the other three. The plan for a fourth year, written to settle the one question the third could not, turns up in Part 8. This series explains every term in that last sentence, so that by the end you can read a methods section, or write one, without guessing what the words mean.

No maths beyond arithmetic is needed. Where a formula would help I will show one, but every idea here can be understood by shuffling cards, and most of the demos are exactly that.

The vocabulary, with a study attached

A study is built out of a small number of objects, and the words for them are used loosely in conversation and precisely in write-ups. Here they are, each attached to the teacher's study so that they are concrete.

When you see a report quote an average for a group, the first thing to look for is the n beside it. An average of 48 pupils and an average of 2 pupils are printed with the same number of decimal places and mean entirely different things.

The twenty-four pupils

The whole series uses one small dataset, small enough to print and to shuffle by hand: the first year of the study. Two schools took part. School A has a strong intake and its marks run high. School B's marks run low. At each school twelve pupils took part, six names were drawn from a hat to get the guide, and six were not. All twenty-four sat the same exam, and the papers were marked by teachers who did not know which pupils had the guide. Twenty-four marks out of 100.

The numbers are invented, but they were invented to behave like real ones: School A scores about 30 marks above School B, the guide is worth about 12 marks on average, and pupils at the same school with the same preparation still land 20 or 30 marks apart. Add up the twelve marks with the guide and divide by twelve and you get 73.7. The twelve without come to 61.8. The gap is 11.9 marks, which I will call 12 from here on.

The same preparation, different pupils

Look at the six pupils at School A who had the guide: 88, 91, 79, 84, 95, 86. Nothing differed between them that the teacher controlled. Same school, same guide, same fortnight, same exam paper, same markers. Their marks still range over 16 points, because a pupil is a person: one slept badly, one guessed well on the last question, one happened to have covered the geometry topic the week before. The School B pupils without the guide range from 37 to 58, a spread of 21 marks, for the same reasons.

This is not a defect of the study. It is a property of the thing being measured, and it has a name: variability, or in everyday language, noise. Every measurement of a person has it, and so does nearly every measurement of anything, which is why the whole apparatus of statistics exists. The teacher's answer to it in the first year was to have six pupils in every group rather than one, so that no single lucky or unlucky pupil could stand in for a condition. By the third year she had at least six in every cell at every one of twelve schools.

Now the uncomfortable consequence. If six pupils with the same preparation can differ by 20 marks, then the average of six pupils is also a noisy number. Test six different pupils and the average moves. How much it moves is the question that decides whether a 12-mark gap between two conditions is a finding or a coincidence, and it is the question that the demo below lets you feel before the next parts put numbers on it.

Try it: test different pupils

The demo draws pupils from two imaginary groups whose true averages you set, adds pupil-to-pupil spread of the size you set, and reports the gap between the two group averages. Press the button a few times. Then change the number of pupils per group and press it again.

Three things to try. Leave the true gap at 12 and the spread at 15, which is roughly what the twenty-four pupils look like, and press Do it 20 times with 6 pupils per group. The measured gap will have ranged from close to zero to over 20. Then set the true gap to 0 and do it 20 times: you will see gaps of 10 or more appear from nothing, which is what "the guide works, they scored 12 higher" has to be defended against. Finally set pupils per group to 100 and watch the measured gaps huddle around the true one. Nothing about the guide changed between those three pictures. Only the amount of evidence did.

The three questions

Everything that follows in this series is a way of answering one of three questions about a gap like the 12 marks above, and it helps to know which is which before the vocabulary arrives.

How big is it? Not the number itself, which is 12, but 12 in units that mean something: marks on an exam that a parent can picture, or a multiple of the pupil-to-pupil spread, or a pass rate that went from 28 per cent to 89 per cent. This is the effect size, and it is the question most write-ups answer first. Part 2 covers the ways of summarising many pupils into one number, and Part 6 covers effect sizes properly.

Could chance alone have done it? If the guide did nothing, how often would shuffling twenty-four pupils into two piles of twelve give a gap of 12 or more? That question has an exact answer, obtained by actually doing the shuffling, and the answer is the p-value. Parts 3, 4 and 5 are about it, and Part 7 is about what happens to it when you ask six questions at once.

How sure are you of the size? A gap of 12 measured on six pupils a side could really be 2 or 22. A confidence interval is the range of gaps that the data cannot rule out, and Part 6 shows how to build one by resampling. Part 8 is about the awkward case where the interval includes zero, which is to say a null result, and how to defend one.

Then there are two questions that are not about arithmetic at all, and that decide whether any of the above is believed: was the study designed so that the gap could not have been caused by something other than the guide, and were the choices about how to analyse it made before the results were in view? Those are Parts 9 and 10, and they are where the teacher's study changed most between its first year and its third.

In Python

Everything in this part fits in a dozen lines. The first block computes the gap. The second repeats the study with fresh pupils, twenty times, at three sample sizes, which is the demo above in code.

import numpy as np

# The twenty-four pupils: exam mark out of 100
with_guide    = [88, 91, 79, 84, 95, 86,   # School A
                 62, 58, 71, 49, 66, 55]   # School B
without_guide = [78, 84, 70, 75, 89, 62,   # School A
                 46, 53, 40, 58, 49, 37]   # School B

gap = np.mean(with_guide) - np.mean(without_guide)
print(f"mean with {np.mean(with_guide):.1f}, without {np.mean(without_guide):.1f}, gap {gap:+.1f}")

# Repeat the study with fresh pupils, drawn from a world where the guide is
# really worth 12 marks and pupils with the same preparation differ by about 15
rng = np.random.default_rng(1)
for n in (6, 24, 100):
    gaps = [rng.normal(74, 15, n).mean() - rng.normal(62, 15, n).mean() for _ in range(20)]
    print(f"{n:>3} pupils a side: 20 measured gaps ranged from {min(gaps):+.1f} to {max(gaps):+.1f}")

which prints:

mean with 73.7, without 61.8, gap +11.9
  6 pupils a side: 20 measured gaps ranged from -9.9 to +21.1
 24 pupils a side: 20 measured gaps ranged from -0.2 to +18.8
100 pupils a side: 20 measured gaps ranged from +7.3 to +15.0

Where this leaves us

A number from a study is a sample from a spread, and the spread is often wider than the gap you are trying to see. "The pupils with the guide scored 12 marks higher" is the start of an argument, not the end of one. The rest of the series turns it into one, using the same twenty-four pupils at every step, so that by the last part you can look at the six-row results table from the twelve-school trial, with its effects, intervals, corrected p-values and the words "supported" and "not supported", and know exactly what each entry cost to earn.

Next: Middles, Spreads and Rates: the mean, the median, why a list of tutoring bills needs the median, spreads and box plots, and what a pass rate of 16 out of 18 does and does not tell you, including the interval the textbook formula gets wrong and the ones that get it right.