The Classical Toolbox

The permutation test and the bootstrap need a computer. For most of the twentieth century there was no computer, and the tests that were invented instead work by assuming a shape for the noise, usually the bell curve, and looking up the answer in a table. Those tests are still what most write-ups report, and they are worth knowing for two reasons: you will read them constantly, and when their assumptions hold they give the same answer as the resampling tests in a thousandth of the time. This part goes through the ones you will meet, each with the question it answers, the assumption it makes, and the resampling test it stands in for.

Two independent groups: the t-test

The t-test compares two means. It computes the gap, divides it by the standard error from Part 6, the typical distance between a measured gap and the true one, and asks how often a ratio that large would arise if the true gap were zero and the noise were bell-shaped. On the twenty-four pupils, pooled: gap 11.9, standard error 6.7, ratio 1.77, p = 0.091. The permutation test of Part 4 gave 0.093 on the same numbers. When the bell holds, they agree, and here it holds well enough.

There are two versions and the choice matters more than textbooks admit. Student's t-test assumes the two groups have the same spread. Welch's t-test does not, and costs almost nothing when they do. Two conditions of a classroom study very often have different spreads: a guide that stops pupils leaving whole questions blank removes the very low marks from one group and not the other, and Student's test then gives the wrong p-value in a direction that depends on which group is larger. Use Welch by default, and treat a write-up that says "t-test" without saying which as having probably used Student's, since that is what most software defaults to.

The t-test's assumption is bell-shaped noise, or a sample large enough that the average is bell-shaped regardless, which happens surprisingly quickly, by thirty or so pupils a side for anything that is not wildly skewed. Tutoring bills are wildly skewed. Exam marks usually are not.

Two paired groups: the paired t-test

Part 11 introduced the seven topics of the exam paper, each with an average mark for the pupils with the guide and an average for those without, and showed the paired bootstrap giving an interval five times narrower than the unpaired one. The paired t-test is the formula version. Take the seven differences, one per topic, and run a one-sample t-test on them against zero. It never sees the fourteen original averages, only the seven gaps, so the topic difficulty that varied from 52 to 84 has already cancelled.

Same numbers. The wrong test says "probably real", the right one says "no doubt", because the wrong test is fighting noise that the design already removed. This is the single most common mistake in small classroom studies, and the fix is always the same question: were these numbers paired, by topic, by pupil or by school? If they were, the differences are the data.

When the shape is wrong: rank tests

For skewed outcomes and small samples the bell assumption fails, and the classical answer is to replace the marks with their ranks, first to twenty-fourth, and test those. Ranks cannot be skewed, so the shape problem disappears, at the cost of throwing away how far apart the marks were.

The Mann-Whitney U test (also called the Wilcoxon rank-sum test) is the rank version of the two-group t-test, and it is, exactly, a permutation test on ranks. Its natural effect size is Cliff's delta from Part 6, the chance that a pupil drawn from one group outscores a pupil drawn from the other, minus the reverse. On the twenty-four pupils it gives p = 0.088, close to the others because the marks were not badly skewed. On the tutoring bills of Part 2 it would be the right choice and the t-test would not.

The Wilcoxon signed-rank test is the rank version of the paired t-test. On the seven topics it gives p = 0.0156, and that number is worth a pause: with seven pairs that all point the same way, 0.0156 is the smallest p-value the test can produce, because there are only 2⁷ = 128 ways the signs could have fallen and the two most extreme are the observed ones. Rank tests with tiny samples have a floor, and a write-up reporting p = 0.0156 from seven pairs has hit it, not measured it.

Counts: chi-square and Fisher

For a two-by-two table of counts, School C's 16 of 18 passing with the guide against 5 of 18 without, there are two tests. Fisher's exact test, from Part 5, enumerates every possible table and is right at any size. The chi-square test compares the observed counts with the counts you would expect if the two rows were the same, and looks the discrepancy up in a table. It is an approximation, and the standard warning is that it needs an expected count of at least five in every cell. The 18-versus-18 table has a smallest expected count of 7.5, so both are fine, and they give 0.0007 and 0.0005. With three of eighteen against one of eighteen the chi-square approximation breaks and Fisher is the only sound choice. For large tables, hundreds of pupils, the two are indistinguishable and chi-square is what you will see.

Three or more groups: ANOVA

The twelve-school trial had several versions of the guide and a free one-page revision sheet, eight conditions in all, not two. Testing every pair with a t-test would be twenty-eight tests, and Part 7 explained what that does to the false-positive rate. The classical first step is analysis of variance (ANOVA), which asks one question of all the groups at once: is there any difference among these means, or could they all have come from one population? It compares the spread between group means with the spread within groups, and reports an F statistic and a p-value.

Add a third condition to the twenty-four pupils, twelve more who were given only a free one-page revision sheet and whose marks land between the other two groups, and the one-way ANOVA across the three gives F = 1.75, p = 0.189. Not significant, because pooled across the two schools the noise is too large, which is the lesson of Part 4 again: the classical version of stratifying is to add the school as a second factor, a two-way ANOVA, and the school's variance stops counting against the guide. The rank version of one-way ANOVA is the Kruskal-Wallis test, p = 0.179 here.

ANOVA says whether anything differs, not what. The follow-up is pairwise comparisons, and they need the correction of Part 7. The classical package is Tukey's HSD, which is built for exactly this, and Holm across the pairs does the same job. On the three conditions, the three pairwise Welch tests give raw p-values of 0.091, 0.219 and 0.581, and Holm turns them into 0.272, 0.438 and 0.581, none significant, in line with the ANOVA. The twelve-school trial did not use ANOVA. It wrote down six predictions in advance and corrected across them, which is the cleaner design when you know before the exam which comparisons matter.

Which test

Two rules cover most of the table. If the numbers came in pairs, use the paired row. If the outcome is skewed or the sample is small, use the rank row or a resampling test. Everything else is Welch.

Try it: which test

Answer three questions and the box names the test, its resampling equivalent, and the effect size to put beside it.

In Python

SciPy has every test in the table. The script runs each on the numbers from this part: the twenty-four pupils, the seven paired topics, School C's 16-of-18 table, and the third group of twelve pupils who had only the revision sheet.

import numpy as np
from scipy import stats
from statsmodels.stats.multitest import multipletests

with_guide    = np.array([88, 91, 79, 84, 95, 86, 62, 58, 71, 49, 66, 55])
without_guide = np.array([78, 84, 70, 75, 89, 62, 46, 53, 40, 58, 49, 37])

# Two independent groups: Student assumes equal spreads, Welch does not. Welch is the default.
print(f"Student t: p = {stats.ttest_ind(with_guide, without_guide).pvalue:.3f}")
print(f"Welch t:   p = {stats.ttest_ind(with_guide, without_guide, equal_var=False).pvalue:.3f}")
print(f"Mann-Whitney U: p = {stats.mannwhitneyu(with_guide, without_guide).pvalue:.3f}")

# Two paired groups: the seven topics of Part 11, each averaged under both conditions
with_by_topic    = np.array([81, 74, 66, 79, 70, 62, 84])
without_by_topic = np.array([70, 61, 57, 68, 55, 52, 75])
print(f"paired t: p = {stats.ttest_rel(with_by_topic, without_by_topic).pvalue:.1e}   "
      f"(unpaired Welch on the same numbers: p = {stats.ttest_ind(with_by_topic, without_by_topic, equal_var=False).pvalue:.3f})")
print(f"Wilcoxon signed-rank: p = {stats.wilcoxon(with_by_topic, without_by_topic).pvalue:.4f}")

# Counts: chi-square wants expected counts of 5 or more in every cell; Fisher does not care
table = [[16, 2], [5, 13]]   # School C: passed and failed, with the guide and without
chi2, p_chi, _, expected = stats.chi2_contingency(table)
print(f"chi-square: p = {p_chi:.4f} (smallest expected count {expected.min():.1f}); Fisher exact: p = {stats.fisher_exact(table).pvalue:.4f}")

# Three or more groups: add a third condition, twelve pupils with only the one-page sheet, and ask whether the three differ at all
sheet_only = np.array([84, 86, 77, 80, 91, 83, 58, 55, 66, 47, 63, 52])
f, p_anova = stats.f_oneway(with_guide, sheet_only, without_guide)
print(f"one-way ANOVA across three conditions: F = {f:.2f}, p = {p_anova:.3f}")
print(f"Kruskal-Wallis: p = {stats.kruskal(with_guide, sheet_only, without_guide).pvalue:.3f}")

# Then which pairs differ, with Holm across the three comparisons
pairs = {"guide vs nothing": (with_guide, without_guide), "sheet vs nothing": (sheet_only, without_guide), "guide vs sheet": (with_guide, sheet_only)}
raw = [stats.ttest_ind(a, b, equal_var=False).pvalue for a, b in pairs.values()]
_, holm, _, _ = multipletests(raw, method="holm")
for name, p, h in zip(pairs, raw, holm):
    print(f"  {name}: raw p = {p:.3f}, Holm p = {h:.3f}")

which prints:

Student t: p = 0.091
Welch t:   p = 0.091
Mann-Whitney U: p = 0.088
paired t: p = 1.0e-05   (unpaired Welch on the same numbers: p = 0.028)
Wilcoxon signed-rank: p = 0.0156
chi-square: p = 0.0007 (smallest expected count 7.5); Fisher exact: p = 0.0005
one-way ANOVA across three conditions: F = 1.75, p = 0.189
Kruskal-Wallis: p = 0.179
  guide vs nothing: raw p = 0.091, Holm p = 0.272
  sheet vs nothing: raw p = 0.219, Holm p = 0.438
  guide vs sheet: raw p = 0.581, Holm p = 0.581

Where this leaves us

The classical tests are formulas for the resampling tests, valid when the noise has the shape they assume, and they are what you will read in most write-ups. Welch's t for two groups, the paired t when the observations came in pairs, ranks when the outcome is skewed, Fisher for small counts, ANOVA when there are several groups followed by corrected pairwise comparisons. The paired row is the one to remember, because pairing turned 0.028 into 0.00001 on the same fourteen numbers. Every one of these tests compares groups. The next part is about the other shape a question can take: not "do these two differ" but "how does one thing change with another", which is where lines, correlations, and the regression that cross-checked the twelve-school trial come in.

Next: Lines Through the Data: correlation and what it does not mean, fitting a line and reading its slope, regression as the general form of every comparison so far, the mixed-effects model that gave each school its own baseline, and why P5 was an interaction.