STATE 05 · REFERENCE

Inference and tests

Deciding between hypotheses without losing sight of the probabilities.

Null and alternative

A test of significance begins with two hypotheses. The null hypothesis is a precise claim about the law that generates the data, often a claim of no effect. The alternative hypothesis is the collection of possibilities we are prepared to accept when the null is rejected. The test is a rule that divides the possible samples into a region of rejection and its complement, the region of acceptance.

The choice of the rejection region fixes the behaviour of the rule. Two errors are possible. A first-kind error occurs when the null is true and the rule nevertheless rejects it. A second-kind error occurs when the null is false and the rule nevertheless accepts it. The two errors pull in opposite directions, and no rule can reduce both at once without more data.

Significance and p-values

The significance level of a test is the probability of a first-kind error under the null, chosen in advance and held fixed. The p-value is the probability, obtained under the null, of obtaining a result at least as extreme as the one observed. A small p-value indicates that the data are improbable under the null, and the null is then rejected in favour of the alternative.

The p-value is a statement about the data given the null, not a statement about the null given the data. This direction of conditioning is the most frequently misunderstood point in the whole subject. A p-value of a given size does not mean that the null has that probability of being true; it means that data of the observed extremity would arise with that probability if the null held.

Choosing a rule

Among rules with a fixed significance level, the most powerful is the one with the smallest second-kind error probability across the whole alternative. For simple hypotheses the Neyman–Pearson lemma identifies that most powerful rule as a likelihood ratio test. For composite alternatives the theory is richer, and the generalised likelihood ratio often supplies an acceptable rule whose distribution under the null is, in large samples, a chi-squared law.

Power depends on the sample size, the size of the true effect, and the noise in the data. A test with little power may fail to detect a real effect, and the resulting acceptance of the null is not evidence that the null is true. The careful analyst therefore reports the power of the test alongside the result, and treats a failure to reject as a statement about the evidence rather than a demonstration of absence.

Multiple comparisons

When many hypotheses are tested at once, the chance of at least one spurious rejection rises quickly. A family of twenty independent tests, each run at a five per cent level, will produce at least one false rejection with a probability near two thirds. The remedy is to control a family-wide error rate, either by tightening each individual level or by using a procedure that adjusts the p-values together.

The same reasoning underlies the reporting of intervals rather than of bare decisions. An interval communicates the width of our uncertainty, while a bare accept or reject hides it. The reference therefore recommends that estimates be reported with their intervals, and that tests be read as one part of a larger argument rather than as an end in themselves. A reader new to the vocabulary may consult the glossary for a compact restatement of these terms.