STATE 01 · FOUNDATIONS

Probability foundations

The measure that turns uncertainty into arithmetic.

Sample spaces and events

A probabilistic model begins with a set Ω, called the sample space, whose elements are the possible outcomes of an experiment. A die, a coin, the waiting time until a particle decays — each has its own sample space, chosen to reflect what we intend to observe. A subset of Ω is called an event, and we think of the event as occurring when the outcome lies within that subset. The empty subset and the whole space are events as well, standing for the impossible and the certain.

We do not usually allow every conceivable subset to count as an event. Instead we select a family ℱ of subsets, closed under complement and under countable union, and call the members of ℱ the measurable events. This family is a sigma-algebra. The restriction protects us from pathological subsets while leaving every set that arises in practice inside the family.

The probability measure

A probability measure is a function P from ℱ to the interval from zero to one. It assigns the empty event the value zero and the whole space the value one, and it is countably additive: the measure of a disjoint union of events equals the sum of their measures. These three axioms are short, but they imply a great deal. The complement rule, the monotonicity of the measure, and the addition rule for two overlapping events all follow as immediate consequences.

From additivity we obtain the union bound, sometimes called Boole’s inequality, which states that the measure of a union never exceeds the sum of the individual measures. The bound is crude, yet it decides many questions about convergence, and it reappears later in the study of sums of random variables.

Conditional probability and independence

Suppose we learn that an event B has occurred. Our assessment of another event A is then revised to the conditional measure P(A | B), defined as the measure of the intersection of A and B divided by the measure of B, provided that B has positive measure. Conditioning is the central operation of the whole subject. Every inference of a parameter from an observation, and every statement about a future value given the past, is a conditional statement in disguise.

Two events are independent when the measure of their intersection equals the product of their measures. Equivalently, learning that one has occurred leaves our assessment of the other unchanged. Independence extends to families of events, and the extension is subtler than it first appears: pairwise independence does not imply mutual independence. A triple of events may have each pair independent while the three together violate the product rule.

Bayes’ rule

The rule of Bayes rewrites P(A | B) in terms of P(B | A), P(A) and P(B). Its power lies in the direction of the argument. We often know how likely an observation is under each of several competing explanations, and we wish to know how likely each explanation is given the observation we have. The rule, together with the law of total probability, supplies exactly that reversal. When the competing explanations form a partition of the sample space, the denominator becomes a weighted sum over the parts.

Figure 1. Two events within a sample space, with their intersection shaded.

Histogram of a relative frequency distribution over a small range — Figure 2. A relative frequency histogram, the empirical shadow of a probability mass function.

These ideas are enough to support everything that follows. The reader who wishes a longer treatment of the algebra of events may continue to random variables and moments.