STATE 01 · FOUNDATIONS

Random variables and moments

From sets of outcomes to numerical summaries.

The random variable

A random variable is a function that assigns a real number to each outcome of an experiment. We insist only that the function be measurable, so that statements such as “the value is below three” correspond to genuine events. The distribution of the variable is then the measure that the original probability induces on the real line. Two variables with the same distribution are interchangeable for every probabilistic purpose, even if they are built on quite different sample spaces.

Variables divide into two broad families. A discrete variable takes its values on a countable set, and its law is given by a probability mass function that assigns a weight to each value. A continuous variable takes values in an interval, and its law is given by a density function whose integral over a region returns the probability of that region. Some variables are of neither pure kind; they place mass on a single point and spread the rest continuously. The cumulative distribution function unifies all cases, since it exists for every variable and can never decrease.

Expectation

The expectation of a variable is its probability-weighted average, written as the integral of the variable against its distribution. For a discrete variable the integral becomes a sum; for a continuous variable it becomes an ordinary Riemann or Lebesgue integral. The expectation is linear: the expectation of a sum equals the sum of the expectations, and a constant multiplier passes through unchanged. Linearity holds whether or not the variables are independent, which is precisely why the operation is so useful.

The expectation of a function of a variable is obtained by integrating that function against the same distribution, a statement sometimes given the name of the law of the unconscious statistician. The law spares us the work of finding the distribution of the function before averaging.

Variance and higher moments

The variance measures the spread of a variable about its mean. It is the expectation of the squared deviation, and it equals the expectation of the square minus the square of the expectation. The square root of the variance is the standard deviation, which carries the same units as the variable itself. A constant has zero variance; adding a constant leaves the variance unchanged; multiplying by a constant multiplies the variance by the square of that constant.

The moments of a variable are the expectations of its successive powers. The first moment is the mean, the second central moment is the variance, and the third and fourth central moments describe asymmetry and the weight of the tails. The skewness standardises the third moment, and the kurtosis standardises the fourth. For the Gaussian law the skewness vanishes and the kurtosis takes a fixed value, a fact that gives the Gaussian family its characteristic bell shape.

Generating functions

A generating function packages a whole sequence of moments or probabilities into a single object. The moment generating function is the expectation of the exponential of a scaled variable, and its derivatives at the origin return the moments in order. The characteristic function replaces the real scale by an imaginary one, which guarantees convergence for every distribution and makes the characteristic function the more robust instrument of the two.

Generating functions are especially valuable when variables are added. If two independent variables are summed, their generating functions multiply, and the distribution of the sum may often be read directly from the product. This feature explains why sums of independent Gaussian variables remain Gaussian, and why sums of independent counts remain counts of the same family. The reader may turn to continuous densities for the next step.