STATE 02 · THE GAUSSIAN

Bivariate density matrices

Two Gaussian variables, one joint density, and the matrix that ties them together.

The joint density

Two variables observed together are described by a joint density on the plane. For the bivariate Gaussian law the joint density has a symmetric form built from a two-by-two matrix of variances and covariances. The density decreases as a quadratic form increases, so its level sets are ellipses centred on the pair of means. The shape of those ellipses, their orientation and their eccentricity, carries the entire dependence between the two variables.

When the two variables are independent the ellipses stand upright and the joint density factors into a product of two one-dimensional densities. When they are dependent the ellipses tilt, and the tilt encodes the sign and strength of the association. A positive association tilts the ellipses up and to the right; a negative association tilts them down and to the right.

Quadratic forms

The exponent of the bivariate Gaussian density is a quadratic form in the two standardised deviations, and it is written with a two-by-two matrix that has inverses of the variances on its diagonal and the covariance in its off-diagonal entries, after a division by that factor. This matrix is symmetric, and it is positive definite whenever the two variables are not perfectly dependent. Positive definiteness is what guarantees that the level sets are genuine ellipses rather than open curves or empty sets.

A single scalar factor appears twice in the density: once inside the quadratic form and once as a normalising factor in front. A small factor means a large spread, a tall normalising factor and unusually fat ellipses. A factor that approaches zero means that the two variables are almost perfectly dependent and the ellipses collapse towards a line.

Figure 1. A cluster of points with nested elliptical contours of equal joint density.

Contour view of a two-dimensional frequency surface — Figure 2. A contour view of a two-dimensional frequency surface, the empirical counterpart of the level sets.

Marginal and conditional densities

If we ignore one of the two variables we obtain the marginal density of the other, found by integrating the joint density over the whole range of the discarded variable. For the bivariate Gaussian the marginals are themselves Gaussian. The mean and variance of each marginal are simply the corresponding entries of the two-by-two table, a fact that makes the bivariate law especially easy to read.

If instead we fix the value of one variable and look at the conditional density of the other, we again find a Gaussian. Its mean moves linearly with the fixed value, and the slope of that movement is the regression coefficient. Its variance is smaller than the marginal variance, because knowledge of one variable reduces our uncertainty about the other. When the two variables are independent, fixing one leaves the conditional density equal to the marginal and the regression slope vanishes.

The conditional view is the foundation of prediction: given the past, the future is described by a conditional density, and its mean is the best forecast in the sense of squared error. The following page, covariance structures, generalises the two-by-two table to many variables at once.