STOCHASTIC SERIES LABstochastic-series-lab.org
This reference keeps only necessary storage in your browser so that its pages behave as expected between visits. Read the Cookie Policy.

STATE 05 · REFERENCE

Estimation theory

Learning an unknown quantity from observation.

Estimators

An estimator is a function of the observations intended to approximate an unknown quantity of the law. The quantity is called a parameter, and the estimator is itself a random variable, since it is built from random data. The realised number that the estimator takes on a given set of observations is the estimate. Keeping the two words apart, the estimator and the estimate, avoids a great deal of confusion.

A good estimator should be close to the parameter it targets, and the notion of closeness can be made precise in more than one way. We may ask that the estimator be unbiased, so that its expectation equals the parameter. We may ask that its variance be small, so that its values cluster tightly. We may ask that it be consistent, so that it converges to the parameter as the number of observations grows without bound.

Bias and variance

The bias of an estimator is the difference between its expectation and the true parameter. An unbiased estimator centres on the truth, though it may still scatter widely. The variance measures that scatter, and the average squared error of the estimator decomposes exactly into the bias squared plus the variance. This decomposition is the first identity of the subject, because it separates two distinct sources of error that are often in tension.

A slight bias is sometimes worth accepting if it buys a substantial reduction in variance. The compromise is quantified by the mean squared error, and estimators that sacrifice unbiasedness for a smaller total error are common and often preferable. The classical example is the shrinkage of an estimate towards a central value, which introduces bias but stabilises the result.

true value A B
Figure 1. Two estimators about a true value: one centred but spread, one offset but tight.

Consistency

An estimator is consistent when it converges in probability to the parameter as the sample grows. Consistency is the weakest demand we ordinarily make of an estimator, yet it is also the most important, because an inconsistent estimator will not improve however much data is gathered. A sufficient condition is that the bias and the variance both vanish in the limit, though the condition is not necessary.

The law of large numbers is the source behind most consistency claims. It states that the average of many independent observations with a common law converges to the mean of that law. Whenever an estimator can be written as such an average, or as a smooth function of such averages, consistency follows by a short argument. The technique of expanding a smooth function about its limit extends the conclusion far beyond simple averages.

Maximum likelihood

The likelihood of a parameter is the joint density of the observed data, read as a function of the parameter with the data held fixed. The maximum likelihood estimate is the parameter value under which the observed data are most plausible. The technique is general, requiring only a specified family of laws, and it enjoys a powerful asymptotic theory: under regularity, the maximum likelihood estimator is consistent, asymptotically unbiased, and asymptotically of smallest variance among a broad class.

The curvature of the log-likelihood at its peak supplies an estimate of the precision of the estimate, and the inverse of that curvature approximates the asymptotic variance. This link between curvature and precision is the bridge from estimation to the intervals of the next section.

Confidence intervals

A confidence interval is a random interval, built from the data, that covers the unknown parameter with a stated long-run frequency. The frequency refers to the repeated construction of the interval, not to any single realisation. Once an interval is formed it either contains the parameter or it does not, and the probability language applies to the construction rather than to the realised outcome.

The width of a confidence interval shrinks as the sample grows, at a rate set by the square root of the sample size. Two intervals with the same coverage may differ in expected length, and the shorter is preferred. The next page turns from estimation to the closely related task of inference and tests.