Law of large numbers

In probability theory, several laws of large numbers say that the average of a sequence of random variables with a common distribution converges (in the senses given below) to their common expectation, in the limit as the size of the sequence goes to infinity. Various formulations of the law of large numbers, and their associated conditions, specify convergence in different ways.

When the random variables have a finite variance, the central limit theorem extends our understanding of the convergence of their average by describing the distribution of the standardised difference between the sum of the random variables and the expectation of this sum. Regardless of the underlying distribution of the random variables, this standardised difference converges in distribution to a standard normal random variable.

The phrase "law of large numbers" is also sometimes used to refer to the principle that unlikely outcomes become likely when an event is repeated a large number of times. For example, the odds that you will win the lottery are very low; however, the odds that someone will win the lottery are quite good, provided that a large enough number of people purchased lottery tickets.

The weak law

The weak law of large numbers states that if X₁, X₂, X₃, ... is an infinite sequence of random variables, all of which have the same expected value μ and are uncorrelated (i.e., the correlation between any two of them is zero), then the sample average

A consequence of the weak law of large numbers is the asymptotic equipartition property.

The strong law

The strong law of large numbers states that if X₁, X₂, X₃, ... is an infinite sequence of random variables that are independent and identically distributed with common expected value μ and finite variance, and if  E(|X_i|) < ∞, then

If we replace the finite expectation condition with a finite second moment condition,  E(X₁²) < ∞, then we obtain both almost sure convergence and convergence in mean square. In either case, these conditions also imply the consequent of the weak law of large numbers, since almost sure convergence implies convergence in probability (as, indeed, does convergence in mean square).

This law justifies the intuitive interpretation of the expected value of a random variable as the "long-term average when sampling repeatedly".

A weaker law and proof

Proofs of the above weak and strong laws of large numbers are rather involved. The consequent of the slightly weaker form below is implied by the weak law above (since convergence in distribution is implied by convergence in probability), but has a simpler proof.

Theorem. Let X₁, X₂, X₃, ... be a sequence of random variables, independent and identically distributed with common mean μ < ∞, and define the partial sum S_n := X₁ + X₂ + ... +X_n. Then,  S_n / n converges in distribution to μ.

Proof. (See [1], p. 174) By Taylor's theorem for complex functions, the characteristic function of any random variable, X, with finite mean μ, can be written as

References