Foundations of probability · Section 5 of 5
Central limit theorem
Why sample means tend to a bell curve, and what the laws of large numbers and the central limit theorem each say.
about 13 min
Start from a problem
An instrument measures the same quantity repeatedly, with a random error in every reading. Averaging many readings reduces the error: if each error has variance , the average of readings has variance . This is what the law of large numbers told us.
But there is a more precise question: what does the distribution of the average error look like? If a single error is highly skewed, taking only the values or , will the average be skewed too? And why do many entirely different random phenomena exhibit a similar bell curve after enough independent repetitions?
Do not read the theorem yet. Choose Bernoulli() with . At the sample mean is just one observation, so the histogram has only two bars at and . Move through and watch the shape and width change. Now switch to the uniform and exponential laws, and finally press “Standardised”.
Whose distribution is changing? Do the original data become normal as n grows?
- Simulated histogram
- Normal approximation
Source: draw each Xᵢ independently
Distribution of 5000 sample means; the dashed curve is the normal approximation
What did you observe?Play first, then answer. Read the explanation only after you have thought it through.
- 1.Choose Bernoulli(p) and look at the histogram at n = 1, then at n = 10, 30 and 100. How do its shape and width change? Is this the same as the original data becoming normal?
- 2.Keep n fixed and switch between the uniform and exponential laws. Which quantity shrinks like 1/√n? Press “Standardised”: what do the three histograms have in common?
- 3.Move Bernoulli p from 0.1 to 0.5. How does the skewness change? Now set p back to 0.1: why does the normal approximation become useful again as n grows?
The sample mean remains centred at the original expectation , while its spread shrinks like . Its shape approaches a normal curve as n grows. After subtracting and dividing by , every source law seems to approach the same curve.
The first statement concerns centre and spread; the second concerns shape. They belong respectively to the law of large numbers and the central limit theorem, and should not be confused.
Definitions
Random variables are independent and identically distributed (i.i.d.) if they are independent and all have the same distribution. Write for their common expectation and
By linearity and the additivity of variance for independent variables, from mean and variance,
Whether the sample mean is approximately normal is therefore a separate question from whether the source law is normal.
Let be the distribution functions of and that of . If, at every continuity point of ,
we say that converges in distribution to , written . Requiring convergence only at continuity points avoids the jumps of discrete laws; it does not require to approach at every outcome.
The characteristic function of a random variable is
The expectation always exists because . The characteristic function of is .
Theorems and proofs
For constants ,
If and are independent, then
Using properties of the exponential and the fact that independence splits the expectation of a product,
If , then for every fixed ,
where and
Taylor expansion gives
with the standard estimates and . Hence
It remains to show . Given any , since , choose such that
Dividing by and letting gives
If
This is what makes characteristic functions useful for limits of distributions. We cite Lévy's continuity theorem as an analytic tool; to prove the central limit theorem below, it will suffice to check pointwise convergence of characteristic functions.
Let be i.i.d. with and finite
In other words, for every real ,
Write . By the properties above and independence,
For fixed , as , so the second-order expansion gives
Therefore
Here we used the standard limit when ; for sufficiently large , take the principal logarithm, so
The theorem does not assume a normal source, a density, a third moment or boundedness. Being i.i.d. with finite variance is the key hypothesis. The standardisation also has a precise meaning: subtract and divide by , not by .
The law of large numbers and the central limit theorem are different. The law of large numbers says tends to in probability, with typical deviation of order . The central limit theorem multiplies this deviation by and examines its shape, obtaining the standard normal law. One describes where the average settles; the other describes the shape of its standardised fluctuation.
The source law does not become normal as n grows. Each still has its original Bernoulli, uniform or exponential distribution. It is the distribution of that approaches normality. Discrete sources also have lattice effects: a Bernoulli sample mean can only take values spaced by , so the normal curve is a continuous approximation, not a replacement for point probabilities.
The i.i.d. hypothesis cannot simply be dropped. If every error contains a common component, averaging cannot remove it even when every variance is finite; the previous section's example
Applications
If , then is a sum of independent indicators, with and
Because takes integer values, shift each boundary by when computing an interval probability; this is called continuity correction:
For example, if , then and , so
Since and are symmetric around , the corrections give .
If is large and is small, counts many rare successes, and the Poisson approximation is often more natural: the count remains a non-negative integer and its law may be visibly right-skewed. If is away from and and is large, the normal approximation is usually better. The choice depends on shape and parameters, not only on the size of .
With independent repeated measurements, has standard deviation , and the central limit theorem further gives an approximate confidence interval. For example,
So repeated measurement not only reduces the error, but also lets us estimate the probability that the error lies within a given range. A common bias or strong correlation invalidates direct use of this formula.
In the plotter, draw exp(-x^2/2)/sqrt(2*pi), the density of . Now draw exp(-(x-1)^2/(2*0.25))/(0.5*sqrt(2*pi)) and compare the centre and width: this is the normal density with mean and standard deviation . Standardisation in the experiment turns the distributions of different sample means into the first shape.
The central limit theorem explains why bell curves often appear when small independent errors accumulate, but it does not say every random phenomenon is normal. It concerns sums of i.i.d. terms. Joining independent increments continuously leads to random walks and Brownian motion, one route from probability towards analysis and differential equations.
Exercises
0 of 4 done
✓ All exercises in this section are done.
Discussion
Found a mistake, have a question or a suggestion? You can also write to 620026600@qq.com
No comments yet. Share your understanding, ask a question, or point out an error on this page.
Log in to join the discussion