LeoMath

Foundations of probability · Section 5 of 5

Central limit theorem

Why sample means tend to a bell curve, and what the laws of large numbers and the central limit theorem each say.

about 13 min

Start from a problem

Problem

An instrument measures the same quantity repeatedly, with a random error in every reading. Averaging many readings reduces the error: if each error has variance σ2, the average of n readings has variance σ2/n. This is what the law of large numbers told us.

But there is a more precise question: what does the distribution of the average error look like? If a single error is highly skewed, taking only the values 0 or 1, will the average be skewed too? And why do many entirely different random phenomena exhibit a similar bell curve after enough independent repetitions?

Observe

Do not read the theorem yet. Choose Bernoulli(p) with p=0.1. At n=1 the sample mean is just one observation, so the histogram has only two bars at 0 and 1. Move through n=10,30,100 and watch the shape and width change. Now switch to the uniform and exponential laws, and finally press “Standardised”.

Whose distribution is changing? Do the original data become normal as n grows?

Interactive experimentHow sample means become bell-shaped
  • Simulated histogram
  • Normal approximation

Source: draw each Xᵢ independently

Distribution of 5000 sample means; the dashed curve is the normal approximation

Sample size n
Mean of the means 0.1SD of the means 0.093Theoretical 0.1 ± 0.095
Source mean μ 0.1SD of the mean σ/√n 0.095
X‾n=X1+⋯+XnnZn=X‾n−μσ/nVar⁡(X‾n)=σ2n

What did you observe?Play first, then answer. Read the explanation only after you have thought it through.

  1. 1.Choose Bernoulli(p) and look at the histogram at n = 1, then at n = 10, 30 and 100. How do its shape and width change? Is this the same as the original data becoming normal?
  2. 2.Keep n fixed and switch between the uniform and exponential laws. Which quantity shrinks like 1/√n? Press “Standardised”: what do the three histograms have in common?
  3. 3.Move Bernoulli p from 0.1 to 0.5. How does the skewness change? Now set p back to 0.1: why does the normal approximation become useful again as n grows?
Conjecture

The sample mean remains centred at the original expectation μ, while its spread shrinks like σ/n. Its shape approaches a normal curve as n grows. After subtracting μ and dividing by σ/n, every source law seems to approach the same N(0,1) curve.

The first statement concerns centre and spread; the second concerns shape. They belong respectively to the law of large numbers and the central limit theorem, and should not be confused.

Definitions

Definition 5.1Independent, identically distributed variables and the sample mean

Random variables X1,X2,… are independent and identically distributed (i.i.d.) if they are independent and all have the same distribution. Write μ=EXi for their common expectation and σ2=Var⁡Xi for their common variance, and suppose σ2<∞. The sample mean of the first n variables is

X‾n=X1+⋯+Xnn.

By linearity and the additivity of variance for independent variables, from mean and variance,

EX‾n=μ,Var⁡X‾n=σ2n.

Whether the sample mean is approximately normal is therefore a separate question from whether the source law is normal.

Definition 5.2Convergence in distribution

Let Fn be the distribution functions of Zn and F that of Z. If, at every continuity point x of F,

Fn(x)⟶F(x),

we say that Zn converges in distribution to Z, written Zn⇒Z. Requiring convergence only at continuity points avoids the jumps of discrete laws; it does not require Zn(ω) to approach Z(ω) at every outcome.

Definition 5.3Characteristic function

The characteristic function of a random variable X is

φX(t)=E[eitX],t∈ℝ.

The expectation always exists because |eitX|=1. The characteristic function of N(0,1) is e−t2/2.

Theorems and proofs

Proposition 5.1Basic properties of characteristic functions

For constants a,b,

φaX+b(t)=eitbφX(at).

If X and Y are independent, then

φX+Y(t)=φX(t)φY(t).
Proof

Using properties of the exponential and the fact that independence splits the expectation of a product,

φaX+b(t)=E[eit(aX+b)]=eitbE[eiatX]=eitbφX(at),
φX+Y(t)=E[eitXeitY]=E[eitX]E[eitY]=φX(t)φY(t).
Proposition 5.2A second-order expansion with finite variance

If EX2<∞, then for every fixed t,

φX(t)=1+iμt−σ2+μ22t2+o(t2)(t→0),

where μ=EX and σ2=Var⁡X. In particular, if EX=0, then φX(t)=1−σ22t2+o(t2).

Proof

Taylor expansion gives

eitx=1+itx−t2x22+R(t,x),

with the standard estimates |R(t,x)|≤|t|3|x|3/6 and |R(t,x)|≤t2x2. Hence

φX(t)=1+iμt−EX22t2+E[R(t,X)].

It remains to show E[R(t,X)]=o(t2). Given any ε>0, since EX2<∞, choose M such that E[X2𝟏|X|>M]<ε. Use the first estimate on |X|≤M and the second on |X|>M:

E|R(t,X)|≤|t|3M36+t2ε.

Dividing by t2 and letting t→0 gives lim supt→0⁡|E[R(t,X)]|/t2≤ε. As ε was arbitrary, the remainder is o(t2). Finally substitute EX2=σ2+μ2. Only the second moment is used; no third moment is required.

Proposition 5.3Lévy's continuity theorem

If φZn(t)→φZ(t) for every real t, then Zn⇒Z.

This is what makes characteristic functions useful for limits of distributions. We cite Lévy's continuity theorem as an analytic tool; to prove the central limit theorem below, it will suffice to check pointwise convergence of characteristic functions.

Theorem 5.4The central limit theorem for independent, identically distributed variables

Let X1,X2,… be i.i.d. with EXi=μ and finite Var⁡Xi=σ2>0. Then

Zn=X‾n−μσ/n ⟹ N(0,1).

In other words, for every real x,

P(Zn≤x)⟶Φ(x)=∫−∞xe−u2/22πdu.
Proof

Write Sn=X1+⋯+Xn. By the properties above and independence,

φZn(t)=e−itμn/σ[φX(tσn)]n.

For fixed t, t/(σn)→0 as n→∞, so the second-order expansion gives

φX(tσn)=1−t22n+o(1n).

Therefore

φZn(t)=[1+−t2/2+o(1)n]n⟶e−t2/2.

Here we used the standard limit (1+zn/n)n→ez when zn→z; for sufficiently large n, take the principal logarithm, so log⁡(1+zn/n)=zn/n+O(1/n2). The right-hand side is the characteristic function of N(0,1), and Lévy's continuity theorem gives Zn⇒N(0,1).

The theorem does not assume a normal source, a density, a third moment or boundedness. Being i.i.d. with finite variance is the key hypothesis. The standardisation also has a precise meaning: subtract μ and divide by σ/n, not by σ/n.

Common mistake

The law of large numbers and the central limit theorem are different. The law of large numbers says X‾n−μ tends to 0 in probability, with typical deviation of order σ/n. The central limit theorem multiplies this deviation by n/σ and examines its shape, obtaining the standard normal law. One describes where the average settles; the other describes the shape of its standardised fluctuation.

Common mistake

The source law does not become normal as n grows. Each Xi still has its original Bernoulli, uniform or exponential distribution. It is the distribution of X‾n that approaches normality. Discrete sources also have lattice effects: a Bernoulli sample mean can only take values spaced by 1/n, so the normal curve is a continuous approximation, not a replacement for point probabilities.

Common mistake

The i.i.d. hypothesis cannot simply be dropped. If every error contains a common component, averaging cannot remove it even when every variance is finite; the previous section's example Var⁡X‾n→ρσ2 illustrates this. With infinite variance, the second-order expansion and this proof no longer hold, and a different kind of limiting law may appear.

Applications

ApplicationNormal approximation to the binomial law and continuity correction

If X∼B(n,p), then X is a sum of n independent indicators, with EX=np and Var⁡X=np(1−p). When n is large and np(1−p) is not too small, we can approximate

X−npnp(1−p)≈N(0,1).

Because X takes integer values, shift each boundary by 0.5 when computing an interval probability; this is called continuity correction:

P(a≤X≤b)≈Φ(b+0.5−npnp(1−p))−Φ(a−0.5−npnp(1−p)).

For example, if X∼B(100,0.5), then EX=50 and Var⁡X=25, so

P(45≤X≤55)≈Φ(1.1)−Φ(−1.1)≈0.7286.

Since 45 and 55 are symmetric around 50, the corrections give z=(55.5−50)/5=1.1.

ApplicationNormal approximation or Poisson approximation?

If n is large and p is small, X∼B(n,p) counts many rare successes, and the Poisson approximation is often more natural: the count remains a non-negative integer and its law may be visibly right-skewed. If p is away from 0 and 1 and np(1−p) is large, the normal approximation is usually better. The choice depends on shape and parameters, not only on the size of n.

ApplicationErrors in repeated measurements

With independent repeated measurements, X‾n−μ has standard deviation σ/n, and the central limit theorem further gives an approximate confidence interval. For example,

P(|X‾n−μ|≤1.96σn)≈0.95.

So repeated measurement not only reduces the error, but also lets us estimate the probability that the error lies within a given range. A common bias or strong correlation invalidates direct use of this formula.

Try it in the plotter →

In the plotter, draw exp(-x^2/2)/sqrt(2*pi), the density of N(0,1). Now draw exp(-(x-1)^2/(2*0.25))/(0.5*sqrt(2*pi)) and compare the centre and width: this is the normal density with mean 1 and standard deviation 0.5. Standardisation in the experiment turns the distributions of different sample means into the first shape.

Remark

The central limit theorem explains why bell curves often appear when small independent errors accumulate, but it does not say every random phenomenon is normal. It concerns sums of i.i.d. terms. Joining independent increments continuously leads to random walks and Brownian motion, one route from probability towards analysis and differential equations.

Exercises

0 of 4 done

1
Independent, identically distributed variables each have mean 2 and standard deviation 3. What is the standard deviation of the mean of 36 variables?

Enter the exact value; a fraction such as 1/3 is fine.

2
Let X∼B(100,0.5). Estimate P(45≤X≤55) using a normal approximation with continuity correction, to three decimals.

Decimals or fractions; anything within 0.0006 of the answer counts as correct.

3
For independent, identically distributed variables with finite variance, which statement is true?
Answer to exercise 3
4
Let X∼B(1000,0.005). Which approximation is more appropriate for the distribution of X?
Answer to exercise 4

Discussion

Found a mistake, have a question or a suggestion? You can also write to 620026600@qq.com

No comments yet. Share your understanding, ask a question, or point out an error on this page.

Log in to join the discussion