LeoMath

Foundations of probability · Section 4 of 5

Mean & variance

Average and spread made precise; covariance is an inner product, so correlation lies in [−1, 1].

about 30 min

Start from a problem

Problem

An instrument measures a distance; the reading is the true value plus a random error. The error is sometimes positive, sometimes negative, zero on average, but every reading wobbles, and the wobble has a typical size, call it σ.

To do better, take two instruments, one reading each, and average the two readings. How much smaller than σ is the error of the average? If both instruments sit in the same room and feel the same temperature, so that their errors tend to be too large together and too small together, does averaging still help? And if the two errors could be made to always have opposite signs, how good could it get?

To answer this we first need two things made precise: the "average value" of a random variable and the "size of its wobble". These are the expectation and the variance. As for how much two errors are "too large together", the answer lands somewhere familiar: the angle between two vectors.

Observe

Play first, read later. The six points on the left are the six outcomes of one die roll, each with probability 16; the point of outcome ω has coordinates (X(ω),Y(ω)), the values of two random variables for that outcome. The cross marks the two means, and a rectangle joins the cross to each point. On the right, X and Y with their means removed are drawn as two vectors.

Drag the points and watch Cov⁡ and the angle θ on the right change together. Can ρ ever exceed 1? Then press "Negative": σX2+σY2 is about 6.2, yet Var⁡(X+Y) is below 0.8. Which arrow on the right explains why?

Interactive experimentCorrelation is the cosine of an angle
  • outcome ω (probability 1/6 each)
  • mean (EX, EY)
  • best line
  • product > 0
  • product < 0

Scatter: each outcome ω is a point (X(ω), Y(ω))

Vectors: X and Y with their means removed

  • X − EX
  • Y − EY
  • projection
  • sum (X+Y) − E(X+Y)
EX
−0.17
EY
0.17
σX
2.05
σY
1.18
Cov(X,Y)
1.99
ρ = cos θ
0.82
θ
34.9°
Var(X+Y)
9.58
Cov⁡(X,Y)=E[(X−EX)(Y−EY)]ρ=Cov⁡(X,Y)σXσY=cos⁡θVar⁡(X+Y)=σX2+σY2+2Cov⁡(X,Y)
Arrow keys move

What did you observe?Play first, then answer. Read the explanation only after you have thought it through.

  1. 1.Press “On a line”. What are ρ and the angle θ? Now drag point 6 down, off the line. How does ρ change? Can any drag push ρ outside [−1, 1]?
  2. 2.Press “Uncorrelated, yet related”. Here Y is completely determined by X, yet Cov is 0. How do the positive and negative areas on the left cancel?
  3. 3.Press “One outlier” and note ρ; then drag point 6 to (0, 0). What does ρ become? Why can one point matter so much?
Conjecture

Counting blue rectangles as positive and red ones as negative, the average area seems to measure how much X and Y are "too large together, too small together"; it looks like the same number as the inner product of the two vectors on the right, and ρ looks like the cosine of the angle between them, so it always lies between −1 and 1. Var⁡(X+Y) looks like the squared length of the sum of the two vectors: the closer the angle is to 180∘, the shorter the sum.

If all this holds, the opening question is answered: the wider the angle between the two instruments' errors, the more averaging helps. Below we first define "average" and "wobble", then prove these guesses.

Definitions

Throughout, random variables live on a finite or countable probability space (Ω,P), and P(ω) is short for P({ω}).

Definition 4.1Expectation

Let X be a discrete random variable. If ∑x|x|P(X=x)<∞, the number

EX=∑xxP(X=x),

where x runs over the possible values of X, is the expectation (mean) of X; otherwise X has no expectation.

The expectation is the average weighted by probability. For a fair die the expected score is 1+2+⋯+66=3.5: the expectation need not be a value X can take. With infinitely many values we ask for absolute convergence, so that grouping and reordering the terms cannot change the sum; this is used right away.

Proposition 4.1Expectation as a sum over outcomes

If EX exists, then

EX=∑ω∈ΩX(ω)P(ω),

and the right-hand side converges absolutely. More generally, for any function g with ∑x|g(x)|P(X=x)<∞, Eg(X)=∑xg(x)P(X=x).

Proof

The events {X=x} for different x are disjoint with union Ω, so by countable additivity ∑ω:X(ω)=xP(ω)=P(X=x). Grouping the non-negative terms |X(ω)|P(ω) by the value of X gives ∑ω|X(ω)|P(ω)=∑x|x|P(X=x)<∞, so the right-hand side converges absolutely. An absolutely convergent series may be grouped at will:

∑ωX(ω)P(ω)=∑x ∑ω:X(ω)=xxP(ω)=∑xxP(X=x)=EX.

Applying this to the random variable g(X) and grouping the outcomes by the value of X, the same computation gives the second formula.

Example 4.1Indicator variables

For an event A, let 𝟏A(ω)=1 if ω∈A and 0 otherwise. 𝟏A takes only the values 0 and 1, so

E𝟏A=0⋅P(Ac)+1⋅P(A)=P(A).

A probability is a special expectation. Writing a count as a sum of indicators is the most useful trick in what follows.

Definition 4.2Variance and standard deviation

Suppose EX2<∞. The number

Var⁡X=E[(X−EX)2]

is the variance of X, and σX=Var⁡X its standard deviation.

Since |x|≤1+x22, EX2<∞ guarantees that EX exists, so the variance makes sense. The variance is the average of the squared distance from the mean. Squares rather than |X−EX|, because squares expand and the expansion can be computed with linearity below; squares are also what make it the square of a length. The standard deviation has the same unit as X: if errors are in millimetres, so is σ. The "typical size of the wobble" from the opening is the standard deviation.

Definition 4.3Covariance and correlation

Suppose EX2<∞ and EY2<∞. The number

Cov⁡(X,Y)=E[(X−EX)(Y−EY)]

is the covariance of X and Y. If σX>0 and σY>0,

ρ(X,Y)=Cov⁡(X,Y)σXσY

is their correlation. When Cov⁡(X,Y)=0, X and Y are called uncorrelated.

Since |xy|≤x2+y22, the covariance always exists, and Cov⁡(X,X)=Var⁡X. In the experiment, the signed area of rectangle ω is exactly (X(ω)−EX)(Y(ω)−EY): positive when the point is up-right or down-left of the cross, negative when up-left or down-right. By the sum-over-outcomes formula, Cov⁡(X,Y) is the average of these six signed areas.

Definition 4.4Independent random variables

Discrete random variables X and Y are independent if for all x,y

P(X=x, Y=y)=P(X=x)P(Y=y),

that is, the events {X=x} and {Y=y} are always independent. Several random variables are independent if, for every choice of values, the corresponding events are mutually independent.

Theorems and proofs

Theorem 4.2Linearity of expectation

If EX and EY exist, then for all real a,b the expectation E(aX+bY) exists and

E(aX+bY)=aEX+bEY.
Proof

Since |aX(ω)+bY(ω)|≤|a||X(ω)|+|b||Y(ω)|, the series ∑ω(aX(ω)+bY(ω))P(ω) converges absolutely. Summing over outcomes,

E(aX+bY)=∑ω(aX(ω)+bY(ω))P(ω)=a∑ωX(ω)P(ω)+b∑ωY(ω)P(ω)=aEX+bEY.

Linearity does not need independence: X and Y may be related in any way at all. It is the most useful tool of this section, and the hat problem below shows how much work it saves.

Proposition 4.3Computing variances
Cov⁡(X,Y)=E(XY)−EX⋅EY,Var⁡X=EX2−(EX)2,Var⁡(aX+b)=a2Var⁡X.
Proof

Write μ=EX and ν=EY. Expand (X−μ)(Y−ν)=XY−νX−μY+μν and use linearity (a constant is its own expectation):

Cov⁡(X,Y)=E(XY)−νμ−μν+μν=E(XY)−μν.

Taking Y=X gives the second formula. Since E(aX+b)=aμ+b, we have (aX+b)−E(aX+b)=a(X−μ); squaring and taking expectations gives a2Var⁡X.

A shift leaves the variance unchanged, since shifting is not wobbling; scaling by a multiplies the variance by a2 and the standard deviation by |a|.

Proposition 4.4Independence splits the expectation of a product

If X and Y are independent and EX, EY exist, then E(XY)=EX⋅EY. Hence independent random variables with finite variance are uncorrelated.

Proof

Apply the sum-over-outcomes formula to the pair (X,Y), grouping by its values, then use independence:

E(XY)=∑x,yxyP(X=x,Y=y)=∑x,yxyP(X=x)P(Y=y)=(∑xxP(X=x))(∑yyP(Y=y)).

With |xy| in place of xy the same computation gives E|X|⋅E|Y|<∞, so the double series converges absolutely and the grouping and splitting above are valid.

Now the heart of the section: covariance is an inner product.

Theorem 4.5Covariance is an inner product; correlation lies between −1 and 1

On random variables with EX2<∞, ⟨X,Y⟩=E(XY) is symmetric and bilinear, and ⟨X,X⟩≥0 with equality exactly when P(X=0)=1. The covariance is this inner product after removing the means: Cov⁡(X,Y)=⟨X−EX, Y−EY⟩. Consequently

|Cov⁡(X,Y)|≤σXσY,−1≤ρ(X,Y)≤1.

|ρ|=1 exactly when there are constants a≠0 and b with P(Y=aX+b)=1; then ρ has the sign of a.

Proof

Symmetry is clear and bilinearity is linearity of expectation. EX2=∑xx2P(X=x)≥0, with equality exactly when P(X=x)=0 for every x≠0, that is, P(X=0)=1.

Write X~=X−EX and Y~=Y−EY, so EX~2=σX2, EY~2=σY2 and E(X~Y~)=Cov⁡(X,Y). Now repeat the Cauchy–Schwarz proof from inner products: remove from Y~ its projection onto the direction of X~; what is left has non-negative length. For every real t,

0≤E[(Y~−tX~)2]=σY2−2tCov⁡(X,Y)+t2σX2.

If σX=0, the right-hand side is linear in t and non-negative for all t only if Cov⁡(X,Y)=0, so the inequality holds. If σX>0, take the projection coefficient t=Cov⁡(X,Y)/σX2:

0≤σY2−Cov⁡(X,Y)2σX2,

that is, Cov⁡(X,Y)2≤σX2σY2; dividing by σXσY gives |ρ|≤1.

|ρ|=1 exactly when equality holds, i.e. E[(Y~−tX~)2]=0, i.e. P(Y~=tX~)=1: with probability 1, Y=tX+(EY−tEX). Then Cov⁡(X,Y)≠0, so a=t≠0 and has the sign of Cov⁡ and of ρ. Conversely, if P(Y=aX+b)=1 then Y~=aX~ with probability 1, Cov⁡(X,Y)=aσX2 and σY=|a|σX, so ρ=a/|a|=±1.

So ρ=cos⁡θ now makes sense: θ is the angle between the "vectors" X~ and Y~, and the right-hand picture in the experiment is the plane they span. The projection coefficient t=Cov⁡(X,Y)/σX2 appears on the left too: it is the slope of the line drawn there, and an application below explains why that line is the "best".

Remark

Strictly speaking, E(XY) satisfies only "half" of the last inner-product axiom: ⟨X,X⟩=0 gives P(X=0)=1, not X=0 everywhere; X may be non-zero on an outcome of probability 0. Treating random variables that are equal with probability 1 as the same, it is a genuine inner product. The inequality above uses only ⟨X,X⟩≥0, so it is unaffected; the equality case can only be stated "with probability 1".

Theorem 4.6Variance of a sum: the law of cosines
Var⁡(X+Y)=Var⁡X+Var⁡Y+2Cov⁡(X,Y).

If X1,…,Xn are pairwise uncorrelated (for instance, independent), then

Var⁡(X1+⋯+Xn)=Var⁡X1+⋯+Var⁡Xn.
Proof

(X+Y)−E(X+Y)=X~+Y~, and by bilinearity

E[(X~+Y~)2]=EX~2+2E(X~Y~)+EY~2.

In general Var⁡(∑iXi)=E[(∑iX~i)2]=∑i,jCov⁡(Xi,Xj); for uncorrelated variables the terms with i≠j vanish, leaving ∑iVar⁡Xi.

Written as σX+Y2=σX2+σY2+2σXσYcos⁡θ, this is the law of cosines for the diagonal of a parallelogram; the sum vector on the right of the experiment is that diagonal. For uncorrelated variables θ=90∘ and it becomes Pythagoras: variances of independent errors add; standard deviations do not. When θ is close to 180∘ the two vectors nearly cancel, which is why Var⁡(X+Y) drops below 0.8 in the "Negative" preset.

Theorem 4.7Chebyshev's inequality and the law of large numbers

If Var⁡X<∞, then for every ε>0

P(|X−EX|≥ε)≤Var⁡Xε2.

If X1,…,Xn are independent, each with mean μ and variance σ2, and X‾n=1n(X1+⋯+Xn), then EX‾n=μ, Var⁡X‾n=σ2n, and for every ε>0

P(|X‾n−μ|≥ε)≤σ2nε2 ⟶ 0(n→∞).
Proof

Let A={|X−EX|≥ε}. For each ω: if ω∈A, then (X(ω)−EX)2≥ε2=ε2𝟏A(ω); if ω∉A, the right-hand side is 0 and the inequality holds too. So the random variable (X−EX)2−ε2𝟏A is non-negative everywhere, its expectation is non-negative, and by linearity and E𝟏A=P(A),

Var⁡X−ε2P(A)≥0.

For X‾n: linearity gives EX‾n=1n⋅nμ=μ; Var⁡(aX)=a2Var⁡X and additivity for independent variables give Var⁡X‾n=1n2⋅nσ2=σ2n. Chebyshev's inequality for X‾n finishes the proof.

The second half is the weak law of large numbers: measure independently again and again and average; the probability that the average misses the expectation by more than any fixed ε tends to 0. This is where the word "expectation" earns its name: it is where averages of many repetitions end up. The standard deviation of the average is σ/n, not σ/n: halving the error takes 4 measurements, cutting it to a tenth takes 100.

Common mistake

Uncorrelated does not mean independent. Let X take −1,0,1 with equal probability and Y=X2. Then EX=0 and E(XY)=EX3=0, so Cov⁡(X,Y)=0. Yet Y is completely determined by X: P(X=0,Y=0)=13, while P(X=0)P(Y=0)=19. The experiment's "Uncorrelated, yet related" is the same thing: points symmetric left to right make the positive and negative areas cancel in pairs. Covariance measures only linear dependence; independence implies uncorrelated, not the other way round.

Common mistake

Variance is not linear. For independent X and Y, Var⁡(X−Y)=Var⁡X+Var⁡Y, with a plus sign: the variance of −Y is (−1)2Var⁡Y. Subtracting two errors only piles up the wobble. Likewise Var⁡(2X)=4Var⁡X, while the sum X1+X2 of two independent copies has only 2Var⁡X: doubling one measurement and adding two independent measurements are entirely different things.

Common mistake

Eg(X) is generally not g(EX), and the expectation may not exist. From Var⁡X=EX2−(EX)2≥0 we always have EX2≥(EX)2, with equality only when the variance is 0; E(1/X) and 1/EX generally differ as well. As for existence: if P(X=2k)=2−k for k=1,2,…, then ∑k2k⋅2−k=1+1+⋯ diverges and X has no expectation. Then there is no variance or covariance either, and none of the theorems here apply.

Applications

ApplicationBack to the opening: two instruments

Let the two errors X, Y have EX=EY=0, σX=σY=σ and correlation ρ. By the variance of a sum,

Var⁡X+Y2=σ2+σ2+2ρσ24=1+ρ2σ2.

For independent errors (ρ=0) the average has standard deviation σ/2≈0.71σ; in a shared room with ρ=0.9 it is σ0.95≈0.97σ, hardly better; at ρ=1 averaging is useless; at ρ=−1 the errors cancel completely. That ρ cannot go below −1 is exactly the Cauchy–Schwarz inequality.

With n instruments whose errors all have pairwise correlation ρ, the same expansion gives

Var⁡X‾n=1n2(nσ2+n(n−1)ρσ2)=σ2(ρ+1−ρn).

As n→∞ this tends to ρσ2, not 0. The part of the error the instruments share cannot be averaged away; only the independent part dies out like 1/n.

ApplicationMean and variance of the binomial and Poisson laws

Let X∼B(n,p) and let Ai be "trial i succeeds", so X=𝟏A1+⋯+𝟏An. Each 𝟏Ai has mean p; since 𝟏Ai2=𝟏Ai, its variance is p−p2=p(1−p). By linearity and additivity of variance for independent variables,

EX=np,Var⁡X=np(1−p),

without a single binomial sum.

Let Y∼Poisson(λ). Drop the term k=0 and put j=k−1:

EY=∑k≥1ke−λλkk!=λ∑j≥0e−λλjj!=λ.

In the same way E[Y(Y−1)]=λ2, so Var⁡Y=E[Y(Y−1)]+EY−(EY)2=λ. This settles the claim made in random variables: the variances of B(n,p) and Poisson(np) differ only by the factor 1−p, so the two are close only when p is small.

ApplicationThe hat problem: linearity without independence

n≥2 people put their hats in a pile and each takes one at random, all n! ways equally likely. How many people get their own hat back, on average? Let Ai be "person i gets their own hat", so P(Ai)=(n−1)!n!=1n. With N=∑i𝟏Ai, linearity gives

EN=n⋅1n=1,

whatever n is. The events are not independent (if the first n−1 people all got their own hats, so does the last), but linearity does not need independence.

The variance can be computed too: P(Ai∩Aj)=(n−2)!n!=1n(n−1), so for i≠j, Cov⁡(𝟏Ai,𝟏Aj)=1n(n−1)−1n2=1n2(n−1), and

Var⁡N=n⋅1n(1−1n)+n(n−1)⋅1n2(n−1)=1.

Mean 1 and variance 1, exactly those of Poisson(1). The "many nearly independent rare events" of the previous section are back: indeed P(N=k)→e−1/k! as n→∞.

ApplicationPredicting Y from X: the best line

We look for constants α,β that minimise the mean squared error E[(Y−α−βX)2]. Write Y−α−βX=(Y~−βX~)+c with the constant c=EY−α−βEX. Since EX~=EY~=0, the cross term has expectation 0:

E[(Y−α−βX)2]=E[(Y~−βX~)2]+c2=σY2−2βCov⁡(X,Y)+β2σX2+c2.

Choosing α=EY−βEX makes c=0; minimising over β (with σX>0) gives

β=Cov⁡(X,Y)σX2,min⁡E[(Y−α−βX)2]=σY2(1−ρ2).

β is exactly the projection coefficient of Y~ onto X~, and the least error is the squared length of what is left over. This is the line y=EY+β(x−EX) in the left half of the experiment. ρ2 can be read as "the share of the variance of Y that X explains linearly": all of it when ρ=±1; when ρ=0 the best line is the horizontal y=EY and X is no help at all.

Try it in the plotter →

In the plotter, draw r+(1−r)/x and drag the parameter r to about 0.2: this is the variance (in units of σ2) of the average of n instruments with pairwise correlation r. As x grows from 1 the curve drops steeply, then flattens onto the horizontal line y=r; at r=0 it is 1/x and goes all the way down to 0.

Remark

The law of large numbers says that the average X‾n approaches μ, with deviations on the scale σ/n. But Chebyshev's inequality only bounds the deviation; it says nothing about the shape of its distribution. Magnify the deviation by n and it tends to a bell curve, whatever the distribution of the Xi. That is the subject of the next section, the central limit theorem (in progress).

Exercises

0 of 4 done

1
Roll two fair dice. What is the variance of the total? (Three decimals, or a fraction.)

Decimals or fractions; anything within 0.0006 of the answer counts as correct.

2
Two instruments have errors with mean 0, standard deviation 2 mm each, and correlation 0.5. Averaging the two readings, what is the standard deviation of the error of the average, in mm? (Three decimals.)

Decimals or fractions; anything within 0.0006 of the answer counts as correct.

3
X takes −1,0,1 with equal probability, and Y=X2. Which statement is true?
Answer to exercise 3
4
10 people put their hats in a pile and each takes one at random. What is the expected number of people who get their own hat back?
Answer to exercise 4

Discussion

Found a mistake, have a question or a suggestion? You can also write to 620026600@qq.com

No comments yet. Share your understanding, ask a question, or point out an error on this page.

Log in to join the discussion