Foundations of probability · Section 4 of 5
Mean & variance
Average and spread made precise; covariance is an inner product, so correlation lies in [−1, 1].
about 30 min
Start from a problem
An instrument measures a distance; the reading is the true value plus a random error. The error is sometimes positive, sometimes negative, zero on average, but every reading wobbles, and the wobble has a typical size, call it .
To do better, take two instruments, one reading each, and average the two readings. How much smaller than is the error of the average? If both instruments sit in the same room and feel the same temperature, so that their errors tend to be too large together and too small together, does averaging still help? And if the two errors could be made to always have opposite signs, how good could it get?
To answer this we first need two things made precise: the "average value" of a random variable and the "size of its wobble". These are the expectation and the variance. As for how much two errors are "too large together", the answer lands somewhere familiar: the angle between two vectors.
Play first, read later. The six points on the left are the six outcomes of one die roll, each with probability ; the point of outcome has coordinates , the values of two random variables for that outcome. The cross marks the two means, and a rectangle joins the cross to each point. On the right, and with their means removed are drawn as two vectors.
Drag the points and watch and the angle on the right change together. Can ever exceed ? Then press "Negative": is about , yet is below . Which arrow on the right explains why?
- outcome ω (probability 1/6 each)
- mean (EX, EY)
- best line
- product > 0
- product < 0
Scatter: each outcome ω is a point (X(ω), Y(ω))
Vectors: X and Y with their means removed
- X − EX
- Y − EY
- projection
- sum (X+Y) − E(X+Y)
- EX
- −0.17
- EY
- 0.17
- σX
- 2.05
- σY
- 1.18
- Cov(X,Y)
- 1.99
- ρ = cos θ
- 0.82
- θ
- 34.9°
- Var(X+Y)
- 9.58
What did you observe?Play first, then answer. Read the explanation only after you have thought it through.
- 1.Press “On a line”. What are ρ and the angle θ? Now drag point 6 down, off the line. How does ρ change? Can any drag push ρ outside [−1, 1]?
- 2.Press “Uncorrelated, yet related”. Here Y is completely determined by X, yet Cov is 0. How do the positive and negative areas on the left cancel?
- 3.Press “One outlier” and note ρ; then drag point 6 to (0, 0). What does ρ become? Why can one point matter so much?
Counting blue rectangles as positive and red ones as negative, the average area seems to measure how much and are "too large together, too small together"; it looks like the same number as the inner product of the two vectors on the right, and looks like the cosine of the angle between them, so it always lies between and . looks like the squared length of the sum of the two vectors: the closer the angle is to , the shorter the sum.
If all this holds, the opening question is answered: the wider the angle between the two instruments' errors, the more averaging helps. Below we first define "average" and "wobble", then prove these guesses.
Definitions
Throughout, random variables live on a finite or countable probability space , and is short for .
Let be a discrete random variable. If
where runs over the possible values of , is the expectation (mean) of ; otherwise has no expectation.
The expectation is the average weighted by probability. For a fair die the expected score is : the expectation need not be a value can take. With infinitely many values we ask for absolute convergence, so that grouping and reordering the terms cannot change the sum; this is used right away.
If exists, then
and the right-hand side converges absolutely. More generally, for any function with
The events for different are disjoint with union , so by countable additivity
Applying this to the random variable and grouping the outcomes by the value of , the same computation gives the second formula.
For an event , let if and otherwise. takes only the values and , so
A probability is a special expectation. Writing a count as a sum of indicators is the most useful trick in what follows.
Suppose . The number
is the variance of , and
Since , guarantees that exists, so the variance makes sense. The variance is the average of the squared distance from the mean. Squares rather than , because squares expand and the expansion can be computed with linearity below; squares are also what make it the square of a length. The standard deviation has the same unit as : if errors are in millimetres, so is . The "typical size of the wobble" from the opening is the standard deviation.
Suppose and . The number
is the covariance of and . If and ,
is their correlation. When
Since , the covariance always exists, and
Discrete random variables and are independent if for all
that is, the events and are always independent. Several random variables are independent if, for every choice of values, the corresponding events are mutually independent.
Theorems and proofs
If and exist, then for all real the expectation exists and
Since
Linearity does not need independence: and may be related in any way at all. It is the most useful tool of this section, and the hat problem below shows how much work it saves.
Write and . Expand
Taking gives the second formula. Since , we have ; squaring and taking expectations gives .
A shift leaves the variance unchanged, since shifting is not wobbling; scaling by multiplies the variance by and the standard deviation by .
If and are independent and , exist, then . Hence independent random variables with finite variance are uncorrelated.
Apply the sum-over-outcomes formula to the pair , grouping by its values, then use independence:
With in place of the same computation gives , so the double series converges absolutely and the grouping and splitting above are valid.
Now the heart of the section: covariance is an inner product.
On random variables with , is symmetric and bilinear, and with equality exactly when . The covariance is this inner product after removing the means:
exactly when there are constants and with ; then has the sign of .
Symmetry is clear and bilinearity is linearity of expectation.
Write and , so , and
If , the right-hand side is linear in and non-negative for all only if
that is,
exactly when equality holds, i.e.
So now makes sense: is the angle between the "vectors" and , and the right-hand picture in the experiment is the plane they span. The projection coefficient
Strictly speaking, satisfies only "half" of the last inner-product axiom: gives , not everywhere; may be non-zero on an outcome of probability . Treating random variables that are equal with probability as the same, it is a genuine inner product. The inequality above uses only , so it is unaffected; the equality case can only be stated "with probability ".
If are pairwise uncorrelated (for instance, independent), then
In general
Written as
If
If are independent, each with mean and variance , and
Let
For : linearity gives
The second half is the weak law of large numbers: measure independently again and again and average; the probability that the average misses the expectation by more than any fixed tends to . This is where the word "expectation" earns its name: it is where averages of many repetitions end up. The standard deviation of the average is , not : halving the error takes measurements, cutting it to a tenth takes .
Uncorrelated does not mean independent. Let take with equal probability and . Then and , so
Variance is not linear. For independent and ,
is generally not , and the expectation may not exist. From
Applications
Let the two errors , have ,
For independent errors () the average has standard deviation
With instruments whose errors all have pairwise correlation , the same expansion gives
As this tends to , not . The part of the error the instruments share cannot be averaged away; only the independent part dies out like .
Let and let be "trial succeeds", so
without a single binomial sum.
Let
In the same way
people put their hats in a pile and each takes one at random, all ways equally likely. How many people get their own hat back, on average? Let be "person gets their own hat", so
whatever is. The events are not independent (if the first people all got their own hats, so does the last), but linearity does not need independence.
The variance can be computed too:
Mean and variance , exactly those of . The "many nearly independent rare events" of the previous section are back: indeed as .
We look for constants that minimise the mean squared error . Write
Choosing makes ; minimising over (with ) gives
is exactly the projection coefficient of onto , and the least error is the squared length of what is left over. This is the line in the left half of the experiment. can be read as "the share of the variance of that explains linearly": all of it when ; when the best line is the horizontal and is no help at all.
In the plotter, draw and drag the parameter to about : this is the variance (in units of ) of the average of instruments with pairwise correlation . As grows from the curve drops steeply, then flattens onto the horizontal line ; at it is and goes all the way down to .
The law of large numbers says that the average approaches , with deviations on the scale . But Chebyshev's inequality only bounds the deviation; it says nothing about the shape of its distribution. Magnify the deviation by and it tends to a bell curve, whatever the distribution of the . That is the subject of the next section, the central limit theorem (in progress).
Exercises
0 of 4 done
✓ All exercises in this section are done.
Next section: Central limit theorem →
Discussion
Found a mistake, have a question or a suggestion? You can also write to 620026600@qq.com
No comments yet. Share your understanding, ask a question, or point out an error on this page.
Log in to join the discussion