Foundations of probability · Section 2 of 3
Conditioning & Bayes
Knowing an event occurred shrinks the sample space to it; then reason from effects back to causes.
about 12 min
Start from a problem
A disease affects of a population. A test is positive for of the ill, and also, falsely, for of the healthy. Your result is positive. What is the probability that you are ill?
Many people, doctors included, answer "about ". The true answer is about .
"The test was positive" is new information. It does not change the world; it changes which part of the world we should look at. How to write "we know this event occurred" into a probability space is the question of this section.
Play first, read later. Below are a thousand people: red squares are ill, grey circles healthy; filled means a positive test, hollow a negative one. Count how many of the filled marks are red squares.
- ill · positive
- healthy · positive
- ill · negative
- healthy · negative
Experiment idea from: github.com/huzhuofan1020-svg/ybs
What did you observe?Play first, then answer. Read the explanation only after you have thought it through.
- 1.With the default settings, how many of the positives are actually ill? Why is that share so low when the sensitivity is 95%?
- 2.Drag the prevalence from 1% to 20%. How does the posterior change? The test itself has not changed.
- 3.Turn on “Retest the positives”. Why does the same test, taken a second time, give a much more reliable answer?
Among the positives only about are really ill, while about are healthy people with a false alarm. So the answer should be the share of the ill among the positives:
Knowing "positive", we have in effect shrunk the thousand people to these and recomputed proportions inside that smaller group.
Definition
Let . The conditional probability of given is
Read it exactly as in the conjecture: given that occurred, the sample space shrinks from to ; what remains of is ; dividing by makes the new whole space have probability again.
- event A (left column)
- A ∩ B
- B outside A
- height P(B)
Experiment idea from: github.com/huzhuofan1020-svg/Venn
What did you observe?Play first, then answer. Read the explanation only after you have thought it through.
- 1.Press “Independent”. What shape does B take? How do P(B | A) and P(B) compare?
- 2.Press “Mutually exclusive”. A and B look “unrelated”. Are they independent?
- 3.Turn on “Condition on B”. Which two areas does P(A | B) compare? Is it usually equal to P(B | A)?
What we get after this renormalisation is again a probability satisfying every axiom.
If , then satisfies Kolmogorov's axioms.
Non-negativity: and . Normalisation:
So every property proved in the previous section (complements, monotonicity, the addition rule) holds for with no new proof. Multiplying the definition by gives the form used most often:
Events and are independent if .
When this is equivalent to : knowing that occurred does not change our judgement of at all. In the experiment this is the case where has the same height inside and outside and forms a horizontal band. The definition uses a product rather than a conditional probability because the product is symmetric in and still makes sense when .
Independent is not mutually exclusive. Exclusive means : if one occurs, the other cannot. That is strong dependence. If and are exclusive, then , so they are never independent. Press "Mutually exclusive" in the experiment: has zero height in the column of , nothing like a horizontal band.
Theorems and proofs
Back to the opening problem: we know and want . Two theorems connect them.
Let be pairwise exclusive with and every . Then for every event ,
, a union of pairwise exclusive events, so . Apply the multiplication rule
The law of total probability is "splitting into cases": to find the total chance of , split by non-overlapping causes , compute each, and take the weighted sum.
Under the same assumptions, if , then for each
By the definition and the multiplication rule,
Bayes' theorem turns a "cause → effect" probability into an "effect → cause" probability . The turn requires multiplying by , our estimate of the cause before seeing the effect, called the prior; the result is the posterior.
Take "ill", "healthy", "positive". The prior is , with and .
The false-alarm term in the denominator is more than eight times the true-positive term : the ill are so rare that the absolute number of false alarms swamps the true ones.
and are in general different, and confusing them is the most common and most dangerous mistake. "An ill person tests positive with probability " is , not . Courts see the same error, the "prosecutor's fallacy": "an innocent person matches this evidence with probability one in a million" does not mean "a person who matches is innocent with probability one in a million". The latter also depends on how many people could match, that is, on the prior.
Applications
After a first positive, take the same test again and get another positive. Now the "prior" is no longer the population's but the from the first test:
One test moves to ; two move it to . This assumes the two tests are conditionally independent given the true state (false alarms are not caused by some persistent trait of the person); if false alarms tend to repeat, retesting is worth much less. Turn on "Retest the positives" in the experiment to see this step.
In the plotter, draw with the prior (prevalence) on the horizontal axis and the posterior after a positive on the vertical one. Near the posterior is about : the curve rises quickly at first, but as long as the prevalence is around or below, the posterior stays small. That is why a positive screen for a rare disease must be confirmed.
Further: Bayesian neural networks
Replace "ill or not" by a neural network's weights and "the test result" by the training data . Bayes' theorem does not change by a single symbol:
(Here is a probability density; for now read it as "probability for continuous quantities". The section on random variables will define it.)
Ordinary training finds one best set of weights. A Bayesian neural network keeps the whole posterior : the weights are no longer one fixed set of numbers but a joint distribution over all of them, saying how plausible each setting is. Predictions average over all plausible weights,
so where data are scarce the predictions spread out: the model knows when it does not know. That matters wherever "I'm not sure" beats a confident mistake, as in medicine or self-driving cars.
An unexpected connection: take only the single most probable set of weights and give each weight an independent normal prior . Then maximising is the same as minimising
The real difficulty is that the posterior cannot be computed: both the denominator and the predictive integral run over all weights, often millions of dimensions. In practice the posterior is approximated by variational inference or Monte Carlo dropout, or sampled directly with MCMC. That is a story for another course.
In this section events have only two states, occurring or not. The next section, random variables and distributions, turns outcomes into numbers: counts of tests, numbers of positives, waiting times. The law of total probability and Bayes' theorem become sums and integrals over distributions there.
Screening experiment idea from: github.com/huzhuofan1020-svg/ybs
Event-relations experiment idea from: github.com/huzhuofan1020-svg/Venn
Exercises
0 of 4 done
✓ All exercises in this section are done.
Next section: Random variables →
Discussion
Found a mistake, have a question or a suggestion? You can also write to 620026600@qq.com
No comments yet. Share your understanding, ask a question, or point out an error on this page.
Log in to join the discussion