LeoMath

Foundations of probability · Section 2 of 3

Conditioning & Bayes

Knowing an event occurred shrinks the sample space to it; then reason from effects back to causes.

about 12 min

Start from a problem

Problem

A disease affects 1% of a population. A test is positive for 95% of the ill, and also, falsely, for 8% of the healthy. Your result is positive. What is the probability that you are ill?

Many people, doctors included, answer "about 95%". The true answer is about 11%.

"The test was positive" is new information. It does not change the world; it changes which part of the world we should look at. How to write "we know this event occurred" into a probability space is the question of this section.

Observe

Play first, read later. Below are a thousand people: red squares are ill, grey circles healthy; filled means a positive test, hollow a negative one. Count how many of the filled marks are red squares.

Interactive experimentBayes in screening
  • ill · positive
  • healthy · positive
  • ill · negative
  • healthy · negative
positives 89of them ill 10share (whole people) 10/89 ≈ 0.112
P(ill|+)=P(+|ill)P(ill)P(+)prior → posterior 0.010 → 0.107

Experiment idea from: github.com/huzhuofan1020-svg/ybs

What did you observe?Play first, then answer. Read the explanation only after you have thought it through.

  1. 1.With the default settings, how many of the positives are actually ill? Why is that share so low when the sensitivity is 95%?
  2. 2.Drag the prevalence from 1% to 20%. How does the posterior change? The test itself has not changed.
  3. 3.Turn on “Retest the positives”. Why does the same test, taken a second time, give a much more reliable answer?
Conjecture

Among the positives only about 10 are really ill, while about 79 are healthy people with a false alarm. So the answer should be the share of the ill among the positives:

true positivestrue positives+false positives≈1089.

Knowing "positive", we have in effect shrunk the thousand people to these 89 and recomputed proportions inside that smaller group.

Definition

Definition 2.1Conditional probability

Let P(B)>0. The conditional probability of A given B is

P(A|B)=P(A∩B)P(B).

Read it exactly as in the conjecture: given that B occurred, the sample space shrinks from Ω to B; what remains of A is A∩B; dividing by P(B) makes the new whole space B have probability 1 again.

Interactive experimentConditioning as a ratio of areas
  • event A (left column)
  • A ∩ B
  • B outside A
  • height P(B)
P(A∪B) 0.600P(B|A) 0.600P(B|Aᶜ) 0.200P(A|B) 0.750P(A∩B) − P(A)P(B) 0.100 · positively related

Experiment idea from: github.com/huzhuofan1020-svg/Venn

What did you observe?Play first, then answer. Read the explanation only after you have thought it through.

  1. 1.Press “Independent”. What shape does B take? How do P(B | A) and P(B) compare?
  2. 2.Press “Mutually exclusive”. A and B look “unrelated”. Are they independent?
  3. 3.Turn on “Condition on B”. Which two areas does P(A | B) compare? Is it usually equal to P(B | A)?

What we get after this renormalisation is again a probability satisfying every axiom.

Proposition 2.1Conditional probability is a probability

If P(B)>0, then Q(A)=P(A|B) satisfies Kolmogorov's axioms.

Proof

Non-negativity: P(A∩B)≥0 and P(B)>0. Normalisation: Q(Ω)=P(Ω∩B)/P(B)=1. Countable additivity: if A1,A2,… are pairwise exclusive, so are A1∩B,A2∩B,…, and (⋃iAi)∩B=⋃i(Ai∩B), hence

Q(⋃iAi)=∑iP(Ai∩B)P(B)=∑iQ(Ai).

So every property proved in the previous section (complements, monotonicity, the addition rule) holds for P(⋅|B) with no new proof. Multiplying the definition by P(B) gives the form used most often:

Corollary 2.2Multiplication rule

P(A∩B)=P(B)P(A|B)=P(A)P(B|A) (whenever the conditional probabilities are defined).

Definition 2.2Independence

Events A and B are independent if P(A∩B)=P(A)P(B).

When P(B)>0 this is equivalent to P(A|B)=P(A): knowing that B occurred does not change our judgement of A at all. In the experiment this is the case where B has the same height inside and outside A and forms a horizontal band. The definition uses a product rather than a conditional probability because the product is symmetric in A,B and still makes sense when P(B)=0.

Common mistake

Independent is not mutually exclusive. Exclusive means A∩B=⌀: if one occurs, the other cannot. That is strong dependence. If P(A),P(B)>0 and A,B are exclusive, then P(A∩B)=0≠P(A)P(B), so they are never independent. Press "Mutually exclusive" in the experiment: B has zero height in the column of A, nothing like a horizontal band.

Theorems and proofs

Back to the opening problem: we know P(+|ill) and want P(ill|+). Two theorems connect them.

Theorem 2.3Law of total probability

Let B1,…,Bn be pairwise exclusive with ⋃iBi=Ω and every P(Bi)>0. Then for every event A,

P(A)=∑i=1nP(A|Bi)P(Bi).
Proof

A=⋃i(A∩Bi), a union of pairwise exclusive events, so P(A)=∑iP(A∩Bi). Apply the multiplication rule P(A∩Bi)=P(A|Bi)P(Bi) to each term.

The law of total probability is "splitting into cases": to find the total chance of A, split by non-overlapping causes Bi, compute each, and take the weighted sum.

Theorem 2.4Bayes' theorem

Under the same assumptions, if P(A)>0, then for each k

P(Bk|A)=P(A|Bk)P(Bk)∑i=1nP(A|Bi)P(Bi).
Proof

By the definition and the multiplication rule, P(Bk|A)=P(Bk∩A)P(A)=P(A|Bk)P(Bk)P(A); expand the denominator by the law of total probability.

Bayes' theorem turns a "cause → effect" probability P(A|Bk) into an "effect → cause" probability P(Bk|A). The turn requires multiplying by P(Bk), our estimate of the cause before seeing the effect, called the prior; the result P(Bk|A) is the posterior.

Example 2.1Solving the opening problem

Take B1= "ill", B2= "healthy", A= "positive". The prior is P(B1)=0.01, with P(A|B1)=0.95 and P(A|B2)=0.08.

P(ill|+)=0.95×0.010.95×0.01+0.08×0.99=0.00950.0887≈0.107.

The false-alarm term 0.0792 in the denominator is more than eight times the true-positive term 0.0095: the ill are so rare that the absolute number of false alarms swamps the true ones.

Common mistake

P(A|B) and P(B|A) are in general different, and confusing them is the most common and most dangerous mistake. "An ill person tests positive with probability 95%" is P(+|ill), not P(ill|+). Courts see the same error, the "prosecutor's fallacy": "an innocent person matches this evidence with probability one in a million" does not mean "a person who matches is innocent with probability one in a million". The latter also depends on how many people could match, that is, on the prior.

Applications

ApplicationRetesting: the posterior becomes the new prior

After a first positive, take the same test again and get another positive. Now the "prior" is no longer the population's 1% but the 0.107 from the first test:

P(ill|++)=0.95×0.1070.95×0.107+0.08×0.893≈0.59.

One test moves 1% to 11%; two move it to 59%. This assumes the two tests are conditionally independent given the true state (false alarms are not caused by some persistent trait of the person); if false alarms tend to repeat, retesting is worth much less. Turn on "Retest the positives" in the experiment to see this step.

Try it in the plotter →

In the plotter, draw 0.95x0.95x+0.08(1−x) with the prior (prevalence) x on the horizontal axis and the posterior after a positive on the vertical one. Near x=0 the posterior is about 12x: the curve rises quickly at first, but as long as the prevalence is around 1% or below, the posterior stays small. That is why a positive screen for a rare disease must be confirmed.

Further: Bayesian neural networks

Replace "ill or not" by a neural network's weights w and "the test result" by the training data D. Bayes' theorem does not change by a single symbol:

p(w|D)=p(D|w)p(w)p(D).

(Here p is a probability density; for now read it as "probability for continuous quantities". The section on random variables will define it.)

Ordinary training finds one best set of weights. A Bayesian neural network keeps the whole posterior p(w|D): the weights are no longer one fixed set of numbers but a joint distribution over all of them, saying how plausible each setting is. Predictions average over all plausible weights,

p(y|x,D)=∫p(y|x,w)p(w|D)dw,

so where data are scarce the predictions spread out: the model knows when it does not know. That matters wherever "I'm not sure" beats a confident mistake, as in medicine or self-driving cars.

An unexpected connection: take only the single most probable set of weights and give each weight an independent normal prior N(0,σ2). Then maximising log⁡p(D|w)+log⁡p(w) is the same as minimising −log⁡p(D|w)+12σ2‖w‖2: the negative log-likelihood plus an L2 penalty. (If the loss is averaged over examples or otherwise rescaled, the coefficient changes accordingly; with plain SGD, L2 regularisation is weight decay.) Regularisation is a prior.

The real difficulty is that the posterior cannot be computed: both the denominator p(D) and the predictive integral run over all weights, often millions of dimensions. In practice the posterior is approximated by variational inference or Monte Carlo dropout, or sampled directly with MCMC. That is a story for another course.

Remark

In this section events have only two states, occurring or not. The next section, random variables and distributions, turns outcomes into numbers: counts of tests, numbers of positives, waiting times. The law of total probability and Bayes' theorem become sums and integrals over distributions there.

Screening experiment idea from: github.com/huzhuofan1020-svg/ybs

Event-relations experiment idea from: github.com/huzhuofan1020-svg/Venn

Exercises

0 of 4 done

1
Roll two fair dice. Given that the sum is 8, what is the probability that the first die shows 6? (A fraction is fine.)

Decimals or fractions; anything within 0.001 of the answer counts as correct.

2
Machines A and B make 60% and 40% of the parts, with defect rates 2% and 5%. A randomly chosen part is defective. What is the probability it came from B? (Three decimals.)

Decimals or fractions; anything within 0.0006 of the answer counts as correct.

3
Let P(A)>0, P(B)>0, and let A and B be mutually exclusive. Then A and B are
Answer to exercise 3
4
Use the test from the article: prevalence 1%, sensitivity 95%, false-positive rate 8%. Someone tests positive twice (the tests are conditionally independent given the true state). What is the probability that they are ill? (Two decimals.)

Decimals or fractions; anything within 0.006 of the answer counts as correct.

Discussion

Found a mistake, have a question or a suggestion? You can also write to 620026600@qq.com

No comments yet. Share your understanding, ask a question, or point out an error on this page.

Log in to join the discussion