Numbers That Lie — Critical Thinking 6 (C1)

CRITICAL THINKING · LESSON 6

Numbers That Lie

C1 (B2–C2) · 90 minutes

Guess first

Task Write down an answer to each question before anyone discusses them. A guess is fine.

1

strongly agree

2

mostly agree

3

on the fence

4

mostly disagree

5

strongly disagree

1. A screening test for a disease catches 90 out of every 100 people who have it. It wrongly flags 9 out of every 100 who don't. One person in 100 has the disease. You test positive. What is the chance that you have it? Write a percentage.

2. "A new pill cuts your risk of a heart attack by 50%." Score this statement from 1 to 5: I would start taking it.

3. "People who own more books live longer, so buying books is good for your health." Score from 1 to 5: This is a reasonable conclusion.

READING

Two things together

When two things rise and fall together, they are correlated. A correlation is a fact about data. Causation is a claim about the world: that changing one thing would change the other. Most bad reasoning with numbers happens in the step from the first to the second.

A correlation can mislead in three main ways. The first is a confounder: a third factor that drives both. People who carry a cigarette lighter get lung cancer more often than

people who don't. The lighter is innocent; smoking explains both. Big fires bring more fire engines and cause more damage, but nobody concludes that fire engines cause damage.

The second is reverse causation: the arrow runs the other way. People who exercise a lot tend to be healthier, but people who are already healthy also find it easier to exercise.

The third is chance.

READING · CONTINUED

Two things together

Search enough pairs of variables and some will move together by coincidence. A dataset with 100 variables contains 4,950 possible pairs.

How, then, do researchers get from correlation to cause? The strongest tool is the randomised controlled trial: people are assigned to a treatment group or a control group by chance, so that confounders, known and unknown, are on average spread evenly

between the groups. Where trials are impossible or unethical, as with smoking, scientists build the case from several lines of evidence. In 1965 the epidemiologist Austin Bradford Hill set out a well-known list of such considerations, including a strong and consistent link, the right timing, and a dose-response pattern: the more exposure, the more harm. Lesson 7 returns to how science settles such questions.

READING

The forgotten denominator

Take the test on the opening slide. In a 2007 training session, Gerd Gigerenzer gave a version of this problem, about breast-cancer screening, to 160 gynaecologists. Their answers were scattered widely, and the most common answer was 90%. The correct answer is about 10%.

The mistake is to ignore the base rate: how common the condition is before anyone is tested. Gigerenzer's remedy is to turn percentages into

natural frequencies: counts of imagined people. Picture 1,000 people. About 10 have the disease, and 9 of them test positive. Of the 990 who don't, about 89 also test positive. So 98 people test positive, and only 9 of them are ill: roughly 1 in 10. When a condition is rare, the false alarms among the many healthy people outnumber the true alarms among the few who are ill.

READING · CONTINUED

The forgotten denominator

In experiments by Gigerenzer and his colleagues, doctors and students solved such problems far more often when they were stated in natural frequencies.

A second trap is the gap between relative risk and absolute risk. In October 1995 the UK's Committee on Safety of Medicines warned that newer, "third-generation" contraceptive pills doubled the risk of a dangerous blood clot. That was a relative increase of 100%.

In absolute terms, about 1 in 7,000 women on the older pills had a clot, against about 2 in 7,000 on the newer ones. Many women stopped taking the pill in alarm. Gigerenzer and his colleagues cite an estimate of 13,000 additional abortions in England and Wales the following year.

The lesson of both cases is the same question: out of how many?

Work it out

Task Copy the tree and fill in the five blanks, then answer the questions. Use only counts of people, not percentages.

Part A: a test for Condition K. 5 in every 100 people have Condition K. The test catches 90 out of every 100 people who have it. It wrongly flags 10 out of every 100 people who don't.

1,000 people tested

1

have K

2

do not have K

3

test positive

5 test negative

4

test positive

855 test negative

All positives:

5

Work it out

Task Copy the tree and fill in the five blanks, then answer the questions. Use only counts of people, not percentages.

Part A: a test for Condition K.

Part B: a drug for strokes. In a five-year trial, 4 in every 1,000 people on a placebo had a stroke. On Drug X, 3 in every 1,000 did.

1. How many people test positive altogether?

2. Of those, how many really have K? So what is the chance that a positive result is right?

3. Explain in one sentence why the answer is so much lower than 90%.

4. Now the test is used only on people who already have symptoms, and half of them have K. Redraw the tree for 1,000 such people. What is the chance now?

5. By what percentage did Drug X reduce the risk? (This is the relative risk reduction.)

6. How many fewer strokes per 1,000 people? (This is the absolute risk reduction.)

7. How many people must take Drug X for five years to prevent one stroke? (This is the number needed to treat.)

8. Which of these three numbers would an advert print? Which would you want your doctor to tell you?

READING

Missing planes, bent pictures

Survivorship bias is the error of drawing conclusions only from the cases that made it through a filter, while forgetting the ones that didn't.

Its most famous illustration involves the statistician Abraham Wald, who worked in the Statistical Research Group at Columbia University during the Second World War. In the popular version, officers show Wald the bullet holes on bombers returning from missions and propose adding armour where

the holes cluster. Wald replies that the armour belongs where there are no holes: planes hit there did not come back.

What is documented is more modest. In 1943 Wald wrote a series of technical memoranda on estimating how vulnerable different parts of an aircraft were, using the damage seen on planes that survived. They were published by the Center for Naval Analyses in 1980.

READING · CONTINUED

Missing planes, bent pictures

The insight that survivors are a biased sample is genuinely there, in mathematical form. The dramatic meeting comes from later retellings, and the mathematician Bill Casselman has argued that there is little evidence for its most vivid details.

The bias is everywhere: "old buildings were built better" (the badly built ones fell down long ago); "successful founders took big risks" (so did many who failed and were

never interviewed).

Graphs can mislead without a single false number. A truncated axis, a vertical axis that starts well above zero, makes small differences look huge. A cherry-picked window shows only the stretch of time that tells the story the author wants. The first questions to ask of any chart: where does the axis start, and where does the timeline start and end?

READING

When the parts disagree with the whole

In 1986 a team of London urologists, Charig and colleagues, compared two treatments for kidney stones: open surgery and a less invasive keyhole procedure. Overall, the keyhole procedure succeeded more often. Yet open surgery did better for small stones and for large stones. Doctors had tended to give open surgery to the harder cases, the large stones, so stone size was a confounder.

This is Simpson's paradox: a

pattern that holds in every group reverses when the groups are combined. The best-known case is graduate admissions at the University of California, Berkeley, in 1973: 44% of male applicants were admitted, against 35% of women. Department by department, Bickel, Hammel and O'Connell (1975) found no evidence of bias against women. Women had applied more often to departments that rejected most of their applicants.

When the parts disagree with the whole

Open surgery

Keyhole procedure

Small stones (under 2 cm)

81 of 87 = 93%

234 of 270 = 87%

Large stones (2 cm or more)

192 of 263 = 73%

55 of 80 = 69%

All patients

273 of 350 = 78%

289 of 350 = 83%

1. You have a kidney stone. Which treatment would you choose, and what else would you want to know?

2. A newspaper prints only the bottom row. Is that a lie?

3. How do you know when you have broken the data down far enough?

Data detective: cases A and B

Task For each chart, write three things: (1) the trick, (2) what the data really shows, in one sentence, and (3) one question you would ask the source.

Case A. Headline: "Customers three times happier with SkyVela!"

Case B. Headline: "Burglary rising every year since 2020 under Mayor Quint"

Customer satisfaction, 2025 survey

Recorded burglaries, city of Tarsel

From the city's open-data website, all years:

2014 = 3,200

2015 = 3,050

2016 = 2,900

2017 = 2,750

2018 = 2,600

2019 = 2,450

2020 = 2,100

2021 = 2,200

2022 = 2,300

2023 = 2,350.

84%

2,400

83%

83%

2,300

82%

2,200

81%

81%

2,100

80%

Airline Ostra

SkyVela

2,000

2020

2021

2022

2023

Data detective: cases C and D

Task Same three things for each chart: the trick, what the data really shows, and one question for the source.

Case C. Headline: "Want to live longer? Move near a café"

Case D. Advert: "Our managed funds beat the market. Switch today!"

Cafés and life expectancy, eight districts of one city, 2024

Average yearly return, 2015–2024

Average life expectancy in years

8%

82

6%

7.2%

80

6.0%

78

4%

76

2%

74

0%

72

Our funds

Market index

0

Cafés per 10,000 residents

5

10

15

20

Based on the 62 funds in our range still open on 31 December 2024.

Rewrite the headline

Task Rewrite each headline so that it is honest but still worth reading, in 15 words or fewer. Then name the trap in a word or two.

"Eating breakfast makes children smarter"

1

The facts: a survey of 3,000 pupils found that those who ate breakfast scored higher in school tests on average. There was no experiment.

"New drug slashes risk of hip fracture by 40%!"

2

The facts: in a three-year trial, 5 in 1,000 people on a placebo broke a hip, against 3 in 1,000 on the drug.

"Exports fall for the third month running"

3

The facts: exports fell slightly in June, July and August, by 2% in total. Summer is always the slowest season, and exports were 11% higher than in the same three months last year.

Rewrite the headline

Task Rewrite each headline so that it is honest but still worth reading, in 15 words or fewer. Then name the trap in a word or two.

"Happy people earn more: proof that money buys joy"

4

The facts: in a national survey, people with higher incomes reported higher life satisfaction on average.

"Test for Condition M is 95% accurate: get tested now!"

5

The facts: the test catches 95 in 100 people who have M and wrongly flags 5 in 100 who don't. About 1 in 1,000 people have M.

"Pay rockets 8% this year"

6

The facts: average (mean) pay rose 8%, mostly because of large bonuses at the top. Typical (median) pay rose 1%, while prices rose 4%.

Talking it over

Task Choose three or four questions. Spend most of your time on the one your group disagrees about.

1. Should the law require health stories and adverts to give absolute risk, not just relative risk?

2. Is it acceptable to frame numbers dramatically if it gets people to stop smoking?

3. Whose numbers do people trust most where you live: the government, scientists or journalists?

4. Wald's bomber story is partly legend. Does it matter if a good teaching story isn't quite true?

5. Should schools teach risk and statistics instead of some of the algebra?

6. Can "correlation is not causation" become an excuse to dismiss any evidence you dislike?

Private reflection

Task Choose one prompt and write privately. Nobody will read what you write. Sharing afterwards is optional.

Prompt 1

Think of a number you once believed about health, money or crime that you would now question. Where did it come from, and why did it convince you?

Prompt 2 (lighter)

Think of a statistic you saw this week, in an advert, the news or your feed. Which question from today would you now ask about it?

Back to your guesses

Task Look again at your three answers from the start of the lesson. Change any score you would now give differently.

1. For Question 2, what did you need to know that the headline didn't tell you?

2. For Question 3, what is the most likely confounder?

Before Lesson 7

This week, find one number in the news or an advert and ask three questions: out of how many, compared with what, and who is missing?

1

Bring the number and your three answers to the next lesson.

2

Next lesson: How Science Knows. Experiments, peer review, and why some famous findings fail to replicate.

3

TEACHER'S REFERENCE · SKIP OR DELETE BEFORE CLASS

Answer key (1)

GUESS FIRST

1 About 10%. Of 1,000 people, 10 are ill and 9 of them test positive; of the 990 who are well, about 89 test positive. So 9 of 98 positives are ill: roughly 1 in 10.

2 No single answer. The headline gives only the relative risk; the decision depends on the absolute risk (50% of what?), the side effects and the cost.

3 No single answer, but the conclusion is weak: a likely confounder is wealth or education, which goes with both owning books and living longer.

WORK IT OUT

• Tree: blank 1 = 50 have K; blank 2 = 950 do not; blank 3 = 45 true positives; blank 4 = 95 false positives; blank 5 = 140 positives in all.

1 140.

2 45 of 140, about 32%, or roughly 1 in 3.

3 Because K is fairly rare, the 10% false alarms among the 950 healthy people (95) outnumber the true alarms among the 50 who are ill (45).

4 500 have K and 450 of them test positive; 500 don't and 50 of them test positive. 450 of 500 positives are right: 90%. Same test, different base rate.

5 From 4 to 3 in 1,000: a 25% relative risk reduction.

6 1 in 1,000 fewer strokes (0.1 percentage points).

7 1,000 people for five years to prevent one stroke.

8 An advert would print the 25%. Open question: most people would want all three, plus the side effects. (Gigerenzer reports the same proportions for mammography screening and breast-cancer deaths: about 4 versus 3 per 1,000 women over roughly ten years.)

TEACHER'S REFERENCE · SKIP OR DELETE BEFORE CLASS

Answer key (2)

DATA DETECTIVE suggested

• A. Truncated axis. The gap is 2 percentage points, 81% against 83%; the axis starting at 80% makes it look threefold. Ask: how many people were surveyed, and is a 2-point gap larger than the survey's margin of error?

• B. Cherry-picked window (plus a truncated axis). The four years shown do rise, but 2020 was the lowest year in the series; in 2023 burglaries were still about a quarter below the 2014 level (2,350 against 3,200). The headline also credits the change to the mayor, a causal claim the data can't support. Ask: why start in 2020, and was 2020 unusual for other reasons?

• C. Correlation presented as causation. Likely confounders: income, wealth, education, the age of the residents. Cafés open where people have money to spend; money and its effects also lengthen lives. Moving near a café would not change any of that. Ask: do the districts with more cafés also have higher incomes?

• D. Survivorship bias. Funds that closed, often because they did badly, are left out of the average, so the survivors look better than the range as a whole did. Ask: how many funds did you have in 2015, and what returns did the closed ones make?

REWRITE THE HEADLINE suggested; many good rewrites are possible

1 "Children who eat breakfast score higher in tests, survey finds." Trap: correlation and causation (family income or routine could explain both).

2 "New drug prevents about 2 hip fractures in every 1,000 people over three years." Trap: relative versus absolute risk (500 people treated for three years to prevent one fracture).

3 "Exports dip over summer but are 11% up on last year." Trap: cherry-picked window (and seasonality).

4 "Richer people report more life satisfaction, national survey finds." Trap: causation claimed from a correlation, and the headline's two halves point in opposite directions (reverse causation).

TEACHER'S REFERENCE · SKIP OR DELETE BEFORE CLASS

Answer key (3)

5 "Rare-disease test flags many healthy people: most positives are false alarms." Trap: base rate. Of 10,000 tested, about 10 have M (and about 9 or 10 test positive), while about 500 healthy people also test positive: roughly 1 positive in 50 is real.

6 "Typical pay up 1% as prices rise 4%." Trap: mean versus median, and ignoring inflation (typical pay fell in real terms).