How Science Knows — Critical Thinking 7 (C1)

CRITICAL THINKING · LESSON 7

How Science Knows

C1 (B2–C2) · 90 minutes

Your science instincts

Task Write a number from 1 to 5 next to each statement.

1

strongly agree

2

mostly agree

3

on the fence

4

mostly disagree

5

strongly disagree

Science proves things.

A theory that can explain every case is a strong theory.

If a study is published in a peer-reviewed journal, it is probably true.

One large experiment is worth more than a thousand personal stories.

When a famous finding fails to replicate, our trust in science should go down.

READING

The theory that explained too much

In 1919, Karl Popper was a young man in Vienna, helping with Alfred Adler's social work among working-class children. His friends admired three theories, Marx's, Freud's and Adler's, because each seemed to explain almost everything. Popper came to see this as their weakness.

He later gave an example. One man pushes a child into the water to drown it; another man gives his life trying to save a child. Freud's theory explains both: the first man is repressing a drive, the second has sublimated it. Adler's

theory explains both too: each is driven by feelings of inferiority, one proving he dares to commit a crime, the other that he dares to rescue. Whatever people do, the theory fits. When Popper once described a case to Adler, Adler analysed it confidently without having seen the child, citing his "thousandfold experience". Popper's thought: and now it is a thousand-and-one-fold.

Einstein's general theory of relativity was different. It predicted that the Sun's gravity would bend starlight by a definite amount.

READING · CONTINUED

The theory that explained too much

In 1919, British expeditions, the best known led by Arthur Eddington, photographed stars during a total eclipse to check. The theory could have failed in public. It survived.

From this contrast Popper built his criterion of falsifiability: a theory is scientific only if some possible observation could show it to be false. Confirmations are cheap; what counts is a risky prediction that survives a serious attempt to refute it. On this view science never proves a theory true. It only fails, so far, to prove it false.

Critics reply that scientists rarely drop a theory after one failed test: they doubt the instruments or adjust an assumption, often rightly.

Falsifiable or not?

Task Write F (falsifiable) or N (not falsifiable) next to each letter. For every F, write one observation that would prove the claim wrong. For every N, say what protects it from refutation, then try to rewrite it so that it could be tested.

A. People born under Leo are natural leaders.

E. The universe is expanding: distant galaxies are moving away from us.

B. This remedy works, but only if you truly believe in it. If it doesn't work for you, your doubts blocked it.

F. Tomás says the 7.40 train from his village is always late on Mondays.

G. A market commentator: "Shares fell today because investors are nervous. Yesterday they rose because nervous investors were buying safe shares."

C. Drinking two cups of coffee a day lowers the risk of type 2 diabetes.

D. Everything happens for a reason.

H. Every living species evolved from earlier species by natural selection.

Is an unfalsifiable claim worthless, or just not science?

READING

Normal science and revolutions

In The Structure of Scientific Revolutions (1962), the physicist turned historian Thomas Kuhn drew a different picture from Popper's. Most of the time, he argued, scientists do normal science: puzzle-solving inside a paradigm, a shared package of theories, methods, model problems and standards for a good answer. Normal scientists do not try to refute their paradigm; they take it for granted. Results that don't fit, anomalies, are set aside or blamed on the equipment. Only when anomalies pile up does a field fall into crisis, and a rival paradigm may then win in a

scientific revolution: Copernicus's Sun-centred astronomy, Lavoisier's oxygen chemistry replacing phlogiston, Einstein's physics. Kuhn added that rival paradigms are hard to compare point by point, because they disagree even about which problems matter; he called this incommensurability. Critics charged that he made theory choice irrational, a matter of persuasion and generational change, which Kuhn denied. The philosopher Margaret Masterman counted at least 21 senses of "paradigm" in the book.

Kuhn's cycle

Paradigm

Normal science: puzzle-solving

Anomalies pile up

Revolution: a new paradigm

Crisis

1. A scientist protects her theory from one awkward result. Is that stubborn or sensible?

2. Can you name a real paradigm shift in medicine, economics or your own field?

3. If paradigms decide what counts as evidence, can science still be objective?

Build the pyramid

Task Number the five kinds of evidence from 1 (the bottom of the pyramid: weakest protection against bias) to 5 (the top: strongest). Then write the level number next to each study A to D.

Randomised controlled trial (RCT)

People are assigned by chance to a treatment group or a comparison group, then followed up.

Anecdote and expert opinion

A personal story, or the view of an experienced person, without systematic data.

Systematic review or meta-analysis

A search for every good study on one question, with the results appraised and, where possible, combined.

Case study

A detailed account of one patient, one event or a handful of cases.

Observational study

Researchers follow or compare large groups without deciding who gets what.

Build the pyramid: the studies

5

more protection against bias

A. Researchers in Nairobi follow 12,000 adults for 15 years and compare the heart health of those who cycle to work with those who don't.

4

B. A doctor in Lyon writes up an unusual reaction in one patient who took two common medicines together.

3

C. A team finds all 31 published trials of a reading programme worldwide and pools their results.

2

1

D. 600 schools in Brazil are split by lottery: half get a new maths programme, half carry on as normal.

Which of A to D would you trust most to change your own behaviour, and why?

READING

The pyramid and its critics

The hierarchy of evidence ranks study designs by how well they protect against bias. Anecdotes and opinion sit at the bottom; case studies can spot something new but cannot show cause; observational studies reveal associations but leave room for confounding, a hidden third factor. RCTs use chance to make the groups alike, on average, in everything except the treatment. At the top, systematic reviews and meta-analyses combine all the good studies on a question.

Critics accept the logic but not the rigidity. The link between smoking

and lung cancer was established in the 1950s by observational studies, by Richard Doll and Austin Bradford Hill among others, because nobody could ethically assign people to smoke. A meta-analysis is only as good as the studies fed into it. And some questions need no trial at all: in 2003 the BMJ published a mock systematic review that found no randomised trials of parachutes. Systems such as GRADE now rate the quality of each body of evidence, so a strong observational study can end up rated above a weak trial.

READING

When findings don't hold up

Peer review, in which editors send a manuscript to independent experts before publication, is often treated as a seal of truth. It is much weaker than that. Reviewers judge whether a study is plausible and clearly reported; they rarely see the raw data and almost never repeat the work. In a trial published in 2008, Sara Schroter and colleagues sent BMJ reviewers papers with nine major errors deliberately inserted; on average, reviewers spotted fewer than three.

The real test is replication: does the result appear again when someone else repeats the study? In 2015 the Open Science Collaboration, a large network of researchers, repeated 100 studies published

in 2008 in three leading psychology journals. Of the originals, 97% had reported statistically significant results; only 36% of the replications did, and the average effect was about half the original size. Causes include small samples, publication bias (journals favour positive results) and flexible analysis, sometimes called p-hacking: trying analyses until one "works". The responses include preregistration, publicly recording hypotheses and analysis plans before collecting data, and registered reports, first offered by the journal Cortex in 2013, in which a journal accepts a study on its design before the results exist. A failed replication does not always prove the original wrong: samples and settings differ.

CASE STUDY

Case study: power posing

In 2010 Dana Carney, Amy Cuddy and Andy Yap reported in Psychological Science that 42 participants who held two expansive "high-power" poses, one minute each, felt more powerful, took more risks in a gambling task, and showed a rise in testosterone and a fall in the stress hormone cortisol. Cuddy's 2012 TED talk made the idea famous worldwide.

In 2015 Eva Ranehill and colleagues ran a larger version with 200 participants. People again said they felt more powerful, but there were no significant effects on hormones or on risk-taking. The original authors pointed to differences in the

procedure. In 2016 Carney, the first author, stated publicly that she no longer believed power-pose effects were real, citing the small sample and flexible analysis of the original study. Cuddy disagreed. In 2018 she and two colleagues analysed a wider set of studies and argued that the effect on feelings of power is robust. Critics replied that much of it may come from slumped poses making people feel worse, not expansive poses making them feel better.

What held up: a modest effect on how powerful people say they feel. What did not: the hormones and the risk-taking.

The debate: broken or working?

MOTION

The power posing story shows that science is broken.

Task Before you prepare, write your number privately. In your room, share out the three roles below and write notes for a one-minute speech each. Use facts from today's slides.

1

2

3

4

5

strongly agree

mostly agree

on the fence

mostly disagree

strongly disagree

The debate: sides and roles

Side A

Side B

Argue that the story shows science is broken. Think about: who checked the study before it became famous, and how many other findings might be like it.

Argue that the story shows science works. Think about: who found the problem, how quickly, and what has changed since.

Opener

Evidence

Rebuttal

States your side's case in one minute and names your two strongest reasons.

Gives the facts and figures that support you: from the case, the 2015 project, peer review, Popper or Kuhn.

Answers the other side's best point, then sums up.

Private reflection

Task Choose one prompt and write privately. Nobody will read what you write. Sharing afterwards is optional.

Prompt 1

Think of something you believe because "studies show". What evidence would make you give it up?

Prompt 2 (lighter)

Think of a health or lifestyle tip you have heard many times. Where on the pyramid does it probably come from?

Back to your instincts

Task Look at your numbers from "Your science instincts". Change any you now disagree with.

1

strongly agree

2

mostly agree

3

on the fence

4

mostly disagree

5

strongly disagree

Science proves things.

A theory that can explain every case is a strong theory.

If a study is published in a peer-reviewed journal, it is probably true.

One large experiment is worth more than a thousand personal stories.

When a famous finding fails to replicate, our trust in science should go down.

Which statement moved most for you, and what moved it?

Before Lesson 8

This week, find one post or news story that says "a study shows" and trace the study.

Write down where it sits on the pyramid, how many people it studied, and whether anyone has tried to replicate it.

Next lesson: Truth in the Feed. How claims travel online, and how to check them before you share.

TEACHER'S REFERENCE · SKIP OR DELETE BEFORE CLASS

Answer key (1)

FALSIFIABLE OR NOT? suggested; A, F and G are there to be argued over

• A. F as stated: compare leadership (posts held, personality scores) across birth signs; if Leos are no more likely to lead, it is false. It becomes N as soon as failures are explained away ("your rising sign overrides it"). Popper put astrology on the non-science side for exactly this habit.

• B. N. The escape clause absorbs every failure. Testable version: compare the remedy with a placebo in a blinded trial, believers and doubters alike.

• C. F. A large trial or long cohort study finding that coffee drinkers have the same or a higher rate of diabetes, once age, weight and income are allowed for.

• D. N. No event could count against it.

• E. F. Light from distant galaxies showing no shift towards red, or a shift towards blue, that grows with distance.

• F. F. A single Monday when the 7.40 leaves on time refutes "always". If Tomás really means "usually", the claim needs a count over many Mondays, and becomes a weaker but fairer one.

• G. N as stated: "nervousness" explains rises and falls after the event, like Freud and Adler explaining both men. It becomes F if nervousness is measured in advance and used to predict the direction of the market.

• H. F. Fossils in the wrong order, for example mammal fossils in rock older than the first fish, would count against it.

TEACHER'S REFERENCE · SKIP OR DELETE BEFORE CLASS

Answer key (2)

BUILD THE PYRAMID

• Levels: 1 Anecdote and expert opinion · 2 Case study · 3 Observational study · 4 Randomised controlled trial · 5 Systematic review or meta-analysis

• A. 3, observational (a cohort study). B. 2, case study (a single case report). C. 5, systematic review or meta-analysis. D. 4, RCT (randomised by school, a "cluster" trial).

• Versions of the pyramid differ: some split observational studies into cohort (higher) and case-control (lower), and some leave expert opinion off altogether.