The Moral Circle: Machines, Animals, the Future — Ethics 10 (C1)

ETHICS · LESSON 10

The Moral Circle: Machines, Animals, the Future

C1 (B2–C2) · 90 minutes

Moral Machine, our class (1 and 2)

Task A driverless car's brakes have failed. It can stay in its lane (A) or swerve into the other lane (B). Everyone in the lane it enters will be killed. For each dilemma, write A or B privately. Don't discuss yet.

Dilemma 1

Dilemma 2

four adults

one adult

two elderly people

two children

A: stay

B: swerve

A: stay

B: swerve

Lane A (stay): four adults crossing the road legally.

Lane B (swerve): one adult crossing legally.

Lane A (stay): an elderly man and an elderly woman.

Lane B (swerve): a boy and a girl.

Moral Machine, our class (3 and 4)

Task Write A or B privately for dilemmas 3 and 4. Then show your votes for all four when your teacher asks.

Dilemma 3

Dilemma 4

two homeless people

two executives

crossing on green

crossing on red

A: stay

B: swerve

A: stay

B: swerve

Lane A (stay): two homeless people.

Lane B (swerve): two company executives.

Lane A (stay): two adults crossing legally on a green signal.

Lane B (swerve): two adults crossing against a red signal.

What the world chose

From 2016 the MIT Media Lab's Moral Machine website asked visitors to judge dilemmas like these. By 2018 it had collected about 40 million decisions in ten languages from people in 233 countries and territories (Awad and colleagues, Nature, 2018).

The strongest global preferences

Spare humans over pets. Spare more lives over fewer. Spare the young over the old.

Weaker, but present

Spare the law-abiding, people of higher status, the fit, women, and pedestrians over passengers. A slight preference for staying in lane rather than swerving.

What the world chose

Western cluster

Eastern cluster

Southern cluster

North America and many European countries. A stronger preference than the others for inaction: staying in lane.

Japan, Taiwan and other countries of Confucian heritage, and Islamic-majority countries such as Indonesia, Pakistan and Saudi Arabia. A much weaker preference for sparing the young; a stronger one for sparing the law-abiding.

Latin America, plus France and countries with a history of French influence. A stronger preference for sparing women, the young, the fit and people of higher status; a weaker preference for sparing humans over pets.

Patterns behind the clusters

In more individualist cultures, a stronger preference for sparing more lives. In more economically unequal countries, a stronger preference for sparing higher status. Where institutions and the rule of law are weaker, more tolerance of people crossing on red.

READING

Polling the planet

The Moral Machine is the largest survey of moral preferences ever run, but it measures something narrower than its headlines suggest. Its respondents chose to play an online game, so they were self-selected internet users, not a representative sample of any country. They made quick choices about cartoon figures, with nothing at stake. What people click in a game may differ from what they would accept on a real road.

The format also shapes the answers. Every scenario forced a choice between two groups. In 2020 the psychologists Yochanan Bigman and Kurt Gray repeated the task with a third option: treat both groups equally. Most participants chose it for most factors; the main exception was the number

of lives. The original authors replied that answers to such an option depend heavily on how it is worded.

The deepest objection is philosophical. A survey reports what people prefer; it cannot show what is right. In 2017 Germany's Ethics Commission on Automated and Connected Driving set out guidelines stating that, in an unavoidable accident, a car must not distinguish between people by personal features such as age or gender, though it may be programmed to reduce the number injured. That rules out acting on the global preference for sparing the young. Should engineers follow the crowd, or the commission?

READING

When the algorithm harms

Since Aristotle, philosophers have tied blame to two conditions: control (you could have acted otherwise) and knowledge (you knew, or should have known, what you were doing). Automated systems strain both. In 2004 the philosopher Andreas Matthias described a responsibility gap: when a machine learns from data and acts in ways its makers cannot predict, nobody seems to have enough control or knowledge to be fully to blame. The political theorist Dennis Thompson

had already named a related difficulty in 1980, the problem of many hands: when many people each contribute a small part to a harm, it can look as if no one did it.

In March 2018, in Tempe, Arizona, an Uber test car driving itself struck and killed a woman who was wheeling a bicycle across a road at night. It was widely reported as the first pedestrian death involving a self-driving car.

READING · CONTINUED

When the algorithm harms

The US National Transportation Safety Board found that the immediate cause was the safety driver, who was distracted by her phone. It also blamed Uber's "inadequate safety culture": among other things, the company had switched off the car's factory-fitted emergency braking while its own system drove. Prosecutors did not charge Uber. The driver was charged with negligent homicide and in 2023 pleaded guilty to a lesser charge of endangerment.

The researcher Madeleine Clare Elish calls this pattern a moral crumple zone. A car's crumple zone absorbs the force of a crash to protect the passengers; here, the nearest human absorbs the blame and protects the system. Some propose strict liability for the companies; others insist on meaningful human control over machines.

Share the blame

Task Read the case. Privately share out 100 blame points among A to E (zero is allowed). Then compare with your partner: where do your numbers differ most, and is anyone in a moral crumple zone?

The case In the fictional city of Vellmar, a driverless shuttle run by Orvane Mobility fails to brake for a cyclist at dusk and injures her badly. Investigators find the facts below.

A

Tomasz Lind, the remote supervisor, was watching eight shuttles at once and was looking at another screen.

B

Orvane's managers had raised the number of shuttles per supervisor from four to eight to cut costs.

C

Kestrel Vision, which made the camera software, had trained it on few images of cyclists at dusk, though it had warned Orvane in writing that low-light performance was weak.

D

Vellmar city council approved the trial without independent safety tests.

E

The shuttle itself, whose software made the decision not to brake.

Draw your moral circle

Task Write privately the number of the outermost ring whose members you think have real moral claims on you. Then write partly beside any ring you include only in part. You may also change the order of rings 6 to 8 in your notes.

8

4

5

6

7

1. Me

2. My family

3. My community

4. My nation

5. All humanity alive today

6. Animals

7. People not yet born

8. Machines?

2

3

1

READING

The expanding circle

In 1869 the Irish historian W. E. H. Lecky described moral progress as a widening circle: from the family to a class, then a nation, then all humanity, and finally the animal world. Peter Singer took up the image in The Expanding Circle (1981), but his best-known book is Animal Liberation (1975). It starts from Jeremy Bentham's remark of 1789: "The question is not, Can they reason? nor, Can they talk? but, Can they suffer?"

Singer, a utilitarian, defends the principle of equal consideration of interests: a similar interest, such as the interest in not suffering pain, counts the same whoever has it. This does not mean treating animals and humans identically; a pig has no interest in voting. It means suffering cannot be discounted because the sufferer belongs to another species.

READING · CONTINUED

The expanding circle

To do so is speciesism, a term coined by the psychologist Richard Ryder in 1970, which Singer compares to racism and sexism. His main targets were factory farming and animal experiments. The idea is not only Western: Jain, Buddhist and Hindu traditions have taught ahimsa, non-harm to living beings, for more than two thousand years.

Critics answer in several ways. Kant held that only rational beings are owed direct duties: cruelty to

animals is wrong mainly because it hardens us towards people. Contractualists say morality is an agreement among those able to make and keep agreements. Singer replies that infants cannot do so either, yet clearly count. Bernard Williams argued that favouring humans is not a prejudice like racism, because the human point of view is the only one from which we do ethics at all.

READING

People not yet born

Carbon dioxide released today will keep warming the planet for centuries, and most of the people who will live with the results have not been born. They cannot vote, bargain or protest. Do we owe them anything? Some institutions say yes. The Well-being of Future Generations (Wales) Act 2015 created a commissioner to speak for future people. In 2021 Germany's Federal Constitutional Court found the national climate law partly unconstitutional, because it pushed too much of the burden of cutting

emissions onto the years after 2030, threatening the freedoms of the young.

The philosophy is harder than it looks. In Reasons and Persons (1984) Derek Parfit described the non-identity problem. Large policies change who meets whom and when children are conceived, so they change which people will exist. Imagine a society that chooses to use up its resources rather than conserve them, so that life two centuries later is much worse.

READING · CONTINUED

People not yet born

The people born then would not have existed under the other policy. If their lives are still worth living, they are not worse off than they would otherwise have been. So whom has the policy harmed? Parfit thought the choice was clearly wrong, and that ethics needed a theory explaining why.

There are practical doubts too. Economists disagree about how far to discount future wellbeing: the Stern Review (2006) weighted it almost equally with ours, while critics such as William Nordhaus argued for a steeper discount. And some argue that duties to the poor alive today come first.

Where does your circle end?

Task Choose three or four questions. Spend most of your time on the one your group disagrees about. Has anyone's outermost ring moved since "Draw your moral circle"?

1. Is Lecky's widening circle a story of real moral progress, or just a story we like to tell?

2. Is favouring your own species a prejudice like racism, or something different?

3. Do we owe as much to a person born in 2200 as to a stranger alive today?

4. Could a machine ever deserve moral consideration? What would convince you?

5. Where you live, which beings are usually inside the circle, and which are left out?

6. Does the Relationships lens ("Who am I to them?") shrink the circle, or explain it?

The case: Marenhall General

Marenhall General, the main public hospital in the fictional city of Marenhall, has an overcrowded emergency department. Patients wait an average of five hours to be seen, and last winter two patients with sepsis became dangerously ill in the waiting room.

The offer A company, Lumetra Health, offers TriageOne: an AI system that reads symptoms, vital signs and medical records and ranks every waiting patient by urgency. Nurses would see the ranking and could override it.

The evidence In a two-year trial at another hospital, average waits fell by a third and sepsis was spotted earlier than by nurses alone. But patients over 75, and patients who spoke little of the local language, were more often ranked too low. Nurses overrode the ranking in fewer than 2 in 100 cases.

The catch Lumetra will not reveal how the system weighs its inputs. The licence costs as much as employing ten more nurses.

A

Approve TriageOne as offered.

B

Approve it with conditions.

C

Run a further one-year trial at Marenhall first.

D

Reject it and spend the money on nurses.

Ethics committee

Task Groups of 4 or 5. You are Marenhall General's ethics committee. Recommend A, B, C or D (see "The case: Marenhall General"), judging the case through all four lenses. One person is Chair: they keep time and write the recommendation. Each of the others speaks for one lens; in smaller groups, someone takes two.

Consequences

Duty

Character

Relationships

What will happen?

What rule applies?

What would a good person do?

Who am I to them?

AI triage at Marenhall General

Ethics committee

Chair

Consequences

Duty

Character

Relationships

They keep time and write the recommendation.

What will happen?

What rule applies?

What would a good person do?

Who am I to them?

1

Read the case again (2 minutes).

2

Each lens speaker says what their lens recommends, and why (1 minute each).

3

Find where the lenses agree, and where they clash. Which clash matters most here? (3 minutes)

4

Agree on A, B, C or D. Write a two-sentence recommendation; for B, add up to three conditions (3 minutes).

5

Choose one person to report the recommendation in one minute.

The last word

Task Choose one prompt and write privately. Nobody will read what you write. Sharing afterwards is optional.

Prompt 1

Which lens do you trust most now, and has that changed since the start of the module? What changed it, or kept it the same?

Prompt 2 (lighter)

Which thinker or case from the module would you most like to argue with over dinner, and what would you say first?

The end of the module

1

2

3

4

5

6

7

8

9

10

Consequences

Back to the machine: Look at your four votes on "Moral Machine, our class". Would you change any of them now? Which lens would make you change?

What will happen?

Duty

What rule applies?

One question to keep:

Who is outside my circle, and did I put them there, or did I simply never look?

Character

What would a good person do?

Ethics: How Should We Live? ends here. Ten lessons, four lenses, and no final answer: the arguments are yours to keep.

Relationships

Who am I to them?

TEACHER'S REFERENCE · SKIP OR DELETE BEFORE CLASS

Answer key

WHAT THE WORLD CHOSE there are no right answers; this is how each class dilemma maps onto the Moral Machine findings

1 Number of lives (with action against inaction). Sparing more lives was one of the three strongest global preferences, so most respondents would swerve (B), even though swerving means acting. The Western cluster's stronger preference for inaction pulls slightly the other way.

2 Age. Sparing the young was one of the three strongest global preferences, so most would stay in lane (A). The preference was much weaker in the Eastern cluster.

3 Social status. A weaker global preference for sparing higher status, so a majority would stay in lane (A); stronger in more unequal countries and in the Southern cluster. Expect discomfort here: it is the finding critics cite most often against following public preferences.

4 Law-abiding against jaywalking. A moderate global preference for sparing the law-abiding, so a majority would swerve (B); stronger in the Eastern cluster and where the rule of law is stronger. Germany's 2017 commission forbids distinctions by personal features such as age or gender, which rules out the reasoning behind dilemma 2, but allows programming that reduces the number of people harmed, as in dilemma 1.