Papers I learned from (Part 2: Effective altruism, risk, and human extinction)

Any effort to ensure a long happy future will have to strive both to ensure a long future, by reducing extinction risks, and to ensure it is a happy one, by improving the quality of life. Such an effort might succeed at the former goal without succeeding at the latter. So any effort to ensure a long happy future will increase the chance of such a future, but it will also increase the chance of a long miserable future, even if it increases the latter by a smaller amount. Granted this, if you are risk-averse, or if morality requires you to choose using risk-averse preferences, then you do better to work to bring about humanity’s extinction than to secure its long-term survival. This conclusion will seem troubling to many, though perhaps welcome to some. In this paper, I try to formulate the argument as plausibly and as robustly as I can. I then investigate how those who wish to resist it might do so.

Richard Pettigrew, “Effective altruism, risk, and human extinction“

1. Introduction

This is Part 2 of my series Papers I learned from. The series highlights papers that have informed my own thinking and draws attention to what might follow from them. 

Part 1 looked at Harry Lloyd’s defense of robust temporalism, a form of pure temporal discounting.

Today’s post looks at the relationship between risk aversion and human extinction. It is increasingly common for decision theorists to hold that rational agents may be risk averse, preferring, other things equal, to avoid taking risks. For example, I might avoid placing large bets even at slightly favorable odds.

Does risk aversion advise us to mitigate existential risks? Surprisingly, risk aversion may advise us to do exactly the opposite, acting to hasten rather than prevent human extinction. That’s quite a claim, and the business of today’s post will be untangling why someone might make this claim.

2. Effective altruism, risk, and human extinction

Richard Pettigrew is Professor of Philosophy at the University of Bristol. Pettigrew’s paper “Effective altruism, risk, and human extinction” draws on an approach to risk aversion due to Lara Buchak in order to show how risk aversion may tell in favor of hastening human extinction. This is, to be clear, not meant as an argument for hastening human extinction. It is rather a call to think carefully about how risk aversion bears on the normative case for existential risk mitigation.

In Pettigrew’s first model, a wealthy risk-averse agent chooses between three options. Her first option (HFF) is to donate to the Happy Future Fund, which works to reduce existential risk and improve the value of the future conditional on survival. Her second option (QEF) is to donate to the Quiet End Foundation, which works to hasten human extinction. Her third option (SQ) is to do nothing and retain the status quo.

Consider four ways that the future might go: it might be long and happy (lh); medium and happy (mh); short and mediocre (ext) or long and miserable (lm). Let’s give utilities to these states. Pettigrew suggests an illustrative assignment on which u(lh) = -u(lm) and u(lh) >> u(mh) >> u(ext), arriving at an assignment like this:

	lh	mh	ext	lm
U	1019	1011	104	-1019

What about probabilities? Pettigrew suggests that under the status quo, humanity has a 1/100 probability of going extinct this century (ext), a 10^-5 chance of a long happy future (lh), and 10^-7 chance of a long miserable future (lm), with the remaining probability mopped up by a mediocre future (mh).

	ext	lh	lm	mh
P	10-2	10-5	10-7	1-(10-2+10-5+10-7)

Finally, what effect does donation have on these probabilities? Pettigrew assumes that donating to the Quiet End Foundation will increase the probability of extinction (ext) by the same amount that donation to the Happy Future Fund decreases it, with other probabilities rescaling proportionally. Assuming that the donor is very rich, Pettigrew takes the impact on the probability of extinction to be 10^-5. The resulting conditional probabilities can be found using Bayesian conditionalization: they’re given in Pettigrew’s paper (Figure 1), but I’ll omit them for simplicity.

That’s the setup. But what are you to do?

3. Risk aversion

Suppose you’re evaluating some gamble G. G has some possible outcomes, with utilities u₁, … , u_n, and we’ll order these utilities so that they are increasing. G realizes each outcome with probability p_i, so that G has expected utility EU[G] = p₁u₁ + … + p_nu_n. So far so good.

Each p_i is the probability of getting exactly utility u_i, that is the probability of getting at least utility u_i minus the probability of getting strictly more utility than u_i. Writing the p_i‘s in this way gives an alternative expression for the expected utility of gamble G:

A popular version of risk aversion, Lara Buchak’s risk-weighted expected utility theory (REU), takes this expression as its starting point. Buchak’s view is that risk aversion transforms how probabilities enter into expected utility calculations through a ranking function R. Crucially, R is applied to probabilities in the probability-difference expression above, so that the risk-weighted expected utility of G is:

Let’s fix an exponential risk function R = x^k. Agents are then risk-averse in the case that R is convex, i.e. k > 1, risk-neutral for k = 1, and risk-seeking for k < 1. Pettigrew fixes a moderately risk-averse risk function with k = 1.5.

4. REU and human extinction

What does REU say about the choice between the Happy Future Fund, Quiet End Foundation, and the Status Quo? The thing to note about REU is that it isn’t so concerned about seeking the best possible outcomes. On expected utility theory, the best outcome lh contributes utility to the value of the status quo. But on REU, the best outcome contributes to the value of the status quo. This makes the best outcome 2.5 orders of magnitude less important. By contrast, REU still cares a great deal about avoiding bad outcomes. On EU theory, the worst outcome lm contributes utility to the value of the status quo. And on REU theory, the worst outcome lm contributes utility . This makes the worst outcome about 1.5 times more important.

What about intermediate outcomes, like human extinction? REU isn’t so concerned about intermediate outcomes. On expected utility theory, human extinction contributes utility to the value of the status quo. And on REU, human extinction contributes about 1.5 times this value. But 1.5 times nothing much is still nothing much.

What this means is that on REU, agents should be much more concerned about avoiding a long and miserable future (lm) than they otherwise would be. This doesn’t mean that they are particularly enthusiastic about human extinction, but they should nonetheless look much more fondly on risks of extinction than on risks of long and miserable futures. And that makes all the difference.

In the present example, we have REU(HFF) < REU(SQ) < REU(QEF), suggesting that the agent should be most enthusiastic about hastening human extinction, next-most enthusiastic about preserving the status quo, and least enthusiastic about trying to ensure a long and happy future. Yikes!

5. What we together risk

This discussion took the perspective of an individual risk-averse agent. Do we do better to see that agent as deciding, not based on their own utilities, but instead on behalf of other agents, taking those other agents’ risk attitudes into account?

Pettigrew argues that we don’t. I’ll omit the details of this argument, which might be of interest primarily to philosophers, but roughly the idea is that many present and future agents will be moderately risk-averse. If many of the agents affected by our choices will be moderately risk-averse, then it is plausible that we should ourselves adopt a moderately risk-averse perspective in choosing on their behalf. And that brings us right back to where we started.

6. Has something gone wrong here?

I have to admit, I have a lingering sense that something has gone wrong here. It might be true that risk aversion causes trouble for the value of existential risk mitigation. But it is deeply surprising if moderate levels of risk aversion not only scuttle the case for existential risk mitigation, but also ground a robust case for hastening human extinction.

Has something gone wrong here? I’m not sure. I’d be interested to hear readers’ thoughts.

I am strongly considering following up on this post with a reply to Pettigrew’s paper written by the philosophers Nikhil Venkatesh and Kacper Kowalczyk. I would also be interested in hearing what readers think of that possibility.