What are the philanthropic causes that should be prioritized if one adopts the longtermist perspective of giving equal weight to the welfare of current and future generations? Many have argued that, because human extinction would result in a permanent loss of all future lives, extinction risk mitigation should be the top priority. In this paper, we provide a challenge to this argument. We first introduce a theoretical framework for quantifying cost-effectiveness from a longtermist perspective. We then show that standard population models imply the existence of interventions that produce social benefits that are permanent and large enough to render these interventions at least as cost-effective as extinction risk mitigation from a longtermist perspective.
Maya Eden and Gustav Alexandrie, “Is Extinction Risk Prevention the Unique Longtermist Priority? Not in Standard Population Models”, forthcoming in Essays on longtermism (OUP).
1. Recap
This is Part 5 in my series “Mistakes in moral mathematics” based on a paper of the same name. In this series, I discuss three mistakes that are often made in calculating the value of existential risk mitigation. I show how these mistakes have led the value of existential risk mitigation efforts to be overestimated, and how they have mislocated debates about the value of existential risk mitigation.
Part 1 introduced the series and discussed the first mistake: focusing on cumulative rather than per-unit risk. When existential risk mitigation is framed as an intervention on risk in our own century, rather than risk across many distant centuries, the value of existential risk mitigation drops dramatically. We saw how a leading paper by Nick Bostrom makes the first mistake.
Part 2 discussed a second mistake: ignoring background risks which are not affected by our intervention. We saw how a leading estimate of the cost-effectiveness of biosecurity interventions by Piers Millett and Andrew Snyder-Beattie falls once background risk is incorporated. We also saw how the second mistake compounds with the first in Millett and Snyder-Beattie’s discussion, leading to a larger overestimate.
Part 3 introduced a third mistake: neglecting population dynamics. We saw how standard population models threaten to push the expected number of future lives far below longtermist projections. And we saw that standard models are robust against a number of objections that might be raised against them. Part 4 showed how the third mistake continues to be impactful even under highly optimistic assumptions about technological and population growth.
Today’s post concludes by drawing five lessons from this discussion.
2. Beyond existential risk mitigation
There is something which I very much wish I had stressed in my paper “Existential risk pessimism and the time of perils” and which I must not forget to stress in this paper. That is: this isn’t really a paper about the value of existential risk mitigation. The mistakes in this paper will tend to artificially inflate the value of nearly all longtermist interventions.
Most obviously, the third mistake (ignoring population dynamics), makes longtermist interventions look better by inflating the number of people who are likely to be around to enjoy the future we have left for them. Plausible estimates of the future human population reduce the value of longtermist interventions by reducing the number of people who will benefit from those interventions.
Likewise, the second mistake (ignoring background risk) makes longtermist interventions look better by inflating the likelihood that humanity will be around to reap the benefits of longtermist interventions. On many assumptions about background risk, it is no longer so likely that humanity will be around for billions of years to enjoy what we have left for them.
The first mistake (focusing on cumulative risk) may make longtermist interventions look better in the same way as the first, by making it look relatively less difficult for humanity to live a very long time.
If this is right, then the upshot of this paper is not just that the case for existential risk mitigation has been inflated, but rather that the case for longtermism has been inflated across the board.
3. Demographic policy matters
Suppose that standard demographic models are right, and the future human population is on a course to be depressingly small. This means that demographic policy interventions may be even more important than efforts to mitigate existential risk if they can lead to stable increases in the long-term size of the human population.
This is far from a sure thing. Demographers are currently struggling to find any demographic policies that can bring humanity back even to the replacement fertility level required to avoid a collapse in the size of the future human population.
However, if effective altruists are already willing to bet on highly speculative efforts to mitigate existential risk, then perhaps they should also be willing to bet on less speculative, if less glamorous, demographic research aimed at identifying and implementing feasible strategies for stably increasing the size of the future human population.
How large of a demographic shock is needed? The jury is out, but an excellent forthcoming paper by Gustav Alexandrie (of GPI) and Maya Eden (of Brandeis) sheds some light on the matter. The paper, entitled “Is extinction risk prevention the unique longtermist priority? Not in standard population models” is forthcoming in Essays on longtermism, which should be available (open access) through Oxford University Press in a year or so. Stay tuned!
4. The demographics of digital minds
Now for the wacky bit. Suppose you agree that it is unlikely for humans to spend our days expanding throughout the galaxy as fast as our limited resources will allow. And suppose you think that is a bad thing, for it means that the galaxy will support far fewer future lives than it could support.
Here is one solution: create digital minds instead. We might imagine that a population of digital minds would be constrained primarily by the availability of resources, such as energy and raw materials. And we might imagine that digital minds, or the computers that simulated them, could be endowed with an overriding desire to increase their own population as far as resources allow.
One question is whether the population dynamics of digital minds would indeed follow this path. As far as I know, the population dynamics of digital minds has not been well-studied. Perhaps it is time for that to change.
A second question is whether, if digital minds would really be subject to significantly more growth-oriented population dynamics than future human populations are likely to show, that might not generate pressure to create digital minds, or even to replace ourselves with digital minds.
That’s an uncomfortable thought. I’m not exactly sure what to say about it. I rather dislike the direction that this thought leads us, but I suppose that one must follow an argument where it leads. Is that where this paper leads? I hope not!
5. Intergenerational coordination
Our analysis of the first mistake revealed two things. First, realizing astronomical future value requires reducing cumulative existential risk by an appreciable amount. Second, reducing cumulative risk by an appreciable amount requires massive and consistent reductions in per-century risk. If we want, for example, a mere one-millionth of one percent chance of surviving for a billion years, then we need to drive per-century risk down to about one in a million, and we need to do this consistently, nearly without fail, in every single century.
This means that existential risk reduction is best understood as an intergeneration coordination problem in which all generations must come together to coordinate on a policy of keeping risk very low. This is a terribly difficult coordination problem for any number of reasons.
First, the required level of risk is very low. If effective altruists are right that current generations are running a 10-20% chance of existential catastrophe, then the demanded sacrifice is perhaps five orders of magnitude of risk. How much must each generation give up to reach this level of risk? Perhaps quite a lot.
Second, coordination must be nearly exceptionless. Even a few free-riding generations (such as, perhaps, our own) can scuttle the entire enterprise.
Third, enforcement mechanisms are few and far between. How do you punish past generations for the risk that they exposed you to? What about future generations? What is to stop them from taking on a bit more risk, given that fourth, externalities are high? Each generation bears only a tiny fraction of the total cost of human extinction, so an extreme degree of altruism or else highly effective coordination mechanisms are likely to be needed to bring free-riding generations into line. But there may not be effective coordination mechanisms, and people are not that altruistic.
Can the intergenerational coordination problem be solved? Perhaps. Framing the problem this way tends to incline one towards pessimism. But I am not usually so pessimistic about the chance of existential catastrophe. Perhaps there is a natural way of solving this coordination problem. Or perhaps the risks were never so high to begin with.
6. Fool me once, shame on you. Fool me thrice …
My paper “Existential risk pessimism and the time of perils” brought out a mistake that is made in many estimates of the value of existential risk mitigation: ignoring background risk. I showed how incorporating background risk may significantly lower the value of existential risk mitigation.
This paper grouped the second mistake into a larger pattern. Many authors also make the first mistake (focusing on cumulative rather than per-century risk) and the third mistake (ignoring population dynamics). We saw that each mistake substantially lowers the value of existential risk mitigation.
There comes a point where readers lose confidence in a class of projections and in the authors propounding them. Effective altruists repeatedly, and en masse, offer astoundingly high estimates of the value of existential risk mitigation. We have seen that many of these estimates are inflated by many orders of magnitude, and that they became inflated by making some of the same mistakes across many papers.
The right thing to conclude is not: there are three, and only three mistakes, in existing estimates of the value of existential risk mitigation, and the value of existential risk mitigation can be correctly determined by correcting them.
A better inference is that there are very likely further mistakes in these estimates, and that correcting these mistakes may lead to still more dramatic reductions in the value of existential risk reduction.
How sharp might the drop be? I am not sure. Perhaps I will write further papers chasing down other ways in which estimates become inflated. But at a certain point, one simply begins to lose confidence in the reliability of published figures. How many mistakes are leading estimates allowed to make until we stop believing them? Surely not many more than three.
I would encourage readers to go home, look up some leading estimates, and scratch out the reasoning behind them on their own. What exactly does it take to make these estimates tick? Do you believe them? Are there, perhaps, other systematic errors to be found? Let me know what you find. For my own part, my trust is waning.
Leave a Reply