Long-term consequences swamp short-term ones in total value. And because we generally can’t predict the long-term consequences of our actions, it follows that we generally can’t predict the overall consequences of our actions. But there may be some exceptions. Proponents of longtermism believe that some actions—such as reducing existential risk—have robustly positive expected value in the long term. So, at a minimum, the [claim] needs to be weakened to the claim that we’ve no idea what to do other than work on reducing existential risks.
Richard Chappell, “The cluelessness objection“
1. Introduction
This is Part 5 of my series “The scope of longtermism,” discussing a paper of the same name.
Part 1 introduced the scope question for longtermism: how wide is the class of decision situations for which longtermism holds? I clarified my target, swamping axiological strong longtermism (swamping ASL) and my view: the scope of swamping ASL, while probably nonempty, is smaller than many longtermists suppose.
Specifically, I introduce three scope-limiting factors: probabilistic and decision-theoretic factors that are present in many contemporary decision problems and which, when present, substantially reduce the prospects for swamping ASL to hold of that problem.
Part 2 introduced the first scope-limiting factor: rapid diminution. Part 3 introduced the second scope-limiting factor: washing out. Part 4 discussed the final scope-limiting factor: unawareness.
Longtermists often argue that these and similar challenges can be avoided by turning to existential risk mitigation. If risks are high now, then they may not be so difficult to foresee, change or become aware of. Today’s post argues that this line of response is sometimes correct, but that it is less often correct than it may appear.
2. Cluelessness and existential risk mitigation
Longtermists often concede that we are clueless about the impact that many of our actions will have on the value of distant centuries. However, longtermists often respond that there is at least one class of intervention about which we are not clueless: existential risk mitigation. Even if we are uncertain about the shape that the future will take, so long as we think that the future will probably be worth preserving and that we can have a meaningful impact on the likelihood that it is preserved, then longtermists argue it will be well worth our while to mitigate existential risks.
This argument does gain some traction against each of the three scope-limiting factors mentioned in Parts 2-4.
Rapid diminution occurs when the probabilities of large far-future impacts decrease faster than their values increase. The argument for rapid diminution was that it is hard to make a persistent impact on the long-term future. But mitigating existential risk seems to be an exception to this rule. Not being extinct is a status that can persist for a very long time. Therefore, if we can meaningfully affect the chance of near-future human extinction, it seems that we can impact the long-term future in a persistent way.
Washing out occurs when the probabilities of far-future impacts exhibit significant symmetry about the origin, leading to substantial cancellation between possible positive and negative impacts in taking expected values. The argument for washing out was that it is difficult to forecast the future, and we have very little evidence to go on. However, many longtermists think that levels of existential risk are high in our current century. Even if it is difficult to predict events in future centuries, it may not be so difficult to predict levels of existential risk in our current century or to identify actions which reliably reduce them.
Unawareness occurs when we are unaware of relevant states, acts and outcomes. However, we are not unaware of human extinction as an outcome. Further, longtermists think, we are currently aware of many actions that could be taken to reduce existential risk. For example, we could lobby for AI regulation or a pause on AI development; fund AI safety and biosecurity research; or build a pipeline of young researchers interested in tackling these issues.
If that is right, then it seems that the arguments made in this paper and blog series do not go through against existential risk mitigation. Since existential risk mitigation is one of the very most prominent longtermist interventions, this would mean that my conclusions are consistent with a large portion of longtermist doctrine and practice.
3. Longtermists are not always wrong
The first thing to say is that longtermists are not always wrong. We saw in Part 1 of this series that some longtermists have claimed that the scope of longtermism expands to nearly all decision situations, since all decision situations have some effect on existential risk. We also saw that many opponents of longtermism deny that the scope of longtermism extends to any present, past or future decision situations. Both claims are implausible.
We saw in Part 1 that there is at least one recent decision situation in which longtermism probably held: the United States Congress’ decision to fund the Space Guard Survey. At a cost of less than a hundred million dollars, this survey mapped the vast majority of potentially catastrophic near-Earth objects, leading to very high confidence that no such object will collide with Earth in this century. Coming immediately after the scientific confirmation that just such an asteroid impact did in the dinosaurs, this study allayed a reasonable and potentially actionable fear at a very modest cost.
And indeed, none of the scope-limiting factors seem to get a very good take on the Space Guard Survey. It is, to be fair, quite difficult to deflect an asteroid, but perhaps not so difficult that the value of doing so is outweighed by our low chance of success. While it is often hard to predict the future, astronomy and cosmology are among the few domains in which we can reliably predict the future over a period of centuries. And while many decisionmakers may previously have been unaware of the potential for asteroid impacts to cause human extinction, they were hardly unaware of this fact at the time when the Space Guard Survey was being proposed.
The claim, then, cannot be that the scope of longtermism is empty. The claim is rather that cases such as the Space Guard Survey are more special than they may appear. When we turn our eyes to many other proposed existential risk mitigation efforts, the scope-limiting factors begin to get a more solid take. Let’s focus on risks posed by artificial intelligence.
4. Rapid diminution
The argument for rapid diminution was that because the future is far away and shaped by powerful forces, affecting the future is hard. The present is not far away, but it is affected by powerful forces. As such, even affecting present existential risks may be more difficult than it appears.
Consider effective altruists’ political ambitions regarding artificial intelligence. These political ambitions are ever diminishing, as the difficulty of accomplishing them becomes clear. Many effective altruists would like to halt frontier AI development, at least for a considerable time. That is, it is widely admitted, politically infeasible. Effective altruists then shifted to calling for a pause of perhaps six to twelve months in AI development. Again, a pause is increasingly thought to be politically infeasible. There is a large commercial interest in AI development, an increasing global competition for control over artificial intelligence, and the potential for enormous consumer benefit. These are strong interests that dwarf the power of even a well-targeted billion-dollar opposition.
Now, effective altruists increasingly call to regulate artificial intelligence. But the first wave of regulatory proposals seems to have largely failed. California’s SB-1047 was vetoed. Biden’s AI executive order is under threat. And in many corners, there has been a backlash against any kind of AI regulation. (I wrote these words shortly before the Trump administration began. I left them here as a stark reminder of how much worse the political situation has become. However, I hope that the present situation is politically exceptional in this and other regards).
Further, effective altruists have so far been unable to hold some of their most cherished red lines. One area of concern is that AI agents may act independently and thereby disempower humanity. Effective altruists would like to limit development of AI agents, but AI agents are currently all the rage and widely expected to grow significantly more popular in coming years. Another concern is militarization of AI, with weapons development seen as a risky way of empowering AI agents to act maliciously. But leading AI companies have signed sizable agreements with the United States Defense Department and defense contractors such as Palantir and Carahsoft. Again, these developments are to some degree unsurprising. There is a great deal of interest in agential and military applications of AI.
Effective altruists are doubtless right that mitigating near-future risks may be more tractable than many other ways of improving the far-future. But it should not be assumed that risk mitigation is easy. Developments powerful enough to threaten humanity are backed by powerful forces, and they are not likely to be the kinds of diamonds-in-the-rough that a few well-placed donations can unearth.
5. Washing out
The argument for washing out was that predicting the future effects of our actions is hard. The bulk of my skepticism on this front comes from skepticism about the magnitude of risks posed by artificial intelligence. I don’t think that levels of near-term existential risk from artificial intelligence are anywhere near as high as many effective altruists think they are. Therefore, I think that most of the forecasted benefits of risk-mitigation efforts will not come to pass, because I think the risks they aim to mitigate do not exist.
But I don’t want to push this line of objection, since many readers may disagree. Instead, I want to make a different point. Even granting the existence of severe near-term existential risks posed by artificial intelligence, it is very difficult to foresee how our actions will affect them. We have seen this even on a time-scale of years or decades in the recent history of artificial intelligence.
In 2017, Open Philanthropy committed $30 million to OpenAI in exchange for a board seat, in the belief that “OpenAI’s leadership appears to be highly value-aligned with us.” OpenAI was, at the time, seen as an organization committed to its non-profit mission of producing open-sourced AI in a safety-conscious way.
That wasn’t the way things turned out. OpenAI emerged as one of the very most accelerationist companies, pushing model capacities relentlessly forward beginning with its world-shaking release of Chat-GPT.
Effective altruists tried to use their influence to correct course, at one point playing a role in inducing the board to fire Sam Altman. That didn’t go well. Within days, Altman was back, effective altruists were gone from the board, and the power of a rather less-than-fully-safety-conscious Altman was greater than ever. There was strong backlash against effective altruism in Silicon Valley, as it became clear that effective altruists posed a threat to the value of employee equity, often valued in the millions of dollars per employee.
Since then, much of the safety-focused work that effective altruists hoped to implement within OpenAI has collapsed. Much of OpenAI’s superalignment team resigned in March 2024, with one member using the opportunity to launch a hedge fund which many effective altruists view as dangerously accelerationist. Just months after the superalignment team was fully disbanded in May 2024, the company disbanded its AGI readiness team headed by Miles Brundage in October 2024. Last month saw another high-profile resignation.
It may have seemed, in 2017, that investing in OpenAI was a good way for effective altruists to sponsor a safety-focused AI company and promote AI safety work alongside safety-conscious model development. That does not appear to be what happened.
Another time, perhaps, we might discuss effective altruists’ involvement with Anthropic. I think it is safe to say that at this point, many effective altruists have substantial doubts about Anthropic’s ability to positively contribute to the development of safe artificial intelligence.
What does this show? Not, I think, that effective altruists were unlucky or have poor judgment. It rather shows that predicting the future is hard. Even on a timescale of years and decades, let alone centuries or millennia, it is very hard to know how our actions will affect the future. Acts which may seem to have robustly positive value can cause harm in myriad unforeseen ways. Acts meant to mitigate existential risk are no exception.
6. Unawareness
The problem of unawareness is that we are unaware of relevant actions we could take, states that the world could be in, and outcomes that our actions could have. Begin with act unawareness.
I think that effective altruists would be the first to acknowledge that existential risk mitigation is in its infancy. We have a very poor grasp of the space of conceptually possible mitigation strategies, and it is entirely possible that most of the strategies to have yet been considered are ineffective or even have negative consequences. It could well be that, in hindsight, it will be clear that there are some actions that could have been taken to mitigate risks. But, as with regulating wet markets to prevent the outbreak of pandemics, those acts may not have been on many decisionmakers’ radars.
Likewise, we often do not even consider many of the outcomes that our actions could have. Did those funding OpenAI consider whether members of the safety team at OpenAI would ultimately go on to fund large investments into frontier AI? What about any number of yet-to-be-considered consequences that development of artificial intelligence today will have in the future? Those consequences could prove quite significant, and to that extent our unawareness of them is a substantial limiting factor.
7. Conclusion
This post addressed what is perhaps the most common objection to my paper, “The scope of longtermism,” namely that the scope-limiting factors do not apply to existential risk mitigation efforts. While we saw in Section 3 that this objection is sometimes correct, we saw in Sections 4-6 that the scope-limiting factors appear to get a good take in many other existential risk mitigation contexts. In particular, they appear to get a take on current efforts to mitigate existential risk from artificial intelligence, which are among effective altruists’ key concerns.
If that is right, then I do not think the turn to existential risk mitigation can be regarded as a one-shot solution to the scope-limiting factors. Affecting the future is hard. Forecasting the future is hard. Becoming aware of what the future might be like and what we can do about it is hard. These are enduring problems and they are here to stay. We can, and should, do our best to mitigate these problems and to live within the bounds they set for us. But they are problems that are likely to remain with the human species for as long as we live.
I think that this post will be the conclusion of my series, The scope of longtermism. I may address further objections or applications, but I think it might be more productive to begin a series discussing my paper on instrumental convergence and power-seeking theorems instead.
Leave a Reply