Papers I learned from (Part 7: Essays on longtermism)

Recent years have seen a flurry of interest in longtermism … This volume brings together leading scholars to discuss four sets of overlapping questions raised by longtermism. First, should we accept some version of longtermism? Second, to what extent can we predict and control the far future? Third, which ethical priorities are recommended by longtermism, and how revisionary are they? Finally, what implications would longtermism have for the design or reform of social, political, and legal institutions?

Greaves, Barrett and Thorstad, Essays on longtermism

00:00
1X

1. Introduction

This is Part 7 of my series Papers I learned from. The series highlights papers that have informed my own thinking and draws attention to what might follow from them. 

Part 1 looked at Harry Lloyd’s defense of robust temporalism, a form of pure temporal discounting.

Part 2 looked at an argument by Richard Pettigrew that risk-averse versions of longtermism may recommend hastening human extinction. This was meant not as a recommendation, but rather as a way of putting pressure on standard arguments for longtermism. Part 3 looked at a reply to Pettigrew by Nikhil Venkatesh and Kacper Kowalczyk.

Part 4 looked at a paper by Maarten Boudry and Simon Friederich examining evolutionary arguments for AI risk.

Part 5 looked at a paper by Simon Goldstein and Cameron Domenico Kirk-Giannini on existential risk from language models.

Part 6 looked at a paper by Rhys Southan, Helena Ward and Jen Semler on a timing problem for instrumental convergence.

Today’s post looks at a new volume of essays about longtermism.

2. Introducing the book

Researchers at the Global Priorities Institute thought it was important to put out an academic volume collecting leading scholarly perspectives on longtermism. The result is a scholarly volume, Essays on longtermism, edited by three of us: Hilary Greaves, Jacob Barrett, and yours truly.

The volume has been in the works for some time (since 2021) and has grown to an enormous 31 chapters. It is so long that we had to use a special kind of binding to keep the book together.

This week, Essays on longtermism was published open access with Oxford University Press. You can read the full book here.

The volume has four parts.

Part 1, Evaluating the case for longtermism, considers arguments broadly relevant to the case for and against longtermism.

Part 2, Predicting and evaluating the future looks at questions that arise in predicting and evaluating the future.

Part 3, Ethical priorities, looks at specific ethical issues within longtermism.

Part 4, Institutions and society, takes a group-level institutional and societal lens towards longtermism. (This part of the ebook is currently not displaying on the publisher’s website. I hope that this issue will be fixed very soon.)

I’d like to use the rest of this post to highlight some papers that readers may find especially interesting and some lessons that we may learn from them. Apologies to the authors of the many excellent papers that I have not included in this post.

3. The case for strong longtermism

Hilary Greaves and Will MacAskill‘s paper “The case for strong longtermism” was envisioned as the flagship exposition and defense of the longtermist position. I think it fulfills this charge, and that it is now the best published scholarly defense of longtermism.

Drafts of this paper have been circulating for many years, so some of you may have read this paper already. I would encourage you to take a look at the published version, as it differs in important respects from some earlier drafts.

The paper begins by arguing for:

Axiological strong longtermism (ASL): In the most important decision situations facing agents today,

(i) Every option that is near-best overall is near-best for the far future.

(ii) Every option that is near-best overall delivers much larger benefits in the far future than in the near future.

The argument for axiological strong longtermism relies on:

Benefit ratio (BR): The highest far-future ex ante benefits that are attainable without net near-future harm are many times greater than the highest attainable near-future ex ante benefits.

Benefit ratio is motivated by appeal to the size of the long-term future and the tractability of affecting it.

The paper then addresses a wide range of concerns that have been raised for axiological strong longtermism, which draw on risk aversion, anti-fanaticism, alternative population axiologies, cluelessness, and other familiar concerns. This version of the paper differs from earlier drafts in offering more comprehensive responses to many of those concerns.

The paper concludes by arguing for:

Deontic strong longtermism (DSL): In the most important decision situations facing agents today,

(i) One ought to choose an option that is near-best for the far future.

(ii) One ought to choose an option that delivers much larger benefits in the far future than in the near future.

on the basis of the stakes-sensitivity argument:

(P1) When the axiological stakes are very high, there are no serious side-constraints, and the personal prerogatives are comparatively minor, one ought to choose a near-best option.

(P2) In the most important decision situations facing agents today, the axiological stakes are very high, there are no serious side-constraints, and the personal prerogatives are comparatively minor.

(C) So, in the most important decision situations facing agents today, one ought to choose a near-best option.

I’ve highlighted this paper because it represents a deeply considered attempt by two leading philosophical longtermists to set out the standard arguments for longtermism and to set the groundwork for discussion on a range of common objections to longtermism. I think that this paper succeeds very well in setting out target views (axiological strong longtermism, deontic strong longtermism) and classic arguments (especially the stakes-sensitivity argument) and should be a launching-off point for many future discussions.

4. Depopulation and longtermism

Dean Spears is an economic demographer and the founding director of the Population Wellbeing Initiative at UT-Austin. Mike Geruso is associate professor of economics at UT-Austin and a core faculty member at the Population Wellbeing Initiative.

Much of their research deals with the long-term consequences of declining fertility rates. They recently published an excellent book, After the spike, on these issues. Their paper, “Depopulation and longtermism,” covers a good deal of the same material.

I’ve made no secret of my love for this paper: it forms the core of the third mistake in my paper and blog series “Mistakes in the moral mathematics of existential risk.” I cover Spears and Geruso’s paper more fully in Part 3 of that series. But let me briefly remind readers what the paper says.

Fertility rates are falling across the globe, and in many countries they lie well below replacement. If this trend continues, the future of humanity may not be anywhere near as large as longtermists hope. In fact, if fertility rates stay at almost any reasonable level below replacement, the total human population will look like this:

“The spike” from Spears and Geruso (2025)

That isn’t good. And the bad news, Spears and Geruso stress, is that this is the trajectory that the world appears to be on, and no researchers or governments have found an effective way to reverse it. Spears and Geruso make a detailed case for these demographic trends and address common objections.

I don’t think that readers should take this graph as a literal projection of humanity’s likely future. I am not sure that even Spears and Geruso would ask for the graph to be taken too literally in this sense. However, I do hope that this graph serves as a wake-up call to longtermists relying on the idea that the future may be very large. In Spears and Geruso’s models, the entire human future contains about 20-30 billion humans. That may be an underestimate, but reckoning with it is likely to do a good deal to talk us down from higher longtermist estimates of the future human population.

A large future population is not necessary for longtermism to be true, but it certainly helps. By the same token, a smaller future population is not a death knell for longtermism, but it certainly hurts.

Some of these implications for longtermism are explored in more detail in another of my favorite papers from the volume, “Is extinction risk mitigation uniquely cost-effective? Not in standard population models,” by Zurich economist Maya Eden and economist and philosopher Gustav Alexandrie. I highly recommend this paper for much the same reasons I recommend Geruso and Spears’ paper.

5. Existential risk from power-seeking AI

Joe Carlsmith is a philosopher and an advisor at Open Philanthropy. Many of you may be familiar with his paper “Is power-seeking AI an existential risk?,” perhaps familiar enough to simply call it, as I do, the Carlsmith report. This report has been enormously influential in longtermist thinking about existential risk from artificial intelligence.

The Carlsmith report has both strengths and weaknesses. In Parts 6, 7 and 8 of my series Exaggerating the risks, I talk about some of them. One drawback I don’t mention there is that the report is very long. That is a very good thing where rigor is concerned, but it has posed an obstacle for many readers who would like a more succinct presentation of some of the main ideas of the report.

“Existential risk from power-seeking AI” fills this need by presenting, at article-length, a case for concern about existential risk from power-seeking AI that is substantially inspired by the original report.

Carlsmith argues that by 2070:

1. It will become possible and financially feasible to build relevantly powerful and agentic AI systems.

2. There will be strong incentives to do so, conditional on (1).

3. It will be much harder to build aligned (and relevantly powerful/agentic) AI systems than to build misaligned (and relevantly powerful/agentic) AI systems that are still superficially attractive to deploy, conditional on (1) and (2).

4. Some such misaligned systems will seek power over humans in high-impact ways, conditional on (1)–(3).

5. This problem will scale to the full disempowerment of humanity, conditional on (1)–(4).

6. Such disempowerment will constitute an existential catastrophe, conditional on (1)–(5).

The paper concludes with Carlsmith’s updated probability assignment to the conjunction of (1)-(6), which he doubles to 10% from his earlier 5% estimate.

I hope that this paper will provide a tractable jumping-off point for a larger number of readers to engage with Carlsmith’s work, which continues to be influential in ongoing debates.

6. Defining longtermism

Some of us from the Global Priorities Institute have become famous (or perhaps infamous) for our tendency to nag people to say exactly what they mean by longtermism. We certainly nagged our authors about this, and we do so again in the introduction with a brief reminder of the many ways in which the term is used:

Contributors to this volume (where they use the terms at all) define the terms ‘longtermism’ and ‘longtermist’ in different ways. All are at least loosely related to the general idea that ‘positively influencing the long-term future is a key moral priority of our time’ (MacAskill 2022: 4). But under and around this umbrella, potentially important distinctions abound. Weak longtermist theses (like the one just quoted) assert only that the far future is important; strong longtermist theses make comparative claims, to the effect that considerations of the far future are in some specified sense the most important. Axiological longtermist theses concern the connections between the far future on the one hand, and on the other hand, which actions lead to better vs. worse outcomes than which others. Deontic longtermist theses concern not (or not directly) the question of what is better or worse, but rather the question of what a given agent, in a given decision situation, ought or is morally obligated to do. (While these two issues are of course connected, non-consequentialist accounts of moral obligation hold that they are also importantly distinct.) Some authors use ‘longtermism’ to denote a thesis, while others use the same term to denote an evaluative perspective (for example, a perspective that embodies a zero rate of pure time preference, or one according to which considerations of the far future are very important). The adjective ‘longtermist’, similarly, denotes sometimes a property of persons (that of believing a given thesis, adopting a particular perspective or engaging in a particular project), and sometimes a property of perspectives or projects. In the present state of the discussion, this flexibility is helpful, so we have made no attempt to regiment the use of the terms ‘longtermism’ and ‘longtermist’ throughout the present volume. Nor will we attempt to state any canonical definitions in this introduction. On this terminological point, each essay must be taken on its own terms.

We hope that this brief and highly non-exhaustive survey will help to illuminate some of the relevant distinctions and encourage folks to be as clear as possible about them. Consider yourselves nagged! I’ll do it again in my book Beyond longtermism. Apologies to those who are tired of our nagging.

7. Conclusion

I hope that this brief survey will encourage you to take a look at some of the many excellent essays included in the volume.

Stay tuned for an essay competition themed around the volume in collaboration with the EA Forum team.

Posted

in

by

David Thorstad

Tags:

, , , , , , , , , , , , ,

Comments

3 responses to “Papers I learned from (Part 7: Essays on longtermism)”

  1. yarrow Avatar
    yarrow

    It’s interesting you highlighted the two papers on depopulation from Essays on Longtermism, because those are also the two papers that stood out to me! Those two papers meaningfully changed the way I think about the long-term future, which none of the other papers in the anthology did. (Rather, my reaction to most of the anthology was deepening exasperation that, despite the amount of hype, attention, and research funding around longtermism over the last eight years, I strained to find a single novel insight I could agree with anywhere in the anthology. Having been exposed to scholarship on existential risk and global catastrophic risks prior to the coining of “longtermism”, the longtermist ideas I agree with — guarding against even low-probability catastrophes, such as asteroids, due to high expected value — are not novel to me and the novel longtermist ideas, like patient philanthropy, I don’t agree with.)

    I have two good things to say about the two depopulation papers. First, they hammered home how worrying it is that population growth is running out of steam around the globe. I care about there being a large number of future lives, but even people who don’t should still worry. The Geruso and Spears paper (“Depopulation and Longtermism”) raises the spectre that depopulation could spell the end of technological progress and economic growth. In South Korea and Japan, there are concerning early signs of how depopulation might lead to a decline in quality of life, not just quantity of lives.

    Also, many people want to have kids but feel they can’t, which is a tragedy in its own right. If governments support people in becoming parents, that increases their positive liberty, rather than diminishing their autonomy. (However, if people decided, en masse, they didn’t want to have kids due to widespread depression, nihilism, or pessimism, I would also think that is a problem worth trying to solve, even though the autonomy argument isn’t as clean there.) Similarly, to the extent gender inequality is a factor in declining fertility, the pursuit of greater gender equality is independently a worthy endeavour.

    The second good thing: these two papers, especially the Alexandrie and Eden paper on extinction risk and population models, made me question a foundational implicit assumption that seems to be lurking around a lot of catastrophic risks discourse. The assumption is that if 10% or 1% or 0.1% of the human population survives a global catastrophe, then there is some significant probability that humanity will recover to its pre-catastrophe state, both in terms of population and technological development. In fact, people often seem to talk as if this outcome is assured, although I may be misunderstanding them. This framing of non-extinction catastrophic risk vs. extinction risk goes all the way back to Derek Parfit’s discussion of the topic in Reasons and Persons in 1984. Yet it seems to be a mistake. (Or, at least, partially a mistake.)

    Thinking about population growth also brought to mind another crucial consideration: humanity’s one-time allotment of cheaply and easily accessible fossil fuels. We’ve used it up, and that could very plausibly prevent humanity from ever recovering from a global catastrophe. Somehow, despite being aware of this problem previously, this slipped my mind until I started thinking about population growth.

    I don’t know enough about the catastrophic risks literature to know what sort of probabilities different people assign to humanity eventually recovering from a catastrophe, and whether that makes any difference, in the end, to what sort of risks they want to prioritize. To me, intuitively, it reshuffles the way I think about catastrophic risks, such that extinction risks no longer stand out as astronomically worse than non-extinction catastrophic risks. An asteroid that would kill 90% of the human population might have nearly as much expected disvalue as an asteroid that would kill 100%. However, I haven’t done any math on this. (I wrote a post on the Effective Altruism Forum about this topic, titled “Irrecoverable collapse”.)

    However, there’s a big difference between applying standard population models to a post-catastrophe scenario and applying them to an optimistic scenario. In an optimistic scenario, we might imagine that the sort of economic growth and progress in science and technology we’ve seen over the last 200 years continues for another 1,000+ years, with no major, long-term disruption from a global catastrophe. I don’t know that you should assume the future population will expand in a Malthusian way and exhaust all available resources, but I also don’t know that you should assume standard population models will apply, either.

    I think the third argument in “Three mistakes in the moral mathematics of existential risk”, the population argument, is an important challenge to an unsupported assumption in imagined far future scenarios, but I’m skeptical that we can replace that Malthusian assumption with the assumption that standard population models will apply millennia in the future. In “Three mistakes in the moral mathematics of existential risk”, you imagine that, even in a distant future where interstellar space travel is commonplace, historical patterns of migration and population growth will still apply. Why would they?

    I don’t claim to know what the patterns would look like in such a scenario. I suspect, though, they would look like what we want them to look like. A scenario in which humanity expands throughout the galaxy on interstellar spaceships many millennia into the future implies a level of scientific and technological advancement in which none of the economic, practical, or biological limitations to migration and population growth that have applied historically will still hold. I’m not sure that it makes sense to forecast the far future population in such a scenario based on current models of migration or population anymore than it would to forecast it based on the amount of arable land in the Milky Way Galaxy.

    Star Trek: The Next Generation had many scenes of hardscrabble interplanetary colonists cracking rocks with pickaxes and growing wheat by the sweat of their brow. But this was always an amusing contrivance in a setting with unlimited energy from matter-antimatter reactors and replicators that could instantly and cheaply turn energy into any pre-programmed configuration of matter. Not to mention artificial general intelligence (Data!) and other wonders.

    I don’t feel comfortable with proponents of longtermist interventions simply imagining whatever future they like and working backwards from there to consequential recommendations about what we should do today. The space of possible futures is essentially unlimited, and using the highly uncertain particulars of what one imagines about the distant future to motivate present action seems perilously close to treating science fiction as non-fiction. However, I don’t think current population models have much predictive value when extrapolated many millennia forward into an optimistic, interstellar future.

    For instance, if people no longer age, and accidental deaths are much more rare, population models would have to be adjusted accordingly. If all medical problems related to fertility are solved, and if per capita GWP is in the millions or billions, that would call for adjustments as well. The evolution of human culture over a timespan much longer than writing has existed seems impossible to predict. To the extent that cultural factors influence fertility, that matters too. How many more adjustments to current models would we have to make, or at least consider?

    One of the people who has influenced my thinking about the far future the most is the physicist David Deutsch. In his book The Beginning of Infinity, he makes forays into epistemology, philosophy of science, and other philosophical topics. I find many of his ideas disarmingly creative and beautiful, but I don’t know if they stand up under scrutiny. Unfortunately, I can find nearly no discussion of the book by academic philosophers (just a review in the New York Times and a short review in Philosophy Now). If I had the sort of money that the funders of longtermist research have, I would be tempted to pay philosophers to write an Essays on Longtermism-esque anthology about the book. Two relevant parts of the book here are Deutsch’s ideas about the fundamental unpredictability of the content of future scientific discoveries and about the pitfalls of inductivism and “explanationless theories” that simply extrapolate statistical trends forward. Those ideas have wide-reaching implications for forecasting and futurism.

    Reply
    1. David Thorstad Avatar
      David Thorstad

      Thanks Yarrow!

      Yes, I’m a big fan of those papers. I’ve also collaborated regularly with their authors and I’m currently preparing a grant proposal with Alexandrie/Eden. These are serious economists and demographers who already have plenty of prospects for more conventional grant funding sources, so I’m glad that work has been done to keep them in the fold and I hope that the community will continue to reach out. (For example, the Population Wellbeing Initiative where Spears and Geruso work is running towards the end of its initial funding period and does not currently have EA-related funding lined up, which could cause staff shortages or even an institute closure without more support.)

      You’re certainly right that given uncertainty, we should not only care about the conclusions of standard population models. Those models lie almost at the opposite side of the optimism/pessimism spectrum from standard longtermist models, and the truth is often in the middle. I think it’s important to note that even many intermediate models are much worse for the longtermist than their favored models are, and that uncertainty over a wide range of models is likewise not particularly good for the longtermist if substantial credence pools away from the very most favorable models.

      Demographers are quite sympathetic (as am I) to your skepticism about projecting present population models forward for many centuries or millennia. The last person who did that impactfully was Malthus, and he didn’t exactly get things right. That’s why the longest projections that I cite here go out to the year 2300, and truthfully I think readers should be a lot more skeptical of those projections than they are of projections out to 2100. I think that the authors of the projections would mostly agree with this claim.

      I’ll take a look at the Deutsch book. The NYT review that you mentioned is by a very eminent philosopher of physics, so I am inclined to give it a lot of weight.

      Reply
      1. Yarrow Avatar
        Yarrow

        Thanks for the reply. 🙂

        I hope Spears’ and Geruso’s organization will be able to get their funding!

        I wasn’t sure to what extent in the “Three mistakes” paper you were using standard population models and models of human migration to make an epistemological or methodological challenge to the longtermist argument, i.e. that it’s sensitive to assumptions that can reasonably be disputed, versus saying those models actually have predictive value in a distant, interstellar future. Thanks for your helpful clarification on this.

        Here’s perhaps my most important thought about extrapolating historical data and models forward more than a million years into the future when, it’s stipulated, people will be zooming around on interstellar spacecraft. Humans have existed for 300,000 years; hominins for 5-7 million; apes for 25 million; primates for 60-90 million; mammals for 160 million; complex life for 540 million. Last year, CRISPR was used to give a baby in Philadelphia a life-saving cure to a genetic disease. At least a dozen people with quadriplegia have experimental brain implants that allow them to use computers and play video games. We already have nascent means to biologically and mechanically modify the human brain and body. Surely, if progress continues, eventually we’ll be able to do this much faster than natural evolution. In a million years, or a billion years, will people orbiting distant stars really be just like people alive today?

        I broadly agree with the overall gist of that New York Times review of David Deutsch’s book. When Albert Brooks calls the book “brilliant and exhilarating and profoundly eccentric” and says Deutsch is “so smart, and so strange, and so creative, and so inexhaustibly curious, and so vividly intellectually alive”, I agree — I read the book years ago and still can’t stop thinking about it. When Brooks says “the chutzpah of this guy is almost beyond belief” and notes, skeptically, that Deutsch is “positively bubbling over with inviolable principles”, I share Brooks’ skepticism. We should be suspicious of anyone simultaneously expounding grand, sweeping theories of science, history, morality, education, and art. (Even if they’re allegedly all somehow part of one big theory.) Surely, at least some of what Deutsch says must be wrong.

        I have no opinion on which interpretation of quantum mechanics is correct. However, I can believe Brooks is right when he says Deutsch is too brazen in dismissing alternatives to the Everettian or many worlds interpretation. Deutsch is a respected quantum physicist, so perhaps he’s entitled to strong views on that topic, but his overall tendency is to be about equally brazen on every topic. If you can look past that, I think it’s easier to enjoy Deutsch’s work.

        The one part of Brooks’ review I think is really wrong is when Brooks says, “What he adores, what he is convinced contains the salvation of the world, is, in every sense of the word, The Market.” Deutsch doesn’t use this terminology. And I don’t see how it’s helpful or clarifying to use the term “The Market” to refer to academic science, liberal democratic institutions, or cultural evolution. Deutsch criticized libertarianism in an interview with Tyler Cowen. (To be fair, that was a decade after the review was published.) So, this feels like a weird political potshot out of nowhere from Brooks, in an otherwise artful, fair, and eloquent review.

        Reply

Leave a Reply

Discover more from Reflective altruism

Subscribe now to keep reading and get access to the full archive.

Continue reading