In the limit of technological progress, as it becomes feasible to create a nearly perfectly plastic world—one compliant with our whims and wishes and undemanding of our labor—how can a utopia be conceived in which values such as purpose, interestingness, richness, fulfillment, and meaning are not undermined to such an extent as to largely deprive the resulting condition of its desirability (and without sacrificing unacceptable amounts of other values)?
Nick Bostrom, Deep Utopia
1. The post I didn’t write
I meant to write a different post today. My aim was to take a look at the METR task-completion time horizon report. Given that many reports in this area, such as the AI 2027 report tend to be fairly unreliable and based largely on subjective forecasts, I thought the same might be true for the METR report.
To a large extent, I found that this was not the case. The modeling is reasonably rigorous, and with the exception of some human completion-time data for large time-horizon tasks, most of the data comes from experiments rather than forecasts. While there remain some challenges with the report, the team at METR has been admirably forthright about admitting and when possible addressing them. More to the point, METR’s predictions have held up fairly well, and to a large extent the benchmark appears to have saturated, making further discussion less valuable. I don’t know that I would say the report holds up to the highest academic research standards, but it is far, far better than many reports in this space.
(This is not to say that I approve of the conclusions that many have drawn from the graph, but that is another topic entirely.)
I write a good deal about things that I do not like within this community, so I think it is important to give credit where credit is due. I do not think the METR report is open to anything like the kind of criticism typically leveled at reports within this community.
2. Introduction
This is Part 3 of my series, Deep Utopia, discussing Nick Bostrom’s book of the same name. The book is structured around a series of six hypothetical lectures delivered by Bostrom himself on six successive days.
Part 1 discussed the first lecture, Monday. Monday’s lecture explored two inter-linked sets of questions: whether humans might work and earn incomes in the future; and what shape future human population dynamics will take.
Part 2 discussed the second lecture, Tuesday. Tuesday’s lecture discussed our cosmic resource endowment and barriers to reaping the full endowment.
Part 3 discussed the third lecture, Wednesday. Wednesday’s lecture was structured around two versions of the purpose problem of how we might find purpose in a solved world.
Today’s post discusses the fourth lecture, Thursday. Thursday’s lecture asks how a solved world could contain not only purpose but also interestingness. Wouldn’t a solved world be boring?
The lecture is the longest yet, at a whopping 114 pages. I will do my best to compress its main ideas in what follows, then discuss what is going well and what could go better.
3. Plasticity and redundancy
The structuring concern of Thursday’s lecture is that future societies would have two characteristics. First, they would be plastic: it would be relatively easy to achieve any local configuration of matter that we desired. Second, humans would be autopotent, able to modify themselves as they chose.
This threatens not only purpose in life, but also things such as interest, fulfillment and meaning, leading to a new formulation of the redundancy concern:
In the limit of technological progress, as it becomes feasible to create a nearly perfectly plastic world—one compliant with our whims and wishes and undemanding of our labor—how can a utopia be conceived in which values such as purpose, interestingness, richness, fulfillment, and meaning are not undermined to such an extent as to largely deprive the resulting condition of its desirability (and without sacrificing unacceptable amounts of other values)?
Thursday’s lecture is structured around responding to this redundancy concern, focusing largely on the problem of preserving interestingness.
4. Objective and subjective interestingness
Bostrom distinguishes between the subjective feeling of boredom and the objective condition of things being boring. Feeling bored is an emotion which does not exhaust (and need not coincide with) objective facts about whether conditions are boring, though it is hard to say exactly what objective boringness amounts to. Taking interest to be the opposite of boredom gives an analogous distinction between subjective and objective interestingness.
In some ways, Bostrom notes, it is easy to get rid of subjective feelings of boredom. We could do this through any number of artificial means, and even employ affective prosthetics meant to alert us to the conditions that boredom signals without the unpleasant feelings of boredom. At the same time, Bostrom warns, we should be cautious about monkeying with human nature. Further, on some views boredom is a fitting response to objectively boring circumstances, so it would not do to purge subjective feelings of boredom unless we also eliminated objectively boring circumstances.
Bostrom notes some complexities that arise in speaking of objective interestingness. It is not easy to say how objectively interesting Shakespeare, a Rubik’s cube, a rattle, or a radio telescope is. It is not even easy to see whether something like manufacturing 200,000 table legs could be as objectively interesting as an interesting mathematical problem.
Further, Bostrom emphasizes, we should not discount the possibility of aesthetic experiences being as interesting as intellectual experiences. At the same time, the value of aesthetic experiences may have something to do with the capacities of observers to appreciate them, which would seem to place a hard cap on how much interesting aesthetic experience an observer with limited appreciative capacities can have.
5. Further discussion of interestingness
Bostrom considers four ways in which our desire to live an interesting life might come about. It could be (a) a way of promoting learning and cognition; (b) a strategy for social signaling of having engaged in interesting experiences; (c) a rut-avoidance method for avoiding purposeless repetitive activities or (d) a spandrel that arises as a byproduct of other concerns.
Bostrom thinks that (b) and (c) provide decent prospects for interestingness in the far future, since social signaling and rut-avoidance will be live concerns. Bostrom is more pessimistic about (a), since opportunities for learning may diminish. Interest under (d) depends entirely on the values that give rise to concern for interestingness.
Bostrom distinguishes between the contained interestingness within a thing, and the contributed interestingness that it brings to a whole. For example, if I live an interesting life, then cloning me may produce an object with high contained interestingness, since its life would be as interesting in itself as mine, but low contributed interestingness, since it would not add much interest to the world beyond what I had already added. Generalizing, Bostrom suggests that in a large universe our lives may contain a great deal of interestingness while often contributing little interestingness to the world. However, Bostrom suggests, we may contribute more interestingness in a local or parochial perspective, such as a community.
6. Trajectories
As in previous chapters, Bostrom urges that we think about the interestingness of human trajectories and not merely about the interestingness of the end states that these trajectories produce.
One reason why trajectories matter is that personal identity, or other quantities that matter for survival, may decay as chains of mental connection between our current and future selves erode, or with the passage of time. If we want to survive and not merely create a great future full of people, we may want to take things slow. Bostrom also suggests that some steps could be taken to preserve personal identity over time by preserving identity-relevant quantities such as memory.
Another reason why trajectories matter is that we might care to live in a future whose direction we shaped ourselves. For this reason, Bostrom suggests, we might send AI outriders to secure the broad shape of the future, but allow humans to determine its contours and how we get there.
7. Feodor the Fox
The lecture concludes with the customary series of epistles by Feodor the Fox. Feodor returns home to learn that wolves have killed his friend Pignolius. Feodor flees his home and encounters a doe, Clara, with an injured leg. Sympathetic to Clara’s plight, Feodor feeds her out of his limited supplies, for which Feodor gets a scolding from his friend Rey.
Rey takes umbrage at the waste of scarce resources and insists that Clara commit to a lifetime of indentured work in exchange for food. Rey then attempts to “hook [Feodor] up with a nice vixen” in order to kickstart their project of selectively breeding the population for cooperativeness. Feodor is skeptical and manages to talk Clara’s indenture down to a year.
The epistles conclude with Feodor watching the sunset, reflecting on how the world can be at once beautiful and terrible.
8. Philosophical contributions
Bostrom is certainly right to note that a solved world threatens not only purpose, but also other quantities such as interestingness. He is also right to distinguish the subjective feelings of interest and boredom from objective states of living in an interesting or boring condition. And he is certainly right to consider that ensuring objective interestingness may be more difficult than simply ridding the world of feelings of boredom.
Probably the most significant philosophical contribution of the chapter is the distinction between contained and contributed interestingness. This is not a common distinction, and it advances the debate by revealing one way in which ensuring interestingness can be challenging. Even if individual lives within a solved world contained many interesting experiences, they might contribute little interestingness to the whole. The paradoxical result is that a solved world could be quite uninteresting, even if the lives lived within it were more interesting than our lives today. Indeed, many of the high-welfare scenarios envisioned by longtermists, such as universes suffused with flourishing digital minds, might be relatively uninteresting if these minds lived similar lives.
9. Ensuring objective interestingness
One strange thing about the chapter is that it does not offer much in the way of telling the reader how future societies could be made objectively interesting. For the most part, it is rather the opposite: we are told that assessing interestingness is difficult, and that natural routes to interestingness such as learning and aesthetic experience may not ground as much interestingness as they seem to.
The main positive contribution is the idea that the future could probably be made interesting on two stories about the origin of our subjective desire for interestingness. On the first story, the desire for interestingness arises from social signaling. On the second story, the desire for interestingness arises from attempts to avoid ruts of repetitive activities.
In each case, Bostrom argues that the relevant causes of our subjective desire for interestingness would be alive and kicking in the future. We would, Bostrom argues, still have ample need for social signaling as well as to avoid ruts of repetitive activities.
The problem is that none of this tells us why the world would be objectively interesting. A world in which people spend their days socially signaling to one another does not seem especially objectively interesting. And the mere avoidance of ruts seems like a way to avoid objectively uninteresting experiences, but not yet a way to produce objectively interesting ones.
The chapter is rather long. Readers might have expected a strong positive proposal about how to ensure objective interestingness. I don’t find one in this chapter.
10. Feodor revisited
I have to confess, I don’t know what to make of Feodor. I’m not sure why we are told that Pignolius was eaten by wolves, or what the allegorical exploitation of Clara is supposed to mean. I’m certainly not sure why Rey feels the need to make his offer to hook Feodor up with a nice vixen. And the closing reflection on the mixed beauty and horror of the world, while Feodor watches a sunset, is poignant but hard to place.
I don’t know what Bostrom is doing with these epistles, and I think it would have been rather a better book if Bostrom had done more to explain himself here.
11. Personal identity
I’m also struck by the large amount of space devoted to discussions of personal identity. It is executed well enough. But it is relevant only on a story on which we ourselves anticipate living for a very long time, such that it will be not only our descendants but also we ourselves who find ourselves occupying a solved world.
As in previous chapters, one is tempted to think that the discussion here rests on strong unargued assumptions. Perhaps those assumptions involve methods of life extension such as brain uploading, or perhaps they involve the assumption that a solved world is quite temporally near.
If these assumptions are being made, it would be nice to have them stated and argued for. And if they are not made — or even if they are, given that they are relatively niche concerns — this may have been a place where the book could have benefited from a judicious trimming.
12. Taking stock
Today’s post discussed the fourth lecture of Deep utopia, Thursday. The lecture extends the redundancy concern from lack of purpose to lack of interestingness and other related phenomena. After an extended discussion of interestingness, the lecture closes with further epistles by Feodor the Fox.
Like Wednesday’s chapter, Thursday’s chapter is in many ways one of Bostrom’s richer philosophical contributions, and we saw that there are clear insights to be taken away from it. It is also, we saw, quite long. It may not offer a strong positive solution to the problem of interestingness. And some of the discussions, such as the treatment of personal identity and the epistles of Feodor the Fox, could perhaps do with a judicious trimming.
This is not to say that there is nothing to be gained from the chapter. To be honest, I wish that more readers had made it this far into the book. But at the same time, it may be hard to justify spending over a hundred pages in exchange for the gains made.
Leave a Reply