Look on our Works, ye Mighty! What an indictment, these long lines of motorists – thousands upon thousands, glum-faced behind their steering wheels, honking and swearing at one another, inhaling each others’ exhaust fumes, each one by his very existence making life a bit worse for everyone else. Every day, twice a year, year in and year out!
Nick Bostrom, Deep utopia
1. Introduction
This is Part 2 of my series, Deep Utopia, discussing Nick Bostrom’s book of the same name. The book is structured around a series of six hypothetical lectures delivered by Bostrom himself on six successive days.
Part 1 discussed the first lecture, Monday. Monday’s lecture explored two inter-linked sets of questions: whether humans might work and earn incomes in the future; and what shape future human population dynamics will take.
Today’s post discusses the second lecture, Tuesday.
2. Our cosmic endowment
Tuesday’s lecture is concerned with our cosmic endowment: the resources available to Earth-originating civilization. The lecture has two parts. The first, and shortest, part discusses the potential size of our cosmic endowment. The second, and longest, part discusses limiting factors that may prevent us from realizing the totality of our cosmic endowment.
Bostrom’s positive account of our cosmic endowment is a bit underwhelming, because it is short and re-uses some previous materials from earlier works. Bostrom begins with the remark that our cosmic endowment is probably finite, and that, absent strange technologies or utility functions, this likely places a finite constraint on the achievable size of the economy. Bostrom then repeats a discussion from his earlier book, Superintelligence, of how the size of the cosmic endowment might be calculated in terms of human lives or performable computational operations under various assumptions.
Assuming a civilization that sends von Neumann probes in all directions at 50% of the speed of light, Bostrom calculates that 6*1018 stars would be reached, rising to 2*1020 stars at 99% of the speed of light.
If 10% of these stars had planets, each of which could support a population of a billion humans for a billion years, then on the high end we might expect 1035 human lives, rising if larger populations could be supported.
If stars could be turned into Dyson spheres, Bostrom estimates, about 1067 computational operations per second could be performed at the high end, leading to perhaps 1058 simulated human lives of 100 years each.
To reach our cosmic endowment, Bostrom argues, we would need to approach technological maturity: a level of control over nature close to the maximum that could be achieved in the fullness of time.
Bostrom provides a lower bound on the level of technology needed to approach technological maturity by listing some necessary advances in manufacturing and robotics, artificial intelligence, transportation and aerospace, VR and computation, medicine and biology, mind engineering, sensors and security.
3. Barriers
The rest of Tuesday’s lecture discusses barriers to reaching technological maturity and exploiting our cosmic endowment.
The first barrier is coordination problems. Bostrom suggests that technology may worsen some coordination problems, for example by aiding criminals and dictators.
The second barrier is prudential barriers. It may not always be prudent to pursue our cosmic endowment as quickly as possible, for example because the pursuit of progress may exacerbate existential risks.
The third barrier is axiological contours: limits related to the shape of our values. We may value positional goods, making ourselves miserable in the endless pursuit of status or failing to uplift others as a means of enhancing our relative status. We may value impact and novelty, finding it hard to make a significant impact or engage in novel pursuits in a largely solved world. Our preferences may saturate as we grow wealthier, and it may be increasingly hard to meaningfully improve our lot. And there may be moral constraints on methods of reaching our cosmic endowment, because the means used could be unacceptable.
The fourth barrier is metaphysics. It may turn out to be metaphysically impossible to create some otherwise good things. For example, it may turn out to be impossible to create digital minds that are sentient or have moral status, undermining the more ambitious estimates of our cosmic endowment that drew on the value of digital minds.
The fifth barrier is limits to automation. Humans may resist automation in any number of ways, limiting technological progress. Regulation may slow technological development. Purchasing human-produced goods and services may become a status symbol, or a way to show solidarity with fellow humans. Humans may be uniquely desirable in certain roles, including the clergy, leaders of customs and ceremonies, and professional athletes. And humans may distrust AI and other forms of automation.
4. Feodor the fox
Tuesday’s lecture closes with another reading from the chronicles of Feodor the Fox. This reading takes the form of ten numbered epistles in which Feodor writes to his dear Uncle Pasternaught.
As for substantive discussions, Feodor discusses problems of ensuring that various individuals contribute to the common good, the need for systems of knowledge-sharing and the difficulty of creating them, and barriers to civilizational improvement including a lack of consistent surplus, overpopulation, defection from population control measures, and foreign invasion. There are also some brief musings on the differences of norms across cultures.
As for narrative, the dialogue involves many curious elements including an extended house stay with Pignolious the Pig, a new board game, a dream about a mangy lamb, and a trip to the sea.
I don’t know what to make of this reading, so I will not discuss it further. The rest of this post discusses two concerns with the discussion in Tuesday’s lecture.
5. Undercurrents: Surveillance and happy slaves
The first concern is that the discussion sneaks in, without significant argument, two themes that have landed Bostrom in trouble before: a fondness for surveillance and an openness to creating “happy slaves” who would genuinely enjoy being subservient.
5.1. Surveillance
In two parts of the lecture, Bostrom expresses unusual fondness for government surveillance.
First, in his discussion of lower-bound technologies required for technological maturity, Bostrom includes a category of “sensors and security” technologies. These include “ubiquitous, fine-grained, real-time multi-sensor monitoring” and “aligned police-bots and automatic treaty enforcement.”
Second, Bostrom makes similar remarks in discussing how technology could solve coordination problems: “Improved surveillance could make it easier to prevent certain kinds of crime” and “treaty bots” could enable countries to more credibly commit to nonaggression pacts.
This appeal to ubiquitous real-time monitoring has landed Bostrom in hot water before. In Part 2 of my series Harms, we saw that Bostrom elsewhere expresses a fondness for the following scenario in response to high levels of existential risk:
(High-tech Panopticon) Everybody is fitted with a `freedom tag’ … worn around the neck and bedecked with multidirectional cameras and microphones. Encrypted video and audio is continuously uploaded from the device to the cloud and machine-interpreted in real time. AI algorithms classify the activities of the wearer, his hand movements, nearby objects, and other situational cues. If suspicious activity is detected, the feed is relayed to one of several patriot monitoring stations. There are vast office complexes, staffed 24/7. There, a freedom officer reviews the video feed on several screens and listens to the audio in headphones. The freedom officer then determines an appropriate action, such as contacting the tag-wearer via an audiolink to ask for explanations or to request a better view. The freedom officer can also dispatch an inspector, a police rapid response unit, or a drone to investigate further. In the small fraction of cases where the wearer refuses to desist from the proscribed activity after repeated warnings, an arrest may be made or other suitable penalties imposed. Citizens are not permitted to remove the freedom tag, except when they are in environments that have been outfitted with adequate external sensors (which however includes most indoor environments and motor vehicles). The system offers fairly sophisticated privacy protections, such as automated blurring of intimate body parts, and it provides the option to redact identity-revealing data such as faces and name tags and release it only when the information is needed for an investigation. Both AI-enabled mechanists and human oversight closely monitor all the actions of the freedom officers to prevent abuse.
Many people would view this kind of panopticon-style surveillance as more akin to a dystopian police state than a utopian future.
5.2. Happy slaves
In his discussion of axiological contours, Bostrom considers the possibility of “happy slaves,” sentient humanoid or artificial creatures who would willingly and happily occupy servile roles. Bostrom holds that although more shallow forms of subservience may be easily rejectable, it is unclear whether there is anything wrong with creating happy slaves who are deeply subservient in that “their natures are, through and through, adapted in such a way that they are genuinely happy and authentically fulfilled by being subservient.” He closes with the cryptic suggestion that “perhaps there are additional moral complications beyond this” first-pass permissibility of creating happy slaves, but does not elaborate further.
Later, while discussing the automation of labor, Bostrom confronts the worry that automated societies full of sentient machines could be likened to slave societies. Again, Bostrom moves past the worry without substantial argument.
Depending on exactly how the robots were constructed, there could be profound ethical differences between the case of human slavery and the case of sentient machines that may be owned by humans and used to perform tasks. These differences might be fundamental enough that it would not be appropriate to refer to the sentient machines as “slaves”. Perhaps new terms will need to be introduced to designate these cases: not automation, not slavery, but some novel third category.
In both cases, Bostrom shows a strong willingness to endorse potentially quite problematic relationships to fellow humans and sentient machines without substantial argument, or even without much beyond a bare characterization of the problem.
6. Current technology and science
A common challenge in predicting the future is that our predictions are often profoundly shaped by current technologies and scientific understanding.
A medieval denizen asked to list the technologies essential for reaching humanity’s potential would likely have handed over a list of agricultural innovations, and perhaps even some engineering innovations if they were especially progressive. They might have pictured utopia as Cockaigne, the Land of Plenty introduced in Monday’s lecture, where agricultural labor has ceased, the walls are made of sausages, the rivers flow with wine, and nuns show their bottoms.
A 19th-century industrialist asked to list the technologies essential for reaching humanity’s potential would likely have handed over a list of mechanical innovations, emphasizing improvements in factory production, and perhaps the discovery of deep piles of coal or coal substitutes. They might have pictured utopia as a world full of automated factories providing goods on command.
Most of us think that these individuals had at best a highly incomplete understanding of the technologies needed to reach humanity’s potential, and the worlds that might arrive once our potential was reached.
Bostrom’s discussion of the cosmic endowment presents a vision of humanity’s potential that is firmly grounded in 21st-century science and technology. Growth comes from space settlement, limited primarily by sub-light travel speeds. The most important resources to be seized are energy, and the most important use for energy is to convert that energy into computational power that can be used to simulate minds.
These visions of the future can seem attractive, or at least plausible, from the present imagination. But I would urge readers to reflect on the speed with which past visions of the future have come to look naive, grounded in incomplete scientific understanding and limited technological vision.
We no longer think that a utopian future is likely to resemble a land where the walls are made of sausages and the river flows with wine. Why, then, should we place much stock in a future where sub-light space settlement leads to conversion of stars into Dyson spheres and the use of energy to simulate minds?
Here, Bostrom may object that even if our understanding of the future is radically wrong, it at least provides a lower bound on the value of possible futures. With each increase in understanding, Bostrom might continue, we realize not that the futures we previously envisioned were impossible, but rather that we can construct an even better future than before.
Perhaps that is right. I am not sure. But note that it speaks at most to the limited aim of characterizing the value of the future. When we turn more concretely towards thinking what the future may be like and what we should do to prepare for it, matters are otherwise.
Consider, for example, the list of technologies that Bostrom submits as lower-bound requirements for technological maturity. These include advances in manufacturing and robots (such as atomically precise manufacturing); artificial intelligence (such as machine superintelligence); transportation and aerospace (such as von Neumann probes); virtual reality (such as realistic mental simulations); medicine and biology (such as anti-aging); mind engineering (such as brain uploading); and sensors and security (such as ubiquitous surveillance).
Perhaps some of these elements, such as medical advances, will remain stable across visions of the future. But most of the elements on this list seem just as arbitrarily rooted in particular, historically grounded understandings of science and technology as any that would have been submitted by a medieval lord or a 19th-century industrialist.
If this is right, then we might be cautious in leaning too much on Bostrom’s particular technological vision of what the future might be like and how we could progress towards that future.
7. Taking stock
Today’s post discussed the second (Tuesday’s) lecture from Deep utopia. This lecture concerned the potential size of our cosmic endowment, and limiting factors that may prevent us from realizing the totality of our cosmic endowment.
After surveying the contents of the lecture, we looked at two sets of concerns.
First, Bostrom expresses at least guarded enthusiasm for two kinds of situations that many readers may balk at: ubiquitous government surveillance, and happy slaves bred to enjoy servitude. These situations are advanced with surprisingly little argument, generating concern that Bostrom is allocating space for reflection and interrogation on the wrong issues within the chapter.
Second, Bostrom’s vision of our cosmic endowment and progress towards technological maturity is in many ways situated within a particular 21st-century understanding of science and technology. This is not Bostrom’s fault, but a situation that all futurists find themselves in. But once we realize the extent to which our visions of the far future are situated within current scientific and technological understandings, this realization may give us reason to distrust them as guides to action.
The next post in this series will discuss the third (Wednesday’s) lecture.
Leave a Reply