I have had it. I have had it with crack houses, dictatorships, torture chambers, disease, old age, spinal paralysis, and world hunger. I have had it with a planetary death rate of 150,000 sentient beings per day. I have had it with this planet. I have had it with mortality. None of this is necessary … We can end this … The Singularity will solve these problems. I declare reaching the Singularity as fast as possible to be the Interim Meaning of Life, the temporary definition of Good, and the foundation until further notice of my ethical system.
Eliezer Yudkowsky, “Staring into the singularity“
1. Introduction
This is Part 6 of a series based on my paper “Against the singularity hypothesis“.
The singularity hypothesis begins with the assumption that artificial agents will gain the ability to improve their own intelligence. From there, the singularity hypothesis holds that the intelligence of artificial agents will grow at a rapidly accelerating rate, producing an intelligence explosion: an event in which artificial agents rapidly become orders of magnitude more intelligent than their human creators.
Part 1 introduced and clarified the singularity hypothesis. Part 2 and Part 3 gave five preliminary reasons to doubt the singularity hypothesis. Together, these five reasons for doubt place a strong burden on defenders of the singularity hypothesis to provide significant evidence in favor of their view.
Parts 4-5 argued that leading defenses of the singularity hypothesis do not meet this burden. Part 4 looked at Chalmers’ arguments for the singularity hypothesis and argued that they do not work. Part 5 looked at Bostrom’s arguments for the singularity hypothesis in Superintelligence and drew similar conclusions.
Today’s post concludes by drawing out some implications of this discussion.
2. Optimistic scenarios
Many effective altruists fear that the singularity may bring death or suffering. But it is important to remember that a large number of authors have more optimistic expectations for the post-singularity future.
Ray Kurzweil, who along with Vernor Vinge is widely credited with popularizing the singularity hypothesis, makes this prediction in his book, The singularity is near:
The Singularity will allow us to transcend these limitations of our biological bodies and brains. We will gain power over our fates. Our mortality will be in our own hands. We will be able to live as long as we want (a subtly different statement from saying we will live forever). We will fully understand human thinking and will vastly extend and expand its reach. By the end of this century, the nonbiological portion of our intelligence will be trillions of trillions of times more powerful than unaided human intelligence.
This is, for Kurzweil, not the end of things. Kurzweil thinks that the singularity hypothesis is part of a more general `law of accelerating returns’ in which many or most growth processes show accelerating returns. Kurzweil predicts that the law of accelerating returns will continue for a good while beyond the end of this century:
The law of accelerating returns will continue until nonbiological intelligence comes close to “saturating” the matter and energy in our vicinity of the universe with our human-machine intelligence. By saturating, I mean utilizing the matter and energy patterns for computation to an optimal degree, based on our understanding of the physics of computation.
With such optimistic expectations for the post-singularity future, it is natural to think that the singularity could be a solution to many human ills. Indeed, the young Eliezer Yudkowsky held just that. Yudkowsky wrote:
I have had it. I have had it with crack houses, dictatorships, torture chambers, disease, old age, spinal paralysis, and world hunger. I have had it with a planetary death rate of 150,000 sentient beings per day. I have had it with this planet. I have had it with mortality. None of this is necessary. The time has come to stop turning away from the mugging on the corner, the beggar on the street. It is no longer necessary to look nervously away, repeating the mantra: “I can’t solve all the problems of the world.” We can. We can end this.
This is a reasonable sentiment to express. One might imagine that Yudkowsky, like early effective altruists, would go on to propose funding proven, evidence-based interventions to address poverty, disease, and other problems facing humanity today. But Yudkowsky has a different solution in mind. He continues:
The Singularity will solve these problems. I declare reaching the Singularity as fast as possible to be the Interim Meaning of Life, the temporary definition of Good, and the foundation until further notice of my ethical system.
Here the early Yudkowsky’s optimistic belief in the nearness of the singularity and the radical transformation brought about by the post-singularity future led him to turn away from short-term, evidence-based solutions to global ills, and instead to declare reaching the singularity to be “the interim meaning of life”. On this basis, Yudkowsky founded the Singularity Institute (which would later become MIRI) with the explicit goal of bringing about the singularity faster. (The singularity would, according to the young Yudkowsky, come about anyways in 2021, but it would be beneficial to bring it about sooner).
Even some leading scholars expressed quite optimistic expectations for the progress of self-improving artificial agents. Artificial intelligence pioneer I.J. Good, who introduced the singularity hypothesis, held that:
It is more probable than not that, within the twentieth century, an ultraintelligent machine will be built and that it will be the last invention that man need make.
The arguments in this paper suggest that the above scenarios are less likely than Good, Kurzweil and Yudkowsky took them to be.
This is important because it should make us less likely to gamble the solutions to the world’s social, political and technological problems on the coming singularity. If the singularity hypothesis is false, it becomes relatively less attractive to work towards accelerating the growth of artificial intelligence, and relatively more attractive to work directly to solve the world’s current problems.
3. Weakening definitions
We saw in Part 1 that it is very important to be clear about the meaning of the singularity hypothesis. A singularity hypothesis (1) specifies a quantity, (2) claims that this quantity will experience a sustained period of accelerating growth, until (3) a fundamental discontinuity in human history is reached. A specific version of the singularity results from specifying the quantity in question, the rate and duration of accelerating growth, and the nature of the hypothesized continuity.
This paper targets a common version of the singularity hypothesis on which the quantity in question is the general intelligence of artificial agents. Growth will be accelerating, usually hyperbolic, until the general intelligence of artificial agents is orders of magnitude beyond that of a typical human.
We saw In Part 4 that one prominent author, Chalmers, weakens the singularity hypothesis. Chalmers argues for a sustained period of exponential, rather than hyperbolic growth. We saw there that exponential growth is most naturally regarded as a form of constant rather than accelerating growth, and that this is important, because exponential growth may not necessarily have the same consequences as hyperbolic growth.
A recent survey article by Adam Bales, William D’Alessandro and Cameron Domenico Kirk-Gianni weakens the singularity hypothesis further. Their most direct definition of the singularity hypothesis is as follows:
The Singularity Hypothesis is the hypothesis that there will be a period of rapid recursive improvement in the capabilities of AI systems following the point at which AI systems become able to contribute to research on AI.
This definition weakens the singularity hypothesis in several important ways. First, it abandons claim (3), since it does not speak of a discontinuity. It may also weaken claim (2), insofar as rapid growth might not necessarily be accelerating. (Note for reference that this definition also treats a different version of the singularity hypothesis than that discussed in this paper, since the quantity given in (1) is the capability of AI systems, rather than their intelligence.)
Elsewhere, Bales and colleagues do not entirely jettison claim (3). The abstract of their paper defines the singularity hypothesis as the claim that:
The development of human‐level AI will unlock rapid further progress, culminating in AI systems far more capable than any human.
And in the introduction, they take the singularity hypothesis to hold that:
Rapid improvements, culminating in capabilities far beyond our species’ level, will follow on the heels of human‐level AI.
This is good to see. However, both readings continue to weaken claim (2), since they speak of rapid rather than accelerating growth. And on many natural readings, they have weakened the discontinuity from claim (3): systems “far more capable” than any human might not be orders of magnitude more capable than any human, and might not bring about an event that can be naturally compared to crossing the event horizon of a black hole.
What evidence do Bales and colleagues give to support their definition? As far as I can tell, they do not give any evidence at all.
It is established scholarly practice to show strong deference to established definitions, especially when set out by the editors of leading anthologies. If deference is to be avoided, it must be avoided on the basis of detailed textual analysis of leading texts. What must be avoided is a situation in which the meaning of the singularity hypothesis is weakened without argument during the course of discussion. This will, as we saw above, make it difficult to have a precise conversation, and also make it harder to see how the claimed implications of the singularity hypothesis are meant to follow from it.
4. Motivations are not everything
One reason why the singularity hypothesis has been popular in arguments for existential risk from artificial agents is that the singularity hypothesis simplifies the discussion. If the singularity hypothesis is right, then there is little question that artificial agents will become powerful enough to do what they wish with us. The only remaining questions are about motivations: what will artificial agents be motivated to do with us?
It is precisely for this reason that authors such as Bostrom and Yudkowsky focused so much on two claims about the motivations of artificial agents. The first is the orthogonality thesis, which on Bostrom’s formulation holds:
(Orthogonality Thesis) Intelligence and final goals are orthogonal axes along which possible agents can freely vary. In other words, more or less any level of intelligence could in principle be combined with more or less any final goal.
If the orthogonality thesis is correct, then it is in principle possible for superintelligent agents to hold any number of goals. But what goals are they likely to have?
This question is answered by a second claim, the instrumental convergence thesis. On Bostrom’s formulation, this holds:
(Instrumental Convergence Thesis) Several instrumental values can be identified which are convergent in the sense that their attainment would increase the chances of the agent’s goal being realized for a wide range of final goals and a wide range of situations, implying that these instrumental values are likely to be pursued by many intelligent agents.
Given the instrumental convergence thesis, many of the possible goals a superintelligent agent might have would lead the agent to pursue instrumental goals, typically argued to include resources and power.
Crucially, if the singularity hypothesis is correct, then the argument ends there. If a radically superintelligent agent emerges too rapidly to be controlled and comes to desire power and resources, it will seize them. If humans get in the way, then we will be pushed out of it. To borrow a phrase from Yudkowsky, one does not bargain with ants.
However, if the singularity hypothesis is false, then we cannot assume without argument that the next few centuries will bring agents so radically superintelligent that they can do with us as they will. We must either find another argument for attributing such capacities to superintelligent agents, or more likely, study more than the motivations of artificial agents. We need, that is, to say not only what agents will want to do, but how they might do it.
Recent power-seeking arguments have made some preliminary efforts at completing this task. I hope that future attempts will be more numerous and more detailed. Bringing about an existential catastrophe is no mean feat, and before sinking a good fraction of resources previously reserved for the poor into an all-out effort at preventing catastrophe, we are owed a good explanation of how that catastrophe might plausibly come about.
5. Why takeoff speed matters
I am often asked why the speed of takeoff to superintelligence should make a difference to discussions of existential risk. Above, we saw one reason why takeoff speed may make a difference: without accelerating growth, it is rather less likely that radically superintelligent agents will emerge any time soon. If that is right, then the agents we have to contend with, while still quite impressive, may not obviously be able to do as they wish with us.
Another reason why takeoff speed matters is that gradual threats are much easier to detect and prepare for. Indeed, Nick Bostrom argues extensively for this view in Superintelligence:
The main reason why the question of takeoff speed is important [is] not because it matters exactly when a particular outcome happens, but because the speed of the takeoff may make a big difference to what the outcome will be. With a fast or medium takeoff, it is likely that one project will get a decisive strategic advantage … A superintelligence with a decisive strategic advantage would have immense powers, enough that it could form a stable singleton – a singleton that could determine the disposition of humanity’s cosmic endowment.
Chapter 5 of Superintelligence argued that a fast takeoff would likely allow a single project to gain a decisive strategic advantage and form a singleton. That singleton could then do as it wished with us. However, we saw above that Bostrom thinks it is not so clear that a slow takeoff would lead any project to gain a decisive strategic advantage and form a singleton. Why does Bostrom think this?
Here, I think that Bostrom may be worth quoting at length:
During a slow takeoff, there would be plenty of time for the news to get out. In a moderate takeoff, by contrast, it is possible that developments would be kept secret as they unfold. Knowledge might be restricted to a small group of insiders, as in a covert state-sponsored military research program. Commercial projects, small academic teams, and “nine hackers in a basement” outfits might also be clandestine – though, if the prospect of an intelligence explosion were “on the radar” of state intelligence agencies as a national security priority, then the most promising private projects would seem to have a good chance of being under surveillance. The host state (or a dominant foreign power) would then have the option of nationalizing or shutting down any project that showed signs of commencing takeoff. Fast takeoffs would happen so quickly that there would not be much time for word to get out or for anybody to mount a meaningful reaction if it did. But an outsider might intervene before the onset of the takeoff if they believed a particular project to be closing in on success.
Moderate takeoff scenarios could lead to geopolitical, social, and economic turbulence as individuals and groups jockey to position themselves to gain from the unfolding transformation. Such upheaval, should it occur, might impede efforts to orchestrate a well-composed response; alternatively, it might enable solutions more radical than calmer circumstances would permit. For instance, in a moderate takeoff scenario where cheap and capable emulations or other digital minds gradually flood labor markets over a period of years, one could imagine mass protests by laid-off workers pressuring governments to increase unemployment benefits or institute a living wage guarantee to all human citizens, or to levy special taxes or impose minimum wage requirements on employers who use emulation workers. In order for any relief derived from such policies to be more than fleeting, support for them would somehow have to be cemented into permanent power structures. Similar issues can arise if the takeoff is slow rather than moderate, but the disequilibrium in moderate scenarios may present special opportunities for small groups to wield disproportionate influence.
This is not all of Bostrom’s discussion of the importance of takeoff speeds. That discussion spans the better part of Chapter 5, as well as portions of Chapters 4-8. But I think it is important to bear in mind that one of the best-known advocates of taking existential risk from artificial agents seriously has argued explicitly and at length for the importance of takeoff speeds to existential risk concerns.
If Bostrom is right, then slower takeoff speeds should lessen many existential risk concerns, not just because they make radical superintelligence less likely to emerge, but instead because they change how humanity is likely to react along the way.
6. Extraordinary claims require extraordinary evidence
Extraordinary claims require extraordinary evidence. We saw in Part 2 of this series that the singularity hypothesis is an extraordinary claim, positing a sustained period of growth at a rate rarely found in nature. As such, it should require correspondingly extraordinary evidence.
We saw in Parts 4-5 that the evidence given by key authors is somewhat less than extraordinary. We saw in Part 4 that Chalmers gives little evidence for the singularity hypothesis. We saw in Part 5 that Bostrom gives comparatively more evidence, but that this evidence breaks into three categories: plausible scenarios that have been over-interpreted to support the singularity hypothesis, restatements of the core hope behind the singularity hypothesis, and mis-interpretations of historical trends. This is considerably less extraordinary than the evidence required to assign significant plausibility to the singularity hypothesis.
This isn’t the first time that we’ve discussed instances of extraordinary claims being advanced by effective altruists on less-than-extraordinary evidence. Part 6 discussed Eliezer Yudkowsky’s claim to have learned that the Bank of Japan was leaving trillions of dollars on the table by perusing economics blogs. We also discussed Robin Hanson’s claim that only a modest fraction of UFO sightings by otherwise-reliable witnesses can be explained as errors or hallucinations, which Hanson supports by holding that “If you’ve also looked but can’t see this, I just don’t know what to say. Pay more attention?”
We have also seen a great number of extraordinary claims about the level of existential risk facing humanity in my series Exaggerating the Risks. There, we have seen evidence that climate risk, biorisk, and AI risk may have been exaggerated. In many cases, the arguments for key claims were simply missing. We found no more than a few sentences in The Precipice which could plausibly be interpreted as an argument for Ord’s claim that there is a 1 in 1,000 risk of irreversible existential catastrophe from climate change by 2100, that the Carlsmith report gives no meaningful argument for instrumental convergence, and that effective altruists have repeatedly refused to say in detail what is driving their fears about existential biorisk.
The singularity hypothesis should be understood as one more in a growing line of extraordinary claims advanced by effective altruists on the basis of rather less than extraordinary evidence. This should decrease our confidence in the reliability of other claims made by effective altruists, especially extraordinary claims. It should also be seen by effective altruists as a reason to provide clearer and stronger evidence behind other extraordinary claims, in order to allay suspicion that these claims, too, are being made on the basis of insufficient evidence.
7. Conclusion
This concludes my discussion of the paper “Against the singularity hypothesis.” This paper has now been published in a special issue of Philosophical Studies on AI Safety. It can be read on an open-access basis here.
Here are some more resources available to explore the paper:
Summary: Global Priorities Institute Paper Summaries; EA Forum
Podcast: The Gradient
Discussion (Video): AI safety reading group
Infographic: Here
I hope that readers have enjoyed this series. I hope to soon begin a series on my most recently published paper, “The scope of longtermism“, forthcoming in Australasian Journal of Philosophy.
Leave a Reply