It might appear to some readers that … [a] slow takeoff is the most probable, [a] moderate takeoff is less probable, and [a] fast takeoff is utterly implausible. It could seem fanciful to suppose that the world could be radically transformed and humanity deposed from its position as apex cogitator over the course of an hour or two … Nevertheless … [I] present some reasons for thinking that the slow takeoff transition scenario is implausible. If and when a takeoff occurs, it will likely be explosive.
Nick Bostrom, Superintelligence
1. Introduction
This is Part 5 of a series based on my paper “Against the singularity hypothesis“.
The singularity hypothesis begins with the assumption that artificial agents will gain the ability to improve their own intelligence. From there, the singularity hypothesis holds that the intelligence of artificial agents will grow at a rapidly accelerating rate, producing an intelligence explosion: an event in which artificial agents rapidly become orders of magnitude more intelligent than their human creators.
Part 1 introduced and clarified the singularity hypothesis. Part 2 and Part 3 gave five preliminary reasons to doubt the singularity hypothesis. Together, these five reasons for doubt place a strong burden on defenders of the singularity hypothesis to provide significant evidence in favor of their view. The next task is to argue that this burden has not been met.
Part 4 looked at Chalmers’ arguments for the singularity hypothesis and argued that they do not work. Today’s post looks at Bostrom’s arguments for the singularity hypothesis in Superintelligence.
2. Stage setting
Nick Bostrom is Principal Researcher at the Macrostrategy Research Initiative and a leading figure in the effective altruism movement. Bostrom’s book, Superintelligence, is widely regarded as a foundational contribution to the study of existential risk from artificial agents, and many of Bostrom’s papers have been deeply influential as well.
Chapter 4 of Superintelligence, “The kinetics of an intelligence explosion” argues that takeoff to superintelligence is likely to occur on a `fast’ timescale of minutes, hours or days, or a `moderate’ timescale of months or years. Bostrom’s argument is significantly deeper, more extensive, and more scientific than most other arguments in this space – a good sign. Nevertheless, I will argue that Bostrom’s argument does not work.
Bostrom structures his discussion around the notions of recalcitrance and optimization power. Optimization power measures the quality-weighted design effort put into improving artificial systems at any given time. Recalcitrance measures the amount of optimization power needed to produce a unit improvement in intelligence. With this definition in place, we can express the rate of change of intelligence (ΔI) as a function of recalcitrance and optimization power, as ΔI = Optimization Power / Recalcitrance.
To argue for a fast or moderate takeoff speed, Bostrom needs to argue that we will experience a sustained period of high optimization power coupled with low recalcitrance.
By my best count, Chapter 4 of Superintelligence gives six arguments for a sustained period of high optimization power coupled with low recalcitrance. I look at each of these arguments below. I argue that they break into three categories.
Category 1: Plausible but over-interpreted scenarios: Arguments in this category point to somewhat plausible scenarios that have been substantially over-interpreted to provide support for the singularity hypothesis, when they in fact support much weaker conclusions.
Category 2: Restating the core hope: Arguments in this category do not go substantially beyond restatements of the core hope of the singularity hypothesis. These will therefore not offer much independent support for the singularity hypothesis.
Category 3: Mis-interpreting history: Arguments in this category mis-interpret historical trends. When these mis-interpretations are corrected, the arguments tell against rather than in favor of the singularity hypothesis.
3. Plausible but over-interpreted scenarios
Bostrom points to three different scenarios which, though somewhat plausible, do not support anything like the singularity hypothesis.
3.1: Whole-brain emulation
Suppose that human-level artificial intelligence is reached through whole-brain emulation. Although it may be hard to produce the first whole-brain emulation, once this is done, we may make quick progress by improving or duplicating the emulation. Bostrom writes:
The difficulties involved in creating the first human emulation are of quite a different kind from those involved in enhancing an existing emulation. Creating a first emulation involves huge technological challenges … By contrast, enhancing the quality of an existing emulation involves tweaking algorithms and data structures: essentially a software problem, and one that could turn out to be much easier than perfecting the imaging technology needed to create the original emulation … Another way to improve the code base once the first emulation has been produced is to scan additional brains with different or superior skills and talents. Productivity growth would also occur as a consequence of adapting organizational structures and workflows to the unique attributes of digital minds. (pp. 68-69).
Bostrom is quite correct that whole-brain emulation, like many technologies, may be relatively difficult to create, but then may allow for a series of low-effort early improvements to the first successful model. For example, Bostrom suggests we could improve algorithms and data structures, scan additional brains, or improve organizational structures through which digital minds relate to one another.
But precisely because this can be said of most technologies, it does not suggest anything like the level of growth needed to ground the singularity hypothesis. Indeed, Bostrom himself goes on to list at least two reasons why recalcitrance should soon begin to rise on this path to superintelligence.
Even Chalmers throws Bostrom under the bus here. Letting AI+ be AI that is at least as intelligent as the smartest human (but not superintelligent), Chalmers denies that whole brain emulation is easily extendible to produce AI+, let alone superintelligence, from human-level AI:
Another method that is not obviously extendible is brain emulation. Beyond a certain point, it is not the case that if we simply emulate brains better, then we will produce more intelligent systems. So brain emulation on its own is not clearly a path to AI+. It may nonetheless be that brain emulation speeds up the path to AI+. For example, emulated brains running on faster hardware or in large clusters might create AI+ much faster than we could without them. We might also be able to modify emulated brains in significant ways to increase their intelligence. We might use brain simulations to greatly increase our understanding of the human brain and of cognitive processing in general, thereby leading to AI+. But brain emulation will not on its own suffice for AI+: if it plays a role, some other path to AI+ will be required to supplement it. (Chalmers 2010, p. 18).
If Chalmers is on the right track, then bare appeal to whole brain emulation won’t be enough to ground the singularity hypothesis.
3.2: Content improvements
Second, Bostrom argues that we may make content improvements: improvements to software that go beyond improvements to central algorithms. Bostrom writes:
Consider content improvements. By “content” we here mean those parts of a system’s software assets that do not make up its core algorithmic architecture … Consider a contemporary AI system such as TextRunner (a research project at the University of Washington) or IBM’s Watson (the system that won the Jeopardy! quiz show) … Now imagine a remote descendent of such a system that has acquired the ability to read with as much understanding as a human ten-year-old … So we are imagining a system that thinks much faster and has much better memory than a human adult, but knows much less, and perhaps the net effect of this is roughly human-level equivalent in its general problem-solving ability. But its content recalcitrance is very low – low enough to precipitate a takeoff. Within a few weeks, the system has read and mastered all the content contained in the Library of Congress. Now the system knows much more than any human being and thinks vastly faster: it has become (at least) weakly superintelligent. (pp. 70-71).
Here Bostrom makes an interesting suggestion that history has borne out: training AI systems on vast amounts of text (say, the Library of Congress or the Common Crawl) is a good way to quickly improve those systems. For the most part, this has already come to pass. But we didn’t see an explosion to superintelligence, and we are running out of data to feed into our models to precipitate a `fast’ or `moderate’ takeoff along these routes.
At the very end of this passage, Bostrom makes the suggestion that there is a weak sense in which such a system is superintelligent. I think that this remark is mostly a distracting and uncharacteristic move by Bostrom to weaken the target of the chapter, which is to explore speeds of takeoff towards a type of superintelligence sufficiently powerful to ground the arguments of the rest of the book. I think that this remark is most charitably ignored.
3.3: Hardware improvements
Bostrom also suggests that fast or moderate takeoff might be induced by improving hardware.
In the short term, computing power should scale roughly linearly with funding: twice the funding buys twice the number of computers, enabling twice as many instances of the software to be run simultaneously … In the slightly longer term, the cost of acquiring additional hardware may be driven up as a growing portion of the world’s installed capacity is being used … Over a somewhat longer timescale, the supply of computing power will grow as new capacity is installed … Historically, the rate of improvement of computing technology has been described by the famous Moore’s law, which in one of its variations states that computing power per dollar doubles every 18 months or so. Although one cannot bank on this rate of improvement continuing up to the development of human-level machine intelligence, yet until fundamental physical limits are reached there will remain room for advances in computing technology. (pp. 72-73).
There are three things to say here. The first is that this isn’t what Bostrom needs: the singularity hypothesis requires hyperbolic growth, whereas Moore’s law provides only exponential growth. We will see later that Bostrom attempts to fill the gap by positing falling recalcitrance: as time goes on, improvements in the intelligence of artificial systems become easier, not harder to make. (We saw in Part 2 in our discussion of diminishing research productivity that this is unlikely).
Second, we saw in Part 2 that most theorists expect Moore’s law to end in this decade, if it has not ended already. The reason is that, as we saw in Part 2, Moore’s law has been sustained across rapidly diminishing research productivity by throwing more and more resources at the problem, and we are quickly running out of resources to add to the pile.
Third, we saw in Part 3 that exponential gains in hardware capacities need not translate into exponential gains in machine intelligence. On many natural measures, exponential hardware gains have yielded only linear intelligence gains. This means that even an indefinite continuation of Moore’s law may not lead even to exponential intelligence growth.
4. Restating the core hope
Bostrom’s next two arguments mostly restate the core hope behind the singularity hypothesis. As such, they do not provide much independent support for the singularity hypothesis.
First, Bostrom suggests that a few programmers may make a software discovery that brings us to superintelligence overnight.
In some situations, recalcitrance could be extremely low. For example, if human-level AI is delayed because one key insight long eludes programmers, then when the final breakthrough occurs, the AI might leapfrog from below to radically above human level, without even touching the intermediary rungs. (p. 69).
This is a less plausible restatement of the core hope behind the singularity hypothesis. It is less plausible because it removes the core mechanism, recursive self-improvement, that is supposed to help us bootstrap our way to superintelligence. Instead, it insists, a single software insight may do the trick. It is noteworthy that many advocates of fast AI timelines have dropped this line of argument, opting instead for appeals to recursive self-improvement or, more recently, scaling laws, all of which have a gradualist character and the last of which gives a key role to hardware as well as software.
Second, Bostrom suggests that the intelligence explosion may take off once the intelligence of AI systems becomes predominantly domain-general rather than domain-specific:
Another situation in which recalcitrance could turn out to be extremely low is that of an AI system that can achieve intelligent capability via two different modes of processing. To illustrate this possibility, suppose an AI is composed of two subsystems, one possessing domain-specific problem-solving techniques, the other possessing general-purpose reasoning ability. It could then be the case that while the second subsystem remains below a certain capacity threshold, it contributes nothing to the system’s overall performance, because the solutions it generates are always inferior to those generated by the domain-specific subsystem. … Then, once the capacity of the general-purpose subsystem crosses the threshold where its solutions start to beat those of the domain-specific subsystem, the overall system’s performance suddenly begins to improve at the same brisk pace as the general-purpose subsystem, even as the amount of optimization power applied stays constant: the system’s recalcitrance has plummeted.
Here, Bostrom suggests that the reason why we have not yet seen an intelligence explosion may be that AI systems (in 2014) were not, in the first instance, general-purpose reasoners. We might therefore expect an intelligence explosion to take place after AI systems are dominated by their general-purpose capacities, a threshold that we may perhaps have crossed already.
Well, that is certainly the hope behind the singularity hypothesis. The hypothesis says that a self-improving artificial agent will grow at an accelerating rate until a fundamental discontinuity is reached, and it is hard to see how that system could fail to have a significant amount of general-purpose reasoning ability. But that means Bostrom’s second proposal is little more than a restatement of one of the most plausible versions of the singularity hypothesis. We aren’t getting a significantly new argument for the singularity hypothesis here.
5. Mis-interpreting history
One of the most frustrating facts about AI risk arguments is that it is rare for arguments to be made in a way that subjects them to feasible empirical test. This means that it is important to seize on empirically testable claims when they are made as an indicator of the empirical status that other claims are likely to have.
Bostrom’s last argument makes some empirical claims about Moore’s law to support his belief in fast or moderate takeoff. Unfortunately, those empirical claims are demonstrably false, and when they are corrected the argument tells against the singularity hypothesis. That is not good, and it would be a bit eyebrow-raising if Bostrom were to withdraw the argument on this basis. Why not follow the argument where it leads?
Nick Bostrom makes two claims about historical hardware growth rates: (1) that hardware capacities have grown exponentially, and (2) that this growth was produced through constant optimization power, i.e. constant quality-weighted hardware research effort. Bostrom takes (1) and (2) to indicate that recalcitrance (of hardware) grew inversely with hardware capacities.
If (3) we take hardware capacities to track system intelligence, then this implies that (4) system intelligence has grown exponentially and (5) this growth was produced through constant optimization power. (4) and (5) would indicate that recalcitrance (of system intelligence) grew inversely with system intelligence. If we project (4) and (5) into the future, the result is (6) hyperbolic growth sustained for as long as we project (4) and (5) to hold together.
These claims emerge in the following passage. Claim (3) is left tacit, unless we are to accuse Bostrom of equivocating between claims about hardware and claims about intelligence. The passage is unaltered except for annotations where I take each claim to occur in Bostrom’s text:
Suppose that (5) the optimization power applied to the system is roughly constant … prior to the system becoming capable of contributing substantially to its own design, and that … (4) this leads to the system doubling in capacity every 18 months. ((1,2) This would be roughly in line with historical improvement rates from Moore’s law combined with software advances.) (6) This rate of improvement, if achieved by means of roughly constant optimization power, entails recalcitrance declining as the inverse of system power (p. 76).
Bostrom gives the following illustrative model of the growth trajectory entailed by (6):
What are we to make of this argument?
Claim (1), that hardware capacities have grown exponentially, is at least misleading. We saw in Part 2 that many experts think Moore’s law has ended, and the rest think it is likely to end by 2030. And certainly Bostrom’s claim of an 18-month doubling can’t be right: Moore’s originally proposed 1-year doubling has long since been revised down to a 2-year doubling. But that is a minor point.
Claim (2), that hardware growth was produced through constant optimization power, is a whopper. Are we really meant to believe that the amount of quality-weighted effort used worldwide to design chips hasn’t increased since the 1960s, even as the number of researchers has grown sharply along with their training and equipment? Indeed, we saw in Part 2 that Claim (2) is badly false: a leading study suggests that recalcitrance has increased by a factor of 18 through the history of Moore’s law.
Denying Claim (2) would be bad enough for Bostrom: replacing decreasing hardware recalcitrance with increasing hardware recalcitrance and leaving exponential hardware growth intact would predict sub-exponential hardware growth in the future. Denying Claim (1) as well would predict slower hardware growth yet.
Claim (3), that hardware capacities track system intelligence, was seen to be doubtful in Part 3 of this series. There, I suggested that exponential growth in hardware capacities has tended in the past to produce only linear growth in system intelligence. If that is right, then the failure of Claim (3) would be enough to scuttle Bostrom’s argument for hyperbolic intelligence growth, even granting Claims (1) and (2).
Claims (4) and (5) can be evaluated in light of their counterparts, Claims (1) and (2), together with the linking Claim (3). As the counterpart and linking claims fail, so too will Claims (4) and (5) begin to fail, or at least require alternative support.
If Claims (4) and (5) fail, then Claim (6) will be false for the same reasons that the failure of Claims (1) and (2) put pressure on hyperbolic hardware growth projections. As before, if Claim (5) fails, then even Claim (4) grounds only subexponential hardware growth. And if Claim (4) fails, the predicted growth is slower still.
There is only one natural way for Bostrom to wriggle out of the argument. That is for Bostrom to recognize that while it may be permissible to project past trends into the near future, it is a Malthusian mistake to project past growth trends into the distant future. Aggressive growth rates tend to slow for exactly the reasons discussed in Parts 2 and 3 of this series, and in general we don’t have any reason to suspect that trends will continue indefinitely. I am sympathetic to this view. However, it would be odd for Bostrom to recant at this point, having already committed himself to exactly this sort of projection.
If Bostrom remains committed to this method of forecasting future intelligence growth, then the most plausible scenario is one in which both hardware and the intelligence of artificial systems grows at a subexponential rate. That’s hardly a singularity.
6. Conclusion
In this post, we looked at six arguments by Nick Bostrom for fast or moderate takeoff to superintelligence, in a period of minutes, hours, days, months or years. We saw that those arguments break into three categories: somewhat-plausible scenarios that have been over-interpreted to support the singularity hypothesis; restatements of the core hope behind the singularity hypothesis; and mis-interpretations of history that, once corrected, do not even ground exponential growth.
In Parts 2 and 3 of this series, we met five reasons for skepticism about the singularity hypothesis. We saw that these reasons for skepticism place a substantial burden on defenders of the singularity hypothesis to produce many excellent arguments in favor of their view.
In Part 4 and today’s post, we saw that leading defenses of the singularity hypothesis by Dave Chalmers and Nick Bostrom fall considerably short of meeting this burden. Unless stronger arguments can be produced, this suggests that it would be inappropriate to place substantial credence in the singularity hypothesis at this time.
The remainder of this series will explore implications of skepticism about the singularity hypothesis, and perhaps answer objections.
Leave a Reply