Exaggerating the risks (Part 17: Biorisk, taking stock)

Widespread concerns about new technologies—whether they be novels, radios, or smartphones—are repeatedly found throughout history. Although tales of past panics are often met with amusement today, current concerns routinely engender large research investments and policy debate. What we learn from studying past technological panics, however, is that these investments are often inefficient and ineffective.

Amy Orben, “The Sisyphean cycle of technology panics“

1. Introduction

This is Part 17 of my series Exaggerating the risks. In this series, I look at some places where leading estimates of existential risk look to have been exaggerated.

Part 1 introduced the series. Parts 2-5 (sub-series: “Climate risk”) looked at climate risk. Parts 6-8 (sub-series: “AI risk”) looked at the Carlsmith report on power-seeking AI.

Parts 9, 10 and 11 began a new sub-series on biorisk. In Part 9, we saw that many leading effective altruists give estimates between 1.0-3.3% for the risk of existential catastrophe from biological causes by 2100. I think these estimates are a bit too high.

Because I have had a hard time getting effective altruists to tell me directly what the threat is supposed to be, my approach was to first survey the reasons why many biosecurity experts, public health experts, and policymakers are skeptical of high levels of near-term existential biorisk. Parts 9, 10 and 11 gave a dozen preliminary reasons for doubt, surveyed at the end of Part 11.

The second half of my approach was to show that initial arguments by effective altruists do not overcome the case for skepticism. Part 12 examined a series of risk estimates by Piers Millett and Andrew Snyder-Beattie. Part 13 looked at Ord’s arguments in The precipice. Part 14 looked at MacAskill’s arguments in What we owe the future. Part 15 and Part 16 looked at evidence for the more targeted claim that large language models exacerbate existential biorisk.

Today’s post concludes by drawing out some important lessons from this discussion.

2. Reasons for doubt

The first thing to emphasize is that there are good prima facie reasons to doubt high estimates of existential biorisk. Parts 9, 10 and 11 developed a dozen such reasons:

1. Engineering existential biothreats is a very difficult problem: the pathogen would need to reach everyone, be virtually undetectable, unprecedently lethal and infectious, and circumvent basic public health measures. Then you would need a group with the sophistication and motivation to pull off an attack.
2. Human history suggests surprising resilience: Human history is replete with examples of resilience not only to highly destructive pandemics, but also to the combination of pandemics with numerous other catastrophes.
3. Mammalian history: Throughout history, there is only one recorded instance in which disease has led to the extinction of a mammalian species. The victim was a species of rat living on a single island.
4. Everything must come together at once: Producing a virus with surprising features of many types (lethality, infectiousness, detectability, …) is much harder than producing a virus which is surprising along a few dimensions.
5. Biological threats can be met with effective public health responses. These may be as simple as the now-familiar package of non-pharmaceutical interventions such as masking, social distancing and travel restrictions. But they can also be as complex as the increasingly rapid and sophisticated techniques for bringing vaccines to market in time to counteract a novel pandemic.
6. Motivational obstacles:  it is hard to find credible examples of sophisticated actors with both the capability and the motivation to carry out omnicidal attacks. Many groups wish to cause harm to some fraction of humanity, but far fewer groups wish to kill all of us.
7. Expert skepticism: experts are generally skeptical of existential biorisk claims. Experts do not doubt the possibility or seriousness of small-scale biological attacks, and indeed such attacks have occurred in the past. But experts are largely unconvinced that there is a serious risk of large-scale biological attacks, particularly on a scale that could lead to existential catastrophe.
8. Inscrutability: the biorisks being appealed to are relatively inscrutable in two ways: they involve speculation about far-off future technologies, and much of this speculation is hidden from public view. 
9. Superforecaster skepticism: Superforecasters are uniquely skeptical of existential biorisk claims, and become much more skeptical after taking a careful look at the arguments.
10. Knowledge and expertise: Even sophisticated groups often lack the knowledge and expertise to make effective use of bioweapons. Recent groups, such as Aum Shinrikyo, bungled their bioweapons programs rather badly.
11. Lack of mechanisms: Making an effective case for existential biorisk requires exhibiting the mechanisms by which biological threats might produce existential catastrophe. This has, in many cases, not been done.
12. They might not believe it either: Leading EA organizations often focus their public defenses on catastrophic rather than existential biorisk, and several prominent problem profiles openly contradict published high estimates of existential biorisk.

While these arguments do not decisively prove that levels of near-term existential biorisk are low, they do present a high bar for defenders of high biorisk estimates to clear. In this series, I have argued that the bar has not been cleared.

3. Regression to the inscrutable

I have complained elsewhere about what I term a regression to the inscrutable in discussions of existential risk. Both within and across risk areas, the most evidentially tractable risk claims tend to fare poorly. As this happens, discussion increasingly shifts to areas that are less tractable through current evidence (chiefly, AI risk and biorisk) and to the least tractable discussions within these areas.

One good way to make the case for existential biorisk more scrutable would be for those concerned about existential biorisk to say directly what they are concerned about. We saw in Part 9 that, for the most part, they have refused to do this, saying that any detailed case for concern about existential biorisk would be an infohazard.

This does not do much to improve the scrutability of existential biorisk claims, nor to convince those justifiably moved by considerations similar to the reasons for doubt surveyed in Parts 9–11 of this series. It also strains belief: are we meant to believe that a small group of effective altruists have discovered grave concerns that the entire community of global experts have missed, and that these concerns are so dangerous they could not be shared even with leading academic experts, or perhaps the CIA?

I suppose that one’s reaction to such scenarios might depend on how much trust they antecedently place in effective altruists. But taking unspoken biorisk concerns seriously requires a great deal of trust, and if effective altruists’ track record of exaggerating other risks surveyed in this series is any indication, that trust may be misplaced.

4. Scrutable failures

One of the best ways to assess the credibility of an inscrutable series of claims is to seize upon the few elements of the view that are relatively scrutable using current evidence and see how they do. If evidence tends to support the more scrutable claims, then that is some reason to be favorably disposed towards other, less scrutable claims. On the other hand, if evidence tends not to support the more scrutable claims, then that is some reason to worry that the less scrutable claims are not well grounded.

Parts 15–16 of this series looked at recent claims that existential biorisk is exacerbated by current large language models. Part 15 looked at a report by the Center for the Governance of AI (GovAI) on the risk of open-source LLMs. This report has been widely cited as one of the best available arguments that current LLMs exacerbate biorisk, at least if they are open sourced.

We saw, however, that the report did not provide much evidence for existential biorisk from LLMs. Because the report did not deliver substantially new arguments for existential biorisk from LLMs, the approach in Part 15 was to check whether the evidence cited in the GovAI report might support its claims. Following the outlines of a LessWrong investigation, which was less complementary even than my own assessment, we saw that these citations break into three categories: background materials, which are not primarily argumentative; OpenAI and Anthropic materials which make strong claims but suffer from evidential deficits similar to the GovAI report; and two scientific papers which, though they raise a few good points, hardly provide enough evidence to support the GovAI report’s main claims.

Even if existing arguments for existential biorisk from LLMs are not compelling, their conclusions could still be correct. Part 16 then looked at a recent red-teaming study from the RAND corporation aiming to assess whether LLMs exacerbate biorisk. The study found “that using the existing generation of LLMs did not measurably change the operational risk” of biological weapons attacks, even those falling considerably short of existential catastrophe. And the authors of this study can hardly be accused of bias against effective altruists: the study was written this year, at a time when RAND is overseen by a CEO who identifies as an effective altruist and has received millions of dollars of funding from effective altruists to pursue similar research projects.

In sum, this case study of existential biorisk from current LLMs suggests both that arguments offered in favor of existential biorisk from current LLMs are unsuccessful, and that there is some evidence against the idea that current LLMs exacerbate biorisk. When this type of scrutable and evidentially-tractable case study tells against existential risk claims, there is some temptation to worry that the less-scrutable claims may also rest on shaky ground.

5. Limited positive evidence

Those of us who are not disposed to take risk estimates on trust will want to see some evidence for these estimates, particularly given the failure of some scrutable risk claims. While effective altruists have refused to say directly what they are concerned about, they have given some direct arguments for existential biorisk estimates. However, we saw in Parts 12–14 that those arguments do not go very far.

Part 12 examined a series of risk estimates by Piers Millett and Andrew Snyder-Beattie. We saw that those estimates are orders of magnitude below the estimates given by leading effective altruists. We also saw that the estimates rely on relatively unreliable forms of extrapolation, namely an informal poll of the audience at a conference hosted by the Future of Humanity Institute; an unevidenced and largely unrelated extension of an otherwise-credible government report on risks of gain-of-function research; and a power-law extrapolation of fatalities from warfare and bioterrorism, pushing power law models far beyond the inferences that can be supported by the motivating data.

Part 13 looked at Toby Ord’s arguments for existential biorisk in The precipice. We found that Ord draws largely on a range of familiar facts about biological risk which are common ground between Ord and the skeptical expert consensus. We saw that Ord gives few detailed arguments in favor of his risk estimates, and that those arguments given fall a good deal short of Ord’s argumentative burden. We also saw that Ord estimates a 1/10,000 chance of irreversible existential catastrophe by 2100 from natural pandemics. Again, we saw that very little support is provided for this estimate.

Part 14 looked at Will MacAskill’s arguments for existential biorisk in What we owe the future. We saw that MacAskill’s claim that experts he knows typically assign a probability of around 1% to existential biorisk in this century mischaracterizes expert consensus. We saw that MacAskill’s argument largely repeats basic facts familiar to experts and cited by Ord and other effective altruists. And we saw that although MacAskill makes a few new arguments, these arguments are neither detailed nor convincing.

If effective altruists really are hiding a smoking gun up their sleeves, then that is one matter. But if effective altruists, when pressed to give positive arguments, give arguments that fall far short of their argumentative burdens, then that is some reason to suspect that effective altruists’ judgment on the matter may not be wholly reliable. It is at least tempting for skeptical readers to judge unspoken arguments by the success of spoken arguments, and the arguments that have so far been spoken are not especially promising.

6. Expert disagreement

We saw in Parts 9-11 that experts are resoundingly skeptical of high estimates of existential biorisk. While MacAskill might claim that experts he knows assign a probability of around 1% to existential biorisk in this century, this is probably best interpreted as a sign of epistemic insularity rather than an apt characterization of the opinion of the scientific community.

This is not the first time that effective altruists have shown comfort in challenging scientific consensus. Any number of claims, from widespread claims such as the singularity hypothesis to more particular claims, such as Yudkowsky’s claim to have discovered by reading economics blogs that the Bank of Japan was leaving trillions of dollars on the table, try the patience of largely skeptical experts.

Increasingly, the reaction of effective altruists is to push back against the legitimacy of expert consensus or even, at times, against the importance of engaging with experts. For example, in response to a summary of my paper “Against the singularity hypothesis“ on the EA Forum, several users asked me to read and engage with a report on the economic singularity hypothesis by Epoch AI. I suggested that leading economists are resoundingly skeptical of the economic singularity hypothesis, and that if staff members at Epoch AI were serious about gathering evidence for the economic singularity hypothesis, they might do well to hire staff members with PhDs in economics and to publish their results in an economics journal where they could engage with and be evaluated by the community of experts.

This was not my most popular comment (I was, admittedly, a bit rude), and one commentator even suggested that they had given up hope in talking to economists about the issue, since economists would never believe them anyways. No one showed up to argue in favor of deference to experts, or even in favor of engagement with experts.

This type of willingness to stand outside of expert consensus, without possessing equivalent expertise or providing a credible and detailed replacement, does effective altruists few favors. It makes existential risk arguments appear to be what many will have already assumed them to be: fringe arguments pushed by groups without the evidence to make them, or the credentials to assert them.

7. But what about AI risk?

Many readers have expressed impatience with the progress of this series. Yes, they tell me, you may be right that climate change is not going to lead to existential catastrophe. And yes, it may be that biorisks are greatly exaggerated. Some other risks, such as purported existential threats posed by self-replicating nanobots (which Yudkowsky previously aimed to stop by accelerating the singularity) may also have proven to be exaggerated. But what about existential risk from artificial intelligence? Surely, they say, you cannot deny this risk. And since with each passing year, effective altruists place larger fractions of their risk estimates into AI risk, why does this series not just focus on AI risk?

Throughout this series, we have observed the following trends: scrutable risk claims by effective altruists look to be exaggerated. There is an increasing trend both within and across cause areas to shift discussion towards less-scrutable claims. When we do discover some subpart of these claims that is relatively scrutable by evidence, the evidence does not look good for high risk estimates.

Now we come to hypothesized risks from artificial agents, which are by far the least detailed, scrutable, and scientifically tractable of the bunch. When we look at leading arguments, such as the Carlsmith report, they often fail to argue in detail for key premises. What, then, are we to make of the case for AI risk?

I must say, the trajectory is not looking good. There is considerable temptation for skeptical readers to suspect that effective altruists often exaggerate existential risks, and accept high risk estimates on the basis of arguments that are not very conclusive. There is, in the absence of strong evidence bearing on the nature of and risks posed by future AI systems, considerable temptation to think that discussions of AI risk will prove to be more of the same.

I can and sometimes do engage, against my better judgment, in speculation about the nature and possibility of radically transformative AI systems. Perhaps with enough urging I can be talked into repeating the act in this series. But as we gaze through a dense fog into the landscape of AI futures, recognizing our limited ability to penetrate the fog, we should approach the matter with a healthy suspicion grounded in the failures of existential risk claims in other, more scrutable areas.

There are, of course, diamonds in the rough. AI risk could, against all odds, be the one type of existential risk to which it is appropriate to give high levels of credence. But most rough is not populated with diamonds, and so far we have come home empty-handed.