Harms (Part 2: Surveillance)

There are two general methods that we could use to achieve the ability to stabilize the world against the whole spectrum of possible vulnerabilities, and we probably would need both. One is an extremely effective ability to do preventive policing, such that … if anybody started to do [a] dangerous thing, you could intercept them in real time and stop them. So this would require ubiquitous surveillance. Everybody would be monitored all the time. You would have maybe AI algorithms, big freedom centers that were reviewing this, etc. etc. … Imagine [a] kind of necklace that you would have to wear at all times with multidirectional cameras. But, to make it go down better, just call it the “freedom tag” or something like that.

Nick Bostrom, “How civilization could destroy itself — and 4 ways we could prevent it“

1. Introduction

This is Part 2 in my series Harms. Risk mitigation has potential harms, as well as benefits. This series aims to chronicle some of the harms that existential risk mitigation may bring about, so that the value of risk mitigation efforts can be properly assessed.

Part 1 looked at the risk of distraction. In particular, I argued that a focus on existential risks posed by artificial intelligence may distract from addressing the many real harms that artificial intelligence is causing today.

Today’s post looks at surveillance. Leading longtermists have been open about the possibility that extreme forms of surveillance may be needed to bring existential risk within reasonable levels. Many readers may, understandably, find such surveillance disquieting, and that disquiet should be tallied against the benefits of surveillance in reducing existential risk.

So that I cannot be accused of inventing the problem, I will try whenever possible to quote the words of leading longtermists. In particular, I will structure this article around a reading of Nick Bostrom’s paper, “The vulnerable world hypothesis“.

2. The vulnerable world hypothesis

Nick Bostrom is Professor of Philosophy at the University of Oxford and a leading figure in the effective altruism movement. Bostrom’s book, Superintelligence, is widely regarded as a foundational contribution to the study of existential risk from artificial agents, and many of Bostrom’s papers have been deeply influential as well.

Though I have had my differences with Bostrom, one thing I (usually) admire about Bostrom is his commitment to saying exactly what he means even when the truth may not be entirely rosy. This makes Bostrom’s work especially suitable for a case study of surveillance, since many other authors shy away from mentioning the full extent of surveillance that may be needed to hit existential risk mitigation targets.

Bostrom’s paper, “The vulnerable world hypothesis“, considers the possibility that humanity lives in a vulnerable world where technologically-driven disaster is likely unless changes are made. Bostrom argues that both extreme surveillance and strong global governance are likely to be necessary to exit the vulnerable world.

In more detail, Bostrom considers:

(Vulnerable World Hypothesis) If technological development continues then a set of capabilities will at some point be attained that make the devastation of civilization extremely likely, unless civilization sufficiently exits the semi-anarchic default condition.

What is the semi-anarchic default condition? One in which:

(1) Humanity has limited capacity for preventive policing: “States do not have sufficiently reliable means of real-time surveillance and interception to make it virtually impossible for any individual or small group within their territory to carry out illegal actions”.

(2) Humanity has limited capacity for global governance: “There is no reliable mechanism for solving global coordination problems and protecting global commons”.

(3) Humans have diverse motivations, and in particular “there are some actors … who would act in ways that destroy civilization even at high cost to themselves”.

Bostrom argues that humans are unlikely to solve the problem of diverse motivations. Sufficiently exiting the semi-anarchic default condition, Bostrom argues, then plausibly requires both strong preventive policing and strong global governance.

The focus of my concern will be Bostrom’s remarks about preventive policing, enabled by ubiquitous surveillance. Let’s look at what Bostrom proposes (Sections 3,4). Then let’s look at why other longtermists may have to make similar proposals (Section 5), and sum up the cost of these proposals (Section 6).

3. Surveillance

Let’s look at Bostrom’s discussion of what he calls `Type-1′ vulnerabilities, in which it is relatively easy to cause mass destruction. For example, perhaps it becomes cheap and easy to produce nuclear weapons. This discussion occupies most of Bostrom’s remarks about surveillance, and much of what Bostrom says here generalizes to many other scenarios, with some exceptions.

In this scenario, Bostrom argues:

What would be required … is an extremely well-developed preventive policing capacity. States would need the ability to monitor their citizens closely enough to allow them to intercept anybody who begins preparing an act of mass destruction.

Just what kind of preventive policing does Bostrom have in mind? Bostrom offers the following example, which he repeats elsewhere:

(High-tech Panopticon) Everybody is fitted with a `freedom tag’ … worn around the neck and bedecked with multidirectional cameras and microphones. Encrypted video and audio is continuously uploaded from the device to the cloud and machine-interpreted in real time. AI algorithms classify the activities of the wearer, his hand movements, nearby objects, and other situational cues. If suspicious activity is detected, the feed is relayed to one of several patriot monitoring stations. There are vast office complexes, staffed 24/7. There, a freedom officer reviews the video feed on several screens and listens to the audio in headphones. The freedom officer then determines an appropriate action, such as contacting the tag-wearer via an audiolink to ask for explanations or to request a better view. The freedom officer can also dispatch an inspector, a police rapid response unit, or a drone to investigate further. In the small fraction of cases where the wearer refuses to desist from the proscribed activity after repeated warnings, an arrest may be made or other suitable penalties imposed. Citizens are not permitted to remove the freedom tag, except when they are in environments that have been outfitted with adequate external sensors (which however includes most indoor environments and motor vehicles). The system offers fairly sophisticated privacy protections, such as automated blurring of intimate body parts, and it provides the option to redact identity-revealing data such as faces and name tags and release it only when the information is needed for an investigation. Both AI-enabled mechanists and human oversight closely monitor all the actions of the freedom officers to prevent abuse.

This is not a caricature of Bostrom’s proposal. It is the actual proposal, or at least Bostrom’s primary illustration of what he might be proposing. Fallback proposals are no more promising: for example, Bostrom suggests that in the face of serious threats, until we are able to implement a digital panopticon we may need:

A policy of preemptive incarceration, say whenever some set of unreliable indicators suggest a greater than 1 per cent probability that some individual will attempt a city-destroying act or worse.

These are strong policies, and Bostrom rightly questions whether citizens would be willing to accept them. Bostrom does not directly tally the costs of these policies, a question to which we will return in Section 6. But I think Bostrom would be the first to agree that they are significant.

4. Timing

It may be comforting to think of the draconian policies suggested by Bostrom as nestled into some hazy point of the indefinite future. But, as Bostrom notes, that may not be good enough. Bostrom proposes that we may well have to develop or implement such a system now, or in the near future.

Bostrom makes this proposal for two reasons: it takes time to develop and build political support for a system of mass surveillance. We may not have time to develop and build support for the system once a specific threat presents itself. Bostrom writes:

One could take the position that we should not develop improved methods of surveillance and social control unless and until a specific civilizational vulnerability comes clearly into view … [But] a highly sophisticated surveillance and response system, like the one depicted in `High-tech Panopticon’, cannot be conjured up and made fully reliable overnight. Realistically, from our current starting point, it would take many years to implement such a system, not to mention the time required to build political support. Yet the vulnerabilities against which such a system might be needed may not offer us much advanced warning … Intense social control may need to be switched on almost immediately. In an unfavorable scenario, the lead time could be as short as hours or days. It would then be too late to start developing a surveillance architecture when the vulnerability comes clearly into view. If devastation is to be avoided, the mechanism for stabilization would need to have been put in place beforehand.

The suggestion that surveillance may soon be needed is not offered as a speculative possibility. It is, chillingly, offered as a piece of advice. The concluding words of Bostrom’s piece are, for all their hedges, a call to action:

If one finds oneself in a position to influence the microparameters of preventive policing capacity or global governance capacity, one should consider that fundamental changes in those domains may be the only way to achieve a general ability to stabilize our civilization against emerging technological abilities.

And given what we have just read, there is every reason to believe that Bostrom would endorse a much stronger call to action than this.

5. The need for surveillance

So far, we have seen that Bostrom thinks a system of extreme surveillance and preventive policing will eventually be needed to deal with existential risks. We have also seen that Bostrom thinks we may well need to develop, or even implement such a system now in order to be prepared when risks strike.

That is a strong enough view that we should ask whether Bostrom needed to hold it. Can we identify general reasons why longtermists may need to advocate such a view?

Certainly, I am not the first to suggest that longtermists may have paradigm-internal reasons to push for surveillance. The point is ably pressed by Zoe Cremer and Luke Kemp (See also Hobson and Corry 2023). Cremer and Kemp write:

Any approach to understanding and mitigating existential risks runs the risk of becoming securitised. Securitisation refers to a discursive manoeuvre that moves an issue from the arena of normal politics to that of national security, making it more likely to permit emergency powers and be placed under the control of unelected military and intelligence officials.

Cremer and Kemp suggest that the underlying problem lies in a tendency of standard approaches to longtermism (which they call the techno-utopian approach) to justify enormous sacrifice in pursuit of existential risk mitigation. Cremer and Kemp write:

There are reasons to expect that the [techno-utopian approach] is particularly vulnerable to misuse … If the world is viewed from the [techno-utopian approach]’s lens of existential risk, then we run the risk that almost any action is justified if it is believed to improve our chance of surviving to expand beyond Earth. Problems which are not considered to be an existential risk dwindle into irrelevance, as other values are sacrificed on the altar of expected astronomical value.

In broad outline, the problem is as Cremer and Kemp describe: given the enormous value that many longtermists assign to existential risk mitigation, even draconian forms of surveillance are likely to seem justifiable so long as they yield a net reduction in existential risk. However, that is a bit too broad: for example, I have argued that if existential risk is very high, and likely to remain high for a while, then we should care a great deal about improving the quality of life today.

Let us then ask, in more detail, why many longtermists should be expected to find Bostromian forms of surveillance theoretically appealing. At least three reasons suggest themselves.

The first is the magnitude of needed risk reduction. I argued in my paper and blog series “Existential risk pessimism and the time of perils” that for existential risk mitigation efforts to produce astronomical value, they need to drive risk down by perhaps as many as 4-5 orders of magnitude, and leave it there, without fail, for a very long time.

Moderate risk reductions can be achieved in many ways, but dramatic risk reductions require us to leave very little to chance. A society which desires, for example, to reduce the risk of bioterrorism not by two or five times, but by ten or a hundred thousand times, will need to do everything in its power to cripple the capacity of individuals or groups to mount bioterrorist attacks. And it is hard to see what, short of the most draconian forms of universal surveillance and predictive policing, could achieve this.

Second, existential risk reduction may be more difficult than reducing the chance that an individual attack succeeds. In a world where biological attacks are easy to mount, a policy that stopped the vast majority of attacks might still provide little protection against the chance that one attack or another would succeed. As Bostrom writes:

Note that an intervention that halves the size of the apocalyptic residual [actors who would act in ways that destroy civilization] would not (at least not through any first-order effect) reduce the expected risk … by anywhere near as much. A reduction of 5 percent or 10 per cent of … risk from halving the apocalyptic residue would be more plausible. The reason is that there is wide uncertainty about how destructive some new black-ball technology would be, and we should arguably use a fairly uniform prior in log space (over several orders of magnitude) over the size of apocalyptic residual that would be required in order for civilizational devastation to occur conditional on a … vulnerability arising.

If this is right, then a reduction of, say, 4-5 orders of magnitude in existential risk may require an even greater reduction in quantities such as the number of individuals motivated and positioned to carry out a successful attack. With so little room for error, universal surveillance begins to look increasingly necessary to achieve risk targets.

Finally, many risks may be posed by future technologies. For any given risk, such as bioterrorism enabled by synthetic biology, there may be effective solutions that do not require universal surveillance. For example, we might halt progress in synthetic biology (though Bostrom thinks this is implausible) or severely restrict access to genetic synthesis technologies. However, leading longtermists hold that any of a number of technologies pose significant existential risks within this century, and given the pace of technological change, many think that the number of dangerous technologies is likely to greatly increase. If that is right, then a strategy of managing risks individually looks less attractive, and holistic strategies such as universal surveillance become more attractive, since they target most or all risks at once.

Bostrom is a smart guy. Bostrom is not making the unpopular suggestion that society should soon consider universal surveillance because he enjoys provoking outrage. Bostrom makes this suggestion because he is clear about the size of the reduction in existential risk that his view requires, and thinks that universal surveillance is plausibly necessary to bring about that reduction. In this section, I have argued that Bostrom may well be right.

6. The cost of universal surveillance

Perhaps Bostrom is right that universal surveillance should soon be developed, or even implemented. Perhaps Bostrom is wrong. But everyone, regardless of their views, should acknowledge two serious costs of this proposal.

First, surveillance imposes direct and immediate harms on those surveilled. The world has received a small taste of Bostrom’s proposal in the system of post-9/11 surveillance developed in the United States, including the arbitrary arrest, killing and indefinite detention of terror suspects without trial. We have seen how the harms of surveillance fall hardest upon religious and ethnic minorities, exacerbating already serious tensions within the country.

Bostrom is quite sanguine about the types of harms that may be imposed. For example, in what Bostrom calls the “optimistic scenario” in response to a hypothesized existential threat, Bostrom supposes that:

Possession of proscribed materials, or equipment that could be used to make them, would be harshly punished, such as by on-the-spot execution. To enforce these provisions, communities would be subjected to strict surveillance – informant networks incentivized by big rewards, frequent police raids into private quarters, continuous digital monitoring, and so forth.

Try to imagine, really imagine living in a society in which every citizen is fitted with a freedom tag. Or perhaps imagine a society in which mere suspicion on the basis of unreliable evidence were a basis for arrest, perhaps even summary execution. What would it be like to live in a society of this nature? Would you like to live in such a society?

A second cost of surveillance is the risk of totalitarianism. Surveillance and policing are longstanding tools of repressive governments. Extensive surveillance and the associated security mindset may increase the chance that would-be totalitarians come to power, though even without such a system, my own country faces a real threat of electing an authoritarian president by the end of the year.

Extensive surveillance certainly increases the chance that totalitarians stay in power. To date, most totalitarian governments have had a limited lifespan. That lifespan may have been measured in years, decades, or even centuries. But eventually they have fallen.

Longtermists are deeply concerned with the risk of stable totalitarianism: a system of totalitarian government which uses technology to maintain an indefinite grip on power. Longtermists generally classify stable totalitarianism as an existential risk, because it permanently curtails the ability of human civilization to grow, thrive and flourish. The problem is that universal surveillance may be just what totalitarian governments need to become stable. For example, one of the first examples of an existential catastrophe that Toby Ord discusses in The precipice is this one:

Consider a world in chains: in a future reminiscent of George Orwell’s Nineteen Eighty-Four, the entire world has become locked under the rule of an oppressive totalitarian regime, determined to perpetuate itself. Through powerful, technologically enabled indoctrination, surveillance and enforcement, it has become impossible for even a handful of dissidents to find each other, let alone stage an uprising. With everyone on Earth living under such a rule, the regime is stable from threats, internal and external. If such a regime could be maintained indefinitely, then descent into this totalitarian future would also have much in common with extinction: just a narrow range of terrible futures remaining, and no way out.

Bostrom is sanguine about the risk that universal surveillance may lead to totalitarianism, and even to stable totalitarianism. He writes:

Admittedly, constructing such a [surveillance] system and keeping it in standby mode would mean that some of the downsides of actually instituting intense forms of social control would be incurred. In particular, it may make oppressive outcomes more likely … Developing a system for turnkey totalitarianism means incurring a risk, even if one does not intend for the key to be turned.

Bostrom’s honesty here is to his credit. It is all too easy to mutter phrases such as `existential risk mitigation’ and `lowering biorisk’ without thinking through exactly what these interventions would entail. Bostrom has thought as seriously as anyone on this Earth about what they would entail, and the results are a bit disquieting.

Should we accept the consequences anyways? Perhaps. But the consequences should be noted, and noted well.