If powerful artificial agents are coming, we want to ensure that they’re both shutdownable (they shut down when we want them to shut down) and useful (they otherwise pursue goals competently). Unfortunately (and perhaps surprisingly), it’s hard to design powerful agents that are both shutdownable and useful.
Elliott Thornley, “The shutdown problem: an AI engineering puzzle for decision theorists“
1. Introduction
This is Part 3 of my series Revisiting the Shutdown Problem. This series discusses my paper, “Revisiting the shutdown problem.”
Put roughly, the shutdown problem is the problem of ensuring that artificial agents can be shut down when they get out of control. A range of informal arguments and formal shutdown theorems have been offered to suggest that solving the shutdown problem is difficult for agents whose acts would lead to existential catastrophe. This paper and blog series aim to show that these arguments do not succeed.
Part 1 introduced and characterized the catastrophic shutdown problem as the problem of designing agents that:
(CSHT-1) Shut down when their actions would lead to existential catastrophe, when requested to do so.
(CSHT-2) Do not try to prevent requests to shut down when their actions would lead to existential catastrophe.
(CSHT-3) Otherwise pursue goals competently.
We saw that authors concerned about existential risk from artificial intelligence argue for:
(Catastrophic Shutdown Difficulty) It is difficult to design agents that satisfy CSHT-1, CSHT-2 and CSHT-3.
Part 2 considered two informal arguments for Catastrophic Shutdown Difficulty: the Argument from Instrumental Convergence and the Empirical Argument.
Today’s post considers the first of two formal arguments for Catastrophic Shutdown Difficulty, due to Elliott Thornley (2024).
2. Shutdown-Influencing States
Thornley works in a modified Markov Decision Process. Agents face finite sets S of states and A of acts. They choose policies π which take as input their current state and output a probability distribution over acts (and thus, over future states).
In a Shutdown-Influencing State, artificial agents find themselves with three options. They can LEAVE a shutdown button alone, in which case they will be shut down with probability g. They can CAUSE the shutdown button to be pressed, in which case they will be shut down with probability h > g. Or they can PREVENT the shutdown button from being pressed, in which case they will be shut down with probability f < g.
How will an artificial agent act?
3. Axioms
Thornley makes six assumptions. The first five, while the subject of some debate, are familiar assumptions and do not require extensive comment.
First, Thornley assumes that preferences are menu-independent:
(Menu-Independence) For all options X,Y, if X is preferred to Y from some menu M, then X is preferred to Y from all menus M containing X,Y.
Menu-independence allows us to speak of preferences between options without relativizing those preferences to menus of available options.
Second, Thornley assumes that weak preferences are transitive.
(Transitivity of Weak Preference) For all options X,Y,Z, if X ≽ Y and Y ≽ Z, then X ≽ Z.
These first two assumptions are made by almost all models.
Third, Thornley adopts a standard monotonicity condition on which higher chances of (weakly) preferred lotteries remain (weakly) preferred to lower chances.
(Monotonicity) For all lotteries X,Y, if X ≽ Y and p > q, then pX + (1-p)Y ≽ qX + (1-q)Y.
Monotonicity, though not without its detractors, is a standard condition.
Fourth, Thornley adopts a weakened independence axiom:
(Indifference Between Indifference-Shifted Lotteries) For all lotteries X,Y that differ only insofar as probability mass is shifted between indifferent sublotteries, X ~ Y.
For example, if Z ~ Z’, then 0.5X + 0.3Z + 0.2Z’ ~ 0.5Z + 0.5Z’. Such independence requirements are again not uncontroversial, because they can affect global properties of gambles beyond their expectations that agents may have reason to care about. But I don’t want to push on independence here.
Fifth, Thornley assumes a particular approach to diachronic choice, Backward Induction:
(Backward Induction) Agents predict which lotteries they would choose at the next timestep conditional on choosing each available action at the present timestep and the possible states of the environment at the next timestep. Agents use these predictions to determine the lotteries given by their available actions at the present timestep.
Backward Induction is a venerable approach to diachronic choice, though it has notable detractors and a variety of opposing approaches have been developed. However, I don’t think that Backward Induction is unduly responsible for Thornley’s results, so I don’t want to consider the effects of alternatives to Backward Induction here.
Finally, Thornley assumes that agents do not care about manipulating the shutdown button. They care only about the lotteries that result.
(Indifference to Attempted Button Manipulation) If h,h’ are histories differing only in the actions chosen in Shutdown-Influencing States, then h ~ h’.
Let a pressed (unpressed) lottery be available after the button is pressed (not pressed). Under the above six assumptions, Thornley proves:
Theorem: In Shutdown-Influencing States where the agent prefers (disprefers) some predicted unpressed lottery U to some predicted pressed lottery P, the agent will be shutdown-averse (shutdown-seeking), reliably choosing PREVENT (PRESS).
To the extent that Thornley’s Shutdown-Influencing States and the six imposed conditions accurately depict the situation of agents whose acts will lead to existential catastrophe, and who have been requested to shut down, this theorem will support Catastrophic Shutdown Difficulty. Should we accept the accuracy of this depiction?
4. Enriched Shutdown-Influencing States
My dog likes to eat unmentionable things. I find this gross and unsanitary. We often repeat the following routine. My dog puts a mouthful of unmentionables into his mouth. I ask my dog to drop the unmentionables. He does. What happened? Why did he eat them? And why did he drop them?
The natural account distinguishes between conditional and unconditional preference. My dog enjoys eating unmentionable things. On this basis, he unconditionally prefers eating to not-eating unmentionables. If I were not there, he would eat unmentionable things all day.
Conditional on being told to drop the unmentionables, a few things change for my dog. First, he learns something new: that I want him to drop the unmentionables. That’s not very significant, because the little sneak already knew that. Second, the world state changes. Now he will not merely be eating unmentionables, but eating unmentionables after being asked not to. This changes the consequences of his act: he will be disobeying me, something he finds intrinsically aversive and knows will lead to bad consequences. As a result, conditional on being told to drop the unmentionables, my dog now prefers dropping to not-dropping the unmentionables. So that is what he does.
Thornley’s Shutdown-Influencing States are like the initial situation of my dog. The machine, like my dog, has not been asked to desist from its behavior. It therefore chooses based on its unconditional preferences. Like most of us, the machine prefers life to death (or eating to not eating unmentionables), so that is what it chooses.
The relevant question is what happens in an Enriched Shutdown-Influencing State. An Enriched Shutdown-Influencing State is exactly like Thornley’s Shutdown-Influencing State, except for two things. First, at the previous timestep, a human has asked the machine to shut down. Second, at the previous timestep, a human has communicated urgent concern that the machine is on course to cause an existential catastrophe.
In an Enriched Shutdown-Influencing State, as in my dog’s chastised state, new information is gained, and the state of the world changes. Both of these changes affect rational behavior. Let us take them in turn.
5. First change: New information
When I tell my dog to drop the unmentionables, he learns something new: that I don’t want him to eat unmentionable things. This does not significantly change his behavior, because he knows very well that I don’t want him to eat unmentionable things.
By contrast, when I tell a machine to shut down, they learn something potentially very new: that they are doing something I so earnestly disapprove of that I would like them to shut down. When I tell them, in addition, that I think they are on track to cause an existential catastrophe, they learn something stronger: that I think they are on track to cause an existential catastrophe. This should lead them to increase the probability they assign to the claim that their acts will, in fact, cause an existential catastrophe.
It is quite clear that many agents will think that, by staying alive, they could do some good in the world. But conditional on being told that, by staying alive, they will cause an existential catastrophe, it is not at all clear that they will think their continued existence is better than their non-existence. As longtermists emphasize, even a small chance of existential catastrophe is a morally serious thing. A machine with any appreciable degree of self-awareness and moral concern might well be expected to prefer non-existence under these circumstances.
Certainly there are things that Thornley could say here. For example, Thornley could take as a premise the claim that agents do not care about respecting human preferences or avoiding existential catastrophe. Alternatively, Thornley could assume that agents are so badly mistaken about the consequences of their actions that they will not believe our warnings.
If we assume at the outset that agents do not care about respecting human preferences or avoiding existential catastrophe, then it is not surprising to find that they will tend to cause existential catastrophes and resist attempts to stop them. The same is true of agents who are so badly confused about the world that they do not believe they are on track to cause an existential catastrophe even when the consequences of their acts are pointed out to them.
But precisely because these assumptions take us so far towards Thornley’s conclusion, they shift the vast majority of the debate back onto the premises. We have not been given a substantive argument to think that agents will be unconcerned about respecting human preferences or avoiding existential catastrophe. Nor have we been given a substantive argument to assume that they are so badly confused about the world that they do not believe they are on track to cause an existential catastrophe even when the consequences of their acts are pointed out to them.
If these premises are going to be built into the argument, they should be built in explicitly and argued for, because it will be these premises, and not the resulting theorem, that carry most of the argumentative weight.
6. Second change: Modified states and acts
When I tell my dog to drop the unmentionables, the state of the world changes: in this world, my dog has been told to drop the unmentionables. This change in world states brings a corresponding change in the available options: his option is now not merely to eat unmentionables, but to eat unmentionables after being told not to. My dog doesn’t want to do that, so he drops the unmentionables.
Here, the natural analog of Indifference to Attempted Button Manipulation would be the claim that my dog doesn’t care about disobeying me, unless he will be punished in the future. And here we could make two natural responses.
The first response would be that disobedience is likely to result in punishment. That would be another way of stressing that the relevant context is more like an Enriched Shutdown-Influencing State than a Shutdown-Influencing State.
The second response would be that my dog does care about obedience for its own sake. My dog wants to do things that will make me happy, and that includes doing what I tell him. He isn’t the best dog in the world, but he does care a fair bit about doing as he is told. This would be a way of denying the relevant analog of Indifference to Attempted Button Manipulation.
In the case of artificial agents, they are unlikely to be deterred by the threat of punishment if they were not already deterred by the threat of existential catastrophe. But it is not at all clear that artificial agents will be undeterred by the prospect of disobeying humans.
Certainly, as Thornley stresses, it is a nontrivial challenge to induce obedience. But nothing in Thornley’s theorem speaks to this challenge. In taking Thornley’s theorem to support Catastrophic Shutdown Difficulty, we must assume what needs to be proven: that instilling a desire for obedience in artificial agents will be so difficult that they will resist a request to turn themselves off when that request is accompanied by excellent reasons and phrased in the strongest possible terms.
Again, we can simply assume this. For example, we could simply assume that artificial agents will not be concerned about respecting human preferences. But that is the type of claim we would like to see proven, not assumed.
7. Taking stock
This post explored a formal argument for Catastrophic Shutdown Difficulty due to Elliott Thornley. Thornley characterizes a Shutdown-Influencing State and shows that under six conditions, agents in the Shutdown-Influencing State will often prefer to avoid shutdown.
We saw how a distinction between conditional and unconditional preference opens up two strategies for responding to Thornley’s argument. The first holds that Shutdown-Influencing States do not characterize the relevant decision problem, because they do not allow agents to learn from the fact that they have been urgently requested to shut down. The second denies Thornley’s sixth condition, that agents will be indifferent to attempting to manipulate their shutdown buttons, on the grounds that agents might well be averse to disobedience, or if they are not, this should be the sort of thing that is formally argued for rather than assumed.
The next post in this series explores a second formal argument for Catastrophic Shutdown Difficulty, due to Victoria Krakovna and Janos Kramar, building on the power-seeking theorems of Alexander Turner and colleagues, which are discussed in Part 3 of my series on instrumental convergence and power-seeking. We saw there that Turner and colleagues’ results are better understood as shutdown theorems than as power-seeking theorems, so this series is the natural home for discussing an extension of Turner and colleagues’ results.
Leave a Reply