There’s never just one cockroach in the kitchen.
1. Introduction
This is Part 5 of my series Revisiting the Shutdown Problem. This series discusses my paper, “Revisiting the shutdown problem.”
Put roughly, the shutdown problem is the problem of ensuring that artificial agents can be shut down when they get out of control. A range of informal arguments and formal shutdown theorems have been offered to suggest that solving the shutdown problem is difficult for agents whose acts would lead to existential catastrophe. This paper and blog series aim to show that these arguments do not succeed.
Part 1 introduced and characterized the catastrophic shutdown problem. Part 2 considered two informal arguments for the difficulty of solving the catastrophic shutdown problem. Part 3 and Part 4 considered two formal arguments for the difficulty of solving the catastrophic shutdown problem. In each case, we saw that leading formal and informal arguments for the difficulty of solving the shutdown problem are not persuasive.
Suppose that is right. What follows?
2. Reduced probability of existential catastrophe
The catastrophic shutdown problem arises in response to an objection. Those concerned about existential risk argue that artificial intelligence will soon pose a significant existential risk to humanity. In response, it is objected, the risk cannot be as large as claimed because malfunctioning artificial agents could be deactivated.
This raises the Catastrophic Shutdown Problem of designing agents that:
(CSHT-1) Shut down when their actions would lead to existential catastrophe, when requested to do so.
(CSHT-2) Do not try to prevent requests to shut down when their actions would lead to existential catastrophe.
(CSHT-3) Otherwise pursue goals competently.
Those concerned about existential risk reply:
(Catastrophic Shutdown Difficulty) It is difficult to design agents that satisfy CSHT-1, CSHT-2 and CSHT-3.
In this paper and blog series, we saw that leading informal and formal arguments for Catastrophic Shutdown Difficulty are not convincing.
If that is right, then it reduces estimates of existential risk from artificial intelligence by revitalizing an objection to leading arguments for taking that risk to be high.
3. New arguments, and new assumptions
To say that current arguments do not significantly support Catastrophic Shutdown Difficulty is not to say that no possible arguments could support Catastrophic Shutdown Difficulty.
One implication, then, is that those concerned about existential risk from artificial intelligence should probably try to identify further arguments in favor of Catastrophic Shutdown Difficulty. If those arguments succeed, the problem will go away. If not, then we might consider stably reducing estimates of existential risk.
Another implication is more subtle. Along the way, we saw that arguments for Catastrophic Shutdown Difficulty often rely on significant unstated or unargued assumptions. In some cases, the assumptions are highly implausible and should probably be jettisoned. For example, we saw in Part 4 of this series that Krakovna and Kramar rely on the assumption that:
Equiprobable Training-Consistent Reward: All training-optimal reward functions are equally likely to be learned.
We saw that Equiprobable Training-Consistent Reward rests on the empirically unmotivated assumption that agents do not learn to generalize in any way outside of their training distribution.
Other assumptions may be worth further investigation. For example, we saw in Part 3 that Thornley’s argument might be revitalized by premises to the effect that artificial systems will be largely unconcerned with satisfying human preferences. And we saw in Part 2 that the empirical argument for Catastrophic Shutdown Difficulty could be strengthened by more, and more relevant, empirical studies of shutdown-resistant behavior.
Not all arguments for Catastrophic Shutdown Difficulty may need to be jettisoned. Sometimes, it may be worth some work to clearly state and justify the needed assumptions, or find and provide the needed empirical evidence. Doing so could be a good alternative to constructing novel arguments.
4. Technical governance solutions
In the full paper corresponding to this blog series, I argue that some technical alignment solutions that have been proposed to solve the shutdown problem have a very high safety tax in the form of reduced model performance.
This should not be surprising. Many ethics- and safety-interventions incur a safety tax. In each case, the question we have to ask ourselves is whether the tax is worth paying.
If Catastrophic Shutdown Difficulty were true, then those who think artificial agents are relatively likely to embark on courses of action that could lead to existential catastrophe might favor paying the tax, whatever the consequences. However, reductions in the plausibility of Catastrophic Shutdown Difficulty should lead to corresponding reductions in our willingness to bear high safety taxes to reduce shutdown-aversion.
We cannot always reduce model performance in the name of safety. Even those who are more concerned about safety than I am must pick and choose where to impose a high safety tax. Putting pressure on Catastrophic Shutdown Difficulty is, among other things, a way of suggesting that shutdown-aversion may not be an area in which we should be especially willing to accept technical alignment solutions with high safety taxes.
5. Foundational weaknesses
This paper is one of several papers in which I have argued that foundational claims in longtermist-adjacent AI safety discussions are less well-supported than their advocates take them to be. I have elsewhere voiced similar concerns about the singularity hypothesis and instrumental convergence and power-seeking, as well as about leading estimates of existential risk and the value of existential risk mitigation.
Taken individually, these papers argue for the surface conclusions that one or another hypothesis or claim should be revisited.
Taken collectively, these papers provide mounting inductive evidence that longtermist-adjacent AI safety foundations may not be in anywhere near as good order as their advocates take them to be. It is, of course, possible that each set of concerns is an isolated incident. But after a certain number of claims turn out to be much less well-defended than they appeared, one could be forgiven for thinking that other neighboring claims may be similarly under-defended.
I hope that readers will take on board my direct arguments that one or another claim is false or undersupported. But I also hope that each successive direct argument may convince readers that there may be quite a bit more foundational issues in longtermist-adjacent AI safety. As Warren Buffett would have it, there’s never just one cockroach in the kitchen.
Leave a Reply