If Open Phil[anthropy] had funded PauseAI [two years ago] and made me feel like activism was part of their portfolio, I would be in a similar motivated reasoning position to the Anthropic crowd … They showed their hand to me — they did not want AI development paused.
Holly Elmore, Director of PauseAI US
1. Introduction
This is Part 5 in my series Harms. Risk mitigation has potential harms, as well as benefits. This series aims to chronicle some of the harms that existential risk mitigation may bring about, so that the value of risk mitigation efforts can be properly assessed.
Part 1 looked at the risk of distraction. Part 2 looked at surveillance. Part 3 looked at the cost of delayed technological development. Part 4 looked at inequality.
Today’s post looks at the complex relationship between effective altruism and frontier AI labs.
2. Support
For all their criticism of frontier AI labs, effective altruists and their allies have been deeply involved with leading companies such as OpenAI and Anthropic, supplying major capital investments, employees, platforms on podcasts and conferences, and support in key papers and job boards. Here are some examples of involvement and support.
2.1. Direct investment
In 2017, Open Philanthropy gave a $30 million grant to OpenAI in exchange for a seat on the board. (This grant would, to their credit, give Open Philanthropy some leverage in the later attempt to remove Sam Altman as CEO).
In 2021, Jaan Tallin (a prominent backer of the Survival and Flourishing Fund. Disclaimer: My research is partly funded by this fund.) led a $124 million Series A funding round for Anthropic. This funding round also saw participation from Dustin Moskovitz and Macroscopic Ventures (under its previous name as the Center for Emerging Risk Research).
In 2021, FTX invested $500 million in Anthropic, giving it a 13.56% stake in the company.
2.2. Employment
The Co-Founder of Open Philanthropy, Holden Karnofsky, is a member of technical staff at Anthropic. Karnofsky’s spouse, Daniela Amodei, is the president and co-founder of Anthropic. Other prominent effective altruists or EA-adjacent individuals employed by Anthropic include Joe Carlsmith and Amanda Askell.
A number of effective altruists and EA-aligned individuals joined OpenAI. To their credit, most of these individuals quit the company (many in quite spectacular fashion, and at risk to their equity stakes) when they did not feel that OpenAI showed sufficient concern for safety. I don’t want to complain about the behavior of these individuals.
2.3. Platforming
The cofounder of Anthropic, Chris Olah, has twice been a guest on the 80,000 Hours podcast (here, here).
Jan Leike, who leads the alignment team at Anthropic and previously led the superalignment team at OpenAI, was also a two-time guest on the 80,000 Hours podcast (here, here).
The CEO of OpenAI, Sam Altman, recently spoke at a conference at Lighthaven hosted by the Roots of Progress Institute.
2.4. Other support
The first (2019) version of “The case for strong longtermism” suggests that among the “things that [altruists] can do now … to improve the probability that … an AI system embodies goals that are more rather than less conducive to well-being” would be to “fund technical and policy work on ensuring the safe development of artificial intelligence at OpenAI”. This is an outright call for donations to OpenAI.
Effective altruist organizations such as 80,000 Hours have advertised jobs at frontier AI labs such as OpenAI and Anthropic. Despite criticism, this practice is ongoing. As of November 7, 2025, 80,000 Hours lists 31 jobs at OpenAI and 25 at Anthropic. The head of the 80,000 Hours job board defends this practice here and here.
3. The case for support
To their credit, effective altruists and their colleagues have provided reasons for supporting frontier AI companies.
Some have argued that it is morally permissible to engage in safety-related work at frontier AI labs while allowing others to advance capabilities.
Others have argued that since the work is likely to be performed anyways, it is best to ensure that safety-conscious people take jobs at frontier AI companies to ensure that the work is performed as responsibly as possible.
Still others argue that if superintelligent artificial systems are to be developed, it is better for these systems to be developed in the United States than in competing nations such as China.
4. The case against support
The first thing to note is that while some of these arguments may rationalize taking employment in AI safety-related roles, few arguments would rationalize the full extent of support for frontier AI laboratories.
Leading effective altruist-aligned individuals and organizations have made large investments in frontier AI labs, at some points taking more than a 10% stake in those companies or leading major funding rounds for them. This kind of investment is not rationalized by the idea that it is permissible to engage in safety-related work, and the idea that the benefits of safety-conscious investors can outweigh such a large investment in AI laboratories strains credibility. Perhaps such investments could be rationalized on a very strong version of the view that superintelligent systems should be developed in the United States, though strong versions of this view need good arguments if they are to transcend xenophobia and become well-grounded security concerns.
The second thing to note is the allure of rationalization. There is a good deal of money and power to be found at frontier AI labs, and all of us (myself included) can be influenced by such things. It is very easy to find rationalizing explanations for decisions that will yield money and power. It is much harder to objectively evaluate the strength of these rationalizing explanations and to find them wanting.
A third concern is a more pernicious cousin of regulatory capture. Given the near-total absence of meaningful regulation on frontier artificial intelligence, leading AI labs are at least as concerned to pacify their critics as they are to pacify regulators. In what we might call critical capture, those companies seek to on-board and in many cases directly hire influential critics in order to tame those hired and mute the criticisms of those who might some day wish to be hired.
In this vein, we should not always accept at face value the desire of companies such as OpenAI and Anthropic to establish safety teams. Those very teams bring on board, and thereby silence, many of the companies’ leading critics, and use those same individuals to help the company demonstrate that it is making meaningful progress towards safety, even when this progress is lacking. To say, then, that it is permissible for safety-conscious individuals to work on improving the safety of frontier AI systems may be to ignore the many ways in which safety teams function to dampen safety-focused criticism.
There is, of course, a good deal more to be said on the case against supporting frontier AI companies. But sometimes brevity is more important than comprehensiveness.
5. Implications
Effective altruists and their allies increasingly express concern about the risks posed by developments in artificial intelligence. At the same time, many effective altruists and their allies have previously and presently supported the very companies developing frontier AI systems.
This situation suggests several conclusions. The first is moderate skepticism about the depth of commitment to conclusions about the level of existential risk posed by artificial agents in the importance of reducing it. While it is not impossible to find rationalizing explanations for supporting companies while also taking these companies to pose a double-digit chance of bringing about human extinction within the next century, this is no easy matter, and one can be forgiven for beginning to fish around for alternative explanations.
The second is a genuine and emerging rift within effective altruism and adjacent communities over support for frontier AI companies. I am far from the first to note or complain about the degree of support provided for these companies. The rift between those who support and oppose frontier AI companies is likely to widen in the coming years, and should be formative in future developments.
A final conclusion is to reiterate the difficulty of predicting and influencing the future. It can seem obvious that if there are definite existential risks to humanity within this century, then there is a clear path to reducing them and that this path should be much more important than any competing intervention. But the reality on the ground is much more complicated. Effective altruists and their allies disagree about matters as simple as whether and when to support frontier AI companies. These disagreements compound throughout almost every question related to the reduction of existential risk from artificial intelligence. If we concede (as I, myself, do not) that artificial intelligence poses a significant existential risk, then it remains a long and complicated road from this concession to any definite conclusions about policy or career prioritization.
Leave a Reply