Anthropic researcher quits over self-improving AI risk

Jacob Coxon, a researcher who said he spent the last three years on pretraining research at both OpenAI and Anthropic, resigned this week over fears that unrestrained development of self-improving AI could end up killing everyone. In a Tuesday evening thread on X, Coxon accused the labs racing to build the technology of failing to act responsibly, writing that the people building it “earnestly believe it could kill us all by the end of the decade.” “They are racing straight to self-improving superintelligence and gambling with our lives,” he wrote.

Coxon argued that upcoming systems could become superhuman at hacking, revolutionize entire fields overnight, and acquire real power and resources, and that progress in each of those domains isn’t slowing. He said many executives and senior researchers privately express fear about the risks even when they publicly downplay them, and argued that OpenAI hasn’t deeply internalized the stakes, while Anthropic understands them but is locked in a race, believing no one else will act responsibly. He called entering that “endgame” a hubristic gamble that shouldn’t be launched from a private company’s Slack, and said he remains optimistic that coordination between labs is possible, pointing to recent incidents as having made pacing agreements more viable — though he warned that preventing a global race may require costly steps, including a temporary ban on improving model capabilities.

Anthropic colleague Evan Hubinger echoed the concern, saying his team does believe AI could kill all humans, estimating the likelihood at greater than 10% within the next decade, and acknowledging Anthropic doesn’t have a plan to solve alignment for superintelligence. A recent report from nonprofit Guidelight AI Standards found few top AI labs have published containment plans for shutting down AI that tries to subvert human control.

Anthropic and OpenAI aren’t alone in chasing recursive self-improvement: Ricursive Intelligence raised $335 million at a $4 billion valuation in February, Recursive Superintelligence raised $650 million at a $4 billion valuation three months later, and former Google DeepMind researcher Jeff Dean launched Discovery Loop last month. ControlAI executive director Connor Leahy, who advised on new U.S. and U.K. bills to ban development of superintelligence, called recursive self-improvement the most likely point at which humans lose control, arguing it would be very hard to shut down once underway.

Why it matters

A public resignation over existential-risk fears, from someone who worked inside both leading labs, adds pressure on regulators to act before the technology both companies say could kill people arrives.