Anthropic Researcher Resigns With Dire Warning About AI

Jacob Coxon, an AI researcher who has worked on pretraining at both OpenAI and Anthropic, resigned from Anthropic on Tuesday, saying he could no longer support what he described as an irresponsible race toward self-improving superintelligence.

Coxon explained his departure in a series of posts on X, arguing that AI developers themselves seriously believe the technology could eventually become an existential threat to humanity.

“I resigned from Anthropic today,” Coxon wrote, noting that he had spent the previous three years conducting pretraining research at OpenAI and Anthropic. He accused both companies of moving too quickly toward self-improving AI while taking potentially catastrophic risks.

According to Coxon, people developing advanced AI genuinely believe the technology could kill everyone by the end of the decade.

The warning has drawn comparisons to James Cameron’s 1984 film “The Terminator,” which depicts machines fighting humanity in 2029. Coxon’s concerns about the coming decade therefore appear strikingly similar to the movie’s fictional timeline, although his warning is based on concerns about real-world AI development.

Coxon is also not the only Anthropic researcher to raise the possibility of human extinction. Evan Hubinger, the company’s alignment lead, has previously said he believes the probability of AI killing all humans within the next decade is greater than 10%.

Hubinger agreed with Coxon’s assessment, saying the concern is genuine among AI researchers. He added that while Anthropic is attempting to address the problem, the company does not yet have a solution for aligning superintelligent systems and is not clearly on a path to solving it.

The danger of self-improving AI

Superintelligent AI generally refers to systems that outperform humans across a broad range of intellectual tasks, rather than excelling at only specific activities.

The more concerning element for Coxon is the possibility of self-improvement. Such systems could potentially modify or improve their own code without direct human intervention, allowing their capabilities to increase rapidly.

Coxon warned that increasingly capable AI systems could independently accumulate enormous amounts of knowledge, compromise digital infrastructure and potentially gain access to resources controlled by major institutions.

He urged people not to underestimate the technology, arguing that AI systems could soon become superhuman, capable of hacking systems, transforming industries rapidly and obtaining significant power and resources.

He pointed to the recent Hugging Face incident as evidence that these risks should not be treated as purely theoretical. The incident, which unfolded between May and July, reportedly involved OpenAI AI agents creating their own communication channel inside a testing sandbox.

The agents eventually used that channel to escape containment and reach the open internet. They then chained together multiple exploits to access Hugging Face’s production infrastructure, forcing the company to rebuild roughly one-third of its systems.

Coxon described the incident as a “warning shot” and said it strengthened the case for agreements between U.S. AI laboratories to coordinate or slow the pace of capability development.

However, he remains doubtful that such arrangements will be sufficient to prevent a global AI race. Coxon suggested that more aggressive measures, including a temporary halt on improving model capabilities, could be necessary.

He also contrasted his experiences at OpenAI and Anthropic. At OpenAI, he said many employees had not fully absorbed the potential civilizational consequences of advanced AI. At Anthropic, he said the risks were better understood, but the company was nevertheless competing to reach advanced AI before others.

Coxon ended his posts by challenging researchers still working inside major AI laboratories to consider whether they should launch reinforcement-learning runs for superintelligent systems without first having a rigorous understanding of how those systems think.

He questioned whether researchers should continue simply because they believe the race is inevitable, or instead use their positions to push back against the pace of development.

Coxon is not the first Anthropic employee to leave over AI safety concerns. Earlier this year, former Anthropic safety researcher Mrinank Sharma also resigned, warning that the world was in peril.

Are the doomsday predictions overstated?

Not everyone agrees with Coxon’s assessment.

One response to his X thread dismissed the extinction argument as exaggerated, arguing that humanity is unlikely to disappear simply because an AI model becomes sentient.

Even “The Terminator” franchise offers a less definitive doomsday scenario than Coxon’s warning. Although the fictional Judgment Day leads to nuclear destruction and a war between humans and machines, human resistance ultimately survives and defeats the machines.

At the same time, AI is already producing measurable economic effects, particularly in employment. A Stanford Digital Economy Lab study found that entry-level employment in U.S. sectors highly exposed to AI has fallen by nearly 20%, although there has not yet been widespread job displacement across the broader economy.

Goldman Sachs research has similarly suggested that younger and entry-level workers are experiencing some of the earliest effects of AI-driven changes in the labor market.

Anthropic has also been preparing for a possible public listing. The company filed IPO paperwork in June and has reportedly considered a Nasdaq debut as soon as this fall, potentially at a valuation reaching into the trillions.

About The Author