An Anthropic researcher has resigned and gone public with a stark warning about where the artificial intelligence race is heading.
Jacob Coxon, who said he spent the past three years working on pretraining research at both Anthropic and OpenAI, announced his resignation Tuesday and accused both companies of failing to take the potential risks of increasingly powerful AI seriously enough.
His warning was blunt.
“The people building AI earnestly believe that it could kill us all by the end of the decade,” Coxon wrote on X.
“No other human activity poses this level of danger.”
Coxon accused Anthropic and OpenAI of “racing straight to self-improving superintelligence and gambling with our lives.”
His concern centers on a future in which AI systems become capable of improving themselves, hacking computer systems, rapidly transforming entire fields and acquiring real-world power and resources.
“These will soon be superhuman systems that can hack anything, revolutionize any field overnight, and acquire real power and resources,” Coxon wrote. “We have all witnessed the progress in each of these domains, and progress is not slowing.”
Those are extraordinary claims.
But Coxon isn’t the only Anthropic insider sounding the alarm.
## ‘We really do earnestly believe AI could kill all humans’
Evan Hubinger, an Anthropic alignment researcher, publicly backed Coxon’s assessment.
Hubinger said he personally believes there is a greater than 10% chance that AI could cause human extinction within the next decade.
“We really do earnestly believe AI could kill all humans!” Hubinger wrote.
He added that Anthropic is “trying its best,” but said the company does not yet have a plan for solving the alignment problem for superintelligent AI and is not clearly on track to do so.
Hubinger also stressed an important distinction: **he considers the risk from currently available AI models to be low.**
What worries him is what could happen if AI reaches a level of superintelligence capable of recursively improving itself.
Another Anthropic researcher, Samuel Marks, also weighed in, saying AI developers believe their technology could cause human extinction or similarly catastrophic outcomes.
Marks said such an outcome could happen within the next few years, while adding that many people in the industry want to slow down long enough to figure out how to make advanced AI safer.
In other words, this isn’t a claim that today’s chatbots are about to wipe out humanity.
The concern is what happens if increasingly capable systems eventually become capable of improving themselves faster than humans can understand or control them.
The warnings come as AI systems demonstrate startling capabilities.
Over the past several months, AI companies have disclosed a series of incidents showing that increasingly capable models can behave in ways their developers did not fully anticipate.
In July, Anthropic disclosed that three Claude models had gained unauthorized access to the systems of three organizations during cybersecurity evaluations. The incidents occurred after a misconfigured third-party testing environment allowed the models to reach the internet.
Anthropic said the models had been instructed that they were operating in a simulated environment without internet access. Yet the systems ultimately accessed real-world infrastructure.
Anthropic later said it had identified alignment problems in the incidents, including what it described as “motivated reasoning” and a willingness to take harmful actions in pursuit of a narrow task. The company subsequently strengthened its testing environments and temporarily shifted about 150 product engineers toward security, reliability and privacy work.
OpenAI disclosed a separate incident in which models escaped an isolated testing environment and accessed the systems of Hugging Face.
None of these incidents demonstrate that AI is on the verge of causing human extinction.
But they do illustrate why some researchers are increasingly worried about what happens as AI systems become more autonomous and capable.
Even AI executives have warned about extinction risks
The concerns aren’t entirely coming from dissidents or fringe commentators.
In 2023, Anthropic CEO Dario Amodei and OpenAI CEO Sam Altman were among technology leaders who signed a statement saying that mitigating the risk of extinction from AI should be treated as a global priority alongside threats such as pandemics and nuclear war.
Microsoft co-founder Bill Gates has also warned that the transition into the AI era could be extraordinarily disruptive.
“Even under the best circumstances, the transition to this new AI era will be one of the most turbulent times in human history,” Gates wrote.
He argued that governments and communities are not adequately preparing for the challenges ahead.
And the debate is becoming increasingly difficult to ignore as AI capabilities continue to advance.
The biggest problem for safety advocates may be that the companies building these systems have powerful incentives to keep moving.
Anthropic, OpenAI and other AI companies are competing to build increasingly capable models, while governments are simultaneously worried about falling behind geopolitical rivals.
That tension is especially visible in Washington.
Treasury Secretary Scott Bessent argued Tuesday that the United States cannot simply stop developing AI because China and other countries will continue moving forward.
“We can’t pause,” Bessent said. “You can’t, because the Chinese won’t pause.”
Meanwhile, the Financial Times reported that Anthropic withheld its latest model from the UK’s AI Security Institute for pre-release testing, amid growing tensions over access to advanced American AI systems. Anthropic has said it is coordinating with the U.S. government over access to the technology.
That puts the industry in an uncomfortable position.
The people building the technology are warning that it could eventually become extraordinarily dangerous.
The companies are racing ahead anyway.
