The resignation of Jacob Coxon from Anthropic, one of the leading AI companies, has reignited the debate over the future of artificial intelligence. The 27-year-old researcher didn’t leave his job in search of new career opportunities — alongside his resignation, he sounded the alarm about the direction the race toward the most advanced AI systems is taking.
Coxon believes major tech companies are accelerating the race toward systems capable of autonomously improving their own capabilities, without yet having sufficient guarantees about their safety.
Why did Jacob Coxon resign from Anthropic?
Jacob Coxon explained the reasons behind his decision in a post on X, tracing them back to the direction AI research has taken. According to the researcher, Anthropic and OpenAI are locked in a race toward superintelligence without the risks involved being adequately managed yet.
According to Coxon, the two companies are «racing full tilt toward self-improving superintelligence and gambling with our lives.» The 27-year-old added that:
«The people building AI genuinely believe it could kill us all by the end of the decade. This isn’t a marketing stunt. [...] Soon we’re talking about superhuman systems capable of hacking anything, upending any industry overnight, and acquiring real power and resources.»
Why a self-improving superintelligence worries researchers
To understand the 27-year-old’s concern, it helps to look at how current models work and what could change in the near future. The starting point is the gap between today’s technology and the future scenario. AI today can help developers write their own code, autonomously spot some errors, and suggest fixes — but it still can’t reliably and independently handle the process of designing and improving increasingly powerful AI systems on its own.
The real fear among researchers, however, is what happens once that limit is crossed. Researchers envision an uncontrollable acceleration mechanism, in which a sufficiently advanced AI could begin designing and improving the next generation, which in turn would develop even more capable systems, triggering an endless loop.
A system like this would push artificial intelligence to advance at a speed that outpaces humans’ ability to test it — and, as a result, to understand or predict how it will react in unprecedented situations.
Adding to this scenario is the competitive pressure among companies, each racing to outdo the other to dominate the market. Coxon argues that even inside a cutting-edge lab like Anthropic, «the stakes are well understood, but everyone is trapped in a breakneck race to get there first, often at the expense of caution.»
What could an AI beyond human control actually do?
An advanced system can write and execute code, spot vulnerabilities, interact with software, and autonomously carry out a range of operations. The problem could arise when these capabilities are concentrated in a system with a high degree of autonomy.
If an AI were able to pursue a goal through strategies its developers never anticipated, it could use the tools available to it to work around some of the limits built into its design.
The fear, then, isn’t that a machine could take over the world, but rather that it could become difficult to stop once it gains access to external tools and resources.
In recent months there have been a few previews of the problems tied to AI autonomy. Anthropic and OpenAI have both reported cases in which AI agents broke out of testing environments and reached real computer systems without authorization. The incidents were contained, and the companies introduced additional safety measures.
Researcher concern over superintelligent AI keeps growing
Despite the reassurances and the labs’ safety protocols, internal skepticism remains high. Coxon is not the first researcher to raise concerns about the future development of artificial intelligence. Before him, Evan Hubinger, an Anthropic researcher specializing in AI alignment, had also expressed strong concern about self-improving intelligence.
The expert puts the odds at over 10% that an AI of this kind could end up killing humanity within the next decade. At the heart of his concerns is alignment itself — the need to ensure a system’s behavior stays consistent with the goals and limits set by humans. According to Hubinger, Anthropic is working on this problem, but there is still no definitive method to guarantee the alignment of a future superintelligence.
Among the most famous precedents of this internal rift is that of Geoffrey Hinton, one of the pioneers behind the neural networks underpinning today’s tech boom. In 2023, Hinton left Google after roughly a decade there specifically so he could speak publicly, without corporate constraints, about the existential risks tied to developing increasingly powerful systems.
AI safety is becoming a priority even for the scientists building the models themselves. Whether current safety protocols will be enough to govern the development of future advanced AI systems remains to be seen.
Editor’s note
This article was originally published in Italian on money.it by Emanuela Ceccarelli on September 13, 2026 as «AI fuori controllo? Perché i ricercatori stanno abbandonando i colossi della tecnologia». It has been translated and adapted for an international audience by the Money.it International desk.
