Extract from The Conversation
Earlier this week, researcher Jacob Coxon quit Anthropic, saying the firm and its competitors are “gambling with our lives”. “We really do earnestly believe AI could kill all humans,” added current Anthropic researcher Evan Hubinger in a post on X.
Coxon isn’t the first to down tools over fears of AI doom. The idea that AI could wipe out humanity, advanced in Nick Bostrom’s 2014 book Superintelligence and the influential LessWrong forum, has long circulated among researchers. There are many scenarios for how this could happen, but the core idea is that AI smarter than humans could escape our control and destroy us.
In 2024, Jan Leike and Daniel Kokotajlo quit OpenAI over safety concerns. This year, Anthropic safety chief Mrinank Sharma departed, warning “the world is in peril”. Alex Turner left Google DeepMind after it signed a deal with the Pentagon permitting “killer drones”.
But Coxon’s resignation has made waves, with more researchers admitting they think AI might kill everyone. So if the people building AI believe it could cause extinction, why keep building it? There are three main reasons.
Some think the risk is worth it
AI leaders acknowledge the risk of losing control and killing everyone. In 2023, the chief executives of OpenAI, Anthropic and Google DeepMind agreed that AI extinction risk should rank alongside pandemics and nuclear war. Anthropic’s Dario Amodei puts the chance of things going “really, really badly” at 10–25%.
Yet Amodei also promises a world without poverty or disease, while Elon Musk speaks of AI-enabled “universal high income”.
This is the first reason for pursuing AI: the belief that the benefits outweigh the risks. Perhaps so, but that decision arguably deserves a more democratic process.
Some say you can’t study the danger from a distance
The second reason: you can’t learn to make dangerous AI safe without building it first – like a spacecraft, you can study safety from afar, but can’t really test it without going to space.
OpenAI’s plan is “iterative deployment”: release each model, learn from its problems, and fix them in the next one. The idea is like getting as close to the cliff edge as possible to see what the jump looks like.
Some feel it’s winner-takes-all
The third and perhaps most important reason is the race. OpenAI’s Sam Altman recently said “we are close to creating a genie that can grant any wish”.
The trouble is everyone wants to hold the lamp – it would be hugely profitable, and each company doubts the judgement of rivals to use their wishes wisely.
So they race, reasoning that if they slow down, someone else will get there anyway, so it’s better to arrive first as the “responsible one”. Some fear even a mutual agreement would be broken in secret. So they press on.
AI making better AI
You might doubt runaway AI is plausible. But when the companies themselves raise the alarm, we should listen.
AI firms already report signs of “recursive self-improvement”, where each model helps to build a better successor. According to OpenAI’s chief scientist, models are improving faster than humans’ ability to control them.
In July, hundreds of AI employees signed an open letter calling for a slowdown. But the dynamics of the race make that hard for any single company – or country – to do alone.
A classic arms race
AI research has the hallmarks of an arms race. OpenAI doesn’t want to lose to Anthropic, and the United States doesn’t want to lose to China.
History offers a template for how to manage a situation like this, with rules binding all players, and enforcement everyone can verify.
Nuclear weapons are the classic case. Treaties and verification systems haven’t eliminated the risk of nuclear war, but they have slowed proliferation, and no nuclear weapon has been used in conflict for 80 years.
Rules for AI
In the US, where most cutting-edge AI research happens, the Trump administration shows little sign of slowing AI development.
In its first week it scrapped the old AI safety rules. Now it is trying to override state-level rules, arguing caution risks losing the race to China.
Some politicians are pushing back. California recently passed laws supporting independent assessment of AI systems. US senator Bernie Sanders introduced a bill to ban superintelligence, and British MP Alex Sobel introduced a similar bill.
Companies have moved too. OpenAI paused its most advanced training after a swarm of its agents hacked another startup in August. The company’s head of policy now says that when safety and speed conflict, safety should win.
Still, without binding rules, we’re relying heavily on the goodwill of a handful of companies.
What happens now?
In mid-2025, researchers published what might be our best guide to the coming years: a detailed scenario called AI 2027. Since then, AI capabilities have advanced faster than predicted.
Unless something changes, staffers who quit over safety will simply be replaced, AI models will help build better AI models, and each generation will grow harder to monitor and control.
Is the situation hopeless? I hold out three hopes.
First, that more people recognise AI escaping human control is a bigger risk than AI’s water use.
Second, that governments listen to their people. In the US, two thirds say AI is moving too fast.
Third, that we have a good plan ready before a crisis hits. The best plan, in my view, looks something like this: delays, transparency and verification to slow the race and keep humans in control.
Insiders at the world’s top AI companies say our current safety plan isn’t good enough. If they’re leaving their jobs over safety fears, we should listen to what they have to say.
