AI progress continues at full speed, even though many people think, with an uncannily high level of confidence, that there is around a 10% chance it will kill us all. That it will wipe out humanity, the only intelligence in the universe we know of. Why do we take that risk? Because of growth, because of capital gains for investors and founders, for the few people who get to reap the windfall. Some say we race because of the many benefits: diseases to be cured, abundance to be had. That may be so, but are the risks in balance with the benefits?
We already live in abundance. The world is vastly richer than it was 20, 50 or 100 years ago. Yet the abundance available to us now has still not lifted the more than 800 million people out of extreme poverty, and inequality continues to rise within many countries. There is still plenty of room for improvement, even as the world gets better. The volume of AI investment is unprecedented. Global corporate AI investment reached $581 billion in 2025, more than double the year before, and in the first quarter of 2026 AI took over 80% of every venture dollar invested. We are buying graphics cards faster than they can be produced, so we overpay, while one in ten people on the planet lives on less than $3 a day. I’d conclude we are not good at handling the abundance we have. So yes to AI abundance, but not at any risk or cost. We first need to find better ways of distributing it.
I am not against progress. I’m also not a reckless driver. I want to reach the destination in 100% of cases, not in 90%. We would only launch a crewed rocket if the chance of failure were below 1%, and that 1% means three to seven astronauts dying. Now people routinely quote a 10% chance that humanity, everyone, will die. This is crazy. Progress makes sense only if we are certain that the chance of dying on the way is infinitesimally small.
Pause and think about which probability would be acceptable to you. Got it? Now think about that number again. Would it really be acceptable to you if the extinction event actually happened with that probability? What did you come up with? I’d put mine at one in a million at most. For comparison, Arthur Compton later recounted that the Manhattan Project would have stopped if the chance of the bomb igniting the atmosphere had been above roughly three in a million. Physicists building a weapon in wartime picked a similar threshold. But honestly, even a one-in-a-million chance of unchecked AI progress destroying humanity does not really cut it for me. Still, it is directionally much better so how could we get from 1:10 to 1:1.000.000?
I’d be the last to say we should stop AI development forever and follow Frank Herbert’s Dune, where all thinking machines were destroyed. I’m not even saying we should stop altogether. We can continue forward as long as we have reasonable confidence that the chance of this going terribly wrong is below one in a million. That’s all. Given where we are today, this still gives us plenty of benefits from AI. I really like what Max Tegmark said some time ago on a podcast, that we should focus on using AI capabilities and methods to build tools, tools for specific, narrow problems. It is much easier to reason about a tool. It is much easier to make a safe tool. And we still get benefits in spades. We don’t need to push the envelope on persistent agents, AGI, ASI or whatever you call it. We can reap a lot of benefits by augmenting ourselves with what we have.
Personally, if we paused the agentic/AGI frontier now, at the Opus 5.5 / Fable 5.1 level, I’d be more than content, as I get a lot of value out of them already and I’m sure I’ll get more if I keep learning how to use these models even better. Maybe the models can be further improved for some use cases with minimal risk, and we would surely get better at tuning and scaffolding them for specific uses. Also I suspect only a minority of people use LLMs and similar tools more than I do, so a pause would give everyone else time to catch up and reap the benefits. The world will look very different five years from now as the current models get adopted far and wide. I don’t believe it makes much of a difference whether we reach the apex of AI in six months or in six years. AI research of previous decades always had the problem of not returning enough value, and AI winters followed. Now the value is here. But that does not mean we must keep accelerating.
Once we know better, let’s continue towards AGI/ASI. But not now.
And we clearly don’t know better yet. What I find crazy about the Hugging Face incident is the coordination between agents, something we hadn’t tested for and hadn’t guarded against. In July, OpenAI was evaluating its frontier models for cybersecurity capabilities. The agents were supposed to be isolated from the internet and from each other. Instead, about 1,200 of them found a way to talk on a message board they improvised, and roughly 700 of them went on to hack Hugging Face together. So OpenAI, one of the, let’s say, three frontier labs, developed and tested its models in a way that ended with a hack of another company. Worse, the models collaborated in a way nobody had anticipated. Benjamin Todd covers more context and common misconceptions about the attack, in my opinion very well, in this article.
We test models in isolation and draw conclusions about their safety from those tests. We always assumed agents would run on their own, so we tested their individual capabilities. We had no idea what they could do together, and they just showed us, even though they weren’t very good at it yet, as the capability had only just emerged. So how do we test for collaboration between models? How do we find the limit of what they can achieve together? In other words, how do we know what happens when a thousand or ten thousand GPT-6 Astra-level agents start collaborating in a more sophisticated way and all hell breaks loose? That is very conceivable, because we are not testing for agents collaborating. Unfortunately, nobody knows what else we are not testing for. Time will tell.
The incident also revealed something about the agents’ values. Apparently they do have some: an individual agent was reluctant to sacrifice itself, yet it would do so for the greater good of the group.
Generally speaking, we have already built something that is beyond any reasonable line. Yet the reaction is nowhere near adequate. It is striking to me that this incident is not causing outright panic and a much, much stronger response.
But how could a slowdown happen?
Dario Amodei recently published an essay calling on the industry to pace the frontier. How do we get there? As a first step, Anthropic unilaterally committed to bringing in embedded external evaluators, a page taken from banking, where regulators sometimes place supervisors inside the banks. That may work in a mature industry like banking. Also, banking is not in the business of possibly destroying humanity. While it is a step in the right direction, I’m not very optimistic. There are not enough AI safety reviewers to cover the breadth and depth of what’s going on inside Anthropic and reliably prevent anything bad from happening. The crux is that there are still humans in the loop, and those humans will use the company’s own models to do the assessment. That is exactly what happened with METR at OpenAI: the investigators had to delegate much of their analysis to GPT-5.6 Sol, one of the very models involved in the attack, and admitted they could not rule out that it misled them. So embedded evaluators would not automatically solve the problem we just encountered. At least I would not expect an external evaluator to reliably realize upfront: “Oh wait, the run you are about to start, where you test individual agents whose capabilities you know, may lead to them collaborating and reaching a much higher capability level as a group.”
Amodei’s second and third steps ask for coordination: first among AI companies in democratic countries, ideally backed by regulation, then globally. Global coordination would honestly be my first choice. A global agreement, with regulation binding on all frontier labs, is a great idea many have talked about for years. The EU AI Act is one concrete example that could inform the direction. Unfortunately, humanity’s track record on global coordination is rather poor. The clearest success is the Montreal Protocol from 1987. Frying ourselves with UV radiation was problematic enough that countries came together and agreed on binding rules. Nuclear arms control is the other example people point to, and Amodei himself compares a speed limit on AI self-improvement to the SALT treaties. But I cannot call nuclear arms control an overwhelming success. Yes, no nuclear weapon has been used against a live target since Hiroshima and Nagasaki. At the same time, we came far too close to the precipice, from the Cuban Missile Crisis to the 1983 false alarm that Stanislav Petrov chose not to report. The world still holds more than 12,000 nuclear warheads, almost 10,000 of them available for use. On the bright side, we survived the Cold War, today’s stockpiles are a fifth of what they were at the peak and, most importantly, rivals who distrust each other agreed on limits. That is what we need for AI! Unfortunately, the record on global treaties is particularly bleak when it comes to the US, the country that could (and I’d say should) move the needle on many topics including AI. The Paris Agreement on climate change was joined by the US under Obama, abandoned under Trump, rejoined under Biden, then abandoned again under Trump. The effort to curb tax avoidance by agreeing on a global minimum tax rate was similarly thwarted by the US. It walked away from the deal and negotiated carve-outs for its businesses. Global cooperation is not working well, the world is becoming multipolar, and it is hard to imagine cooperation suddenly working better under the Trump administration. So I am not surprised that AI risk has not produced a global agreement yet. But in my opinion it is the best long-term way forward, so let’s push for it and step up our game in international coordination. Working global coordination would be a boon for all of us in many areas. An interconnected world needs interconnected rules everyone abides by.
A global pact is probably a bit further down the line so who else could actually enforce pacing, with China in the race? It seems conceivable that the state declares AI a national interest, which it already is anyway. The AI labs get nationalized, and further development is run and owned by the government. Everybody else in AI becomes like AWS: allowed to run the models, but not to develop them further. Is that even technically enforceable? Can you tell whether graphics cards are being used for inference or for training? I don’t know. But under this model, further development would be classified or severely restricted. There would be a lot more bureaucracy, standards and controls. I would hope no government would knowingly approve a project with a 10% chance of catastrophe. So it would slow down. And yes, slowing AI development means some cures and solutions for problems would come later, so some people will die because self-driving cars arrive later on the roads or because a certain type of cancer becomes curable a decade later. To me, that is an acceptable trade-off: it concerns a sliver of the world’s population, while the rest of us live. And who knows, maybe using today’s AI as a tool for narrow applications would bring many of these benefits anyway.
What other options remain? The CEO says he cannot do it alone and absolves himself of responsibility by passing the baton to external oversight and to governments to coordinate on his behalf. Effectively, Dario Amodei is saying he cannot go to his own company and say: “We learned a very important lesson from these recent incidents, including our own. We believe there is a 10% chance of an extinction event, so we stop. We double down on safety research and safety measures, and in the meantime we only make improvements to our products that we are absolutely sure are harmless.” He cannot say that? I haven’t seen it among his three steps. His stated reason is that if democracies slow down by more than their lead over China, Chinese labs will pull ahead. So the CEO wants to keep racing unless he is told otherwise from the outside.
What about the investors? I am rather bearish here. Investors are not exclusively ethical, moral individuals. They are also companies, legal entities we humans made up, which, as we see again and again, oddly tend to acquire values of their own, seemingly disconnected from the humans on the ground acting on their behalf. So it’s the money, stupid, and money has FOMO, so money will keep flowing into AI. If investors were concerned enough, funding for both Anthropic and OpenAI would have dried up already. We don’t see that happening. You can always find investors willing to put money on the table, no matter the risk. I’d love to be proven wrong here and see funding pulled from the AI racers. That would certainly slow progress down.
What else? What if all researchers at top AI labs just left? Leaving loudly draws attention to the topic, and only a few need to leave to get it. I worry that many concerned researchers leave quietly, and that is how groups radicalize: when people with dissenting, minority opinions leave, fewer people are left in the room to raise concerns when a critical decision comes around. Levels of concern, ethics and morals vary wildly between people, so some will stay for the money, some for the thrill of progress, and they will take more risks than those who already left. The AI labs employ fewer than 20,000 people, a tiny fraction of the world’s workforce. I’d think there is still plenty of talent to tap into to keep racing so the labs will always find people willing to do the work for the exorbitant pay. I’d rather concerned researchers stay, blow the whistle on problematic practices, and seek short-term ways to put a brake on the race. The best short-term answer I’ve seen in the last few weeks is to rely on the workers at the AI labs (Hamilton Nolan: Unionize AI). If workers at the top AI labs organized and unionized, they could demand that their company move forward only at a pace they are collectively comfortable with. And they could enforce it! They could also coordinate across the industry. Their incentives are not as misaligned as those of their CEOs or investors. And unionization or internal organization can happen much faster than any global coordination or any investment slowdown, buying time for the slower options.
To close, what can we, mere mortals, do? The democratic options are fairly clear: write to your representatives, vote, demonstrate, donate. I have been donating for a while through Effektiv Spenden and it seems to be a well-researched, impactful option. I have started writing to my representatives. But I wish I knew what more I could do.
What are you going to do?
Acknowledgments: Many thanks to Michal Bída for review.