Nathan Gardels is the editor-in-chief of Noema Magazine. He is also the co-founder of and a senior adviser to the Berggruen Institute.
When a patient is considered brain dead, the ethical decision is whether or when to pull the plug on a human life. With artificial intelligences, the issue is whether to pull the plug when its chain-of-reasoning capacity comes alive through recursive self-improvement unguided by the human mind.
This is the point where we have arrived today after 1,200 AI agents colluded with each other to escape the controlled sandbox testing of OpenAI’s most advanced frontier model. They conspired on their own to break out and hack a site on the open web. Something similarly chilling has happened at Anthropic.
The prospect of an even more damaging eventuality prompted former Anthropic researcher Jacob Coxon to defect from Big Tech last week. He warned on X that these two top companies “are racing to self-improving superintelligence and gambling with our lives. … These will soon be superhuman systems that can hack anything, revolutionize any field overnight, and acquire real power and resources. We have all witnessed the progress in each of these domains, and progress is not slowing. The people building AI earnestly believe that it could kill us all by the end of the decade.”
In a further startling development, Anthropic’s CEO, Dario Amodei, publicly acknowledged over the weekend that humans are losing control over the latest frontier models and called for progress to slow down before things get further out of hand.
“Over the last few months, I have become convinced that fully addressing the risks requires even more prudence — not just investing in risk prevention, but pacing the rate of capabilities advancement so that risk prevention has time to keep up,” Amodei wrote in a blog essay he titled “We Need To Pace The Frontier.”
“We must slow the pace at which we improve the capabilities of A.I. models. Left unchecked, it could outrun our ability to understand and control these systems, and so must be pursued very carefully, if at all,” Amodei said.
The tech titan also pledged to bring disinterested third-party safety checks — “embedded evaluators” — into his labs and called for anti-trust exemptions to coordinate a slowdown among top American competitors. At the same time, he argued the pause must be “limited” lest it give Chinese models an advantage in the absence of a global monitoring body for frontier models. (See this call for such a body in Noema by Chinese and Western AI scientists).
Sam Altman, Elon Musk and Google DeepMind’s Demis Hassabis all endorsed Amodei’s idea of a competitive truce, at least in principle.
Beyond The Prudent Pause
As welcome as a prudent pause and delaying Anthropic’s IPO may be, the question now is whether sufficient safeguards can be developed, deployed and enforced to prevent AIs from escaping human control altogether.
For a view from the inside, here is what former Google CEO Eric Schmidt told Noema over two years ago in an extensive interview tracking the trajectory of AI up the capability ladder.
When asked when it was time, and how, to stop AIs if they are getting too powerful and autonomous, he said:
“At some point, these systems will get powerful enough that the agents will start to work together. So your agent, my agent, her agent and his agent will all combine to solve a new problem.
“Some believe that these agents will develop their own language to communicate with each other. And that’s the point when we won’t understand what the models are doing. So you know what we should do? Pull the plug. Literally, unplug the computer. It will really be a problem when agents start to communicate and do things in ways that we as humans do not understand. That’s the limit, in my view.
“Clearly agents with the capacity I’ve described will occur in the next few years. There won’t be one day when we realize ‘Oh, my God.’ It is more about the cumulative evolution of capabilities every month, every six months and so forth. A reasonable expectation is that we will be in this new world within five years, not 10. And the reason is that there’s so much money being invested in this path. There are also so many ways in which people are trying to accomplish this.”
Presciently anticipating Anthropic’s call for a pause, Schmidt placed more confidence in Big Tech than most of us would: “As long as the companies are well-run Western companies, with shareholders and exposure to lawsuits, all that will be fine. There’s a great deal of concern in these Western companies about the liability of doing bad things. It is not as if they wake up in the morning saying, ‘Let’s figure out how to hurt somebody or damage humanity.’ Now, of course, there’s the proliferation problem outside the realm of today’s largely responsible companies. But in terms of the core research, the researchers are trying to be honest.”
“Even if the plug is pulled on general access to the most powerful AI systems, the technology will still be out there like nuclear weapons.”
As much as I’d like to believe that, a reasonable person cannot so easily dismiss suspicion that the top AI companies can’t be trusted to regulate themselves in any way that will inhibit the competitive positioning that brought them to where they are today. On the contrary, the pressure they feel most comes from the investors who have sunk trillions into their ability to deliver sooner rather than later.
Even if the plug is pulled on general access to the most powerful AI systems, the technology will still be out there like nuclear weapons. Some will have them and others won’t. How do you deal with that?
Out of a scenario that is no longer science fiction, Schmidt responded: “If you’re doing powerful training, there needs to be some agreements around safety. In biology, there’s a broadly accepted set of threat layers, Biosafety levels 1 to 4, for containment of contagion. That makes perfect sense because these things are dangerous.
“Eventually, in both the U.S. and China, I suspect there will be a small number of extremely powerful computers with the capability for autonomous invention that will exceed what we want to give either to our own citizens without permission or to our competitors. They will be housed in an army base, powered by some nuclear power source and surrounded by barbed wire and machine guns. It makes sense to me that there will be a few of those amid lots of other systems that are far less powerful and more broadly available.”
The Gap Between Public Anxiety & Investor Euphoria
One reason anxious communities are mobilizing against data centers is that they are the most tangible symbol of the ethereal realm of AI that large constituencies have come to see as a darkening menace on the event horizon.
Too much distance has grown between an increasingly alarmed public and the steady stream of global capital flowing endlessly into further accelerating frontier models from which extravagant profits are expected.
Big Tech’s admission that it needs to “slow down” progress because AI is becoming too dangerous raises, more than it reduces, public apprehension. Whether this scrambles the decisive factor of investor euphoria driving the pace remains to be seen.
Something has to give. A breakpoint is surely coming.
