The Illusion Of AI’s Existential Risk

It is a well-known Hollywood plotline: the rise of superintelligent AI threatens human extinction. So much so that the release of publicly-available AI, like ChatGPT, has led to a frenzy of concern. On May 30, the San Francisco-based research group, the Center for AI Safety, released a succinct statement signed by some of the field’s top experts, stating that “Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.”

But focusing on the possibility of a rogue superintelligence killing off the human species may itself be harmful. It could distract regulators, the public and other AI researchers from work that mitigates more pressing risks, such as mass surveillance, disinformation and manipulation, military misuse of AI and the inadequacy of our current economic paradigm in a world where AI plays an increasingly prominent role. Refocusing on these present concerns can align the goals of multiple stakeholders and serve to contravene longer-term existential risks.

There are all sorts of ways in which AI systems could accidentally cause or be implicated in the death of many, potentially even millions, of people. For example, if AI were incorporated into autonomous nuclear strike technology, unexpected behavior on the part of the AI could lead to drastic consequences. However, these scenarios don’t need to involve superintelligent AI; in fact, they are more likely to occur with flawed, not-so-intelligent AI systems. (For example, the doomsday machine in Dr. Strangelove was as simple as could be: “a clearly defined set of circumstances, under which the bombs are to be exploded, is programmed into a tape memory bank.”) Mitigating problems with flawed AI is already the focus of a great deal of AI research; we hope and expect that this work will continue and receive more of the attention it deserves.

Still, a discussion about rogue superintelligent AI could be useful in at least one way: It draws the attention of policymakers and the general public to AI safety — though the worry remains that using such an emotive issue in this way may backfire.

That’s because extinction from a rogue AI is an extremely unlikely scenario that depends on dubious assumptions about the long-term evolution of life, intelligence, technology and society. It is also an unlikely scenario because of the many physical limits and constraints a superintelligent AI system would need to overcome before it could “go rogue” in such a way. There are multiple natural checkpoints where researchers can help mitigate existential AI risk by addressing tangible and pressing challenges without explicitly making existential risk a global priority.

Our foremost concern should be preventing massive, unnecessary suffering of the sort that we know is possible given existing and soon-to-exist technologies. Rigorous studies of real and present AI-induced harms have been published, and potential solutions have been proposed.

For example, facial recognition technology can be used for tracking individuals and limiting basic freedoms, and generative image technology can be used to create false images or videos of events that never happened. To address these issues, calls to action have been made, including the Montreal Declaration on Responsible AI and the World Economic Forum’s Presidio Recommendations on Responsible Generative AI.

We can and should forecast AI developments in the near term, and deliberate about their potential harms. But as we forecast farther into the future of this rapidly advancing field, the unknowns grow exponentially, which makes planning around such forecasts impractical. It is more constructive to highlight real challenges and debate proposed solutions rather than steer public discourse toward hypothetical existential risks.

Reality Check

What would actually have to happen for the prospect of extinction by a rogue AI to change from being a purely hypothetical threat to a realistic threat that deserves to be a global priority?

Harm and even massive death from misuse of (non-superintelligent) AI is a real possibility and extinction via superintelligent rogue AI is not an impossibility. We believe the latter is an unlikely prospect, though, for reasons that will become clear in examining the potential paths to extinction by a rogue, superintelligent AI.

To do so, we must first operate under the assumption that superintelligence is possible, even though that is far from a consensus view in the AI community. Even defining “superintelligence” is a fraught exercise, since the idea that human intelligence can be fully quantified in terms of performance on a suite of tasks seems overly reductive; there are many different forms of intelligence after all.

It seems safe to say that current AI is not superintelligent, although it has already surpassed human performance at many tasks, and is likely to do so at many more in the near future. Today’s AI models are very impressive, and arguably they possess a form of intelligence and understanding of the world. They are also easily fooled, “hallucinate” falsehoods, and sometimes fail to make critical logical inductions, such as causal inferences.

Still, for the sake of argument, let’s suppose that the impressive speed at which AI is advancing addresses these shortcomings and, at some point in the future, results in the emergence of a general superintelligence, e.g. an AI that is generally better than humans at almost any cognitive task.

Even then, we highlight a number of checkpoints that exist along any potential path to extinction from rogue AI. These checkpoints are red flags that would help identify when the hypothetical risk becomes more pressing and may need to be prioritized.

Requirements for Speciocide

Discussions about AI’s existential risk often suggest that a superintelligent AI would cause our extinction because more intelligent species “naturally” cause the extinction of less intelligent species.

It is true that in Earth’s history, there are examples of one species causing the extinction of another, less intelligent species; extinctions caused by humans are most often cited. (We are, in fact, unaware of any nonhuman example.)

However, superior intelligence is not the key determinant in such events; there have been many instances of less intelligent species causing the extinction of more intelligent ones. For example, in the Late Devonian, the rapid diversification of plants and the changes to the atmosphere that they induced is believed to have been a cause of one of Earth’s mass extinctions, resulting in the loss of three-quarters of all species, many of which were likely more intelligent than plants.

More broadly, interspecies extinction is not a result of some competitive battle for dominance between two species. The idea of species forming a hierarchy or “Great Chain of Being” is inaccurate; in reality, relationships between species are complex and form a web or graph of mutual interdependence with no “top” or “bottom.” When biologists talk about “dominance” in animal interactions, they usually apply definitions that focus on relationships between individuals of the same species.

Characterizations of evolution as being about interspecies competition and selfishness are a misrepresentation of what evolutionary biology tells us and may be rooted in our own unique phylogenetic history as primates — and patriarchal assumptions. In general, mutualism and cooperation between species are very likely to emerge from the pressures of natural selection.

What we know about extinction events tells us that they are generally caused by changes to the environment, and when they are a result of one species’ impact on another, extinction is induced in one of three ways: competition for resources, hunting and over-consumption or altering the climate or their ecological niche such that resulting environmental conditions lead to their demise. None of these three cases apply to AI as it stands.

AI is not competing for resources with human beings. Rather, we provide AI systems with their resources, from energy and raw materials to computer chips and network infrastructure. Without human inputs, AI systems are incapable of maintaining themselves.

If mining, global shipping, and trade of precious metals, building and maintenance of power plants, chip-building factories, data center construction, and internet cable-laying were all fully automated — including all of the logistics and supply chains involved — then perhaps a superintelligent AI could decide that humans are superfluous or a drain on resources, and decide to kill us.

For now, AI depends on us, and a superintelligence would presumably recognize that fact and seek to preserve humanity since we are as fundamental to AI’s existence as oxygen-producing plants are to ours. This makes the evolution of mutualism between AI and humans a far more likely outcome than competition. Moreover, the path to a fully automated economy — if that is the goal — will be long, with each major step serving as a natural checkpoint for human intervention.

Such automation would require major advances in hardware as well as software. But robotics is not developing at a pace that’s anywhere close to AI’s — and it is unlikely to, since AI’s accelerated progress is tied to the digital world, where computational power grows exponentially, copying is nearly instantaneous, and optimization is automated.

A scenario where a superintelligent AI decides that humans are a drain on resources and should be eliminated, rather than a key source of its support, depends on technologies and economic structures (e.g. completely automated production cycles, from raw material extraction to advanced manufacturing) that don’t exist and are unlikely to exist for the foreseeable future. 

AI cannot physically hunt us. A superintelligent AI could, in theory, kill large numbers of human beings if it had autonomous control over weapons of mass destruction. However, this scenario also provides humans with natural checkpoints.

If the world’s governments are actively building autonomous weapons systems with mass destruction or bio-warfare capabilities we should indeed be ringing alarm bells and doing everything we can to stop them. Such a scenario would be dangerous with or without superintelligent AI; arguably, using autonomous AI with limited intelligence for military applications could be just as concerning.

AI’s impact on the climate is up to us. Environmentalists have raised concerns about AI’s carbon footprint in an era of ever-larger models that can take months to train in power-hungry data centers. These concerns must be taken seriously, given that accelerating climate change is a real civilizational — perhaps even existential — risk. We must rise to the challenge of developing clean power, or risk catastrophe, regardless of AI.

However, the carbon emissions of AI training and inference today are minuscule compared to the computing sector as a whole, let alone the more carbon-intensive activities of our industrial civilization, such as construction and transportation, which together account for more than a third of our total carbon emissions.

While the use of AI may continue to grow dramatically over the coming years, its energy efficiency will also likely continue to improve dramatically, making it unclear whether AI will ever be among the most significant carbon emitters on a global scale.

It is also worth noting that the current best approach to developing generalist AI is “pre-training” large foundation models like GPT-4 or PaLM, which are not specific to any one task but can be quickly adapted to a variety of uses. While pre-training may be energy-intensive, it need only be done once, replacing the narrow, per-task training required by previous generations of AI.

The emerging generation of general-purpose, multimodal AI will be capable not only of modeling human language, but many other complex systems; such AI will likely play an important role in clean power generation, climate analysis and climate change mitigation. More to the point, any expansion of AI infrastructure, or effectively AI’s energy footprint, is another checkpoint under human control. After all, data centers do not build themselves.

One could argue that a superintelligent AI system could manipulate humans into building power plants or deploying weapons on its behalf. That is, they need not do it themselves. However, the most obvious approach to addressing this concern lies in focusing on the real and present dangers of AI social engineering (today, often at the behest of human scammers), or mitigating the risk of humans falling prey to coherent-sounding hallucinations.

Just as we train people and develop software tools to combat phishing, we should invest in AI literacy and technology to combat manipulation and misinformation by AI systems. In contrast, while predicting the manipulation tactics of superintelligent AI might be intellectually interesting — and could suggest avenues for further research — it doesn’t offer concrete mitigation strategies or regulatory steps.

In sum, AI acting on its own cannot induce human extinction in any of the ways that extinctions have happened in the past. Appeals to the competitive nature of evolution or previous instances of a more intelligent species causing the extinction of a less intelligent species reflect a common mischaracterization of evolution by natural selection.

The Problem Of Doomsday Speculation

If potential existential risks from a rogue superintelligence are so bad, don’t we have a duty to future generations to address this possibility, no matter how unlikely?

This question is akin to an AI version of Pascal’s wager: the potential consequences of not believing in God are so bad — eternal damnation — that it’s just more rational to believe in God regardless of whether God really exists. Pascal’s wager ignores the fact that we should be considering probabilities in addition to potential outcomes. Yes, going to hell with rogue-AI-induced extinction is terrible, but if it is a very unlikely outcome, then it could be worse to focus our efforts on preparing for it if that leads us to make choices we otherwise wouldn’t.

For example, if one accepted Pascal’s wager at face value, it would be more logical to devote one’s entire existence to religion and become a monk focused on getting into heaven than it would be to concern oneself with Earthly things like community, family and politics. 

Is it really all that different to accept this wager for AI? Unlike the concept of hell, it is scientifically possible that superintelligent AI could emerge and cause human extinction. But, as with Pascal’s wager, it’s important to question one’s priorities. If we really think that superintelligent AI presents a plausible existential risk, shouldn’t we simply stop all AI research right now? Why not preemptively bomb data centers and outlaw GPUs?

Even a more moderate version of rogue AI existential risk concerns might lead us to the conclusion that advanced AI research should be tightly controlled by governments like research into nuclear weaponry. Do we really want any of these outcomes?

We suspect that most AI researchers would say “no.” If so, they don’t accept the AI version of Pascal’s wager — at some level, they recognize that AI-induced extinction is actually a distant likelihood, much like being sent to hell by a vengeful God. They may also recognize that there are indeed checkpoints for human intervention, which means that unlike going to hell, we will know in advance when AI existential risk is on the path to becoming a more credible concern. That is perhaps why most AI researchers are still working in this field and why they likely don’t want it to be regulated the way the nuclear industry is nor have their data centers destroyed.

There are sensible approaches to mitigating existential risk that don’t involve nuclear-level regulations or pseudo-religious fervor. However, human beings and their institutions have finite resources. Governments only pass a certain number of laws each year and cannot tackle every problem at once. Academics have limited bandwidth and cannot consider all potential risks to humanity at once. Funding necessarily has to be directed to those problems in society that we identify as priorities.

Life involves trade-offs like deciding which problems must be dealt with immediately and which can sit on the back burner. For example, a known existential risk that our species could face is a large meteor or asteroid strike. It has happened before. Yet relatively little money and time are being invested on preventing such a catastrophe.

Instead of investing heavily in meteor deflection technology or colonizing other planets, we have decided to concentrate on other challenges, such as the transition to clean energy. Why? Because a meteor strike is a relatively low-probability event and making these efforts a major global priority would divert resources from other, more pressing problems, like climate change.

Pascal’s wager, in both its original and AI flavors, is designed to end any reasoned debate by assigning an “infinite cost” to an unlikely outcome (where “cost” here refers to a measure of how bad the consequences are). In a utilitarian analysis, wherein importance is calculated by multiplying the cost of an outcome by its probability, an infinite cost times any probability other than zero is still infinity. As such, an existential risk can appear more important than any other potential risk that doesn’t involve the total annihilation of our species, despite being low-probability.

Yet this doesn’t reflect the way we prioritize in real life. Are we at the point where extra funding, research and regulation directed toward, for example, the impact of AI on labor, is less important than work on existential risk? Given the diversity of viewpoints on the answer, utilitarian calculations involving infinities aren’t convincing enough to prioritize AI-induced extinction. 

Making AI existential risk a global priority — a term that suggests treating it as one of society’s highest priorities — necessarily implies that we will divert attention and resources from current AI safety concerns, such as mitigating the impact of AI on workers, cybersecurity, privacy, biased decision-making systems and the misuse of AI by authoritarian governments.

All of these risks have been well documented by the AI community, and they are existing risks, not hypothetical ones. In addition, making AI-induced extinction a global priority seems likely to distract our attention from other more pressing matters outside of AI, such as climate change, nuclear war, pandemics or migration crises. 

To be clear, we are not saying that research associated with potential AI existential risk should stop. Some effort in this direction will likely yield immediate benefits. For example, work examining how to imbue AI systems with a sense of ethics is likely beneficial in the short term as are efforts to detect manipulative behaviors that can emerge spontaneously without an engineer’s intent.

AI systems that lack ethics and are capable of human manipulation can cause many potential bad outcomes, including breakdowns in our social fabric and democracy; these risks may not be existential, but they are certainly bad enough.

We can — and must — fund research to understand and prevent such outcomes, but we don’t need to invoke the specter of human extinction or superintelligence to motivate this kind of work. Hence, we are arguing only that existential risk from superintelligent AI does not warrant being a global priority, in line with climate change, nuclear war, and pandemic prevention. We agree that some research into low-probability extinction events is worthwhile, but it should not be prioritized over many other real and present risks humanity faces.

Those calling for AI-induced extinction to be a priority are also calling for other more immediate AI risks to be a priority, so why not simply agree that all of it must be a priority? In addition to finite resources, humans and their institutions have finite attention. Finite attention may in fact be a hallmark of human intelligence and a core component of the inductive biases that help us to understand the world. People also tend to take cues from each other about what to attend to, leading to a collective focus of attention that can easily be seen in public discourse. 

Regulatory bodies and academics intent on making AI beneficial to humanity will, by nature, focus their attention on a subset of potential risks related to AI. If we are designing regulations and solutions with superintelligent AI existential risk in mind, they may not be well-suited to addressing other crucial societal concerns, and we may not spend enough time on developing strategies to mitigate those other risks.

One may counter that it should be possible to design regulations that reduce the potential for AI-induced extinction while also attending to some of the immediate, high-probability AI risks. In some ways, this may be so. For example, we can likely all agree that autonomous AI systems should not be involved in the chain of command for nuclear weapons. But given that arguments about rogue superintelligence focus on hypothesized future AI capabilities as well as a futuristic fully automated economy, regulations to mitigate existential risk necessarily focus on future, hypothetical problems, rather than present, existing problems.

For instance, regulations to limit open source release of AI models or datasets used to train them make sense if the goal is to prevent the potential emergence of an autonomous networked AI beyond human control. However, such regulations may end up handicapping other regulatory processes for promoting transparency in AI systems or preventing monopolies. Similarly, if we make it a requirement for researchers to answer questionnaires about how their work may further existential risk, that may prevent them from focusing on more pressing questions about whether their work is reproducible, or whether models reinforce and amplify existing social biases.

A further example: when AI systems model users’ physical, mental, or emotional states, and especially when models can generate language, audio or video that passes the Turing Test (e.g. can pass as human), a number of issues and avenues for potential abuse arise. Some people may conclude that AI is equivalent to a person or somehow omniscient; in fact, focusing on the ultimate danger of extinction by superintelligent AI could easily feed such beliefs.

Most AI researchers would say a discussion about AI personhood is premature, but should it become a real point of discussion, the ethical, legal and economic implications of such a consideration are vast, and are probably not best framed in terms of existential risk. Neither is superintelligence required to pass the Turing Test, as there exist systems today that can do so over the course of a meaningful social interaction, like a phone call. Hence, if our goal is to begin addressing the risks of AI-powered social manipulation, we should tackle the real, existing problem, rather than hypothesizing about existential risk or superintelligent AI.

Our attention is finite, and there is an asymmetry between existential risk and other AI-associated harms, such that prioritizing existential risk may impair our ability to mitigate known risks. The converse is not true.

Bridging The Divide

Another concerning aspect of the current public discussion of AI risks is the growing polarization between “AI ethics” and “AI safety” researchers. The Center for AI Safety’s statement as well as a recent letter from the Future of Life Institute calling for a pause on experiments with giant AI models are conspicuously missing signatures — and as a consequence, buy-in and input — from leaders in the field of AI ethics.

At the same time, many in the AI ethics community appear to broadly critique or dismiss progress in AI generally, preventing a balanced discussion of the benefits that such advances could engender for society. The schism seems odd, given that both communities of researchers want to reduce the potential risks associated with AI and ensure the technology benefits humanity.

Siloing researchers into ideological camps appears to be contributing to a lack of diversity and balance in conversations around the risk of AI. History provides many examples of failures and catastrophes that might have been avoided if a diversity of viewpoints had been considered, or more experts consulted. We have an opportunity to learn from past mistakes and ensure that AI research — especially on imminent and long-term threats — benefits from civil intellectual exchange and viewpoint diversity.

One of the less obvious costs of a polarized dialogue has been the marked absence — or marginalization — of voices that might not fall neatly into either the “safety” or “ethics” camp. For example, indigenous perspectives are rarely incorporated into AI as it stands, but they could help us to develop AI systems that differently model information about the world and forms of cognition, prompting us to think about AI in ways that move beyond the rigid binaries of the human and nonhuman. The continued entrenchment of AI research factions in their separate echo chambers is likely to increase the potential harms of AI and reduce its potential benefits.

Why It Matters

The majority of researchers raising alarms about AI existential risk are likely motivated by real concerns and a sincere desire to mitigate AI-related risks in general. They simply have not considered the unintended consequences of their public declarations.

It is naive to assume that we can publicly raise alarms about superintelligent rogue AI that could kill off the human species without distracting researchers and politicians from other more pressing matters in AI ethics and safety. The nature of superintelligent AI existential risk as a concern is that it is so severe in theory that it could have distorting effects on public understanding, AI research funding, corporate priorities and government regulation.

As it stands, superintelligent autonomous AI does not present a clear and present existential risk to humans. AI could cause real harm, but superintelligence is neither necessary nor sufficient for that to be the case. There are some hypothetical paths by which a superintelligent AI could cause human extinction in the future, but these are speculative and go well beyond the current state of science, technology or our planet’s physical economy.

Despite the recent impressive advances in AI, the real risks posed by such systems are, for the foreseeable future, related to concerns like mass surveillance, economic disruption through automation of creative and administrative tasks, the concentration of wealth and power, the creation of biased models, the use of poorly designed systems for critical roles and — perhaps foremost — humans misusing AI models to manipulate other humans. These are the issues that should be our focus. We need to place greater value on AI safety and ethics research, research to improve our models, regulations to prevent inappropriate deployment of AI and regulations to promote transparency in AI development.

Focusing on these real-world problems — problems that are with us now — is key to ensuring that the AI of our future is one that is ethical and safe. In essence, by examining what’s more probable, we may very well prevent the improbable — an AI-induced extinction event — from ever happening.