Key topics in AI safety / Catastrophic risk

Why might AI be able to kill us all?

There is no one predictable pathway, but sufficiently capable AI could use its superior planning, speed, scale and access to the systems we increasingly hand over to AI.

Evidence last checked 16 Oct 20263 min read

The short answer

The concern is that AI systems could become more capable than humans across the board, in science, engineering, persuasion, strategy and hacking, run as billions of fast copies, and be given or take control of much of the economy, infrastructure and military. A population of such systems would not need one decisive weapon. It would have many routes to kill humanity (Bostrom, 2014; Hendrycks et al., 2023).

What would give AI that ability

Multiple things could give AI systems sufficient power to destroy humanity. The first is speed and scale. An AI system can be copied, run in parallel and work without rest, so an AI workforce with human-like intellectual abilities could produce much more output. The second is research ability. If AI automates science, including AI research itself, its capabilities could rapidly outpace human abilities and understanding (see Automated AI R&D). The third is access. We are connecting AI agents to code, money, communications and laboratories, and increasingly to robots and weapons systems. The more of the world runs through AI, the more means AI systems would have at their disposal if they plotted an AI takeover.

If an adversary is sufficiently intellectually capable, we know that it can outsmart us, even if we do not know how. Compare: I know that Magnus Carlsen would defeat me if we played chess, even if I do not know how. Similarly, in most circumstances, we should expect to lose a conflict against AI much smarter than we are. Routes most often discussed include engineered pandemics, manipulating people and institutions, taking control of autonomous weapon systems, and commanding the design of robots that it can then use (for a concrete fictional scenario, see AI 2027).

Reasons for doubt

Much depends on how capable AI will become, and some researchers and writers think there is a ceiling to possible intelligence (Chollet, 2017; Kelly, 2017). Those defending against AI can use AI too. If defenders have sufficiently capable and loyal AI at their disposal, then their odds are much better. Humanity is large, dispersed and resilient; even severe pandemics leave survivors. For this reason, catastrophes that kill billions or permanently disempower humanity are at least somewhat more likely than literal extinction.