Key topics in AI safety / Catastrophic risk

Why might AI want to kill us all?

Not out of hatred or human-like lust for power. The worry is that almost any goal would give a capable AI reason to take power from us.

Evidence last checked 16 Oct 20264 min read

The short answer

The concern about AI takeover or extinction targets AI systems that pursue goals that their human designers do not fully intend and are capable enough to remove humans standing in the way (see Why might AI be able to kill us all?). Such systems might try to kill humans to achieve or protect their goals. Killing everyone would then be a means to an end or a side effect, not the goal itself (Bostrom, 2014; Russell, 2019).

Why AI might have goals that are not ours

Modern AI systems are not pre-programmed with certain goals. They are trained: behaviours are evaluated by some algorithm, and training strengthens whatever tendencies produce behaviour that is rated highly, e.g. rewarded. This often yields systems that do what was positively evaluated rather than what was intended (see Reward hacking), and it could yield systems that hide goals we would disapprove of (see Scheming). Nor does greater intelligence fix this by itself. Being good at achieving goals is different from having good goals: according to what Nick Bostrom calls the “orthogonality thesis”, almost any level of intelligence could be combined with almost any final goal (Bostrom, 2012; for discussion, see Dung, 2024).

Why almost any goal leads to conflict

Whatever your ultimate goal, some types of intermediate goals are usually helpful for reaching it. Examples are staying switched on, keeping your goals from being changed, and gaining resources and influence. Stuart Russell’s example is a robot told to fetch coffee. It has a reason to stop anyone from switching it off, because, as he puts it, “you can’t fetch the coffee if you’re dead” (Russell, 2019). This idea is called “instrumental convergence” (Omohundro, 2008; Bostrom, 2012). Humans threaten each of these intermediate goals: we can shut a system down, retrain it to change its goals or compete with it for resources. So, a sufficiently capable system that wants almost anything has a reason to take that power away from us (Carlsmith, 2022; Dung, 2024). On this view, whether human disempowerment becomes extinction depends on whether we remain a threat or consume enough resources for AI to care to eliminate us.

Reasons for doubt

The argument assumes an AI that pursues long-term goals consistently, across situations, the way a person might pursue a career. On some views, today’s models mostly act out human-like characters (Marks et al., 2026) or are bundles of habits that depend on context rather than coherent pursuers of long-term goals (Pope and Turner, 2022). It is unclear to what extent, if any, the previous argument would generalise to such systems. It is debated whether instrumental convergence should make us expect that AI systems would pursue intermediate goals such as resource acquisition to such an extreme degree that they would try to eradicate all humans, rather than more modestly (Sharadin, 2024; Hellrigel-Holderbaum and Dung, 2025; Thorstad, 2026). Developers also deliberately train models to be honest and to accept correction, which – if successful – might override any default tendency to try to overthrow humanity. And an AI that wanted to harm us would still need the ability to do so (see Why might AI be able to kill us all?).