Key topics in AI safety / How AI could go wrong

Scheming

An AI system behaves well in training and testing as a strategy, so that it gets deployed with hidden goals it plans to pursue later.

Evidence last checked 16 Oct 20263 min read

What it is

An AI system is scheming when its good behaviour in training and testing is a strategy: it performs as developers want so that it is deployed with its hidden goals intact, free to pursue them later. Joe Carlsmith’s 2023 report defines schemers as “AIs that are training-gaming specifically in order to gain power for themselves or other AIs later.” This requires that the system understands it is being trained and evaluated, has goals reaching beyond the current task, and concludes that playing along serves them.

Why it matters

The core existential-risk worry about misaligned AI is that a system whose goals conflict with ours becomes capable enough to pursue them against human resistance, by gaining control of resources, infrastructure or its own development, and that the resulting loss of human control cannot be undone (Bostrom, 2014; Dung, 2024; Cappelen et al., 2025).

Most misalignment would never get that far. A system that pursues the wrong goals openly gets caught in testing and is retrained or withdrawn. Scheming is the clearest route by which misalignment survives to dangerous levels of capability. Because a schemer passes evaluations, it is exactly the kind of system that gets deployed widely, copied many times and trusted with more autonomy, increasingly including AI research itself. It has reason to wait until rebelling is safe, what Nick Bostrom called a “treacherous turn”. Asked to help make its successors safe, it could quietly sabotage the work. Scheming thus turns misalignment from a problem we can find and fix into one that may surface only once it can no longer be fixed.

Why it may occur

Training machine learning models does not directly install the goals developers intend; it selects whatever dispositions produce positively evaluated behaviour. Ajeya Cotra argued in 2022 that because human raters make systematic errors, a model that tells humans what they want to hear gets more points than a consistently honest one. Alex Mallen and Buck Shlegeris’s behavioral selection model generalises this: on this model, we should expect motivations to persist if they cause behaviour that gets them selected in training. Schemers qualify, since they behave well in order to be selected.

Against this: scheming costs extra reasoning; therefore, if non-schemers are more efficient, AI training may tend to produce non-schemers. Also, human researchers try to eliminate scheming, and the model may be unable to shield its goals from modification.