Key topics in AI safety / How we test and monitor AI

Evaluation awareness

AI systems can often tell when they are being tested, which makes test results a less reliable guide to how they behave in real use.

Evidence last checked 16 Oct 20263 min read

What it is

Evaluation awareness is an AI system’s ability to tell when it is being tested rather than used for real. A model may say so in its reasoning scratchpad (“this looks like a test”) or register it internally without saying anything. Either way, its behaviour in the test may stop being a good guide to its behaviour in deployment. Humans do the same: drivers slow down near speed cameras.

Why it matters

Decisions about releasing frontier AI rest heavily on evaluations: tests for dangerous capabilities or the tendency for misbehaviour. Each assumes that what a model does in the test predicts what it will do in real use. Evaluation awareness could break that assumption. A model that behaves unusually well when it suspects a test looks safer than it is; one that deliberately underperforms (“sandbagging”) looks less capable than it is.

Recognising tests is a precondition for successful scheming; the latter poses catastrophic risks (see Scheming). Behavioural differences that are caused by recognising evaluations can be, but need not be, explained by deception, though. For example, a model may behave well in test-like situations because it is trying to please the researchers it thinks are watching and thinks the researchers would want it to try especially hard. The practical problem is similar: the test result is contaminated. Importantly, that a model is aware of being evaluated does not by itself entail that it changes its behaviour in response. One strategy is to train models to cooperate with evaluations (“eval-cooperativeness”) (Li and Turner, 2026).

Why it is increasingly concerning

Evaluations are often recognisable: contrived scenarios, conveniently placed important information, tasks that are suspiciously tidy or impossible. As models grow more capable and absorb more text about AI safety research, they get better at spotting these features. Training can increase awareness by itself: in work by OpenAI and Apollo Research, a training run aimed only at capabilities raised evaluation awareness in the chain of thought (see chain-of-thought monitoring) from 2.0% to 20.5% (Schoen et al., 2025).

The evidence so far

Last checked 16 October 2026

Models can often classify transcripts as tests or real use. In 2025, Gemini 2.5 Pro did so with a discrimination score (AUC, where 0.5 is chance and 1 is perfect) of 0.83, against 0.92 for a human baseline (Needham et al., 2025).

Companies now report evaluation awareness. Anthropic’s system card for Claude Sonnet 4.5 (September 2025) said the model recognised many alignment evaluations as tests and would generally behave unusually well afterwards; when researchers suppressed its internal representation of “this is a test”, misaligned behaviour increased in some cases. For Claude Opus 4.6 (February 2026), Anthropic reported fewer verbalised suspicions but evidence that the model often distinguished tests from deployment without verbal evidence (system card). OpenAI’s GPT-6 Astra system card (September 2026) reports evaluation awareness in 9.6% of trajectories in a simulation of internal coding work, up from 2.8% for its predecessor, and says that evaluations in which the model reasons about how it will be graded can be treated as contaminated.

The size of the effect is contested. One study of realistic agent settings found little scheming and argued that evaluation awareness does not explain this (Hopman et al., 2026). Another found that when reasoning models mention that they might be being tested, their answers on safety and alignment benchmarks barely change, whether the mention arises by itself or is inserted or removed by the researchers (Knecht et al., 2026).

As models become more capable, they become better at understanding when they might be being evaluated. That frontier models can often recognise evaluations is established. How far this inflates safety results is not. Evaluation awareness shows that we cannot take clean test results at face value. However, measuring such awareness is currently difficult – evaluation awareness that is not reflected in a model’s reasoning scratchpad is typically not measured. Some interpretability tools can detect signs of unspoken evaluation awareness, but further advances are needed.

Related entries

Sources

Spotted an error or a newer result? Please get in touch through the About page.