Key topics in AI safety / How we test and monitor AI
Chain-of-thought monitorability
Many AI models write out their reasoning before they act. Reading it is one of the few ways to catch misbehaviour in advance, but this crucial safety tool could disappear.
What it is
Many current AI models “think out loud” before answering. They produce a stream of intermediate reasoning on something like a scratchpad, often long, in something close to ordinary language. This is called a “chain of thought”. Chain-of-thought monitorability is the degree to which reading that reasoning tells us what the model is actually doing and why, including when it plans to cheat, deceive or sabotage.
Why it matters
Most of what happens inside a neural network is opaque. This is why chain-of-thought monitoring is a valuable tool. For hard multi-step tasks, current models cannot do all the reasoning internally and have to write it down, so important information often appears in text that a monitor, human or AI, can read. Models discovered performing undesired behaviour (e.g. reward hacking) in evaluations have often been caught because their reasoning revealed their intentions, sometimes straightforwardly as in words like “Let’s hack” (Baker et al., 2025). A 2025 paper by more than 40 researchers from OpenAI, Anthropic, Google DeepMind, the UK AI Security Institute and elsewhere called chain-of-thought monitorability “a new and fragile opportunity” for AI safety (Korbak et al., 2025). For catastrophic risk, it is one of the few tools that could catch a misaligned system planning something before it acts.
Why it is fragile
Three things could close the window. First, training pressure: OpenAI found that penalising “bad thoughts” during training produced models that kept misbehaving but learned to not reveal this in their chain of thought (Baker et al., 2025). Second, unfaithfulness: models do not always write down what actually drives their answers. Anthropic found that reasoning models given a hint mentioned using it only a minority of the time, 25% for Claude 3.7 Sonnet (Chen et al., 2025). Third, architecture: model designs that let them reason internally in a continuous, non-verbal form, sometimes called “neuralese”, would reduce the need to write reasoning down, perhaps making it unnecessary altogether (Linch’s explainer). Each may be tempting: When the model behaves badly, it seems natural to try to improve it by punishing it for having “bad thoughts”, and neuralese may make models more capable or efficient. But taking these steps would undermine a crucial safety tool.
The evidence so far
Last checked 16 October 2026
In January 2026 the Frontier Model Forum, the industry body of the leading labs, recommended that developers avoid training chains of thought to “look nice”, prefer architectures that require written reasoning until interpretability catches up, and publish monitorability evaluations (FMF brief). These are norms qualified with “all else equal”, not binding commitments.
The first real test came in September 2026. Reporting based on an anonymous source said OpenAI’s GPT-6 Astra uses “recurrent depth”, letting it do more reasoning internally. OpenAI denied it is a neuralese model and said its computational depth is within a factor of two of GPT-4 (Transformer). Its system card reports that, mostly in tests where it was told to evade monitoring, Astra controls its chain of thought better than its predecessor and is less likely to write down incriminating information, though it found no evidence of hidden encoded reasoning. The safety researcher Ryan Greenblatt said that, if the reports were accurate, it might be the worst development for AI safety to date (Transformer).
Research continues to show both promise and limits. Monitors that read the chain of thought catch far more misbehaviour than those that see only actions (Baker et al., 2025). When reasoning is genuinely necessary for a task, models struggle to evade monitors (Emmons et al., 2025), though a recent paper argues monitoring can fail even then (Bürger et al., 2026).
Chain-of-thought monitoring works reasonably well on current models and is a valued layer of defence. Whether it survives depends on design choices that developers have not committed to.
Related entries
Sources
- Wikipedia. Reward hacking.en.wikipedia.org
- Bowen Baker et al. (2025). Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation.arxiv.org
- Tomek Korbak et al. (2025). Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety.arxiv.org
- Yanda Chen et al. (2025). Reasoning Models Don't Always Say What They Think.arxiv.org
- Linch. What is neuralese and why is it bad?linch.substack.com
- Frontier Model Forum (2026). Issue Brief: Chain of Thought Monitorability.frontiermodelforum.org
- Shakeel Hashim (2026). What is neuralese and why is everyone so concerned about it? Transformer.transformernews.ai
- OpenAI (2026). GPT-6 Astra System Card: metagaming and alignment faking in evaluations.deploymentsafety.openai.com
- Scott Emmons et al. (2025). When Chain of Thought is Necessary, Language Models Struggle to Evade Monitors.arxiv.org
- Lennart Bürger et al. (2026). Chain of Thought Monitoring Can Fail Even When CoT Is Necessary.arxiv.org
Spotted an error or a newer result? Please get in touch through the About page.