Key topics in AI safety / Catastrophic risk

Automated AI R&D

AI systems are doing a growing share of the work of building better AI, which could make technical progress much faster, and decouple it from human oversight and response.

Evidence last checked 16 Oct 20263 min read

What it is

Automated AI R&D (research and development) means AI systems doing the work of building better AI systems: writing training code, running experiments, analysing results and proposing new methods. It is not merely a future possibility but also a present trend. The open questions are how far and how fast it goes.

Why it matters

The worry is a feedback loop. If AI can do AI research, each improvement in AI also speeds up the research that produces the next improvement – so-called “recursive self-improvement”. This could lead to a radical increase in the speed of AI development, quickly giving rise to much more capable systems. The mathematician I. J. Good called this an “intelligence explosion” in 1965. A September 2026 paper whose authors include Turing Award winners Geoffrey Hinton and Yoshua Bengio, OpenAI’s chief scientist Jakub Pachocki, and Anthropic co-founder Jack Clark argues it could compress years of progress into months or less (Chan et al., 2026).

The automation of AI R&D matters for catastrophic risk. It shortens the time governments and developers have to notice risks and fix problems. It puts more of AI development in the hands of AI systems. Therefore, if some systems are not aligned with good values, this might corrupt the increasingly more capable models that they design. Misaligned models could also purposefully sabotage human attempts to align AI in accordance with our values. And if an intelligence explosion is reached, whoever leads AI development at this moment could gain a decisive advantage by rapidly outpacing their rivals. This raises the stakes of racing to AI and intensifies risks of AI leading to power concentration (Davidson et al., 2025).

What might stop or slow an intelligence explosion

AI progress needs computing power, not just ideas, and computing power growth may be harder to speed up for automated researchers. In addition, returns to research effort tend to diminish as easy-to-find research ideas have been used up, and some tasks may resist automation for the foreseeable future. While these frictions cannot fully stop the accelerating effects of automating AI R&D, researchers disagree about how much these may slow them, and whether they will prevent AI from rapidly and drastically outpacing human cognitive ability soon.