Anthropic’s research institute has warned that artificial intelligence is advancing so rapidly that future systems may soon be able to improve and redesign themselves without direct human involvement, raising concerns about humanity’s ability to maintain control over the technology.

In a joint article, Anthropic Institute President Marina Favaro and co-founder Jack Clark said so-called “fully recursive self-improvement” could unlock major breakthroughs in science, medicine and technological innovation. However, they cautioned that the same capability could also introduce significant risks if AI systems become capable of creating increasingly advanced successors on their own.

The authors argued that such developments would make AI safety, oversight and behavioral control far more critical, adding that the industry may be closer to achieving self-improving AI than previously expected.

They called on technology companies to consider slowing or temporarily pausing the development of the most advanced AI systems, giving researchers more time to assess potential societal risks and establish reliable mechanisms for human intervention should systems behave unpredictably.

Anthropic launched its new institute earlier this year to study the societal challenges associated with increasingly powerful AI models.

Speaking to CNN, Clark urged the technology sector to build what he described as a “braking mechanism” for AI development. Comparing the industry to a vehicle equipped only with an accelerator, he said future safeguards may be necessary to control the pace of progress when risks emerge.

Clark also highlighted the difficulty of verifying, understanding and trusting the behavior of advanced AI systems, describing the issue as one of the most serious long-term challenges facing the industry.

Despite intense competition among leading AI firms, including Anthropic, OpenAI and other major players, Clark said cooperation on safety standards remains achievable. He pointed to Cold War-era agreements that helped curb the nuclear arms race as an example of how rivals can work together to manage technologies with potentially global consequences.