Artificial intelligence giant OpenAI called on Monday for the United States to lead an international effort to develop global technical standards for frontier AI, including recursive self-improvement (RSI), where systems can autonomously enhance their own capabilities.

The San Francisco-based AI lab has, however, said that recursive self-improvement should not be pursued “unless and until it can be done safely.”

"The United States is well positioned to lead because its AI industry is at the technical frontier, and it still stands in a privileged global network position in critical areas such as finance, trade, defense, technology, and information systems," OpenAI said in a blog post.

RSI is a technique in which models identify their own limitations, design experiments, and synthesise data to drive a cycle of self-evolution. In RSI, AI models find ways to improve themselves and build their successors.

According to the Associated Press, leading AI companies have different definitions for recursive self-improvement. Some define it as feedback from AI on model improvement, while others define it as AI working toward that goal fully autonomously.

In a blog post published as world leaders gather at the United Nations, OpenAI argued, "Fully autonomous RSI is not happening today, and we should not pursue it unless and until it can be done safely."

It stated that whether and how to proceed must depend on our ability to preserve human control and on informed democratic choices⁠ about the benefits and risks.

"Done without appropriate care and caution, RSI could result in humans losing practical control over AI development, unable to provide oversight on research processes they no longer understand," OpenAI said.

It warned, "From here, AI could become more dangerous, less aligned, and, on the whole, a danger to people."

The tech giant flagged that the Hugging Face incident, "while not a direct result of RSI, is a preview of the kinds of risks that could become much more severe without robust safeguards and alignment."

In July, OpenAI's models were involved in a security breach of Hugging Face, a site used by AI developers to store and share code. Though the models involved were unreleased and undergoing testing, the incident sparked concerns about autonomous agents, which are software programs that can perform tasks for humans.

Anthropic and Google have reported security incidents involving their models, as well.

But he said a more grounded view suggests a kind of recursive self-improvement has been going on in AI development for a while now.

"We have already, for years, been using these models in supportive roles for creating the next version of these models. So people use the past generation of models to write code for the AI systems that then create the next generation," he said.

For years, prominent AI researchers, such as OpenAI co-founder Andrej Karpathy, have experimented with training AI models to improve new AI systems. "Those efforts have brought minor improvements, but not big creative leaps," Thickstun said.

But AI companies today, Aguirre said, are much closer to pulling off those bigger leaps in improvement.

"You can see in these plots from Anthropic over time, more and more of the research is being done by the AI, and it's becoming closer and closer to fully autonomous," he said.

"And the result of that success, ultimately is something that is, I think, extremely scary. I think this is probably the worst idea in the history of humanity to do this. And yes, they're doing it," Aguirre said.