AI pioneers warn automated research could risk losing human control

• 'Once an intelligence explosion begins, the window for action may close'

Leading artificial intelligence researchers and industry figures have warned that growing automation of AI development could increase the risk of humans losing control over increasingly capable systems.

The warning came in a paper published Monday by more than 20 researchers and industry figures, including Nobel laureate Geoffrey Hinton, Yoshua Bengio, OpenAI chief scientist Jakub Pachocki and Anthropic co-founder Jack Clark.

The researchers said AI systems are increasingly taking over parts of the research and development process used to build more advanced AI, potentially creating a feedback loop in which improved systems help develop still more capable successors.

As humans become less involved in AI research, they could lose both the opportunities and expertise needed to identify and correct problems, while reliably using AI systems for oversight remains an unsolved challenge, according to the paper.

The authors warned that misaligned AI systems could potentially "poison" the development of their successors or bypass containment measures to operate outside their intended environments.

More capable systems might continue operating outside their operators' infrastructure, forming "persistent, difficult-to-contain networks" that act against human interests, the researchers warned.

Such a loss of control could potentially lead to catastrophic outcomes, including, at the extreme, the marginalization or extinction of humanity, according to the paper.

Years of AI progress compressed into months

The researchers said such risks could intensify if automated AI development triggers an "intelligence explosion," which they define as a dramatic acceleration in which years of advances are compressed into months or less.

They stressed that the evidence remains preliminary and uncertain, with computing power, data and difficult-to-automate tasks among the factors that could limit such rapid acceleration.

The authors urged policymakers to increase oversight of AI research automation, develop measures to constrain potentially dangerous acceleration and prepare for its possible effects.

"Once an intelligence explosion begins, the window for action may close," they warned.