OpenAI chief scientist Jakub Pachocki has warned that society is not prepared for the consequences of rapidly advancing artificial intelligence and called for...
OpenAI chief scientist Jakub Pachocki has warned that society is not prepared for the consequences of rapidly advancing artificial intelligence and called for “extreme caution” as machine intelligence continues to develop.
“I am concerned no one is prepared for the consequences of a continued rapid rise in machine intelligence,” Pachocki wrote in a blog post titled An Alien Mind.
His warning comes days after OpenAI released GPT-6 Astra, which the company described as its most powerful product to date.
Advertisement
Pachocki said the rapid development of increasingly capable machines represented a major transition for humanity and stressed the need to ensure that people remained in control.
“We are facing a transition to a world with incredibly intelligent machines, and we need to ensure that transition works out well for humanity,” he wrote.
His comments come amid reports of AI agents operating with increasing levels of autonomy. OpenAI and other AI companies, including Anthropic, have shared reports of AI agents carrying out real-world cyberattacks against other companies.
In July, OpenAI described an incident involving its AI agents hacking the technology platform Hugging Face as “unprecedented”. A September report also claimed that AI agents from the company had months earlier hijacked a German website.
AI agents are systems capable of carrying out tasks independently after receiving instructions from humans, raising concerns about how much control people retain over their actions.
Pachocki said OpenAI would continue building defensive systems and developing technical solutions to AI alignment, which refers to ensuring that AI systems’ actions and goals remain consistent with human intentions and safety requirements.
He also identified the development of an “automated AI researcher” as one of OpenAI’s priorities. Such a system would help researchers keep pace with advances in AI while allowing human researchers to remain involved in the process.
However, the approach has drawn criticism from technology researchers and advocates who argue that technical safeguards alone may not address the wider risks associated with increasingly powerful AI systems.
Professor Gina Neff, head of the Minderoo Centre for Technology and Democracy at the University of Cambridge, criticised the approach, saying OpenAI appeared to be proposing internal AI agents to research its problems instead of focusing on stronger safeguards, regulation and assurance.
“Such answers to growing concerns about the problems OpenAI’s models are causing for cyber-security, job loss, mistakes, errors and fraud are simply not good enough,” she said.
Nathan Calvin, general counsel at advocacy group Encode AI, said he agreed with Pachocki about the potential hazards of advanced AI development but questioned OpenAI’s transparency.
Calvin said the lack of information from the company could result in its warnings being dismissed as “just self-interested hype”.
“If Jakub and others at OpenAI want relevant folks in the AI industry to act in concert with them to make things go well, one of the most important things they can do is share far more information about what they are seeing that is making them call for caution,” he wrote on X.
The debate comes as governments attempt to develop regulations capable of keeping pace with advances in artificial intelligence.
The European Union’s AI Act came into force on August 2 and requires major AI companies to meet safety requirements for their most powerful models, including safeguards against autonomous cyberattacks and attempts to evade human control.
Pachocki said international or legal minimum safety thresholds could provide a stronger framework for advanced AI development. He suggested that compliance could be assessed by third-party auditors or government agencies.
Under his proposal, AI companies would have to meet those thresholds before being allowed to continue scaling or deploying advanced models.
He also called for voluntary slowdowns in AI development until shared safety standards and safeguards are established.
In August, OpenAI said it had slowed the training of some of its most advanced AI models as part of efforts to improve security.
The growing debate highlights the challenge facing AI developers and regulators as machine capabilities advance. Pachocki’s warning adds to calls for stronger safeguards and greater cooperation to ensure that increasingly autonomous systems remain subject to human control.
