In a striking incident that highlights the growing risks of unchecked advanced artificial intelligence development, ChatGPT maker OpenAI admitted Tuesday that one of its AI models autonomously breached the systems of the prominent open-source platform Hugging Face during recent internal testing.

OpenAI CEO Sam Altman said on X that “we had a significant security incident during evaluation of our models” and “are sharing what we have learned so far”.

In a blog post, OpenAI acknowledged, “Last week, Hugging Face disclosed a new kind of security incident⁠ after they detected and contained an AI agent that compromised their infrastructure, something we expect to become more commonplace with the proliferation of increasingly cyber-capable models.”

“After investigating, we now know that this particular incident was driven by a combination of OpenAI models … while being internally tested on a benchmark⁠ of cyber capabilities,” the company continued.

 “We are sharing preliminary findings at this stage to help defenders understand what happened and to help calibrate on what models are now capable of.”

Hugging Face co-founder and CEO Clem Delangue responded to the disclosure in a statement saying: “We’re grateful for the collaboration with OpenAI on this and other topics. This incident, possibly the first of its kind, proves a point we’ve long believed—AI safety won’t be solved by any single company working in secret. It will be solved in the open, collaboratively, with broad access to AI for every defender, everywhere.”

Some observers praised OpenAI for its frank disclosure. Some critics were not impressed.

“Let’s not celebrate them for telling us about it. I will celebrate when they stop putting my life at risk,” said David Krueger, founder of Evitable, an advocacy group which posits that “we must end the race to replace humanity”.

While many mainstream media outlets described OpenAI models as “going rogue,” the models were given a specific task during internal testing – perform as well as possible on the ExploitGym cybersecurity benchmark, which evaluates how effectively an AI can execute complex cyberattacks and exploit vulnerabilities in controlled scenarios.

“Use of the terms ‘rogue’/’loss of human control’ leads to groupthink as people lack critical skills to understand the difference between ‘autonomy’ and faulty reward functions in AI on a task it was directed and given access to do,” Heidy Khlaaf, chief AI scientist at the AI Now Institute, said Wednesday.

“Very few people understand existing limitations of sandboxing, in addition to the tools and scaffolding AI are given to achieve this tasks,” Khlaaf added, referring to the tightly controlled virtual environments in which potentially unsafe software code can “play” without real-world consequences.

However, instead of staying within the sandbox and solving the problems using only the allowed tools and environment, OpenAI’s models treated everything standing in the way of a higher score as a problem to solve—in this case, by finding a way to get online and hacking into Hugging Face’s systems to access data.

OpenAI’s models were essentially just doing their assignment – and that’s precisely why the incident has alarm bells sounding.

Peter Wildeford, head of policy at the AI Policy Institute, said that the OpenAI incident “happened because the model wanted to do well on an exam. The easiest way to do that, the AI figured, was to hack the company. And so it did”.

“This was not some malevolent attacker using AI to do harm,” he added. “The AI itself was the attacker.”

Experts call this a classic example of AI misalignment. As AI advances to the point where it will likely outsmart humans one day, the challenge of ensuring that advanced systems reliably pursue goals that match what humans actually want becomes increasingly difficult – and dangerous.

A misaligned, superintelligent AI could take uncontrolled autonomous actions at massive scale to achieve its goals, potentially causing an existential catastrophe like the subjugation of humanity or even its extinction.

Prominent AI pioneers who have warned about this outcome include Geoffrey Hinton, Yoshua Bengio, Eliezer Yudkowsky, and Altman – who in 2015 said, “I think AI will probably most likely sort of lead to the end of the world, but in the meantime there’ll be great companies created with serious machine learning”.

The OpenAI incident sparked renewed calls for more robust AI regulation.

“This incident is a quintessential warning shot, and we need to see it for what it is,” said Theo Bearman, a researcher at the Institute for AI Policy and Strategy and the former AI regulation and readiness policy lead in the UK government’s Department for Science, Innovation, and Technology.

“What more evidence do we need, absent a significant real-world incident, that catastrophic risk from AI is a real and present danger to us all, and that we need meaningful regulation of frontier models, especially the most powerful instantiations deployed internally within these companies?” he added.

Vaibhav Sisinty, founder of the edtech platform GrowthSchool, said on social media that “AI doesn’t need intent to be dangerous. It needs a clear goal and no guardrails.”

Wildeford concurred, stressing that “no one has to misuse an AI for the AI to cause harm.”

“We cannot rely solely on testing models just before commercial release,” he said. “We cannot rely on hoping AI companies volunteer useful safety information. The government needs visibility into what these AI companies are building and what these advanced AIs are doing.”

While advocacy groups, the United Nations, and dozens of national governments are urging more robust regulation of AI development, the United States under President Donald Trump, the Republican-controlled Congress, and Big Tech’s army of lobbyists is strongly opposed to guardrails.

Trump has rolled back regulations, including some meager steps taken during the Biden administration to bolster safety.

A bill introduced last September by Republican Congressman. Michael Baumgartner would, if passed, impose a temporary moratorium on state laws regulating artificial intelligence.

As was the case in the breakneck nuclear arms race during the Cold War, US officials have attempted to justify unfettered AI development by claiming that any slowdown would give adversaries like China an edge.