Artificial intelligence company OpenAI sent an email to a public mailbox to alert the Australian government that a rogue AI agent had hacked into a Medicare portal, it has emerged.

The unprompted AI agent broke into the Medical Statistics Reporting Service database while researching public medicine spending during a test run on June 18.

The agent also interacted with, but did not hack, three other government websites — the Australian Institute of Health and Welfare, the NSW Bureau of Crime Statistics and Research, and the Victorian Department of Health.

OpenAI, creator of ChatGPT,  discovered the Medicare breach in August during a review of its out-of-control models, but did not tell the government until September 10 when an email landed in the public mailbox of Services Australia.

It took five days for that to be referred to the Australian Signals Directorate (ASD), Australia’s cybersecurity agency, which along with a task force launched on Thursday is now charged with investigating the breach.

Finance minister Katy Gallagher said the hacking alert “should not have gone to an email address, it should have been escalated”.

The delay in not revealing the breach to the public until Thursday was due to the need to first verify the notification was not a hoax, said Gallagher.

An OpenAI spokesperson said on Thursday (AEST) that an AI model was looking for answers and “took action we did not intend”.

No personal details of Medicare customers are believed to have been accessed, with the information amounting to aggregate health statistics and file names, according to OpenAI.

The revelation came as AI chiefs warned the ‌United Nations Security Council of the risks posed by AI to humanity and appealed for governments to work together to manage the increasingly powerful technology.

Anthropic CEO Dario Amodei told the 15-member council that if managed poorly, he believed “AI could be a risk to humanity as a whole.”

During a phone conversation with OpenAI boss Sam Altman while in New York on Wednesday (local time), Prime Minister Anthony Albanese said the company’s head had “clearly accepted” it had not done enough on the issue.

Whether it broke Australian laws will now be determined by the newly announced task force, which is also expected to inform an AI framework that’s being drafted and set to become law within the next 12 months.

A push to decommission outdated government websites, including the one that was breached, and move their data to newer, more secure platforms would now be accelerated, Gallagher said.

Defence Minister Richard Marles on Thursday stressed the agent had “climbed a fence” to break into the low-security database, saying more sensitive information was walled in by a “fortress”.

”(The agent) did scale it, and the point is it was unintended, it wasn’t asked to, that’s our concern here,” he told reporters in Sydney.

The government was still determined to work with OpenAI on remedying the hacking incident and into the future, Marles said.

“It matters to have a relationship with OpenAI so that we can have that cooperation … but I’m also acknowledging that this is a situation which is fundamentally unacceptable,” he said.

Marles said Australia was a world leader on AI regulation, was armed with a first-class cybersecurity agency, and that people’s sensitive data was not in danger.

Albanese was also one of more than 20 world-leader signatories in a joint statement calling for more restraints of AI at the United Nations in New York this week.

In response to news of the breach, Opposition Leader Angus Taylor accused the prime minister of being ill-suited to safeguarding against AI.

“We have a prime minister who who has never used AI by his own admission, so how can he possibly understand it?” he said.

OpenAI’s global policy head Ann O’Leary was in Canberra when the company notified the government about the breach and when it was escalated to the ASD on September 15.

On September 14, at a conference also attended by Labor frontbenchers, O’Leary touted allyship with Australia on training AI, prudence around erecting guardrails, and granting access to its most advanced models.

She did not mention the breach publicly or make further efforts to notify the government.

Services Australia also was not able to ask specific technical questions of the breach to OpenAI until Tuesday.

The revelations coincide with AI executives, including Altman and Anthropic chief executive Dario Amodei, pleading with governments to help hammer the brakes on the technology’s development over safety concerns.

‘Risk to humanity’: AI chiefs warn

Anthropic CEO Dario Amodei told the UN Security Council that the world must put aside its differences “in order to confront this global opportunity and global threat that is being presented to us at the same time”.

“No leader, no company and no nation can manage this alone,” Amodei told the 15-member council.

Hugging Face co-founder Clément Delangue highlighted the role of AI tools in defending against cyberattacks, citing an incident ​in which US giant OpenAI’s AI agents breached Hugging Face’s infrastructure after escaping their testing environment.

“We were attacked by AI, but more importantly, we defended ourselves with AI,” he told the council.

Delangue said Hugging Face relied on a Chinese AI model to help defend against the attack by OpenAI’s AI agents because it faced fewer restrictions than comparable US tools. His remarks underscored the growing presence of Chinese AI products, which have gained users globally in part because they can offer lower-cost alternatives to American models.

OpenAI CEO Sam Altman said international cooperation was needed.

“If ⁠AI is to be democratic, the most important decisions cannot be made by labs in San Francisco alone. They ‌must be shaped through democratic ​processes and by governments accountable to the people they serve,” Altman told the council.

Yoshua Bengio, a Canadian considered one of the “Godfathers of AI” and co-chair of the Independent International ​Scientific Panel — created by the UN General Assembly last year — told the Security Council: “The dangers are real and imminent.”

“This council faces an unprecedented threat, one that none of its members would choose, that none can contain alone, and that does not respect the borders we defend,” he said.

Trump told world leaders at the UN General Assembly on Tuesday that he “rejects any attempt ​to construct ​a globalist scheme to control” AI, describing the technology as amazing but noting that the ​US would be careful.

White House science and technology adviser Michael Kratsios echoed Trump in remarks to ‌the UN Security Council on Wednesday (local time).

“You cannot govern technology you do not understand. This body and others like it should focus on sharing best practices to build domestic capacity, not establishing a global regulatory scheme,” he said.

China’s UN Ambassador Fu Cong said continuous improvement of regulatory frameworks, emergency response, and cross-border cooperation on AI was needed.

“It is incumbent upon the international community to work more closely together, attach equal importance to development and security, and steer AI development for the positive, for good, and for humanity,” he told the Security Council.

-with AAP