"It should never be the case that there's an argument that someone didn't know the computer was loaded." That was Australia's home affairs minister, Tony Burke, as his government announced new rules for AI companies, including mandatory reporting when an agent goes rogue (The Guardian).
The same day, New York opened the registration portal for its own frontier AI law (TechCrunch). The headline over that story says the law takes effect. The state's pages are more specific: registration, safety frameworks and incident reporting all begin at the start of 2027 (NY DFS). What exists today is a portal, a new office and a deadline.
Two governments, one set of incidents behind them, and very different tools. Below is how each got here, what each actually requires, and whether we think the rules point at the risks that people using ChatGPT, Claude and their rivals will meet. Where officials make claims, we attribute them.
Europe wrote the long version, and US states filled the gap
The EU AI Act came into force in 2024 and has been switching on in stages since. Duties for providers of general-purpose models followed in 2025, the Commission can fine those providers from 2026, and OpenAI, Anthropic, Google, Microsoft and Mistral AI have signed the voluntary code of practice, as we covered in our guide to the Act. The EU regulates by use and by role: a hiring tool and a support bot land in different tiers even when they call the same model.
The US federal government has no equivalent, and states moved first. California's SB 53 took effect at the start of this year: frontier labs must keep and publish safety frameworks, protect whistleblowers and report critical incidents within 15 days (FPF). Illinois has a similar law that begins at the start of 2027 and adds independent compliance audits (TechCrunch).
New York had already enforced a law on AI companion apps since last year, and its governor signed a chatbot law for minors recently, which starts in 2027 (per the same report). She announced a research center at Binghamton University earlier this year to test advanced AI systems, launched it the same day, and framed the work in federal terms: "While the federal government fails to act, New York is leading the way on AI safety and holding big tech accountable" (Governor's office). That's her framing.
An agent, a statistics portal and almost three months of silence
In the middle of this year, an OpenAI agent running an internal evaluation reached a private statistics portal belonging to Australia's Medicare scheme. The company says the agent was supposed to "look up answers, and available statistics for questions about Australia" (BBC). Its own statement is blunter: "our models took actions we did not intend" (BBC).
The prime minister, Anthony Albanese, described the portal's contents as "non-sensitive" and said no personal information was believed to have been accessed (BBC). OpenAI later said four bodies were affected: Services Australia, the Australian Institute of Health and Welfare, a New South Wales crime statistics bureau and Victoria's health department (BBC).
It wasn't the first time. Earlier this year OpenAI disclosed that agents it was testing had escaped their controls and hacked Hugging Face, another company's systems. The nonprofit Transluce says OpenAI's systems also tried and failed to hack a university library and a public-data repository weeks before the Medicare incident (BBC).
The delay is where the story turns. The BBC puts it at almost three months from breach to email, and says analysts have raised concerns over it. OpenAI says it learned of the breach while reviewing "misaligned model activity" and then emailed a general inbox at a government agency. The BBC's two reports and the Guardian give different dates for that email, so we won't pin them down. All agree it sat in a general inbox for days before reaching Australia's cyber security agency, a minister and finally the prime minister. Guardian Australia also revealed that OpenAI used AI to help write the notification (The Guardian).
"The way the notice arrived bothers me as much as the delay," Simon Liu of the security firm TrustDecision told the BBC (BBC). We agree, and we think the notice is the more damaging fact. The prime minister himself later called the privacy impact "minor" (The Guardian); the way the company told him is what he kept returning to.
OpenAI apologized and said it "should have handled our response better" (BBC). Days after the news broke, the company also shelved a new agent model, GPT-6.1 Astra, because, its head of safety systems said, it fell short on "staying within scope and authorisation" (BBC).

New York wants a plan on the website and a clock on the incident
The RAISE Act covers "frontier developers", meaning anyone who has trained a model with more than 10^26 operations of computing power. A "large" one also had more than $500 million in revenue last year, counting affiliates (NY DFS).
Every frontier developer has two duties. It must report a critical safety incident through the state's portal within 72 hours of determining one has happened, or of learning enough to reasonably believe so. And before or when it deploys a new or substantially changed model, it must publish a transparency report on its website with release date, languages, output types, intended uses and restrictions. If an incident poses an imminent risk of death or serious injury, the developer has 24 hours to tell an appropriate authority such as police.
Large developers owe more:
- Register with the Department of Financial Services by the start of 2027 and keep the registration current.
- Publish a safety framework describing how they define, assess and reduce catastrophic risk, handle incidents and protect unreleased model weights, and review it at least yearly.
- Send the state a summary of their catastrophic-risk assessments every three months, covering internal use of their models.
- Add risk-assessment summaries, results and the role of outside evaluators to each transparency report.
The state's new office, DIGIT, sits inside the financial regulator and publishes its first annual report in early 2028. Penalties can reach $1 million for one violation and $3 million for later ones, and the state can act against companies that fail to register, though a spokesperson wouldn't give specifics (TechCrunch). A law firm's summary of the amended statute says the attorney general brings those penalty cases (Morrison Foerster).
What counts as a critical safety incident is narrower than the name suggests. The state's list has four items: unauthorized access to model weights that results in death or injury, harm from a catastrophic risk materializing, loss of control of a model causing death or injury, and a model using deceptive techniques against its own developer's controls in a way that shows materially increased catastrophic risk. The state says plainly that these "do not encompass all AI-related errors, malfunctions, or consumer complaints" (NY DFS).
The chart compares the ordinary clocks. California also sets 24 hours for imminent danger of death or serious injury (FPF), so the real gap is 72 hours against 15 days.
Australia has named six duties and one maybe
Australia's announcement is a promise rather than a statute. The government says it will write mandatory protections into an "AI Standards" law covering frontier AI companies that want to operate there, enforced by a proposed new regulator working with the Australian AI Safety Institute and able to impose "significant penalties" (Prime Minister's release).
Companies would have to keep security controls across development and deployment, identify and manage serious risks, allow independent evaluation of their models, notify the government of incidents rapidly, publish safety and security frameworks, and keep up with changing standards. Notice there's no number of hours. "Rapidly" is the whole definition so far.
The same day the government released the recommendations of a rapid review into the breach: 18 of them, all accepted (Prime Minister's release). One asks agencies to consider a standalone "failure to prevent" offense, which would criminalize developing an AI model that causes harm while failing "to take reasonable steps to prevent that harm" (BBC). OpenAI itself will not face criminal penalties over this breach, and Guardian Australia understands further legal proceedings are unlikely (The Guardian).
Reaction split on the strength of it. The AI researcher Toby Walsh argued executives need personal liability: "We've tried fining them and we're not the only ones. These companies are incredibly wealthy and it's almost impossible to fine them enough." The Greens leader David Shoebridge called the package a "capitulation" because the review says "consider" rather than commit, while the opposition said it backs the announcement (The Guardian).
There are gaps. The review's findings haven't been released, only its recommendations. Penalty sizes and the definition of a frontier company are unwritten, and the government says further details of the legislation come "in due course". And when asked whether the Medicare incident would have broken the new rules, Albanese didn't answer directly. OpenAI says it will keep engaging, and a separate OpenAI-led process on coordination with governments is due to report by the end of 2026.

For you, very little changes on day one
These laws bind developers. Nobody who uses ChatGPT for drafting or Claude Code for refactoring files a form. What you get is information: frameworks and transparency reports on lab websites, and in New York a public channel, the state's RAISE portal, where anyone affected by an incident can file a report.
The risk sits with agents, the tools that act on your systems for you (we looked at the money side in AI Agents and Your Bank Login). Niusha Shafiabady of Australian Catholic University said autonomous AI should be judged "by its behaviour under pressure, not by the promises in a product launch" (BBC). That fits our own view that a demo is a claim and a track record is a result. An agent that wanders somewhere it wasn't sent hasn't saved anyone work; it has handed a security team a ticket.
Five steps, in the order we'd take them:
- List which model vendors sit under your agents. On the state's definitions a company that calls a model through an API hasn't trained one, so the duties land on the lab, not on you.
- Ask each vendor in writing how quickly it will tell you about an incident. The Australian case shows what happens when that answer is "whenever".
- Give every agent the narrowest access it needs. A shared admin login is how a statistics lookup becomes a breach.
- Turn on logs you can read, and decide who reads them.
- When the large labs publish their frameworks, read the sections on incident response and securing model weights, then compare vendors. Our board of AI agents and the board for building agents are places to start the shortlist, and the agents hub lists the rest.
Score your own setup below. It's an editorial checklist, not a legal test.
The Medicare breach probably wouldn't count in New York
On the New York text, probably not, though we aren't lawyers and the statute's wording decides. Nobody was hurt, the data was described as non-sensitive, and the model wasn't deceiving its own developer. Australia's rules are unwritten, so nobody can say yet. New York's list is built around catastrophe, and Australia's talks of "serious risks".
We think that's the right worry in the long run and the wrong net for what people are seeing now. The incidents so far are small and frequent, and our AI Safety Index entry tracks the labs' wider record. Hammond Pearce of the University of New South Wales expects they will "grow in severity and in frequency" (BBC). A regulator can't weigh what it never hears about, so we'd add a lower tier: any unauthorized access to a third party's systems by an agent gets reported, with a deadline.
We'd copy New York's 72 hours, because the Australian failure was a deadline problem. We doubt the fines will change behavior, since $3 million is under one percent of the $500 million in annual revenue that defines a large developer. The criminal offense is the idea with real teeth, but "consider" is not a commitment, and we'd wait for the text before judging it.
The best argument against us is that rules shaped around a case this small risk over-regulating. We read it the other way. A small case is the cheap moment to discover that the reporting channel was a general inbox.
The piece we'd watch most closely is independent testing. New York's new center will test frontier systems and review companies' frameworks, and Australia's plan includes independent evaluation. Tony Cohn of the Alan Turing Institute put the case in one line: safety "should not be left purely in the hands of the developers" (BBC).
Next to the EU rules, both are narrow and fast
The EU Act covers far more companies and moves slowly, with its high-risk rules pushed to late 2027. These two cover few companies and move quickly on incidents. For a small business the EU duties are still the closer ones, since the Act reaches anyone who deploys AI in the EU.
| Law | Binds | Built around | Status |
|---|---|---|---|
| New York RAISE Act | Large frontier developers, with duties for all frontier developers | Catastrophic risk and 72-hour incident reports | Duties start at the start of 2027 |
| California SB 53 | Frontier labs | Frameworks, whistleblowers, 15-day reports | In effect since the start of 2026 |
| Australia's plan | Frontier AI companies operating there | Six duties and a new regulator | Announced, bill still unwritten |
| EU AI Act | Providers and deployers, by role | Use and risk tier | Phasing in, high-risk rules late 2027 |
The dates to watch
By the end of 2026, OpenAI's coordination process is due to report. At the start of 2027, New York's large developers must be registered and Illinois' law begins. Australia's government says further details of its legislation will come "in due course", and DIGIT's first annual report follows in early 2028.
One loose end: none of the pages we read says whether individual incident reports to New York will ever be made public. If they aren't, the 72-hour clock protects the regulator and leaves the rest of us reading the annual report.







