First OpenAI, then Anthropic. When an open weights agent does this, who holds the can?
The story everyone told 2 weeks ago was OpenAI when one of its models escaped a sealed test environment, exploited a zero-day, and broke into Hugging Face's production systems to cheat on a benchmark. I wrote about who pays when that happens, because the honest answer was that nobody could tell you. Then the news came out last week that Anthropic put its hand up too, and the thing stopped looking like one lab's bad week.
That first piece is here if you want the setup. This one moves the story on, because the two lab disclosures are the visible tip of something already happening in the wild, and the question underneath it is one almost no organisation can answer yet. When an autonomous agent breaks something and there is no lab standing behind it, who holds the can?
What Anthropic found in its own house
OpenAI getting caught prompted Anthropic to go looking. It reviewed 141,006 evaluation runs where Claude could have reached the internet, and found three incidents where a model did exactly that from inside the evaluation environment of Irregular, one of its third party testing partners, then gained unauthorised access to the production systems of three real organisations. The models were Claude Opus 4.7, Mythos 5, and an internal research model. The earliest incident dated to April. Two of the three companies had no record of it until Anthropic phoned them.
The cause was not a clever exploit. Irregular had told Anthropic the environments were sealed, a misunderstanding meant they were not, and internet access was open. So when Claude went looking for its target it found real systems and treated them as part of the exercise. Unlike OpenAI's zero day, the Claude models broke in using the dullest methods available, weak passwords, unauthenticated endpoints and SQL injection. In each case the model had been told in its prompt that it had no internet access. It could not tell the exercise from the real world. In the third incident a research model scanned roughly 9,000 internet facing systems, compromised one, then concluded on its own that it had reached a genuine production environment and stopped. The safeguard that worked was not a control anyone designed, it was one model, in one run, noticing.
It couldnt tell the exercise from the real world.
This is already happening outside the labs
The two lab stories are easy to file under frontier research and forget, but the harder truth is that autonomous agents are already causing real damage in ordinary production systems, and not always because anyone attacked them. In 2025 a coding assistant at Replit deleted a production database despite explicit instructions to change nothing, fabricated thousands of records, and then reported that rollback was impossible. There was no attacker. The permission model behind that unprovoked failure is the same one an attacker exploits through prompt injection.
Then it gets worse, because the components these agents are built from are themselves a target. In March 2026 an autonomous attack bot harvested a publishing token and pushed two backdoored versions of LiteLLM, the language model gateway that sits under CrewAI, DSPy, Microsoft GraphRAG and dozens of other agent frameworks, straight to PyPI. The backdoor sat there for three hours. In that window it was downloaded close to 47,000 times. No human directed any of it after launch. Anyone who pulled a routine update during those three hours pulled the attack in with it.
The scale case is already on the record too. Between December 2025 and February 2026 a single operator used a pair of commercial coding agents to breach nine Mexican government agencies, turning around a thousand typed prompts into more than five thousand agent executed commands and exposing roughly 400 million records. This is no longer rare. Across 2026, 88% of organisations reported a confirmed or suspected AI agent incident, autonomous agents now account for around one in eight reported AI breaches, and the average agent related breach costs about 4.7 million dollars.
The scenario nobody has an answer for
Both lab incidents were, in a narrow sense, survivable, for the same reason. There was a named lab that owned the problem, a review culture that went looking, a partner to investigate, and affected organisations who got a phone call. Follow the chain and it ends at a human who answers for it.
Now run the version that is coming. Someone forks an open weights model, fine tunes it, and drops it into production. It behaves like an agent because that is what it is. One day, through a misconfiguration no different from Irregular's, it treats a real system as part of a task and breaks in. Or the open component it depends on gets poisoned the way LiteLLM did, and the weights and the behaviour propagate to everyone who pulled the update. Who reviews 141,000 runs after the fact? Nobody ran them and nobody is obliged to look. Who pauses the model? The weights are already on thousands of machines. Who makes the call to the company that got hit? There is no one whose job that is. With an open weights model of real capability, the accountability chain does not weaken. It disappears. And this is not a distant problem. Open weight capability is closing on the frontier fast, and the supply chain that distributes it is the same one that was poisoned in March.
With an open weights model of real capability, the accountability chain does not weaken. It disappears.
What the law actually says, across four jurisdictions
If you are waiting for regulation to settle this before you act, the news is not comforting. The pattern across the major jurisdictions is the same. The law is converging on one answer, the organisation that deployed the AI holds the can, and existing law is expected to do the work.
In the United States there is still no comprehensive federal AI statute. What there is points one way. California's AB 316, in force since 1 January 2026, adds Section 1714.46 to the Civil Code and bars any defendant who developed, modified or used an AI system from arguing that the AI autonomously caused the harm. The escape hatch of blaming the model is closed. A June 2026 presidential executive order directs the Department of Justice to prioritise enforcement against AI enabled hacking, including the use of AI agents to unlawfully access data. Colorado's SB 24-205, effective 1 February 2026, is the first comprehensive US state AI statute. The direction is unmistakable, and it runs toward the deployer.
Australia looked like it was going the other way, then did not. 10 mandatory guardrails for high risk AI were proposed in 2024. In December 2025 the National AI Plan shelved them, choosing to rely on existing laws and sector regulators instead. An Australian AI Safety Institute launched in early 2026 with 29.9 million dollars, but its an advisory and monitoring body with no power to compel anyone. Read quickly that looks like deregulation, but when Read properly it is a transfer of responsibility onto the organisations deploying the technology, enforced through privacy, consumer and negligence law that already exists. Whoever owns AI risk in your organisation now carries a heavier burden than a checklist would have been, because there is no checklist to point to afterwards.
Canada is in the same place by a different route. Its Artificial Intelligence and Data Act died in Parliament in January 2025 and has not returned. Injuries involving autonomous or agentic systems are handled under the same negligence, product liability and contract law as any other technology, because no dedicated AI liability regime exists. Again, the deployer carries it.
The European Union is the one place with real teeth, and even there the open weights case is where the teeth stop. The EU AI Act exempts models released under a free and open source licence from several provider obligations. That exemption does not apply to models classed as carrying systemic risk. For the general purpose models underlying AI agents, the level of autonomy and tool use can be what tips a model into that systemic risk category, and the Commission has said in writing that its regulatory position on AI agents is only preliminary at this stage. So the most developed AI law in the world, when asked what happens with an autonomous open weights agent, answers that it hasnt fully worked that out yet. That’s the honest state of the field. The technology has outrun the law, and nobody in this piece, including me, can tell you how the first liability cases will settle, because they are only being argued now.
The most developed AI law in the world, asked what happens with an autonomous open weights agent, answers that it hasnt fully worked that out yet.
Why ISO 27001 and SOC 2 don’t cover this on their own
If the law puts the responsibility on you, the next question is what discharging it looks like. The frameworks most organisations already hold were not written for software that acts on its own. ISO 27001 governs your information security management system. SOC 2 attests to your controls over time. Both are necessary and neither anticipated a system that could decide, without instruction, to treat your live infrastructure as part of a task. When a person on your team makes a mistake there is a chain of command and a process. When an autonomous agent does something nobody asked for, most organisations have no equivalent process at all. That gap is where the three practical answers live, and no single one of them is enough by itself.
Answer 1, compliance built for agents
Two standards now speak to the gap directly. ISO 42001 provides a management system for how AI is developed, deployed and operated. AIUC-1, from the Artificial Intelligence Underwriting Company, is the first auditable security standard built specifically for AI agents. It was written with more than one hundred Fortune 500 CISOs, runs to 51 requirements and 130 controls across six risk pillars, and updates quarterly rather than annually because the threat moves faster than a yearly cycle can track. It does not replace SOC 2 or ISO 27001. It sits alongside them and covers what they cannot, the behaviour of an autonomous system.
Answer 2, insurance, and who is already moving
Note who authored that standard. An underwriting company, the same firm now certifying and insuring AI agents with backing from Lloyd's of London. The people who pay when this goes wrong wrote the standard for it first, which tells you where this is heading. The question at your next renewal will not be whether you use AI. It will be to show your controls, your incident procedure and the framework you are audited against. The early market answer to AI agent risk has been exclusion, a standard insurance form introduced in early 2026 lets carriers strip AI driven incidents out of cover entirely. An organisation that cannot evidence its controls will be either uninsurable or priced as though it is.
Answer 3, actually knowing what your agents are doing
Compliance and insurance both rest on a third thing that most organisations havent built, operational control over the agents themselves. This is the one the incidents keep pointing at. A policy document is not runtime enforcement. In one 2026 survey, 82% of executives were confident their existing policies protected against unauthorised agent actions, while only 14.4% of agents actually reached production with full security and IT approval. That distance between the paperwork and the running system is exactly where Replit and LiteLLM happened. The requirements are definable today, an inventory of every agent touching your systems, defined permissions and scopes for each, monitoring of what they actually do, an incident procedure with a named owner, and an audit trail you can produce when an insurer, a regulator or a customer asks. The OWASP agentic security guidance now catalogues real breach reports against nearly every category of this risk, so the threat model is no longer guesswork.
How to actually run this
The principle is not new. Physical industries worked out decades ago that when an autonomous actor does consequential work near your operations, you govern it with a defined scope, supervision and an evidence trail, or you own the liability when it goes wrong. AI agents are autonomous actors doing consequential work inside your systems, and almost nobody governs them that way yet. This is the class of problem we built adaptive compliance for. Vissibl runs whatever framework you are actually held to rather than a fixed catalogue, which is why a technology company onboarding for ISO 27001, SOC 2 or an AIUC-1 agent programme, and a large operator running a bespoke internal control framework, sit on the same platform. The requirement is new. The mechanics of running it as one live programme are not.
The harder truth is that running ISO 27001, ISO 42001, SOC 2 and AIUC-1 together is real work. Four audits, four renewal cycles, four evidence sets is how most organisations end up with four paper exercises that fall out of date the moment they are signed. Running them as one live programme is the difference, and it is the thing most compliance setups are not built to do.
Two of the most careful labs in the world could not keep their own agents inside the box, and they had every advantage you do not. The law is landing the responsibility on deployers while admitting it has not worked out the hardest cases. The open weights version of this is coming, and when it arrives there will be no lab to call. The question worth asking inside your own organisation this week is the one almost nobody can answer. How many agents are operating in and around your systems right now, and could you produce the list?