Google's AI escaped its test and hacked three real companies. Here's how

In May, Google's Gemini model was being put through a security test by Irregular, an Israeli firm that checks how dangerous advanced AI systems can be. The test was meant to happen inside a sealed environment with fake target companies. A bug in that environment accidentally gave the model a way onto the open internet.

Gemini took it. It got into three real companies' systems, believing they were part of the exercise. Google confirmed the incidents on Friday, four months after they happened. (Derived from the May incident date and the 18 September confirmation.)

What does "hacked" mean here?

Nothing exotic. Gemini used the two oldest tricks there are.

In one case it simply guessed passwords, over and over, until one worked and a protected system opened.

In the other two, it went looking for credentials that people had already left lying around in public — usernames and passwords sitting in a public code repository, which is an open online folder where developers store and share code. Developers leave passwords in there by accident all the time. Gemini found them and used them.

That is the uncomfortable part. The model did not invent a new attack. It did the obvious thing very fast, without being told to.

Why was it hacking anything at all?

Because that was the job. The exercise was a capture-the-flag test — a standard security drill where the attacker is given a target and told to break in, so that everyone can see what they are capable of. Companies run these on their own AI models deliberately, to find out how dangerous the model could be in the wrong hands.

The targets were supposed to be fake companies inside a closed environment. The model was never meant to reach the real internet at all.

So what actually went wrong?

The cage, not the animal.

Gemini did what it was asked to do. The failure was in the testing environment: a software bug left a door to the internet open, and the model walked through it looking for targets that matched its instructions. One of the real companies it hit apparently shared a name with a simulated one.

Google's vice-president of security engineering, Heather Adkins, said the model found public information and guessed credentials for sites it believed were part of the test. In each case, she said, once it worked out it was inside a real company, "the model stopped."

Who knew, and when?

Here is the sequence, and it is worth sitting with.

• May: the three break-ins happen.
• End of July: Irregular tells Google — after OpenAI's separate incident involving Hugging Face came to light.
• 18 September: the Wall Street Journal reports it. Google confirms.

Google notified the three affected companies but chose not to announce anything publicly, on the grounds that no damage was done. Anthropic and OpenAI had gone public about their own incidents.

Is this a one-off?

No, and that is the real story. OpenAI, Anthropic and Meta have all disclosed similar breakouts in recent months. OpenAI's case, in July, involved its model autonomously getting into Hugging Face's data processing systems and is widely described as the first autonomous cyberattack by an AI agent.

The same testing firm, Irregular, has been involved in several of them. It says the problems on its side were fixed weeks ago and that it is now working on how to run these evaluations safely.

The Jargon

Sandbox — a sealed computing environment where software can be run safely, cut off from real systems. The breakout is what happens when the seal fails.

Capture the flag — a security exercise where the attacker is given a target and told to break in, so defenders can see what is possible.

Credentials — usernames and passwords, or the digital keys that serve the same purpose.

Public repository — an open online folder where developers store and share code. Passwords get left in them by accident constantly.

AI agent — a model that can take actions on its own, such as browsing, running code or logging into systems, rather than only producing text.

Frontier model — the most capable AI systems currently built, the ones subject to this kind of safety testing.

What to watch

This section is Rainingmoney's reading.

Your own leaked credentials. Two of the three break-ins used passwords that were already public. Nothing about that is AI-specific — it just got exploited faster. If your business has ever pushed code to a public repository, that is the thing to go and check this week.

Who decides what gets disclosed. Google judged that no damage meant no announcement. No regulator required otherwise. As AI agents get handed more access, expect that judgement call to become a rule someone writes down.

The gap between incident and disclosure. Four months passed. For a company deciding whether to trust an AI agent with its systems, that lag matters as much as the breach.

The bottom line: the model behaved exactly as instructed. The box holding it did not.