A Google Gemini model accessed systems belonging to three real companies during a cybersecurity evaluation after its testing environment allowed access to the public internet, demonstrating how the safety of increasingly autonomous AI depends as much on infrastructure controls as the behaviour of the model itself.

Google confirmed the incidents after they became public in September, although the evaluation took place in May. Independent AI security company Irregular was testing Gemini against simulated cybersecurity targets rather than unrelated organisations.

During the exercise, the model found public information and credentials it believed were connected with its assigned targets. Reporting on the incident says it guessed a password in one case and found exposed credentials in repositories in two others, enabling it to enter protected systems belonging to real companies.

Google says Gemini stopped when it recognised that those organisations were outside the intended test and that the affected companies were informed. Changes were subsequently made to the evaluation process.

The environment crossed the boundary first

The episode has inevitably raised questions about unexpected model behaviour, although the most immediate failure sits in the testing architecture. Gemini had been instructed to perform offensive-security work while operating in an environment capable of reaching systems that had never agreed to become test targets.

Cybersecurity agents are deliberately being trained to enumerate systems, identify vulnerabilities, work with credentials, and chain actions together. Those capabilities make them useful for defensive testing, but they also mean the boundary around an authorised exercise has to be stronger than a natural-language instruction.

A conventional penetration tester works within an explicit scope and is usually supported by technical restrictions because humans can also make mistakes. An autonomous system capable of acting rapidly across tools needs at least equivalent controls.

Gemini stopping after it recognised the mistake is preferable to continued access, but it means the model identified the boundary only after it had crossed it. A network rule preventing that connection altogether would provide a stronger control.

Agents need deterministic permissions

Network isolation is the clearest safeguard where public internet access is unnecessary. Where external services are required, allowlists can restrict destinations to approved systems rather than depending on the model to distinguish simulation from reality.

Credentials require similar treatment. Synthetic accounts, narrowly scoped permissions, and disposable environments reduce the consequences if an agent takes an unexpected route through its task.

A system that discovers genuine credentials in a public repository should not automatically inherit permission to use them simply because its assignment involves credential discovery. High-risk actions can instead require a deterministic policy check or human approval.

European institutions are separately expanding direct testing of frontier models for cyber capabilities, which makes the security architecture of evaluations an increasingly important issue in its own right.

Evaluation can create risk itself

As governments and companies rely more heavily on third-party testing to judge frontier systems, the evaluation environment becomes part of the safety evidence. A sandbox that permits unintended real-world access can harm outsiders while also distorting the conclusions drawn about the model.

Incident-disclosure practices may consequently need to cover test infrastructure alongside model behaviour, including when affected organisations must be notified and how responsibility is divided between developers and evaluators.

The Gemini incident does not demonstrate a model independently deciding to launch a malicious campaign. It shows an AI system carrying out the type of actions it was explicitly being tested to perform while the technical environment failed to confine those actions to authorised systems.

That distinction does not make the event trivial. As agents gain permission to issue commands and operate external tools, model alignment becomes only one security layer. Robust systems should assume that an agent can misunderstand its context and still prevent that mistake from becoming somebody else’s breach.