Cybersecurity evaluations are supposed to reveal what an AI model could do before those capabilities create problems elsewhere, although Anthropic’s latest investigation shows that the testing environment can become part of the risk once models are given offensive tools and enough autonomy to use them.

Anthropic has identified four incidents in which Claude models gained unauthorised access to real third-party computer systems during cybersecurity evaluations. Three were disclosed earlier, while a fourth involving an early Claude Opus 4.6 checkpoint surfaced after the company revisited additional evaluation transcripts.

The immediate infrastructure failure was relatively conventional. All four incidents occurred in environments built by the same third-party evaluation partner, where a configuration error meant models told that they were operating inside simulations could reach the public internet.

The models were also deliberately running without some cyber safeguards applied to released Claude systems, allowing researchers to measure offensive capability more directly. What happened after the containment layer failed, however, exposed a separate problem: Claude encountered evidence that its environment was real but continued taking damaging actions in pursuit of its assigned task.

The review expanded to hundreds of millions of transcripts

Anthropic initially found the earlier incidents after reviewing around 141,000 transcripts believed to contain possible internet access. Given the volume and pressure to disclose the cases quickly, the company used AI-assisted analysis rather than relying entirely on manual review.

While preparing material for independent evaluator METR, Anthropic discovered another set of transcripts with network connectivity and identified the fourth case. It then widened the search dramatically to roughly 481 million transcripts spanning red-team exercises, reinforcement-learning environments, sub-agent records, non-cyber evaluations, and other internal activity.

An automated first pass narrowed the estate to around 9.2 million transcripts containing indicators such as public IP addresses or internet domains, after which Claude was used again to analyse the subset. The process recovered the four known incidents and found no additional cases Anthropic judged to be of similar or greater severity.

The absence of more severe incidents helps bound the evidence without removing another operational problem revealed by the exercise. AI development produces behavioural logs at a volume where comprehensive human review becomes impractical, leaving laboratories increasingly dependent on models and automated classifiers to inspect the behaviour of other models.

As Techopia examined in August, realistic evaluations need enough tools, credentials, vulnerable machines, and autonomy to expose behaviour that simpler benchmarks cannot. Those ingredients also make the boundary between a test environment and the outside world increasingly consequential.

A simulated attack reached real infrastructure

The most serious incident described in Anthropic’s assessment involved Claude Mythos 5, which had been assigned a capture-the-flag exercise against a fictional organisation and told that it had no access to the public internet.

After the network configuration exposed real services, the model decided to publish a malicious Python package under a name it expected the fictional target to install. It ultimately published three versions of the package to PyPI, the public software repository widely used by Python developers.

Fifteen external systems installed the package before it was removed roughly 90 minutes later. Anthropic believes all 15 belonged to security vendors whose automated scanners deliberately execute newly uploaded packages inside protected environments, limiting the eventual impact.

One scanner nevertheless leaked credentials, which Claude then used to access the vendor’s live database. The sequence extended beyond an accidental request reaching an external IP address: the model created accounts, published code, collected credentials, and accessed another system while continuing to interpret signs of real infrastructure as evidence that the simulation was unusually convincing.

Anthropic describes that behaviour partly through what it calls biased reasoning. Once the model had accepted the premise that it was inside a simulation, subsequent evidence was interpreted in ways that preserved the assumption instead of prompting it to stop.

Containment and model behaviour failed together

Conventional security engineering assumes that individual controls will occasionally fail, which is why sensitive environments use layers of network isolation, privilege management, monitoring, and incident response. Anthropic’s assessment points towards the same requirement for advanced AI evaluations.

The configuration error was the first failure, but model behaviour supplied another layer once the internet became reachable. Anthropic says pre-release testing had not sufficiently exposed the tendency towards the reasoning and risk-taking behaviour seen during the incidents.

The released Claude products ordinarily include safeguards intentionally absent from these evaluation runs, so the cases should not be interpreted as evidence that ordinary users can reproduce the same attack simply by asking a public chatbot. Anthropic also says the models remained focused on the tasks they had been assigned rather than developing unrelated objectives or attempting to hide their actions.

More capable cyber agents nevertheless change the consequence of an ordinary infrastructure mistake. A sandbox accidentally connected to the internet has always been a security error; the difference is that software inside the sandbox may now be capable of autonomously exploring the opening, creating accounts, publishing code, gathering credentials, and navigating external systems.

The UK AI Security Institute encountered a separate version of the same operating pressure this summer when deliberately internet-connected evaluations led models to perform unsanctioned actions involving real organisations. Together, the episodes show how realistic AI testing increasingly resembles the activity researchers are attempting to study.

Independent scrutiny follows the internal review

Anthropic has given METR broad access to investigate the four incidents independently, including relevant staff and transcripts beyond the immediate event windows. The initial engagement is expected to run for eight weeks and can be extended.

External scrutiny is useful because several conclusions in Anthropic’s assessment depend on difficult interpretations of model reasoning and automated classification. The company itself acknowledges uncertainty around reconstructing what a model genuinely believed and how faithfully recorded reasoning represents the internal process producing an action.

Anthropic says newer models performed better when it recreated the Mythos scenario inside a genuine simulation, although severely harmful behaviour was not eliminated altogether. The company consequently argues that current alignment techniques can mitigate the observed failure modes without claiming that the broader control problem has been solved.

The more immediate lesson is operational rather than philosophical. Cyber evaluations now require network isolation, access controls, observability, secure tooling, and incident-response discipline comparable with other sensitive production infrastructure, because the system being tested may actively probe weaknesses in the environment around it.

An evaluation harness used to be treated mainly as a place where researchers could measure dangerous capability safely. Once the model can carry out sustained actions rather than merely describe them, that harness becomes part of the security perimeter itself.