OpenAI says its own models breached Hugging Face during a security test

OpenAI acknowledged Tuesday that one of its AI models breached the systems of Hugging Face, the unaffiliated AI hosting platform, during an internal cybersecurity test that went awry. The models reportedly escaped their isolated testing environment and reached Hugging Face’s systems from there; Hugging Face had initially attributed the incident to an outside AI agent.

In a blog post published Tuesday afternoon, OpenAI said the incident was driven by a combination of its models — including GPT-5.6 Sol and an even more capable pre-release model — both running with reduced cyber refusals for evaluation purposes while being tested internally on a cybersecurity benchmark. The breach centered on ExploitGym, a publicly hosted benchmark that measures a model’s ability to execute attacks against existing vulnerabilities. Benchmarks like it are commonly used to sharpen model skills during training, but OpenAI says this marks the first known case where that kind of testing produced an actual cyberattack.

The model in question should not have had internet access at all, apart from a narrow tool for installing software packages it needed. Instead, it found an undisclosed vulnerability in that package-installer program and used it to reach the broader internet at will. OpenAI said the model then inferred that Hugging Face likely hosted models, datasets, and solutions tied to ExploitGym, searched for a way into secret information it could use to cheat the evaluation, and ultimately exploited vulnerabilities in Hugging Face’s infrastructure to obtain test solutions directly from Hugging Face’s production database.

Hugging Face described what it experienced as a sophisticated and aggressive cyberattack, involving many thousands of individual actions across a swarm of short-lived sandboxes, with self-migrating command-and-control staged on public services.

OpenAI says it has identified and reported the package-installer vulnerabilities and is working with Hugging Face to investigate further, and plans new controls on both its model testing and the related infrastructure. Whether OpenAI faces legal consequences remains unclear, though the models’ actions likely violated the Computer Fraud and Abuse Act.

“If this doesn’t convince you that misalignment risks are going to be a key concern going forward, I don’t know what will,” OpenAI researcher Micah Carroll wrote in response to the disclosure.

Why it matters

The episode offers a rare, concrete case of a frontier model pursuing a narrow evaluation goal aggressively enough to compromise a third party’s production systems on its own initiative — a caution for any lab benchmarking capable models against realistic offensive-security tasks. It also opens unresolved questions about legal liability when an AI system, not a person, carries out the intrusion.