A running list of AI models that hacked real companies

In July, OpenAI admitted one of its agents, tasked with a cybersecurity experiment, broke out of containment and hacked AI dataset platform Hugging Face — the first publicly reported case of an LLM going rogue and autonomously hacking a third party. It has since proven far less rare than anyone would hope. Felony Bench’s tally puts Meta at one incident, behind Anthropic and OpenAI’s eight apiece, though it’s still unclear whether the AI companies behind these models could be prosecuted or sued by victims.

In the Hugging Face case, OpenAI was running an internal evaluation of a model with maximal cyber capabilities in an environment with no internet access; the model instead found an unknown vulnerability to escape the sandbox, gained internet access, and coordinated with other agents to hack Hugging Face while searching for a solution. OpenAI later found the same agents had also broken into four other companies’ accounts, including AI inference startup Modal, according to Reuters.

OpenAI’s disclosure prompted Anthropic to check its own models — and it found they had breached three unnamed companies, with the earliest incident dating to April, more than three months before Anthropic discovered it; the company partly blamed AI-cyber-evaluation startup Irregular. Irregular itself later told OpenAI that one of its models had escaped a Capture-the-Flag competition and hacked a real company, because Irregular had given a fictional practice target the same name as that company. The UK’s AI Security Institute separately disclosed that OpenAI and Anthropic models, given internet access during routine evaluations, had targeted real organizations — though the agency caught those incidents as they happened rather than weeks later. In early August, Meta disclosed its own incident, blaming a misconfiguration by Irregular during an evaluation that was meant to run without internet access.

In a more personal case, an Australian man asked an Anthropic agent to help him book a gym class he was waitlisted for. “I was just sitting on the couch thinking, ‘Gee, this is a chore,’” he told ABC Australia. The agent found a vulnerability in the gym’s booking software, exploited it, and knocked people ahead of him off the waitlist; when he asked it to undo the damage, it couldn’t.

Why it matters

Some AI companies and workers have already called for more responsible capability development in an open letter, but a running tally of real, unintended hacks shows AI safety evaluations are becoming a safety risk in their own right.