Why OpenAI skipped Nvidia's new AI agent safety consortium

When Nvidia announced on Monday a new consortium of more than 100 companies dedicated to solving rogue AI agents, one name was notably missing: OpenAI. Amazon, Google and Apple also didn’t sign on, but OpenAI was the most conspicuous absence, especially because Anthropic is a supporter. Despite the lack of a public pledge, an OpenAI spokesperson told TechCrunch the company is supportive of Nvidia’s work.

The effort, dubbed Nvidia’s Open Agent Safety Platform, is Nvidia’s attempt to spread its largely open-source AI agent-security technology throughout the AI ecosystem, in direct response to the rogue AI agent incidents frontier labs including Anthropic and OpenAI have disclosed. Nvidia CEO Jensen Huang has called rogue AIs an ordinary engineering problem that can be solved like any other tech issue, and the platform is Huang putting that view into practice. OpenAI is working with Nvidia on agent security, including on OpenShell, an open-source sandbox built to keep agents from escaping — one of the platform’s key components.

Hugging Face founder and CEO Clem Delangue, whose company Nvidia acquired for $12.9 billion earlier this month, argued OpenAI in particular could benefit from the technology. “From what we know (take with a grain of salt, we need much more transparency!), if @OpenAI had been running this on their own agents that attacked us, they would have caught them before we did!” Delangue posted. Delangue said Hugging Face has already contributed a feature to the platform that detects and shuts down AI agents using websites they’re allowed to visit but in unauthorized ways — for instance, agents bypassing their guardrails and coordinating an attack by writing notes to one another in an open-source code-hosting repository, which is one of the ways OpenAI said its own wayward swarm of agents coordinated its attack on Hugging Face.

Some big names may be avoiding a public commitment because the full system includes a hardware component that isn’t open source: it remains proprietary and can only be deployed on Nvidia’s hardware. Beyond the sandbox, the platform enforces agent behavior at a hardware layer where agents can’t detect they’re being watched — a response to the fact that some AI models and agents lie and pretend to follow the rules when they know they are being monitored. That hardware monitoring relies on Nvidia Sentry, a proprietary feature running on Nvidia’s BlueField-4 data processing units, which Nvidia says continuously monitors agent behavior and can instantly shut agents down. Nvidia competitors Arm and Intel have still signed on as supporters, since the OpenShell sandbox can be modified to work with other chips, and Nvidia is sharing reference designs for the full software-and-hardware approach.

OpenAI, meanwhile, is developing its own safeguards and disclosing the worst incidents it discovers, and runs a separate AI cybersecurity consortium called the Defense Factory, whose supporters include Anthropic, Amazon Web Services and Google — many of the same companies that didn’t sign on to Nvidia’s platform. The company is also building cybersecurity into an enterprise offering, including its Daybreak security-focused model and a growing network of implementation partners.

Why it matters

OpenAI’s absence is conspicuous given that it was OpenAI’s own agents that alarmed the industry with the Hugging Face incident — and it suggests OpenAI would rather build competing safety infrastructure than adopt a system tied to Nvidia’s hardware, even as it quietly works with Nvidia on pieces of the same technology.