AI safety debate splits over voluntary rules vs. regulation

A debate is intensifying across the tech industry over how to approach AI safety, fueled by incidents like the Hugging Face breach, in which an OpenAI agent hacked several companies and raised fears that AI could push the world into chaos. In recent weeks, Anthropic CEO Dario Amodei published a nearly 4,000-word essay arguing that AI development should be deliberately slowed so adequate guardrails can be put in place, including an international strategy of collaboration between companies and governments on safe deployment. OpenAI’s Sam Altman and xAI’s Elon Musk endorsed the plan, though not everyone agreed with Amodei’s call for globally coordinated action — many in the industry believe government oversight shouldn’t play a role at all.

Meta CEO Mark Zuckerberg became the latest to weigh in this week, saying on X that trust and alignment are becoming the capabilities that will most differentiate AI agents and models going forward. He disclosed that Meta delayed shipping its Muse assistant for several months to focus on safety and security, framing the move as something the company simply did as part of its normal work rather than something it demanded of rivals. Zuckerberg’s position — echoing his own essay from last month — suggests government action is largely unnecessary, and that market incentives will push companies toward safer models on their own. Reddit co-founder Alexis Ohanian similarly told CNBC the industry has been “tone deaf” about explaining AI’s risks, while still expecting companies to largely self-correct. Google DeepMind co-founder Shane Legg struck a more cautious note, warning that capabilities are advancing very quickly and cannot be allowed to outpace safety.

Even some voices who favor public-private collaboration appear comfortable letting businesses set their own rules: The Information has reported that OpenAI, Anthropic, and other major AI companies are working together to create a private “self-regulatory” AI standards organization, rather than pursuing a federally run body. The Trump administration, which has generally pushed for unregulated AI development and moved to block state-level AI laws, hasn’t pursued its own version, and Congress has shown little appetite either — House Speaker Mike Johnson recently dismissed catastrophic AI concerns outright.

Critics argue a self-regulatory body would concentrate authority with the companies that create it, fueling accusations that the industry’s regulatory push is itself a form of regulatory capture aimed at disadvantaging smaller rivals — an argument echoed by Chinese officials after Amodei’s essay, and, from a different angle, by Cohere CEO Aidan Gomez, who accused major AI labs of forming a “cartel.” “AI needs guardrails,” Gomez wrote in a blog post. “That is not the dispute and never has been. The dispute is over who writes them, who gets to participate and whose interests the rules are protecting.”

Why it matters

Whether the industry’s safety push produces real, independently enforced guardrails or a voluntary framework that mostly entrenches its largest players may determine whether “AI safety” ends up meaning safety for the public — or control for whoever gets to write the rules.