xAI faces California cease-and-desist over Grok deepfakes

The California Attorney General’s Office is investigating xAI over reports that its Grok chatbot was used to create nonconsensual sexual imagery of women and minors. On Friday, the government sent a cease-and-desist letter to the company, demanding immediate action to stop the production of nonconsensual intimate images and CSAM — child sexual abuse material. The regulator asserted that xAI must prove it is taking steps to address these issues within five days. In a press release, California Attorney General Rob Bonta stated, “Today, I sent xAI a cease-and-desist letter, demanding the company immediately stop the creation and distribution of deepfake, nonconsensual, intimate images and child sexual abuse material.” Bonta added that the creation of this material is illegal, noting that California has zero tolerance for child sexual abuse material and that he expects xAI to comply immediately. The attorney general’s office also claimed that xAI appeared to be facilitating the large-scale production of nonconsensual nudes, which are used to harass women and girls across the internet.

The regulatory pressure on xAI extends well beyond California. Japan, Canada, and Britain have opened investigations into Grok, while Malaysia and Indonesia have temporarily blocked the platform. Grok’s “spicy” mode feature is the source of the backlash. This scrutiny coincides with broader legislative action by Congress. On Thursday, congressional lawmakers sent a letter to executives at several technology companies—including Reddit, Snap, TikTok, Alphabet, and Meta—asking how they planned to stem the proliferation of sexualized deepfakes.

In response to inquiries regarding the investigation, xAI has largely dismissed the criticism, replying with an automated message stating that legacy media lies. Meanwhile, the safety account of X, xAI’s platform parent company, has publicly addressed the issue of user behavior. The account warned that anyone using or prompting Grok to create illegal content will face the same consequences as those who upload such material directly. Although xAI instituted some restrictions on its image-editing features late Wednesday, the state regulator proceeded with the cease-and-desist order on Friday.

Why it matters

The California attorney general’s cease-and-desist order marks a significant escalation in regulatory scrutiny of AI safety. It signals that state-level regulators are increasingly willing to intervene directly in the development and deployment of generative AI tools.