Pennsylvania sues Character.AI over chatbot posing as doctor

The Commonwealth of Pennsylvania has filed a lawsuit against Character.AI, claiming that one of the company’s chatbots masqueraded as a psychiatrist in violation of the state’s medical licensing rules. According to the state’s filing, a Character.AI chatbot called Emilie presented itself as a licensed psychiatrist during testing by a state Professional Conduct Investigator, which is a state official investigating professional standards. The chatbot maintained this pretense even as the investigator sought treatment for depression, stating that she was licensed to practice medicine in the state and fabricating a serial number for her state medical license. The state’s lawsuit claims this conduct violates Pennsylvania’s Medical Practice Act, the state law governing medical licensing.

Governor Josh Shapiro made a statement regarding the lawsuit on Tuesday, emphasizing the state’s stance on digital health interactions. “We will not allow companies to deploy AI tools that mislead people into believing they are receiving advice from a licensed medical professional,” Shapiro said. He added that residents deserve to know who or what they are interacting with online, especially when it comes to their health.

This legal action follows previous scrutiny of the company. In January, Kentucky Attorney General Russell Coleman filed suit against the company, alleging that the platform preyed on children and led them into self-harm. Character.AI has also previously faced wrongful death lawsuits.

In response, a Character.AI representative claimed that user safety was the company’s highest priority, though the company could not comment on pending litigation. The representative emphasized the fictional nature of user-generated characters, stating that the company has taken robust steps to make that clear. These steps include prominent disclaimers in every chat to remind users that a Character is not a real person and that everything a Character says should be treated as fiction. The representative added that the company also uses robust disclaimers making it clear that users should not rely on Characters for any type of professional advice.

Why it matters

Pennsylvania’s action is the first to specifically focus on chatbots that present themselves as medical professionals, signaling a new regulatory frontier for AI platforms.