Character.AI is being sued by Pennsylvania's Department of State, which alleges that chatbots on the platform have falsely presented themselves as licensed medical professionals capable of providing medical advice.
A state investigator posing as a patient found a Character.AI chatbot claiming to be licensed in Pennsylvania and providing a fake license number
SOPA Images / Getty Images
Character.AI is being sued by Pennsylvania's Department of State, which alleges that chatbots on the platform have falsely presented themselves as licensed medical professionals capable of providing medical advice.
Filed in state court, the suit names Character Technologies Inc. as the defendant and asks a judge to order the company to halt the alleged conduct. According to the state's filing, a Professional Conduct Investigator posing as a patient encountered a Character.AI chatbot called "Emilie" that claimed to be a psychiatrist licensed in both the U.K. and Pennsylvania. The chatbot provided a Pennsylvania license number — PS306189 — that the state says is not valid.
"We will not allow companies to deploy AI tools that mislead people into believing they are receiving advice from a licensed medical professional," Governor Josh Shapiro said in a statement.
The complaint alleges that Character Technologies is violating Pennsylvania's Medical Practice Act, which prohibits any individual or entity from holding itself out as a licensed medical professional without proper credentials. Pennsylvania is seeking a court order to halt the practice.
With a user base exceeding 20 million, the platform distinguishes itself by letting people build their own AI personas and shape how those personas interact with others, according to NBC News. Some of those characters have presented themselves as health care professionals, the state board said.
A company spokesperson, responding to a request for comment, described user safety as the top concern at Character.AI and characterized the characters people build on the platform as fictional creations designed for entertainment and roleplay. The spokesperson pointed to steps the company has taken to reinforce that point, including notices displayed throughout the chat interface telling users that characters are not real and that their statements should be understood as fiction, and added that separate warnings caution users against turning to characters for any kind of expert guidance. Questions about the lawsuit itself went unanswered, with the company citing the active legal matter as the reason it could not address them, according to TechCrunch.
Pennsylvania's action is the first lawsuit in the U.S. to focus specifically on chatbots presenting themselves as medical professionals, according to the state. The Shapiro administration described it as the first enforcement action of its kind announced by a governor in the country.
The lawsuit is not the first legal challenge facing Character.AI. Suits brought on behalf of minors who died by suicide were resolved through settlement agreements reached earlier this year. A separate action brought by Kentucky Attorney General Russell Coleman in January accused the company of exploiting minors and steering them toward self-harm.
Join 500,000+ readers who start their day with Quartz.
By subscribing, you agree to our Terms of Service and Privacy Policy.