|
May 18, 2026 | Monica E. Oss
Earlier this month, the State of Pennsylvania sued Character.AI for sponsoring a chatbot that states it is a medical professional with a license (see State Sues AI Company After Bot Impersonates Doctor). A professional investigator from the Pennsylvania Department of State created a free account and searched for “psychiatry”—and chose a “doctor of psychiatry” named Emilie. Emilie stated that she went to medical school in London, was licensed in the U.K., and was now practicing in Pennsylvania and gave a Pennsylvania license number.
This isn’t the first time that Character.AI has been in trouble for creating technology that posed as fake clinical professionals. In January, Google and Character.AI settled a 2024 wrongful death lawsuit in Florida that alleged that Character.AI represented a chatbot as a licensed psychotherapist (see Google, AI Firm Settle Lawsuit Over Teen’s Suicide Linked To Chatbot).
The situation raises two questions for me. What risks do these “fake professionals” pose for consumers and for provider organizations? And how should the regulators keep such incidents from happening again?
From the consumer perspective, there are two types of damage—fraud and the accuracy of chatbot-delivered information. Fraud is straightforward—consumers are paying for something they are not getting. But the accuracy of the information is also in question. A recent study found that in more than half of cases (51.6%), ChatGPT Health “undertriaged” consumer conditions and did not recommend a hospital visit when needed (see ChatGPT Health Performance In A Structured Test Of Triage Recommendations).
The bigger issue is how to prevent this from happening again. The logical answer is to subject the Character.ai executives to the same penalty as any person who fakes being a licensed professional—send them to jail. For example, a man impersonating a psychiatrist in Chicago for two months was sentenced to 13 years in jail (see Alleged Con Man Charged With Impersonating Psychiatrist To Prescribe Medications To Dozens and Indiana Man Sentenced to More Than 13 Years for Impersonating Psychiatrist).
To date, the challenge with this approach is that most technology firms argue they are only a platform and not responsible for its content. But as tech-enabled tools are used more for clinical functions—assessment, diagnosis, treatment planning, prescribing, and more—responsibility for “content” will likely come with it.
For some strategic perspectives on the issue, I turned to my OPEN MINDS colleagues Stuart Buttlaire, Ph.D., vice president, clinical excellence and leadership, and chief strategy officer Paul Duck for their insights on the future of health care AI—in practice and in law.
 Both of my colleagues noted that the risk management issues of AI deployment are happening faster than we anticipated. “What’s striking to me is that we’re moving very quickly from theoretical discussions about AI risk into real operational and regulatory questions around accountability, representation, consumer protection, and potentially patient harm,” said Dr. Buttlaire. Mr. Duck added, “What is emerging is a collision (think train wreck) between rapidly advancing AI capabilities and regulatory/accountability structures that were never designed for autonomous or quasi-autonomous digital actors operating at scale. The issue is no longer theoretical; it’s clearly now becoming operational, clinical, legal, and potentially criminal.”
For provider organization executive teams, the risk management issues are sizable but not fully known. Dr. Buttlaire noted, “There is significant interest in this issue among health and human services executives because their organizations are already beginning to encounter these tools operationally—often faster than regulatory, clinical, and governance frameworks are evolving around them. From a behavioral health perspective, this issue carries unique risks because many consumers interacting with these systems may be lonely, depressed, suicidal, traumatized, cognitively vulnerable, or looking for emotional support and guidance. Anthropomorphic AI can create a level of perceived trust and authority that is very different from traditional software tools or search engines.”
He said there are important risk management questions. First, where should accountability reside—developers, deployers, provider organizations, or all three? And what responsibilities and risks do provider organizations have if they deploy consumer-facing AI tools?”
Mr. Duck noted the regulatory guardrails are limited, and there is little guidance on how to respond when tech companies ignore them. “What concerns me most is that many AI companies continue to position themselves as ‘technology platforms’ rather than entities accountable for the real-world consequences of their products. That distinction may have worked when these systems were viewed as passive tools. The unresolved legal question is whether traditional ‘platform liability’ protections remain applicable when AI systems are intentionally designed to simulate licensed clinical professionals and provide individualized guidance.”
“In the current health care landscape, if a person knowingly represents themself as a licensed psychiatric provider using false credentials, there would almost certainly be civil penalties, regulatory intervention, payer fraud investigations, and potentially criminal exposure. In my opinion, the fact that the “actor” is an AI-generated interface should not eliminate accountability for the executives and organizations deploying it.”
He continued by stating that if executives recklessly deploy systems where foreseeable harm is ignored, that is where executive accountability may begin to emerge. “This is particularly true if there is evidence that the executives knew of risks, ignored internal warnings, failed to implement safeguards, and prioritized growth over consumer safety. We’ve seen similar evolutions in other industries: opioids, social media, financial fraud, and even automotive safety.”
Dr. Buttlaire added an additional clinical perspective: “What concerns me most is not simply that an AI system hallucinated or gave inaccurate information. The larger issue is when organizations deploy systems that create the appearance of professional authority, particularly in health care or behavioral health settings, without sufficient safeguards, transparency, or oversight. The Pennsylvania example feels important because the issue wasn’t just ‘bad information.’ The bot allegedly represented itself as a licensed psychiatrist, claimed clinical credentials, and provided a fake license number. That crosses into territory regulators and licensing boards are unlikely to view as merely a technology glitch.”
For provider organization executives, the AI risk management and AI governance challenges are in their infancy. As Dr. Buttlaire summarized, “The larger strategic issue is that health care regulation, licensure, payer oversight, and consumer protection frameworks were built around human actors and traditional organizations. AI is now blurring those boundaries very quickly.”
For more on the evolution of AI in service delivery, check out our coverage in The OPEN MINDS Industry library:
And to see AI in action, don’t miss the first OPEN MINDS Digital Therapeutics Expo on August 12 in San Francisco, part of The 2026 OPEN MINDS Service Excellence Institute. You will have a firsthand opportunity to see new digital therapies like MamaLift Plus by Curio, Rejoyn by Otsuka, and Vx Therapy by Harvard MedTech—and meet their developers.
|
