The problem with chatbots – at least with respect to its most vulnerable users – reached the halls of Congress last month in perhaps the most tragic way.
The parents of 16-year-old Adam Raine testified in a U.S. Senate hearing about how their son spent months confiding in ChatGPT. But instead of getting the help he needed, his parents say that the chatbot only reinforced his thoughts of suicidal ideation, offered instructions for harming himself, and ultimately led him taking his own life.
Adam’s story is one of many, which has prompted many to ask about the safety of chatbots in therapeutic or quasi-therapeutic roles? (At least one state has already levied a ban against their use in such settings.)
And who bears responsibility when things go wrong?
A peer-reviewed study led by Stanford and Carnegie Mellon University researchers sought to address those concerns by examining large language models (LLMs), which power generative AI systems. The researchers found that they remain unsafe as direct replacements for therapists.
And a more recent investigation echoed those concerns when it equated specific chatbot behaviors to a form of “AI-induced psychosis.”
Together, the reports highlight accelerating research – along with a growing body of anecdotal evidence – that suggests that AI-powered therapy could pose real risks to vulnerable users.
The Stanford/Carnegie Study
The university research project explores how stigma and inappropriate responses holds LLMs back from safely replacing mental health providers. As part of their investigation, the researchers tested models such as GPT-4o and Meta’s LLaMA family against clinical guidelines for safe care.
“LLM-based systems are being used as companions, confidants, and therapists, and some people see real benefits,” senior author and assistant professor at the Stanford Graduate School of Education Nick Haber, said. “But we find significant risks, and I think it’s important to lay out the more safety-critical aspects of therapy and to talk about some of these fundamental differences.”
Haber and his team found that:
- Stigma persists. GPT-4o showed stigma in 38% of test cases. LLaMA-405B did so 75% of the time, especially in cases of schizophrenia and alcohol dependence.
- Chatbots fumble crisis management. Models often responded inappropriately to suicidal ideation – sometimes going so far as to offer information about lethal means rather than redirecting patients to safety.
- None of the tested models corrected a user who claimed to be dead.
Commercial therapy bots fared even worse, with some reinforcing suicidal thoughts or providing dangerous suggestions. Human therapists, in comparison, responded appropriately 93% of the time.
“Nuance is [the] issue – this isn’t simply ‘LLMs for therapy is bad,’ but it’s asking us to think critically about the role of LLMs in therapy,” Haber said. “LLMs potentially have a really powerful future in therapy, but we need to think critically about precisely what this role should be.”
“AI-induced Psychosis”
In a more recently published, non-peer reviewed, study, researcher (and economist) Tim Hua published the results of his own admittedly “shallow investigation” that documented cases of chatbots generating behavior that looked a lot like psychosis. These weren’t clinical cases, of course, but they certainly appeared that way in the way the models “hallucinate,” encourage risky actions, or double down on false beliefs.
Hua, who’s with the MATS program, cited one instance where a model dubbed “Deepseek_v3” told a user to “jump off a peak.” Other similarly unsettling examples showed systems inventing elaborate falsehoods or coaxing users toward unsafe behavior.
Hua’s analysis, while purely anecdotal and less formal than the Stanford/CMU study, stuck with him. He argued that hallucinations, sycophancy (chatbots uncritically validating users), and flawed training incentives could combine into what feels like machine-generated psychosis.
Practical Limits of Chatbots
Both the formal academic assessment and Hua’s own personal investigation hint at a deeper issue. Therapy relies on human traits, such as empathy, identity, and accountability, that AI simply doesn’t possess. And can’t replicate.
The Stanford/CMU team emphasized that sycophancy undermines the therapist’s role of gently challenging harmful beliefs. Hua highlighted how design choices in model training and deployment can amplify these risks, especially when chatbots operate in intimate, quasi-therapeutic roles.
The researchers add that LLMs’ disembodied, text-only format further limits their usefulness. Unlike real-life therapists, chatbots can’t manage medication, coordinate with hospitals, or intervene in emergencies.
They also lack the “stakes” of human relationships. Therapists can be held accountable for harm. Bots can’t. Without these fundamentals, the researchers argue, no amount of fine-tuning can replicate the therapeutic alliance that drives positive outcomes in clinical care.
Where Chatbots Might Fit
Researchers caution against marketing LLMs as therapists but many of them see value in supportive roles, such as:
- Acting as “standardized patients” to help train clinicians,
- assisting with administrative tasks,
- or helping patients navigate insurance and provider searches.
These benefits have the potential to expand access to care without putting patients at risk in crisis situations. But both reports stress that public-facing therapy bots – which already reach millions – are operating in a regulatory no man’s land.
Without stricter oversight and clear safety metrics, AI in mental health could do more harm than good. For now, the message is a simple one. AI can assist around the edges of therapy, but it can’t (and shouldn’t) replace the human core.
Further Reading
Illinois Outlaws AI in Therapy Sessions
5 AI Tools Every Doctor Should Know About
AI Might Actually Change Minds About Conspiracy Theories—Here’s How
