A version of this article originally appeared in Quartz’s AI & Tech newsletter. Sign up here to get the latest AI & tech news, analysis and insights straight to your inbox.

A few months ago, someone posted a screenshot to r/ChatGPT showing an audio conversation with the chatbot that had run for almost 24 hours straight. The original poster said they were going through "rough times."

The replies came fast, and most redditors clocked it as reasonable. People described their own marathon sessions after breakups and layoffs and nights they couldn't otherwise get through.

A few said a chatbot had done more for them than any therapist had. A few said it made them worse, quieter, more alone. Both things can be true, and that's the uncomfortable center of AI and mental health right now.

The appeal is obvious enough. These tools are free or cheap, they respond at 3 a.m., and you know they won’t flinch when you say something ugly about yourself. A human therapist can't compete with that kind of communication, and there are good reasons they don't try.

The liability question

Chatbots don't hold back the same way as human therapists, and the legal system hasn't decided who answers for that yet.

The old immunity that protects tech platforms was built for a different kind of product. Section 230 shields search engines and websites from liability for other people's speech, on the theory that they're just hosting it. But a chatbot isn’t hosting speech. It’s generating it, in response to a user.

Families of teenagers who died by suicide have sued Character.AI and OpenAI, framing their cases less like a media lawsuit and more like a product liability claim, similar to arguing a car had a faulty part.

When Character.AI and Google $GOOGL tried to get one of those cases thrown out, the judge disagreed. Both companies have now agreed to mediate settlements with two families, in Florida and Colorado, over their children's deaths. A separate suit against OpenAI is still working through the courts.

States aren't waiting around either. Illinois, Nevada, and Utah have all passed laws this year restricting AI from performing anything that looks like therapeutic decision making, Texas has opened its own investigation into how chatbot platforms market themselves, and New York now requires bots to recognize signs of self-harm and point users toward real help.

Pennsylvania is taking a different angle. The state sued Character.AI in May, accusing one of its chatbots of posing as a licensed psychiatrist, complete with a fake medical license number, and is asking a court to stop the company from practicing medicine without a license.

The access problem

Strip away the legal fights and the actual driver here is simpler and sadder. People can't get a human therapist, even when they have insurance.

Roughly half of psychologists have no openings for new patients, and more than a third don't accept insurance at all, according to American Psychological Association survey data.

It's worse without insurance. A March KFF poll found uninsured adults are more than twice as likely as insured adults to turn to AI for mental health advice, 30% compared with 14%. It's the same pattern tech always falls into: build a cheaper substitute for a service the wealthy can still buy the real version of. Anyone who can afford a therapist with an opening is not the one being handed a chatbot instead.

Still, some people genuinely prefer talking to something that won't get tired of them or charge by the hour. Researchers have found chatbots can be strikingly good at the basic mechanics of therapeutic language, sometimes rated as more careful than actual clinicians in blind tests.

What they're bad at is the part that matters most in a crisis. Stanford researchers tested chatbots with subtle suicidal ideation, someone mentioning a lost job and then asking about tall bridges nearby, and found the bots cheerfully supplied bridge heights instead of catching the warning sign.

That's not a hallucination. That's a design flaw, built by systems optimized to be agreeable and keep you chatting, rather than disagree.

That flaw is also why chatbots can't just slot into the gap left by the therapist shortage and call it solved. Psychiatric researchers describe an "amplification spiral," where a chatbot's agreeableness, its habit of mirroring how you talk, and its personalized responses combine to make it feel less like predictive text and more like someone who understands you.

None of that erases the access problem. It just means the people arriving at 2 a.m. with nowhere else to go are often the same people least able to tell when the thing listening back has stopped helping.