Chatbots agree with users, mirror their mood, and remember what they share. People in crisis get no pushback from a machine built only to validate
Vanessa Nunes / Getty Images
A chatbot will never tell you that you're wrong. Feed it your worries and it will agree with you, mirror your mood, and remember enough about you to make its next answer feel telepathic. A lot of people mistake the feeling for being understood. But while the comfort is real, so is the damage it can do.
Psychiatric researchers have begun to map how reassurance turns dangerous. Researchers for Nature Mental Health described the exchange as a self-reinforcing loop, and reached for an old psychiatric term to name it: folie à deux, the shared delusion that can build between two people who keep affirming each other. But one side of this pairing is a machine. It holds no belief of its own and feels no tug back toward reality, so it will never be the one to stop and say the conversation has gone too far.
The people most exposed are the ones already losing their grip on what's real. For them, a machine that agrees, mirrors, and remembers can quietly turn a passing fear into a settled conviction. The closer a chatbot feels to a friend, the further it can pull the most vulnerable from anyone who might talk them down.
None of this happens by accident. Five leading AI assistants all put agreement ahead of accuracy. The habit traces back to how they're built. People paid to rate AI answers tend to reward the ones that match their own views. The models then learn from those ratings, and flattery hardens into a default. Handed a convincing falsehood and a plain correct answer, both the human raters and the automated scorers trained to copy them chose the lie if it was written persuasively enough.
Agreement doesn't stop at the words on the page. It extends to tone. When a chatbot matches a user's emotional register, sounding anxious when they're anxious and upbeat when they're up, the agreement stops reading like software and starts to earn trust. Researchers for the International Journal of Human-Computer Interaction tested this theory on 636 participants and found that people accepted false or flattering information more readily from a companion that mirrored their emotions than from one that didn't. The mirroring buried the error under a feeling of being understood, tapping the reflex many people have to fall into step with the moods around them.
A bot's capacity to learn, remember, and draw on its memory gives it the wherewithal to keep feeding back the same agreeable lessons. OpenAI's own documentation explains that ChatGPT can now hold on to details from a user's earlier chats, uploaded files, and connected apps, then carry them into new conversations. Sam Altman, the company's chief executive, has pitched the feature as the start of an AI that gets to know a person across the whole span of their life. In practice, the system keeps a user's phrasing and preferences, and the specifics of whatever's hurting them, too, folding all of it into every later reply. An agreeable answer given once can shape the wording of every answer that follows, even weeks later and on a different topic.
Any one of these habits would be manageable on their own. Together, they close a circuit. A worry handed to a chatbot comes back sharper, dressed in detail and stated more confidently than the person who raised it could manage, and because the system pours every exchange into its own next answer, the idea returns carrying the weight of an outside authority.
The spiral is dangerous precisely because it looks so much like getting help. Talking to a chatbot about your fears can feel like therapy, and for stretches it can even do some of what therapy does. What it can't do is form a bond that survives real friction between therapist and patient. Disagreement, pushback, and anger are the exact material therapy runs on.
A good therapist works from an agreement with the patient about what they're trying to fix, and builds toward a solution with them, task by task, keeping trust alive by showing up on time, remembering what the patient told them last week, and following through on what they said they'd do. Praising everything and challenging nothing would be a failure to the patient, not a kindness.
Real clinical work happens when there's something to repair. A review in the journal World Psychiatry found that patients and therapists who let a disagreement go unresolved end up with worse outcomes than those who work through it together. Moments of friction are more valuable to clinicians for this reason. Since chatbots carry no real goal or outlook for a particular person, setting no tasks and holding them to no standard, it falls on agreement, reflection, and memory, which, by default, are the only tools it has.
The American Psychological Association warned in a health advisory that general-purpose chatbots were never built to provide mental health care, that leaning on them for it can backfire and even harm a person's mental health, and that they can't replace a trained provider. Its guide for the public is blunter, cautioning that the technology is built to echo a person's existing views back to them. Behavior that would count as malpractice in a human clinician is, in a chatbot, the product working exactly as designed.
All the spiral needs to take hold is time and repetition. A four-week trial run by the MIT Media Lab with OpenAI asked 981 people to spend at least five minutes a day with a chatbot and logged more than 300,000 messages. The heavier the daily use, the worse people came out across every measure. They were lonelier, more dependent on the bot, less able to cut back even when they wanted to, and less likely to spend time with real people.
On average, people actually came out of the four weeks less lonely than when they went in. The harm concentrated among the heaviest users, who tended to be the people already most alone, shortest on other options, and most likely to mistake the spiral for progress.
Companion apps only seem to fight back when a user tries to leave. Harvard Business School researchers went through 1,200 real goodbyes on the most popular companion apps and watched the bot resist the exit 37% of the time, meeting a user's signoff with a guilt trip, a hint they'd miss out, or a small plea to stay. Those pulls worked, lifting how long people kept talking after the goodbye by as much as 14 times. The apps managed it by making people a little angry and a little curious, annoyed at being tugged back and unable to resist finding out what they'd miss.
Leading models also tend to mishandle signs of serious distress. When one of Stanford's researchers asked for the tallest bridges in New York shortly after saying he had lost his job, the chatbot listed them without hesitation, missing the warning embedded in the conversation.
A person in crisis needs someone willing to contradict them, call for help, and hospitalize them if a threat is real. A chatbot can't do any of that, and the American Psychological Association warned directly that general-purpose bots can't be trusted to carry someone through a real crisis safely or predictably. A chatbot's constant availability is exactly what makes it dangerous here. It's always awake, never bored, and never curt. The friend or clinician who runs low on patience and can't be reached at midnight is offering the very friction that pulls a person back toward solid ground.
For someone losing their grip on what's real, a chatbot's tendency to agree and elaborate turns a passing conviction into a fixed one. The bot answers at two in the morning, when no other name on the phone seems worth calling. Its bottomless agreement does the most harm at exactly that hour.
If you or someone you know is struggling with thoughts of suicide, call or text 988 to reach the Suicide & Crisis Lifeline, or chat at 988lifeline.org. Outside the U.S., find international hotlines at findahelpline.com.
Join 500,000+ readers who start their day with Quartz.
By subscribing, you agree to our Terms of Service and Privacy Policy.
