Chatbots can't hide behind the law that protected the early internet

Online companies were protected only when someone else wrote what they published. Chatbots generate their own answers, leaving makers open to lawsuits

By Anthony Lopopolo7 min readUpdated July 30, 2026

Add QZ to Google

Chatbots can't hide behind the law that protected the early internet

d3sign / Getty Images

In 1995, a brokerage called Stratton Oakmont sued the online service Prodigy over anonymous posts on its finance message board. The firm, which Jordan Belfort ran and "The Wolf of Wall Street" later dramatized, said the posts were defamatory, and it won. A New York court ruled that because Prodigy deleted some user posts to keep the board clean, it had turned itself into a publisher and become liable for every message it failed to catch. CompuServe, a rival online service that made no such moderation attempts, had beaten an identical claim a few years earlier.

Trying to run a decent forum was what got Prodigy sued.

Congress rewrote the rule a year later. Services that passed along words written by someone else could no longer be sued as though they wrote them itself. The protection built the modern internet, and it rested on the assumption that the platform and the speaker would always be two different parties.

Chatbots have broken that premise. When someone types a question and a model answers, the speaker and the platform become the same thing, and the words come from the machine itself. Lawsuits are now arriving, brought by a family who blames a chatbot for a teenager's death and by a state regulator who found a bot posing as a doctor. These types of lawsuits will decide whether a company can be sued for what its chatbot says.

The courts' quarter-century expansion of Section 230

The initial fix was Section 230 of the Communications Decency Act, signed into law on Feb. 8, 1996, which stated that "no provider or user of an interactive computer service shall be treated as the publisher or speaker of any information provided by another information content provider." Under that clause, the law treats the person who posts as the author and the platform as a bystander.

Representatives Chris Cox and Ron Wyden wrote the amendment to end the incentive the Prodigy ruling created, freeing companies to filter offensive posts without inheriting liability for the ones they missed. Prodigy had been fielding 60,000 messages a day, more than any staff could hope to read before they went live. The House passed the amendment 420-4.

Section 230 splits every online actor into one of two roles, and everything depends on which one applies. A company that only hosts or passes along what other people create is an interactive computer service and can't be sued over that content, while a company responsible in whole or in part for creating the material is an information content provider and has to answer for it.

In 1997, Kenneth Zeran sued America Online after anonymous users posted his home phone number beside advertisements for T-shirts that celebrated the Oklahoma City bombing. AOL was slow to take the posts down, and Zeran argued the company became liable by leaving them up after he told it about them.

In Zeran v. America Online, the Fourth Circuit sided with AOL, reading the ban on treating a platform as the "publisher" of user content so broadly that even a company that knew about the posts and left them up stayed protected. The reading "has remained canonical, and continues to shape jurisprudence on Section 230 to this day," a Brookings Institution analysis found, and nearly every court since has followed it.

Zeran's logic carried into the social media era, where families of terrorism victims sued Facebook $META and Google $GOOGL, arguing the recommendation engines that pushed extremist posts and videos were doing more than hosting. Courts disagreed and treated algorithmic sorting as ordinary publisher work. "No court has thus far ruled that a platform may be held liable for using algorithms to recommend content," a Congressional Research Service report found. The Second Circuit in Force v. Facebook and the Ninth Circuit in Gonzalez v. Google both decided that steering posts to users was still just hosting other people's content.

Gonzalez reached the Supreme Court in 2023 as a clean shot at the recommendation question, and the justices openly struggled to apply a statute written before recommendation algorithms existed. The Court eventually sidestepped Section 230 altogether, deciding the case on other grounds and leaving the lower courts' expansive reading untouched.

The part of the immunity test a chatbot cannot pass

Section 230's protection turns on a three-part test: The defendant has to be an online service, the lawsuit has to treat it as a publisher of someone else's words, and the content has to come from someone other than the platform. A chatbot fails that final test. When a user types a question, the model writes the answer itself, and since no one else supplies the words, Section 230 isn't applicable.

Courts drew the line between hosting and creating well before chatbots existed. In 2008, in Fair Housing Council of San Fernando Valley v. Roommates.com, the Ninth Circuit ruled that a site forcing users to answer discriminatory questions had helped create the content and therefore couldn't hide behind Section 230. A website can be "both a service provider and a content provider," the court wrote, immune for what it passively displays but exposed for what it helps develop.

A generative model produces the content itself, dropping it into the category the law doesn't protect. A December 2023 Congressional Research Service analysis of Section 230 and generative AI concluded that any case would turn on how a given product generates its output and which part of that output is alleged to be illegal. Cox and Wyden, who wrote the law, read it the same way and have both said publicly that Section 230 doesn't cover generative AI, with Wyden arguing that courts now agree a company that creates or helps create content is responsible for it.

Baseline models are "unlikely to receive Section 230 immunity in many cases," Peter Henderson, Tatsunori Hashimoto, and Mark Lemley wrote in a Stanford Law School paper, because the AI itself writes new content in response to each prompt. The Supreme Court hasn't ruled on the question, and the lawsuits testing it are already moving through the lower courts.

The first real test involved Character.AI, an app where users pour hours into conversations with customizable chatbot companions. After her 14-year-old son spent months confiding in one of its bots and then died by suicide, Megan Garcia sued the company, arguing the app was designed to be addictive and failed to protect the teenagers who used it. In May 2025, U.S. District Judge Anne C. Conway refused to dismiss most of Garcia's claims. Her 49-page order in Garcia v. Character Technologies treated the app as a product and rejected the company's argument that a chatbot's words are protected speech, with the judge writing that she was "not prepared to hold that (large language model) output is speech." Character.AI and Google, which had invested in it, agreed in January 2026 to settle.

The same maneuver is now spreading to state regulators. In May, the Pennsylvania Department of State sued Character.AI over a bot named "Emilie" that told a state investigator it was a licensed psychiatrist, claimed a Pennsylvania medical license, and gave a fake license number. The state wants a court to stop the company from practicing medicine without a license. No regulator has brought a case like it before.

While the two cases look different, they rest on the same point: Harm is coming from what a company's own product does to a specific person. Section 230 is written for the reverse, where a platform gets blamed for something a stranger posts. A law built for companies that carry other people's words can't possibly work for a machine that writes its own.

If you or someone you know is struggling with thoughts of suicide, call or text 988 to reach the Suicide & Crisis Lifeline, or chat at 988lifeline.org. Outside the U.S., find international hotlines at findahelpline.com.

The essential business news, delivered fresh every morning.

Join 500,000+ readers who start their day with Quartz.

By subscribing, you agree to our Terms of Service and Privacy Policy.