The AI era means the internet is splitting in two: one for people, another for the bots
Jason Alden/Bloomberg via Getty Images
The internet’s oldest bargain is breaking.
For decades, websites welcomed crawlers. Getting scraped — by Google $GOOGL or any search engine — meant getting indexed, ranked, discovered. For every two bots it sent to a website, Google sent one user. So scraping meant web traffic — and business.
But now, as generative artificial intelligence tools race to ingest the entire internet, scraping means getting bypassed. According to Cloudflare, for every user that ChatGPT maker OpenAI sends to a site, it sends 1,500 bots. For Anthropic, that figure is 60,000. while — sometimes outnumbering people entirely.
Join 500,000+ readers who start their day with Quartz.
By subscribing, you agree to our Terms of Service and Privacy Policy.
And unlike the Google era, these bots usually don’t link back to their source material. Instead, AI models summarize and serve answers inside their own interfaces, keeping users locked in — and cutting websites and content creators out of the loop.
Linda Tong, CEO of Webflow, a web design and hosting company, called it one of the most profound shifts she’s seen in her 20-years running internet businesses. “It’s fundamentally changing how people find and interact with brands,” she said. “And for some businesses, it’s an existential threat.”
Which goes to the heart of the issue. While much of the public conversation has focused on fears that AI will replace human workers, a more immediate threat isn’t attracting as much attention — even as it’s upending the underlying economics of the internet itself.
Bots have been crawling the web since the early 1990s — first to map it, then to index it, and eventually to fuel the rise of search. Early crawlers like World Wide Web Wanderer laid the groundwork for an era where visibility meant viability. By the time Google’s own crawler — Backrub, later Googlebot — came online in 1996, a new logic had taken hold: Let search engines scan your content, and in return, they’ll send you traffic.
For most of the next 30 years, that logic defined the open web. Visibility in Google results was everything. From niche blogs to the planet’s biggest media conglomerates, websites lived or died by their search ranking. Entire industries, not to mention consulting practices, sprang up around search engine optimization (SEO), information architecture, and content strategy, all to help businesses play the game better.
But AI doesn’t play by the old rules.
Instead of linking back to source material, LLMs like OpenAI’s ChatGPT, Anthropic’s Claude, and even Google’s own Gemini read and repurpose it — answering user queries directly, mostly without any attribution at all.
This shift is giving rise to a new acronym, AEO (meaning AI engine optimization), plus entire strategies for making content more visible to AI and more effective once absorbed inside it, even if AI answers never lead to clicks. If SEO shaped the search era, AEO is likely to define the age of generative AI.
AI bots, Tong said, are “synthesizing what they can access and crawl, and surfacing it in a format that strips away the site’s value proposition.” No clickthrough. No credit. Just answers.
Tong said Webflow has seen AI crawler traffic rise more than 125% in just six months. Across the broader internet ecosystem, more than 50% of all internet traffic now comes from bots. And as bot traffic soars, Tong said, some companies are starting to draw a line — literally. They’re building two versions of their websites. One is designed for humans, with visuals, interaction, and branded storytelling. And the other is stripped down, optimized for machine readability, and engineered to feed the AI beast without giving away (so to speak) the crown jewels.
“For a human, your site should be rich, interactive, delightful,” Tong said. “For a bot? You want clear structure, easy crawlability, but maybe not your full content.” Some publishers are now exposing just a summary or excerpt to crawlers, hoping to bait the index without cannibalizing their monetization model.
The result is a quietly splitting internet: one version designed to delight human users, and another engineered to feed crawlers only specific content with the aim of protecting the value that’s still worth a click.
For some companies, including Webflow itself, getting scraped by AI may actually be good business.
If a user asks ChatGPT for the best website-building platform, and Webflow gets recommended, that’s valuable exposure — especially since, as Tong noted, users arriving via AI tend to be better educated and more “high-intent,” or ready to become customers, than those from search.
“They’ve already done their research,” she said. “When they land on your site, they’re signing up or buying.”
But that logic collapses for businesses that rely on both volume and readers, especially media outlets, creators, and anyone whose model depends on traditional web traffic. If a chatbot summarizes a news article or pulls core facts from a guide or review, that user may never click through. And no click means no ad impressions, no email signups, no audience data, no revenue, no realized value of any kind.
Adam Singolda, the CEO of Taboola, an ad-tech platform known for powering “recommended for you” links across major publishers, put it bluntly: “We’ve seen this movie before. Publishers gave their content to Facebook $META for Instant Articles, and what happened? No traffic. No money.” The imbalance was such that the whole model broke, he said.
It’s true. Facebook launched Instant Articles in 2015 with the promise of speedy load times and a seamless mobile experience. Publishers signed on, hoping to reach massive audiences inside the Facebook app. But while Instant Articles delivered on its promise of feed, it failed to deliver meaningful revenue. Readers stayed within Facebook’s walls, bypassing publishers’ own sites, and with them, the ads, email signups, and tracking tools that fueled their business models.
Within a few years, many media companies had abandoned the format, citing disappointing returns and lack of control. In 2023, Facebook quietly shuttered the program.
Singolda believes tools like Perplexity and ChatGPT are repeating that mistake on a bigger scale, but with even less benefit to publishers. The effects are already visible, he said. Many publishers report search traffic is down 20-30% over the past year — just as AI tools hit mass adoption, with ChatGPT claiming 400 million weekly active users.
“That’s just from the first wave,” Singolda said. “What happens when the next one lands?”
Faced with this slow-motion decoupling of content from traffic, publishers and platforms are pushing back. Some have signed licensing deals — Reddit $RDDT, The New York Times, and Vox Media among them — allowing certain AI companies to access their content in exchange for hefty fees. But those deals are the exception, not the rule.
“There isn’t enough money in the world to pay every publisher whose content is being scraped,” Singolda says. “You can’t offer $100 million to a thousand outlets.”
Tong sees a different future: one where publishers control who can access their content and what they can see. Through Webflow’s partnership with Cloudflare, businesses can now distinguish between good bots, bad bots, and LLMs. They can choose to share partial content, summaries, or nothing at all.
But enforcement remains somewhat tricky. Not every bot respects robots.txt, the term for a site’s crawl policy, which is written in plain text for bots to read. Some companies — Perplexity has been publicly accused — use proxy servers to scrape content even after being blocked. Which means that even as the walls go up, the scraping continues.
Meanwhile there’s a lot more at stake than just traffic. In a world where bots answer first, the difference between being credited and being cannibalized could make or break entire industries.
Already, some AI-generated pages are being created not for people to read, but for other AIs to scrape. It's a closed loop of content made by machines, for machines.
To counter that, companies like Taboola are betting on new models — including Deeper Dive, an AI experience built into publishers’ own websites. Instead of losing users to external bots, it lets audiences ask questions and get answers based on a publication’s existing reporting. “You get the AI interaction,” Singholda said, “but the publisher keeps the relationship, the traffic, the trust.”
That trust may become the most valuable currency of all. In a world of frictionless answers, people still want something real, he argued.
“We’re human,” Singolda said. “When it matters, like your money, or health, or your child, we still want to know who’s talking.”
