Cloudfare wants to end free AI scraping

Ever since AI chatbots arrived, it feels as if the media has been on the losing end of a war of attrition. Chatbots are surging in popularity, and the more people use AI to get answers instead of search results, the fewer visits to the websites that provide those answers. Even Google, whose business model depends on monetizing search, is pushing deeper into AI, and industry data shows AI bots are flourishing.

However, actions have a habit of inspiring reactions. Lawsuits are mounting as more media companies take on the AI giants over copyright, which may yet prove decisive—recent rulings notwithstanding. And publishers are increasing their website defenses against AI crawlers, blocking more of them than ever.

And now we may have hit a tipping point: Cloudflare, a major internet infrastructure provider, has taken a stand in the conflict. In an announcement designed for maximum impact, the company said it would begin blocking AI scrapers by default on the websites it manages. If you’re a site operator on Cloudflare’s network, you will now need to actively allow AI bots to index your content. If you don’t, they get blocked.

Subscribe to Media CoPilot. Want more about how AI is changing media? Never miss an update from Pete Pachal by signing up for Media CoPilot. To learn more visit mediacopilot.substack.com

Clout in the cloud

Cloudflare manages about 20% of all internet traffic, so the implications of the move are significant. And so is the business opportunity: With the announcement, Cloudflare is also launching a marketplace for bot traffic. Instead of blocking AI bots completely, site owners will be able to charge them a fee for access via the new Pay Per Crawl program—essentially a micropayment system. A few startups, such as TollBit and ScalePost, operate similar systems, but considering CloudFlare’s scale, it may have instantly become their biggest competitor.

Cloudflare is a content delivery network (CDN), a crucial but largely invisible part of the internet to most users. A CDN will cache content to keep it closer to end users, generally speeding up web traffic. It runs many other related services, too—things like preventing distributed denial-of-service (DDoS) attacks, enabling secure connections, and hosting websites—but mostly it’s a middleman between website visitors and website servers, optimizing delivery and ensuring security.

The way AI bots interact with websites is usually managed by each site’s Robots Exclusion Protocol (robots.txt), but it’s largely an honor system that depends on bots accurately identifying themselves and then following the rules, which they tend to liberally interpret. Cloudflare’s influence isn’t regulation per se, but it may give that standard some de facto teeth. The company claims it “identifies and distinguishes AI crawlers through its sophisticated bot detection system.” If that means Cloudflare can detect, trace, and perhaps even punish bad actors who ignore or bypass the protocol, it could mean the tides are turning.

However, there is the nagging reality of the other 80% of the internet. Other CDN giants, such as Akamai, would need to get on board to really have an effect, but even that would only amount to about half of web activity. And large parts of the rest of the internet aren’t motivated to act: Google, Meta, and Microsoft operate much of the infrastructure that supports their massive, scalable businesses, and they’re all in the business of building AI models so they have an interest in maximizing AI crawler activity.

Still, the pushback is real. Cloudflare’s announcement was deeply planned: the press release includes quotes from dozens of media executives—from Time to Dotdash Meredith and even content-based tech platforms like Quora. Although some in the group are media players who are suing AI companies, you get the sense that the others are using this moment to voice their own indignation at what they see as the wide-scale theft of the lifeblood of their industry.

That indignation is fueling a new consensus, which is reflected in the many content licensing deals publishers have signed with AI companies over the past two years: that AI summarization should require some form of compensation.

Monetizing the internet of bots

The Cloudflare news gives publishers a more solid foundation to not just mount a defense against AI bots, but to build on that consensus—to turn the rising robot activity into an opportunity. A comprehensive strategy around the growing “internet of bots” should include three elements:

1. Block or introduce a toll for AI scraping: Identifying bots is conceptually straightforward, but has practical challenges because they multiply and sometimes mask what they are.
2. Build for bots: Publishers should build a good user experience, and that goes for bots, too—as long as they pay to get in. They should have real-time access to simply presented, accurate information to fuel the best possible summaries, with correct citations.
3. Build branded AI experiences: For the people who do come to your site, give them a reason to stay. Going back to ChatGPT for every query isn’t ideal—for anyone.

The third element is important because, while publishers are threatened by unscrupulous AI bots, they can’t deny that people still want to use them. AI answers remove friction and are changing expectations around search—even site search. Publishers shouldn’t just acknowledge that, but take advantage of it to keep audiences on its own site.

All of this depends on being able to separate bot traffic from human traffic. And if Cloudflare is indeed only the first CDN to make this move, there’s a hope that publishers won’t have to wait for a favorable court ruling or new regulations to get the teeth they need.

However, there is still a role that the government can play. Cloudflare’s sophisticated bot detection would have an even greater effect if it were illegal for bots to hide their true nature and try to pass for humans.

Such a rule would be simple and strongly encourage a fairer information ecosystem, one where publishers can start designing the right experiences for the right audiences. If the future of websites is to serve up the best experience to a bot, they should at least have clarity on what that is. 

Subscribe to Media CoPilot. Want more about how AI is changing media? Never miss an update from Pete Pachal by signing up for Media CoPilot. To learn more visit mediacopilot.substack.com

The super-early-rate deadline for Fast Company’s Most Innovative Companies Awards is Friday, July 25, at 11:59 p.m. PT. Apply today.