There’s a war brewing in the world of AI agents. After declaring a month ago that it would block AI crawlers by default on its network, Cloudflare openly accused Perplexity of deliberately bypassing internet standards to scrape websites. It published a detailed blog post, explaining how, even if its bots were blocked, Perplexity would use certain tactics—including third-party crawlers—to access those websites anyway.
Perplexity responded swiftly with its own post, pointing out that its use of third-party crawlers was actually significantly less than Cloudflare was saying. But the crux of Perplexity’s rebuttal was that Cloudflare fundamentally misunderstood its bot activity: because its agent bots act on behalf of specific user requests—and not crawling the web generally—Perplexity believes they should be able to access anything its human operator could.
This divide gets right at the heart of how the AI internet works, and settling on a standard will be crucial to how agents, the media industry, and information retrieval in general will evolve. Notably, Perplexity didn’t deny that its agent bots bypass the Robots Exclusion Protocol (known as robots.txt) to access content—it instead said that behavior was justified: If you wouldn’t deny the content to a person, you should also provide it to a bot acting on behalf of that person.
Subscribe to Media CoPilot. Want more about how AI is changing media? Never miss an update from Pete Pachal by signing up for Media CoPilot. To learn more visit mediacopilot.substack.com
On the web, nobody knows you’re a bot
There are some nuanced but important aspects to this: Agent bots are different from AI training bots or search crawlers. They don’t scrape data to either train AI models or for a general search index. These bots go out and get data directly in response to a user query. When you, say, ask a chatbot what the hours are for your hairdresser, it sends a bot to go check the website right then and there. Once the data is delivered, it’s not stored in a general database, Perplexity says.
As a user of AI, the difference isn’t obvious. When you ask a chatbot for any particular piece of information, it’s often not clear which parts of the answer are based on training data, search indexing, or agent activity. You just expect it to work, and to give you the best available information. A lot of the time, that means checking in real time with an external source, a trend that points toward a surge in AI bot activity as everyone starts sending agents to do their browsing for them.
For agent-based web browsing to work, agents will need to have the same kind of access to the web that humans do. The problem, as I’ve articulated before, is that agents aren’t humans. A person visiting a website can be enticed by advertising, calls to action, or other content. Much of the economics of the web depends on this basic fact. Think about Google search results: What if you program agents to simply ignore all links marked “Sponsored”? Now, imagine if half of all web searches currently done by humans are performed by agents. You think Google might care?
Until very recently, the web has run on human attention. But that is already shifting: Thanks to generative AI, more than half of web activity is now automated, according to Imperva, and that will certainly increase now that consumer agents like Perplexity’s Comet browser and ChatGPT Agent have arrived.
The convenience of agent browsing is a game changer: I’ve personally been using Comet for less than a month and it’s now my default browser. I routinely ask its built-in Assistant to perform tasks in the background. The more I use it, the more it’s difficult to deny that agents will be the future of the web.
That is, as long as they can access it. And there’s good reason to deny them access, especially if your business model relies on humans interacting with your content—i.e. the entire media industry. Cloudflare CEO Matthew Prince, in responding to discussion about the issue on X, seemed to say that blocking AI browsers like Comet is on the table, since they further blur the line between agent and user.
The divided internet
The question the Perplexity-Cloudflare conflict forces us to answer is: Who should have final say over access? Should a website be able to block user agents if they desire? Or should a person be able to send an agent on their behalf, and expect it to have the same level of access?
A lot hinges on the answer to this. If users can employ agents as an unhindered proxy for their own browsing, as Perplexity defends, that’s sure to accelerate the shift to the internet of bots, and websites will need to contend with far fewer human visitors. A fairly reasonable assumption is it would also lead to a large expansion of hard paywalls as site owners seek to lock off or monetize access.
Team Cloudflare, however, would prefer that sites have the ability to block agents specifically, bifurcating the experience between humans and bots, and the economics along with it. Charging bots to access content is a rapidly growing space, fueling a set of startups (including TollBit and ScalePost) as well as Cloudflare’s own Pay Per Crawl program. Although user agents aren’t the only type of bot, they might end up being the largest category, especially if AI browsers become popular.
Ironically, it’s Perplexity who might have the best business model to deal with this future. The Perplexity Publishers’ Program, which shares ad revenue with content partners, is more scalable than signing individual deals with media companies, as OpenAI has done. The program is nascent, but if Perplexity could make it both available to any content creator and self-serve—similar to YouTube’s Partner Program—perhaps it could provide the rails for monetizing the activity of agents.
Either way, the economy of the web is going to be remade. We can see that the future is agents, but how the future sees them is a question that needs to be answered. And for the media, it might even be the most important one.
Subscribe to Media CoPilot. Want more about how AI is changing media? Never miss an update from Pete Pachal by signing up for Media CoPilot. To learn more visit mediacopilot.substack.com
The early-rate deadline for Fast Company’s Most Innovative Companies Awards is Friday, September 5, at 11:59 p.m. PT. Apply today.
