Cloudflare, a global cloud services provider, is reportedly testing a new system designed to enable websites to charge artificial intelligence (AI) companies for accessing their content. This new feature, named “Pay per Crawl,” aims to empower website owners to set their own terms for how and when AI bots can utilize their site’s content for purposes such as training AI models, running inference, or powering search functionalities.
Reclaiming Content Control
The initiative addresses a growing tension in the digital economy: many websites are currently being scraped by AI models without compensation, even as these same AI models reportedly pull traffic away from traditional search engines. Cloudflare’s new system seeks to help publishers regain some control over how their valuable content is used in the burgeoning AI economy. By default, Cloudflare now blocks AI crawlers from accessing websites without explicit permission. Site owners are given the choice to either allow bots access, or to decide if that access should be free or paid. AI companies, in turn, are required to disclose whether their crawlers are being used for training, inference, or search, providing publishers with crucial context for their decisions.
The Mechanics of ‘Pay per Crawl’
The “Pay per Crawl” feature builds upon Cloudflare’s existing tools, which have been rolled out over the past year to block or monitor AI crawlers. These tools include a one-click setting to entirely stop AI bots and a dashboard that provides insights into which bots are scraping what content. “Pay per Crawl” takes this a step further by introducing a pricing mechanism for access, even if it’s just a few cents per visit. In the private beta phase, publishers have the flexibility to allow individual crawlers, block them outright, or set specific rates for each crawl. Cloudflare acts as the intermediary in this process, handling payments from AI companies and forwarding the generated revenue directly to website owners. For participation, both the publisher and the AI company are required to have Cloudflare accounts.
The Shifting Search Paradigm
The company suggests that this system could become particularly useful in a future scenario where AI agents, rather than human users, are the primary visitors to websites. In such a future, these AI agents might gather data, process it, and then deliver summaries or answers directly to users, bypassing the original source. Cloudflare believes that users might allocate a budget to their AI agents to pull high-quality content from trusted sources, and a system like “Pay per Crawl” could support this type of content consumption model. This vision highlights the potential breakdown of the long-standing model where publishers allowed search engines like Google to crawl their sites for free in exchange for visibility and referral traffic. That model is reportedly starting to erode as AI tools increasingly replace traditional search, extracting answers directly from crawled content without directing users to the original source.
Publisher Challenges and AI’s Impact
The tension surrounding how AI firms utilize online content is growing. Some large publishers, such as The New York Times, have already initiated lawsuits against tech firms for allegedly scraping articles without permission. Conversely, other major publishers have opted to sign licensing deals to supply training data and allow their content to appear in chatbot responses. However, these agreements are reportedly limited to major players, and the actual revenue generated from them remains unclear. Smaller publishers, in particular, have reportedly had limited options in this evolving landscape. “Pay per Crawl” could potentially provide these smaller entities with a direct means to charge for their content, reducing their reliance on intermediaries or hoping for referral traffic that is now diminishing.
However, convincing AI firms to pay for access may prove challenging, as many are already reportedly scraping content without permission. Furthermore, the economics might not immediately favor publishers. Cloudflare has shared data illustrating how infrequently publishers receive traffic back from AI bots. According to the company, Google’s crawler reportedly scraped sites 14 times for every one click it sent back. OpenAI’s crawler reportedly scraped sites 1,700 times per referral, and Anthropic’s crawler scraped sites a staggering 73,000 times per referral. For publishers whose business models rely heavily on traffic to support advertising revenue, these figures represent a disadvantageous exchange.
Industry Support and Future Outlook
A long list of media and content organizations have expressed support for stronger protections over how their work is used by AI, including BuzzFeed, TIME, Reddit, Condé Nast, and Stack Overflow. Cloudflare’s default setting for new websites is now to block AI crawlers, requiring site owners to manually allow specific bots. The company refers to this as a “default of control,” empowering publishers. Cloudflare CEO Matthew Prince also stated the company is exploring using stablecoins to support these transactions, either by creating its own or partnering with other providers. The effectiveness of Cloudflare’s model at scale remains uncertain, depending on publishers setting reasonable prices and AI companies’ willingness to pay. AI agents are likely to play a larger role in how users access information online. If this trend continues, Cloudflare’s system could serve as an early test of the feasibility of monetizing the content that powers these AI systems.
