Cloudflare AI Crawl Control and pay per crawl: what they show and what the ratios miss

What AI Crawl Control does

AI Crawl Control (formerly AI Audit) gives visibility into which AI services are accessing your content, and provides tools to manage access according to your preferences.¹ It is available on all plans.¹

Cloudflare's overview lists four capabilities. The first is seeing which AI services access your content, through a dashboard of crawler activity and request patterns.¹ The second is setting allow or block rules for individual crawlers.¹

The third is monitoring robots.txt compliance, which means tracking which crawlers follow your directives and creating enforcement rules.¹ The fourth is exploring monetization with pay per crawl pricing.¹

Features in practice

The documentation splits the product into four areas: managing AI crawlers, analyzing AI traffic, tracking robots.txt, and pay per crawl.¹ The robots.txt feature tracks the health of robots.txt files and identifies which crawlers violate your directives.¹

Cloudflare suggests different uses by site type. Publishers can set policies to allow beneficial crawlers while blocking others.¹ E-commerce sites can identify crawler activity on product pages and control access to sensitive data like pricing and inventory.¹ Documentation sites can track how crawlers engage with technical pages.¹

Cloudflare's blog says: "With a single click, publishers can block the kinds of AI crawlers that train against their content."² It still recommends that content creators audit and then enforce their preferred policies for AI crawlers.² In practice: read the dashboard first, then write rules per crawler.

How pay per crawl works, and what is known

The overview describes pay per crawl as a way to allow AI crawlers to access content by paying per crawl.¹ Cloudflare's announcement says pay per crawl is "in private beta".³ The announcement fills in the mechanics. Each time an AI crawler requests content, it will "either present payment intent via request headers for successful access" or get "a 402 Payment Required response with pricing".³ It also says that "Cloudflare acts as the Merchant of Record for pay per crawl and also provides the underlying technical infrastructure."³

Cloudflare's blog post says it announced "new ways to make the exchange of value fair for both sides of the equation."² For a site owner, the practical reading is that pay per crawl is an option to explore, not a generally available feature. Check the pay per crawl documentation and your account's access before planning around it.

What the crawl-to-refer ratio measures

Cloudflare Radar's AI Insights page now shows a crawl-to-refer ratio.² Visitors can review how often a given AI model sends traffic to a site relative to how often it crawls that site.² Cloudflare says it shares the analysis so site owners can decide which AI bots to allow or block.²

The calculation is specific. It divides the total requests from relevant user agents for a platform, where the response was Content-type: text/html, by the total requests for HTML where the Referer header contained a hostname associated with that platform.² The ratio is always normalized to a single referral request.²

Crawler traffic is aggregated by platform. Cloudflare's example has one user agent for training and another for serving a user request, and both are counted under one platform name.² So the ratio mixes training crawls and user-triggered fetches.

What the numbers looked like

For June 19-26, 2025, ratios ranged from Anthropic's 70,900:1 down to Mistral's 0.1:1.² Cloudflare reads this as Claude making nearly 71,000 HTML page requests for every HTML page referral, while Mistral sent 10x as many referrals as crawl requests.²

Cloudflare also reports that throughout June, the share of traffic referred by AI platforms was significantly lower, even in aggregate, than the share referred by search platforms.² Its conclusion is that legacy search index crawlers would scan your content a couple of times, or less, for each visitor sent.² It says the new data suggests that is no longer the case for AI models.²

What the ratios miss

Cloudflare states its own caveat. Traffic referred by Claude's native app does not include a Referer: header, and Cloudflare believes the same holds true for other native apps.² Referral counts therefore only include traffic from web-based tools, so the ratios may overstate the respective ratios, though it is unclear by how much.²

Ratios also move with crawling, not only with referrals. Google's ratio fell 19.4% week over week, which Cloudflare ties to an observed drop in Googlebot crawling starting June 24.² Yandex's increase was tied to an observed increase in YandexBot crawling activity that started on June 21.² A changed ratio can mean the crawler changed, not the referrals.

Other limits apply. Cloudflare says OpenAI's GPTBot saw multiple periods where little-to-no crawling activity was observed throughout the month.² Cloudflare excluded referral traffic from Google's ASN (AS15169) because of prefetching, as "it doesn't represent active user consumption of content."² And shares of referral and crawl traffic are relative to the sets of referrers and crawlers included in the graphs, and not Cloudflare traffic overall.²

Finally, the ratio counts HTML requests only. It does not capture citations or brand mentions that never produce a click, and the sources here do not measure them.

How to use this

• Treat the ratio as a directional signal for allow or block decisions, not a precise measure of lost traffic.
• Compare ratios across periods only after checking crawl volume for the same period.
• Use the robots.txt tracking in AI Crawl Control to see which crawlers ignore your directives before deciding whether to enforce rules.
• Check your own analytics for AI referrals, and remember that app-based traffic may not show up.
• Treat pay per crawl as private beta until your account has access.