AI crawlers take far more from the web than they give back — but exactly how much more depends entirely on which vendor’s Cloudflare-derived dataset you trust, and those datasets disagree by a factor of more than 20x on the same bots.
The Basic Pattern Everyone Agrees On
Every source surveyed for this post agrees on the direction: AI crawlers like GPTBot, ClaudeBot, and PerplexityBot request vastly more pages than they ever send human visitors back to the sites they crawl. The disagreement is entirely about magnitude, and it’s a big disagreement. SEOmator’s October 2026 GEO data report put ClaudeBot’s crawl-to-refer ratio near 23,951:1 in early 2026 — meaning roughly 24,000 pages crawled for every one visitor referred. A separate dataset cited in the same space shows the ratio for ClaudeBot more than halving across a later measurement window, from 1,917:1 down to 926:1. Those two “ClaudeBot ratio” numbers — 23,951:1 and a range of 926–1,917:1 — are not measuring the same window or methodology, and no source reconciles them; they’re presented here exactly as found, as a demonstration of how unstable these figures currently are.
Cross-Vendor Numbers Disagree Even More
A June 2026 baseline attributed to Cloudflare Radar data and cited by one industry blog gives: Anthropic (ClaudeBot) near 4,580:1, OpenAI (GPTBot) at 848:1, Perplexity at 186:1, and Google at roughly 5:1. A different October 2026 report puts Perplexity’s ratio at 4,056:1 — more than 20 times higher than the 186:1 figure from the same general period. Both claim to derive from crawler-traffic data; neither source in this research reconciled the gap, and the underlying methodology (what counts as a “crawl,” what counts as a “referral,” and over what time window) isn’t published clearly enough in either source to explain it. This kind of unreconciled disagreement across vendor dashboards is the exact pattern flagged in our coverage of how AI engines differ in citation behavior — treat any single crawl-to-refer number as a snapshot from one measurement pipeline, not a settled industry fact.
| Crawler / Engine | Reported ratio (crawl:refer) | Source | Period |
|---|---|---|---|
| ClaudeBot | 23,951:1 | SEOmator | Early 2026 |
| ClaudeBot | 1,917:1 → 926:1 | Unnamed later-period dataset (cited by SEOmator writeup) | Later 2026 window |
| ClaudeBot (Anthropic) | ~4,580:1 | Cloudflare Radar-attributed baseline | June 2026 |
| GPTBot (OpenAI) | 848:1 → ~251:1 | Cloudflare-attributed baseline; later report | June–July 2026 |
| PerplexityBot | 186:1 vs. 4,056:1 | Two disagreeing reports | 2026 (unreconciled) |
| Googlebot | ~5:1 | Cloudflare Radar-attributed baseline | June 2026 |
Referral Shares Are Tiny in Absolute Terms
Even the more generous crawl-to-refer ratios obscure how small AI referral traffic remains in absolute terms. One October 2026 report puts Claude’s overall share of referral traffic at 0.40% — a rounding error next to Google Search’s share of overall referrals. The same reporting notes that ChatGPT’s measured referral share actually fell in some datasets partly because its outbound links stopped reliably carrying a referrer header, which makes click-through measurement unreliable rather than reflecting an actual drop in usage. That measurement artifact is a recurring problem across this entire category of statistic: a tool’s referral share can move in a report not because real user behavior changed, but because the way links are tagged changed. Our post on how to measure AI referral traffic covers the specific tracking pitfalls (referrer stripping, UTM gaps, server-log vs. analytics-tool mismatches) that produce exactly this kind of discrepancy.
Crawler Share Itself Is Shifting Month to Month
Separate from the refer-ratio question, which crawler dominates total AI bot traffic is also moving quickly. Cloudflare-based reporting shows ClaudeBot and GPTBot trading the lead: one report has ClaudeBot at 11.8% of all AI crawler traffic in October, down from 19.5% in July — a drop of more than 7 points in three months, while GPTBot’s reported share moved in the opposite direction over a similar window according to a different source. Anyone building a robots.txt or crawler-access policy based on “which bot matters most” needs to treat that answer as a moving target, re-checked quarterly rather than set once — the same operational lesson covered in our guide to AI crawler permissions.
Why the Numbers Are This Unstable
Three structural issues explain most of the disagreement. First, “crawl” and “refer” aren’t standardized units — some reports count every HTTP request as a crawl event, others deduplicate by URL or by day. Second, referral counting depends on referrer headers that AI products inconsistently send, so a change in a vendor’s link format can swing the denominator without any change in real usage. Third, most of these figures trace back to Cloudflare’s network-level view, which sees traffic across its customer base but not the full web, so smaller or non-Cloudflare-fronted sites aren’t represented in the same proportion as they would be in a true population-level measurement. None of the sources reviewed for this post published raw methodology detailed enough to let an independent analyst reconcile two disagreeing reports — which is itself worth knowing before citing any single crawl-to-refer number as settled fact.
What Publishers Can Actually Do With This
Given the instability in the numbers, the more defensible move for a publisher isn’t chasing a precise crawl-to-refer ratio — it’s tracking the trend direction on your own server logs rather than relying solely on third-party Cloudflare-derived aggregates, which only see traffic for sites behind Cloudflare’s network and may not represent your own setup. Pair raw crawl-hit counts from your own logs (filtered by user-agent for GPTBot, ClaudeBot, PerplexityBot, and Google-Extended) against referral sessions tagged with those same sources in your analytics tool, measured monthly, and you’ll get a ratio that’s internally consistent even if it doesn’t match any published industry number. That approach also sidesteps the referrer-stripping problem described above, since you’re comparing your own crawl logs against your own referral data rather than trying to reconcile two different third-party datasets that may define “crawl” and “refer” differently. This is the same operational discipline behind a proper AI visibility audit: trust your own first-party data over any single vendor’s headline statistic.
FAQ
Which AI crawler takes the most relative to what it gives back? Depending on the source and time window, either ClaudeBot or PerplexityBot has been reported with the highest crawl-to-refer ratio — but the reported figures for both bots vary by more than an order of magnitude across sources in 2026, so no single “worst offender” can be stated with confidence.
Does a high crawl-to-refer ratio mean a bot is “scraping” rather than legitimately indexing for AI answers? Not necessarily — a high ratio is consistent with normal AI-training and retrieval-indexing crawling, where a bot needs to visit many pages to build or refresh an index but any individual answer only cites a small fraction of what it crawled. It becomes a legitimate business concern mainly when the ratio is used to argue that a site’s content is being consumed without proportionate traffic value returned.
How often should a site revisit its robots.txt rules for AI crawlers given how fast these ratios change? At minimum quarterly, given that Cloudflare-based reports show crawler share shifting by 7+ percentage points within a single quarter (per the ClaudeBot July-to-October figures above) and referral-ratio figures from different sources in the same period disagreeing by 20x or more.
Last updated 2026-11-13. This page is refreshed as benchmarks and scores move.