GPTBot, ClaudeBot, and PerplexityBot are not one bot each — they are three separate crawlers with three separate legal fictions of “opt-out,” and blocking the wrong one leaves the others crawling anyway. OpenAI, Anthropic, and Perplexity have all split their web access into a training crawler, a search-indexing crawler, and a user-triggered fetcher, each with its own User-agent line in robots.txt — and each honoring “Disallow” to a different degree. If you manage a site and think one blanket rule handles “AI,” it doesn’t.
The three-bot pattern, and why it exists
All three major AI labs have converged on the same architecture: separate the crawler that builds training data from the crawler that builds a search index from the fetcher that retrieves a page because a live user asked a question. That separation is deliberate — it lets a publisher opt out of model training while still appearing in AI-powered search, or vice versa. According to OpenAI’s own Overview of OpenAI Crawlers, “each setting is independent of the others — for example, a webmaster can allow OAI-SearchBot in order to appear in search results while disallowing GPTBot to indicate that crawled content should not be used for training.”
Anthropic runs the same pattern. Its Claude Help Center article, last updated April 7, 2026, lists three distinct bots — ClaudeBot, Claude-User, and Claude-SearchBot — and states plainly that “Anthropic’s Bots respect ‘do not crawl’ signals by honoring industry standard directives in robots.txt,” including the non-standard Crawl-delay extension.
Perplexity is the outlier, and the exception matters more than the rule. Its official crawler documentation splits access into PerplexityBot (automated indexing, which does honor robots.txt) and Perplexity-User (fetches triggered by a live user question). For Perplexity-User, the docs say: “since a user requested the fetch, this fetcher generally ignores robots.txt rules.” That’s a materially different compliance posture than either OpenAI or Anthropic state for their user-triggered bots — ChatGPT-User’s status is murkier (“robots.txt rules may not apply,” per OpenAI), while Perplexity states its ignore-behavior outright.
The current user-agent reference
| Company | User-agent | Purpose | Robots.txt behavior |
|---|---|---|---|
| OpenAI | GPTBot | Training data collection for foundation models | Honors Disallow |
| OpenAI | OAI-SearchBot | Indexing for ChatGPT search answers | Honors Disallow; independent of GPTBot |
| OpenAI | ChatGPT-User | Live fetch when a user’s question needs a page | “Robots.txt rules may not apply” (user-initiated) |
| OpenAI | OAI-AdsBot | Ad landing-page safety checks | Only visits submitted ad pages |
| Anthropic | ClaudeBot | Training data collection | Honors Disallow and Crawl-delay |
| Anthropic | Claude-SearchBot | Indexing for Claude’s search results | Honors Disallow and Crawl-delay |
| Anthropic | Claude-User | Live fetch for a user’s Claude query | Honors Disallow and Crawl-delay |
| Perplexity | PerplexityBot | Indexing for Perplexity search (not model training) | Honors Disallow |
| Perplexity | Perplexity-User | Live fetch for a user’s Perplexity query | “Generally ignores robots.txt rules” |
Source for each row: OpenAI’s crawler overview, Anthropic’s Claude Help Center, and Perplexity’s crawler docs, all fetched July 2026. Anthropic also publishes a source-IP list at claude.com/crawling/bots.json as a secondary verification method, since it warns that blocking by IP alone “may not work correctly or persistently guarantee an opt-out” if the bot can’t read robots.txt in the first place.
Why “blocking AI” doesn’t mean what people think it means
The practical implication of the three-bot split: disallowing ClaudeBot in robots.txt does not touch Claude-SearchBot or Claude-User — each needs its own User-agent stanza. A site that wants out of AI training but still wants to appear in AI-powered search answers has to write three or more separate rules, correctly, across three companies, and keep them current as new bots ship. This is exactly the granularity Anthropic’s February 2026 documentation update was built to enable, replacing an earlier, coarser single-bot setup.
Meanwhile the underlying protocol these directives sit on top of, the Robots Exclusion Protocol, is not a law — it is a voluntary convention formalized only in September 2022, when the IETF published RFC 9309 (authored by Koster, Illyes, Zeller, and Sassman) as a Proposed Standard, 28 years after Martijn Koster’s original 1994 draft. RFC 9309 standardizes syntax, error handling, and caching rules, but compliance is still entirely at the crawler operator’s discretion. Nothing in the RFC obligates a company to honor it; OpenAI, Anthropic, and Perplexity’s search/training bots do so as policy, not law.
The bigger structural problem: mixed-use bots
Cloudflare, which sits in front of more than 20% of the web by its own account, published a year-one retrospective on its “Content Independence Day” initiative on July 1, 2026, and the headline numbers reframe the whole robots.txt conversation. Per Cloudflare’s report, 52% of crawler requests across its network were for AI training as of June 2026, up from 22% in spring 2025, and crawlers that blend search, agent, and training purposes into a single user-agent now account for more than 36% of crawler activity. Cloudflare states directly that “more than one-third of crawler activity on our network still comes from mixed-use bots that make it impossible for content owners to distinguish crawl intent” — and singles out Google as the clearest example, noting Google “has access to about 2x more information than leading AI companies because Google leverages a mixed-use bot that makes it difficult for customers to participate in Google’s search ecosystem without also participating in Google’s AI ecosystem.”
That’s the opposite design choice from OpenAI, Anthropic, and Perplexity, all of which split search from training into separately blockable bots. It also means the clean per-bot table above is the good case. A meaningful share of AI-adjacent traffic hitting any given site isn’t cleanly attributable to a single declared purpose at all, and no robots.txt line can opt out of “part of what this bot does” — only all of it, or none.
Practical checklist for a site owner
Working from the primary docs above, a robots.txt block aimed at “AI” needs at minimum: separate User-agent stanzas for GPTBot, OAI-SearchBot, and ChatGPT-User if the goal is to distinguish training from search from live fetches; the same three-way split for ClaudeBot, Claude-SearchBot, and Claude-User; and an acknowledgment that blocking PerplexityBot stops indexing but does not stop Perplexity-User from fetching a page on a live user’s behalf, since Perplexity’s own documentation says that fetcher “generally ignores robots.txt rules.” None of this addresses mixed-use crawlers, which by definition can’t be selectively disallowed by purpose.
For sites pursuing the opposite goal — maximizing AI visibility rather than blocking it — the companion approach is llms.txt, a proposed file format for actively inviting AI systems in rather than just declining to block them, and the broader question of how these systems decide what to surface at all is covered in How AI Assistants Decide Which Brands to Recommend. The training-versus-retrieval distinction in this piece also underpins the sourcing problems documented in ChatGPT’s and Claude’s respective limitations pages — both note real-time retrieval and citation quality issues that trace back to which bot a site allowed in the first place.
FAQ
Does blocking GPTBot also block ChatGPT-User and OAI-SearchBot?
No. OpenAI’s documentation states each bot’s robots.txt setting is independent, so a site must add separate Disallow rules for GPTBot, OAI-SearchBot, and OAI-AdsBot to control each function separately; ChatGPT-User is user-triggered and may not fully respect robots.txt regardless.
Can a site actually stop Perplexity-User from fetching a page?
Not reliably through robots.txt. Perplexity’s own docs state Perplexity-User “generally ignores robots.txt rules” because the fetch is initiated by a live user question, not automated crawling.
Is honoring robots.txt legally required for AI crawlers?
No. RFC 9309 formalized the syntax of the Robots Exclusion Protocol as an IETF Proposed Standard in September 2022, but compliance remains voluntary; OpenAI, Anthropic, and Perplexity’s indexing/training bots honor it as stated policy, not because any law compels it.
Last updated July 22, 2026. This page is refreshed as benchmarks and scores move.