A page that returns a 302 instead of a 301, or that takes too long to respond, doesn’t rank lower in AI search — it frequently doesn’t get read at all.
Most technical SEO thinking about status codes was built for crawlers that index now and rank later. AI systems that answer questions in real time — ChatGPT, Perplexity, and increasingly Claude’s search products — don’t work that way. They fetch a candidate set of pages at query time, under a latency budget measured in milliseconds, and a page that isn’t reachable inside that window simply never enters the pool of things that could be cited. The status code your server returns isn’t a ranking signal here. It’s a gate.
The status codes that actually matter
Five outcomes cover almost everything relevant to AI visibility: a clean 200 (fetched, parsed, eligible), a 301 permanent redirect, a 302 temporary redirect, a 404 (or soft 404), and a non-standard but increasingly important 499.
The first four are defined in RFC 9110, the current HTTP semantics spec: a 301 tells a client the resource moved for good; a 302 says it moved for now and will be back at the original URL. Search engines have historically respected that distinction — Googlebot keeps a 302’s original URL indexed rather than replacing it. Whether AI training and search crawlers preserve the same distinction with the same fidelity is far less established, and it’s the crux of why 302s are riskier than they look for AI-facing content.
301 vs. 302: why the difference is sharper for AI crawlers
According to Passionfruit’s March 2026 analysis of redirect chains and AI crawler behavior, AI training crawlers like GPTBot “may not maintain that distinction with the same precision” as Googlebot, and can end up treating a 302’s destination as the canonical version of the content without preserving the context of the original URL. That matters because a page’s identity — the URL a model associates with a claim — is part of what gets baked into training data and reused in citations. Use a 302 for a permanent move and you risk an AI system permanently misattributing content to the wrong address.
The same report, citing a Search Engine Land analysis of ChatGPT agent behavior published in October 2025, found that 63% of first-click ChatGPT agent visits bounce immediately, with HTTP errors, redirects to unexpected URLs, slow load times, and bot-blocking among the leading causes. Redirects are the one item on that list entirely inside a publisher’s control.
Two developments since this post was written change where the 301-vs-302 decision actually gets made. The first is that Cloudflare now enforces canonicalisation at the edge for AI training crawlers: with Redirects for AI Training, introduced April 17, 2026 and available on Pro, Business and Enterprise plans at no extra cost, a verified AI training crawler requesting a page whose <link rel="canonical"> points elsewhere receives a 301 to the canonical version, while humans, search crawlers and AI Search agents continue to see the original page. That is the misattribution risk described above being fixed one layer up from the application — worth knowing before rewriting redirect rules at the origin, and worth checking for, since it changes what your own logs will show for those user agents.
The second is more consequential and runs the other way. On September 15, 2026 Cloudflare set new defaults across the three AI-traffic categories it defined on July 1, 2026: “For all new domains onboarding to Cloudflare, the categories of Training and Agent will be blocked by default on the pages that display ads, while Search will remain allowed by default” (Cloudflare, July 1, 2026). For an ad-monetised page on a newly onboarded domain, the status-code hygiene described in this post is now downstream of a prior gate: an agent fetch can be refused at the edge before the origin ever emits a 200, a 301 or a 499. Cloudflare also confirms that from that date multi-purpose crawlers are judged “according to all of their behaviors” under the most restrictive applicable rule, and names Googlebot, Applebot and BingBot as crawlers blocked for customers who chose to block Training. Cloudflare’s post does not state which HTTP status a blocked crawler receives, so we are not asserting one here — check your own edge logs rather than assuming a 403.
Redirect chains: where patience runs out first
Googlebot will follow roughly ten redirect hops and has decades of crawl-budget infrastructure behind it. AI crawlers don’t have the same slack. Oncrawl’s server-log analysis, published in April 2026 and updated in July 2026, found AI bots behaving erratically around broken paths in the wild — including a case where ChatGPT-User hit a hallucinated, non-existent product URL on a French electronics retailer’s site nearly a thousand times in a few hours, and a travel site where OAI-SearchBot got lost in pagination and started chaining through redirects on its own. Neither behavior was triggered by a site change; it was the crawler’s own path-following logic breaking down.
JavaScript-based redirects are worse than server-side ones for this exact reason: most AI crawlers, including GPTBot and ClaudeBot, do not execute JavaScript, so a redirect implemented client-side is functionally invisible to them — the crawler fetches the shell, finds no server-side redirect, and moves on. This is also why JavaScript-rendered content is broadly invisible to AI crawlers, not just redirects built with it. The fix in both cases is the same: server-side 301s, resolved in a single hop, with no client-side logic in the path.
The 499 blind spot: eligibility happens before ranking
A 499 isn’t in the HTTP spec — RFC 9110 stops at 426. It’s an NGINX-originated code, since adopted by Cloudflare and most modern edge infrastructure, that logs when the client disconnects before the server finishes responding. iPullRank’s Mike King, writing in May 2026, argues this code is quietly becoming one of the most consequential things in AI search that almost no SEO tool surfaces. Working with data Profound shared from a random sample of 700,000 pages analyzed over multiple days in April 2026, King reports that pages with AI-crawler failure rates above 75% received roughly 18x fewer citation events than more reliable pages — in many cases, zero citations at all.
The mechanism is retrieval-augmented generation itself: at query time, a system fans out to candidate documents, fetches them, and only then extracts and ranks passages. A page that doesn’t return content inside the crawler’s patience window is excluded before that pipeline even starts. It isn’t ranked poorly — it’s absent from the candidate set. One iPullRank client reportedly recovered a 22% AI-visibility bump after fixing the backend latency and edge-caching gaps causing a 499 spike, though that figure comes from a single client case study rather than a controlled study, and should be read as directional rather than a general rate.
Where the data disagrees
Two Cloudflare-sourced figures floating around right now don’t reconcile cleanly, and that’s worth flagging rather than smoothing over. Oncrawl cites Cloudflare Radar data for the twelve months to March 2026 putting GPTBot at 12% and ClaudeBot at 9.2% of global bot traffic, both closing in on Bingbot. Cloudflare’s own “From Googlebot to GPTBot” report, covering May 2024 to May 2025, shows GPTBot requests up 305% year-over-year while ClaudeBot requests fell 46% over the same window. These aren’t contradictory once you notice the measurement windows don’t overlap — one is a trailing-twelve-month share snapshot ending in 2026, the other a year-over-year growth comparison ending in 2025 — but the two crawlers are clearly moving in different directions depending on which period you sample, and any AI-visibility report that cites a single crawler’s “share” without a date range attached should be treated with suspicion.
A newer snapshot complicates the picture further rather than resolving it. seomator’s monthly AI-crawler report puts July 2026 Cloudflare-sourced traffic shares at Googlebot 25.5%, ClaudeBot 19.2%, Meta-ExternalAgent 12.7%, GPTBot 9.9%, Bytespider 5.4%, and a now-separate Claude-SearchBot at 3.9% — Anthropic has split its crawler into distinct bots for training (ClaudeBot) and live search retrieval (Claude-SearchBot), so any GPTBot-vs-ClaudeBot comparison that predates that split is no longer measuring the same thing on the Anthropic side. This doesn’t settle the Oncrawl/Cloudflare disagreement described above; it’s a third, more recent, still-shifting data point rather than a tie-breaker. Separately, a BuzzStream analysis of 4 million citations (published March 2026) found that blocking an AI crawler in robots.txt doesn’t reliably stop citations: 70.6% of top-50 news sites that blocked ChatGPT’s bot were still cited in AI answers, even though blocking cut measured referral traffic by roughly 23%. And crawl-to-referral ratios remain wildly lopsided by vendor regardless of status-code hygiene: estimates for Anthropic’s crawl-to-click ratio run from roughly 11,000:1 to 20,000:1 depending on source and method, versus roughly 850:1 to 1,255:1 for OpenAI and closer to 5:1 for Google — a high crawl volume from a given bot says nothing about how much traffic that bot sends back.
A status-code checklist for AI-facing pages
| Status code | What it does to AI visibility | Fix |
|---|---|---|
| 200 (direct) | Eligible for the fetch stage; no penalty | None needed — this is the target state |
| 301 (single hop) | Followed normally; content reachable | Keep chains to one hop |
| 302 on a permanent move | Risk of AI systems citing or training on the wrong canonical URL | Convert to 301 for anything that isn’t genuinely temporary |
| Redirect chain (2+ hops) or JS redirect | Increases odds the crawler abandons the request before reaching content | Flatten to a single server-side 301 |
| 404 / soft 404 | Generates crawler noise; hallucinated links compound the problem | Ensure dead URLs return real 404s, fix broken internal/external links |
| 499 (client gave up) | Content excluded before the retrieval pipeline evaluates it at all | Reduce origin latency; edge-cache HTML for known AI bot user agents |
What to do with this
One cross-site baseline worth attaching a number to: seomator’s 2026 Crawl Waste Report found verified AI bots receive a 200 response only 73.0% of the time versus 33.3% for non-AI bot traffic overall — meaning even AI crawlers built specifically to fetch content are getting redirected, blocked, or erroring out on roughly a quarter of requests, which is the gap this checklist is aimed at closing. None of this requires new tooling most sites don’t already have. Server or CDN logs will show every AI crawler’s status-code distribution if you filter by user agent — the same logs already used for Googlebot analysis. The priority order, based on the evidence above: eliminate JavaScript-only redirects on pages you want cited, flatten any chain longer than one hop, audit 302s for ones that have quietly become permanent, and check for 499/5xx spikes correlated with origin latency. None of this touches whether robots.txt allows or blocks a given crawler in the first place — that’s a separate, prior question — but it determines whether a permitted crawler can actually finish the job once it arrives, which is exactly the layer beneath how AI assistants end up recommending one brand’s page over another’s and beneath how different AI engines choose citations in the first place.
FAQ
Does a 302 redirect hurt AI visibility more than a 301?
For genuinely temporary moves, no — a 302 is the correct code and traditional search engines handle it as designed. The risk is using a 302 for a permanent change: evidence from Passionfruit’s 2026 analysis suggests AI training crawlers may not preserve the temporary/permanent distinction as reliably as Googlebot does, which can result in content being associated with the wrong canonical URL.
What is a 499 status code and why does it matter for AI search?
It’s a non-standard code, originated by NGINX and now used across Cloudflare and other edge infrastructure, logged when a client disconnects before the server finishes responding. It matters because real-time AI search systems fetch pages under strict latency budgets; a page that trips a 499 is excluded from the citation pipeline before ranking is even attempted, per data Profound shared with iPullRank in 2026.
Do AI crawlers follow robots.txt the same way Googlebot does?
Not uniformly. Oncrawl’s 2026 log analysis found bots including PerplexityBot and ClaudeBot fetching URLs explicitly disallowed in robots.txt and receiving 200 or 304 responses — meaning the directive was ignored at the server level, though separate firewall-level blocking may occur invisibly to standard logs. On-demand fetch bots like ChatGPT-User are also explicitly documented by OpenAI as potentially not following robots.txt at all, since the fetch is triggered by a live user request rather than a background crawl.
Status-code hygiene is one layer of a broader framework — see the 7-layer AI visibility audit.
Last updated October 8, 2026. This page is refreshed as benchmarks and scores move.
Pingback: DIY AI Visibility Audit: The 7-Layer Checklist - Tech Blog