AI visibility breaks down into seven checkable layers — crawler access, llms.txt, structured data, server-side rendering, HTTP status hygiene, entity consistency, and citation measurement — and most sites fail more of them than they pass: one 2026 audit tally put the average site at just 31% readiness on the AEO/GEO-specific checks, against 62% on ordinary technical SEO.
That gap is the reason a “DIY audit” is worth doing before paying for a platform. Every layer below has a primary-source check you can run yourself in under an hour, plus the current data on how many sites pass it.
Layer 1: Can AI crawlers reach the page at all?
Start with robots.txt. A Cloudflare-network analysis run on July 27, 2026 found GPTBot named in 633 DISALLOW rules, ClaudeBot in 563, and PerplexityBot in 311 across the sampled sites — and a separate Q1 2026 tracker put GPTBot in 5.52% of all DISALLOW rules, just ahead of ClaudeBot at 4.88% (TechnologyChecker.io). GPTBot blocking has plateaued near 25% of top sites after growing fast from 2023–2024, while roughly 30% of top sites now run a “middle path”: block training bots (GPTBot, CCBot, ClaudeBot, Google-Extended) but allow retrieval bots (OAI-SearchBot, PerplexityBot). That distinction — training crawler vs. search/retrieval crawler — is the single most common audit mistake, and it’s the full subject of our crawler-permissions breakdown.
Layer 2: Does an llms.txt file exist, and does it matter?
Adoption is climbing but still a minority behavior. Rankability’s June 2026 scan found 8.7% of the top 1,000 sites publish an llms.txt; a broader 300,000-domain SE Ranking sample put adoption at 10.13%; among Fortune 500 companies only 7.4% (37 of 500) had one as of March 2026. Total adopting domains reached roughly 39,000 by May 2026 — an 8.8x year-over-year increase off a small base. The catch: a 37,894-domain scan by Trakkr Research found zero measurable citation advantage for sites that publish the file, and a separate tracker reported 97% of llms.txt files receive zero AI requests. Presence isn’t useless as a manifest, but don’t audit it as a citation lever — see our llms.txt deep dive for the full adoption-vs.-impact data.
Layer 3: Is structured data doing real work?
Schema markup is one of the most oversold layers. Ahrefs tracked 1,885 pages that added JSON-LD schema between August 2025 and March 2026 and found no major citation uplift on any platform from adding schema alone. But specificity matters within it: pages with Product or Review schema carrying populated concrete attributes (price, rating count, availability) were cited at 61.7% vs. 41.6% for pages with generic schema types. Google’s own guidance, current as of May 2025, still recommends JSON-LD specifically because AI systems parse it more reliably than microdata or RDFa. Audit for populated fields, not just markup presence — our schema markup post breaks down which types carry weight.
Layer 4: Is the content actually in the HTML, or only in the DOM?
This is the fastest fail to check and the easiest to fix. As of June 2026, none of the major AI crawlers — GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, PerplexityBot, Meta-ExternalAgent, Bytespider — render JavaScript; an analysis of over 500 million GPTBot fetches found zero evidence of JS execution. The audit check is simple: view page source (not the DevTools Elements panel) and confirm your pricing tables, FAQ answers, and comparison content appear in the raw HTML. Gemini is the documented exception, since it can piggyback on Googlebot’s rendering service. Full methodology and more examples are in our JS-parseability piece.
Layer 5: Are status codes and redirects clean?
Crawlers tolerate imperfection differently by task. In 2026 observation, GPTBot, ClaudeBot, and PerplexityBot’s training crawlers will follow up to 5 redirect hops, while the real-time search variants (OAI-SearchBot, Claude-SearchBot, Perplexity-User) give up after the third hop. The bigger issue for most sites is waste, not redirects: one 2026 crawl-log analysis found ChatGPT’s crawler spending 34.82% of its fetches on 404 pages and another 14.36% following redirects, with Claude’s crawler showing a near-identical ~34% 404 rate. Separately, Rutgers/Wharton research (published December 2025) found publishers who fully blocked AI crawlers saw a 23.1% total traffic decline without reliably cutting citation rates — an argument for fixing broken paths rather than blocking wholesale. See our status-codes post for the hop-by-hop data.
Layer 6: Does the AI know who you are?
Entity resolution sits upstream of citation. Google’s Knowledge Graph holds more than 500 billion facts about roughly 5 billion entities, and consistent name/URL/description across Wikidata, LinkedIn, Crunchbase, and your own Organization schema is what triggers an entity merge into that graph. Wikidata has fed the Knowledge Graph since Google retired Freebase, and it has no eligibility bar the way Wikipedia notability does — making it the more accessible starting point for B2B brands. A May 2026 study of 153,425 AI citations found 76.95% of cited URLs sat outside the organic top-10, which is strong evidence that entity recognition, not search rank, is the gate AI engines actually check. Run the check yourself: search your brand name on Wikidata and Google’s Knowledge Graph Search API and see if an entity exists at all. Our Wikidata post and entity-consistency post cover the mechanics.
Layer 7: Can you actually measure any of this?
The last layer is closing the loop — and it’s the leakiest one. Google’s native “AI Assistant” channel in GA4, rolled out in May 2026, still excludes Perplexity and Claude, so a custom regex-based channel group is effectively mandatory for complete tracking. Worse, an estimated 35–70% of AI referral sessions arrive with no referrer header at all and land in “Direct,” meaning actual AI-driven traffic is likely double what GA4 reports natively. Market-share context as of March–April 2026: ChatGPT holds roughly 62.6% of measurable B2B AI referrals, Claude 18.5%, Gemini 10.6%, and Perplexity 7.3%. None of this captures prompts or citations shown without a click. For setup steps, see how to measure AI referral traffic, and for how the citation decision itself gets made, how ChatGPT, Perplexity, and Gemini choose citations differently.
The seven layers at a glance
| Layer | What it checks | Current pass rate / benchmark | Related deep dive |
|---|---|---|---|
| 1. Crawler access | robots.txt allows/blocks GPTBot, ClaudeBot, PerplexityBot | ~25% of top sites block GPTBot; ~30% run a training-vs-search split | AI crawler permissions |
| 2. llms.txt | Machine-readable site manifest exists | 8.7%–10.1% adoption; no measured citation lift | llms.txt explained |
| 3. Structured data | JSON-LD present with populated concrete fields | Populated Product/Review schema: 61.7% vs 41.6% citation rate | Schema markup for AI |
| 4. Server-rendered HTML | Key content visible in raw view-source | 0% of major AI crawlers render JS as of June 2026 | JS content parseability |
| 5. Status code hygiene | Redirect hops and 404 rates within crawler tolerance | ChatGPT crawler: 34.82% of fetches hit 404s | Status codes and AI visibility |
| 6. Entity consistency | Brand resolves as an entity in Wikidata / Knowledge Graph | 76.95% of cited URLs sit outside the organic top-10 | Wikidata for brands |
| 7. Citation/referral measurement | AI traffic is tracked beyond default GA4 channels | 35–70% of AI referral sessions arrive with no referrer | Measuring AI referral traffic |
Taken together, these seven checks are also the practical version of the question our cluster hub asks at a higher level — how AI assistants decide which brands to recommend. Access and parseability (layers 1, 4, 5) determine whether an engine can see you; structured data and entity consistency (layers 3, 6) determine whether it trusts what it sees; llms.txt and measurement (layers 2, 7) are instrumentation, not levers.
FAQ
Do I need to pass all seven layers to be “AI visible”?
No single layer is sufficient or fully necessary on its own, but layers 1, 4, and 5 are gating: if a crawler can’t reach or parse the page, nothing downstream matters. Layers 3 and 6 are the trust layer that most affects whether visible content gets cited. Layers 2 and 7 are diagnostic, not causal.
Is llms.txt worth implementing if it shows no citation lift?
The Trakkr Research scan of 37,894 domains and separate reports of near-zero request rates suggest it isn’t currently a citation lever. It can still function as a low-cost, structured manifest for future tooling, but it shouldn’t be prioritized ahead of crawler access or JS rendering fixes.
Which layer has the most disagreement between studies?
Structured data. BrightEdge reported a 44% citation increase from schema plus FAQ blocks, while Ahrefs’ 1,885-page tracked study found no major uplift from adding schema alone. The likely reconciling factor is specificity — populated, concrete attribute fields appear to matter more than schema’s mere presence — but the studies don’t agree on magnitude.
Last updated September 2, 2026. This page is refreshed as benchmarks and scores move.
Pingback: How to Measure AI Referral Traffic (ChatGPT, Perplexity, Gemini) - Tech Blog
Pingback: Why JavaScript-Rendered Content Is Invisible to AI - Tech Blog