llms.txt is a simple, sensible spec with a measurement problem: roughly one in ten websites now ships the file, yet server-log studies covering 137,000 domains show 97% of those files received zero AI requests in May 2026 — and no major AI lab has committed to reading it in production. That gap between adoption and consumption is the entire story. This post covers what the spec actually says, what the adoption and log data show, and the one use case where the file demonstrably earns its keep.
What the spec actually says
The proposal comes from Jeremy Howard (Answer.AI), published September 3, 2024, and maintained at llmstxt.org. The premise: LLM context windows are too small to ingest whole websites, and converting HTML full of navigation, ads, and JavaScript into clean text is lossy. So a site should publish a single markdown file at /llms.txt acting as a curated table of contents for machines.
The format is deliberately minimal: one H1 with the site name, a blockquote summary, then H2 sections grouping links with one-line descriptions. An optional companion, /llms-full.txt, inlines the full content of every linked page into one file for tools that want everything in a single fetch. That variant was co-developed with Anthropic and documentation platform Mintlify, which began auto-generating both files for all hosted docs in November 2024 — instantly giving sites like Anthropic, Cursor, and Pinecone compliant files with zero configuration.
One thing the spec is not: a standard. It has no W3C or IETF status. It is an informal convention with a GitHub repo, in the same lifecycle stage robots.txt occupied in 1994 — except robots.txt had crawler compliance from day one, and llms.txt, as we’ll see, does not.
Adoption: real, growing, and oddly uniform
Adoption numbers depend heavily on the sample. Three 2026 datasets worth knowing:
- SE Ranking checked roughly 300,000 domains and found a 10.13% adoption rate — and the rate is strangely flat across traffic tiers: 9.88% for near-zero-traffic sites, 10.54% for mid-traffic, 8.27% for high-traffic (100,001+ visits). Adoption is not concentrated where you’d expect sophistication to concentrate.
- Rankability’s June 2026 tracker found 8.7% of the top 1,000 sites shipping the file.
- PPC Land reported adoption rising 8.8x year over year — the headline continues “…but 97% of files get zero AI requests,” which is the correct way to read every number in this section.
Where adoption does cluster meaningfully is developer documentation, because platforms like Mintlify generate the file automatically. Much of the measured “adoption” is a platform default, not a deliberate decision.
Does anything actually read it?
This is the question that matters, and the log data is unambiguous. The largest study to date is Ahrefs’ analysis of server logs across 137,000 domains, published June 2026: 97% of llms.txt files received zero requests in May 2026. Among the few files that were fetched, GPTBot accounted for 4.51% of requests, ClaudeBot 0.80%, and DeepseekBot 0.02%. A separate 48-day single-site log study (February–March 2026, ~12,000 AI bot requests analyzed) reached the same conclusion: no AI crawler looks for the file as a matter of routine.
| Source | Sample | Finding | Date |
|---|---|---|---|
| Ahrefs server-log study | 137,000 domains | 97% of llms.txt files got zero AI requests; GPTBot 4.51% of the requests that did occur, ClaudeBot 0.80% | June 2026 (May 2026 logs) |
| SE Ranking | ~300,000 domains | 10.13% adoption; no correlation with AI citation frequency | 2026 |
| Rankability | Top 1,000 sites | 8.7% adoption | June 2026 |
| Wislr log analysis | 1 site, 48 days, ~12,000 AI bot requests | No AI crawler routinely requests llms.txt | Feb–Mar 2026 |
| Google (John Mueller) | — | No Google Search system reads or acts on llms.txt | 2025, reaffirmed since |
Google’s John Mueller put it bluntly on Reddit: “AFAIK none of the AI services have said they’re using LLMs.TXT (and you can tell when you look at your server logs that they don’t even check for it). To me, it’s comparable to the keywords meta tag.” The comparison stung enough that Search Engine Land ran a rebuttal — but note what Mueller actually claimed: not that the file is spam-prone like meta keywords, just that nothing consumes it, so it doesn’t matter. On the consumption question, the logs side with him.
The correlation question
Could the file help even if crawlers fetch it rarely? SE Ranking tested this directly, feeding llms.txt presence into an XGBoost model predicting AI citation frequency across its 300k-domain dataset. The result, covered by Search Engine Journal: removing the llms.txt variable improved the model’s prediction accuracy. The file added noise, not signal. Both the statistical analysis and the ML model showed no effect on how often a domain gets cited.
This matches what we found when mapping how AI assistants decide which brands to recommend: citation selection runs on retrievability, third-party corroboration, and content structure — not on self-declared manifests. A file where a site owner describes their own importance is, evidentially, worth about what self-declared importance is usually worth.
The one real use case: agentic retrieval
Here is the reconciliation for an apparent paradox: no major lab commits to reading llms.txt, yet Anthropic, Stripe, Cloudflare, and Hugging Face all publish one. The answer is that the file’s real consumer is not a training crawler or a search index — it’s a coding agent at inference time.
When a developer tells Claude Code or Cursor to “read the Anthropic API docs and write the integration,” the agent can fetch docs.anthropic.com/llms.txt as a curated map and traverse only the pages it needs, instead of parsing HTML navigation. In that workflow the file works today, which is why documentation-heavy developer companies ship it and why agent traffic (Claude-Code was the #2 bot in Wislr’s logs) is the traffic worth watching. The same fetch-at-inference-time behavior is a documented workaround for the training-cutoff problems catalogued in our ChatGPT limitations ledger entry.
So the honest framing: llms.txt is not an AI-visibility tool. It is developer-documentation UX for agents.
Should you ship one?
Decision rule, based on the data above: if you run developer docs, an API reference, or content that coding agents fetch at inference time — yes, and generate it automatically rather than by hand. If you run a marketing site, a blog, or an e-commerce catalog hoping the file will get you cited by ChatGPT or Perplexity — the 300k-domain evidence says it will do nothing, and the effort is better spent on the things that measurably move citations. The cost of shipping one is near zero, so it can’t hurt; the error is expecting it to help. Treat any vendor claiming llms.txt “boosts AI visibility” with the same skepticism you’d apply to an unverifiable benchmark score — the measurement mindset from our guide to evaluating LLMs applies to marketing claims too.
FAQ
Is llms.txt an official standard like robots.txt?
No. It’s an informal proposal (Jeremy Howard, September 2024, maintained at llmstxt.org) with no W3C or IETF status. Even robots.txt only became an IETF standard (RFC 9309) in 2022, decades after de facto adoption — llms.txt has the spec but not the de facto adoption by consumers.
Does llms.txt affect Google rankings or AI Overviews?
No. John Mueller has confirmed no Google Search system reads the file, and SE Ranking’s 300k-domain analysis found no correlation between the file’s presence and AI citation frequency in any engine.
What’s the difference between llms.txt and llms-full.txt?
llms.txt is a curated index — links plus one-line descriptions. llms-full.txt inlines the complete content of every linked page into one large file for tools that want everything in a single fetch. Documentation platforms like Mintlify generate both automatically.
Last updated July 20, 2026. This page is refreshed as benchmarks and scores move.