Does llms.txt work? In 66 days of my server logs, AI crawlers fetched my Markdown pages 118 times and llms.txt never

Does llms.txt work? Start with what it was built to do

The idea comes from a single page, llmstxt.org, published by Jeremy Howard on September 3, 2024 and last modified on August 10, 2026. Its opening line is the whole pitch: “We propose adding a /llms.txt markdown file to websites to provide LLM-friendly content.” The file starts with an H1 holding the name of the site, which the page calls “the only required section”, then a blockquote with a short summary, then sections of links with a line of description each.

There is a second half to the proposal that gets less attention. The same page asks sites to “provide a clean markdown version of those pages at the same URL as the original page”, with .md on the end. So the proposal is two things: an index of the site, and a Markdown copy of each page. The index points a language model at what matters, and the copies give it the text without menus, scripts and footers.

The proposal is also clear about who it expected to read the file. “Our expectation was that llms.txt would mainly be useful for inference rather than training,” it says, and “llms.txt information is instead used on demand, when an agent needs information about a topic while assisting a user.” That changes the question. Asking whether a training crawler downloads the file misses the point; the useful question is whether anything that reads the web, for a person or for an index, uses either half.

The status matters too. The page describes itself as “A proposal to standardise on using an /llms.txt file.” Robots.txt, by contrast, is an IETF standard, RFC 9309, published in September 2022. A crawler that follows standards has every reason to fetch robots.txt before anything else. Nothing obliges it to look for llms.txt, and nothing on a normal page tells it the file exists.

This site publishes both halves, which is what makes it a usable test. Its /llms.txt has been at the root since the site’s first day; today it is 3,892 bytes with an H1, a summary, the services and the notes. Every page and note also has a Markdown copy, announced in the page’s own <head> with <link rel="alternate" type="text/markdown">, which I described in the two ways to serve Markdown to agents, and how each one fails quietly. The llms.txt file mentions the .md pattern once, with a single example URL. Everything else in it is ordinary links.

What the published studies found

“Does llms.txt work?” is a question several studies have already put to data, and they are worth reading before a single site’s log, because they measure breadth that one site cannot.

Adoption, and whether it changes citations

SE Ranking analyzed nearly 300,000 domains and found that “only 10.13% had an LLMs.txt file in place.” It then tested whether having the file is associated with being cited, using Spearman correlation and an XGBoost model. Its conclusion, published October 15, 2026, is blunt: “Our analysis of 300,000 domains shows that LLMs.txt doesn’t impact how AI systems see or cite your content today.” SE Ranking sells AI visibility tools, so it is a vendor study, but the method is described and the sample is large.

Requests for the file itself

Ahrefs looked at requests. “Our study focuses on all 137,210 domains in Ahrefs Web Analytics that received traffic in May 2026,” and of the llms.txt files among them, “97% of those files received zero traffic in May 2026.” It also named the requesters it could: “19.5% of fetches came from named AI tools,” and “GPTBot is top and Claude-Code is second, ahead of every AI search and assistant bot.” Keep that last line in mind: GPTBot does fetch llms.txt on some sites.

Otterly ran one experiment site and read its server logs for 90 days: “Out of 62,100+ total AI bot visits over 90 days, just 84 requests targeted the /llms.txt file.” That is about one request in 740.

What Google and the crawler operators say

Google answers the question for its own search directly. Its guide to optimizing for generative AI features lists llms.txt under things you can ignore: “You don’t need to create new machine readable files, AI text files, markup, or Markdown to appear in Google Search (including its generative AI capabilities), as Google Search itself doesn’t use them.” The same guide adds: “It’s completely fine if you decide to create and maintain LLMS.txt files (or other similar files) for other services or systems that use these files. Doing so will neither harm nor help your site’s visibility or rankings in Google Search.” Its page on AI features and your website says the same in fewer words.

The crawler documentation of the AI companies is about user agents, robots.txt and address lists. OpenAI’s page describes three: “GPTBot is used to make our generative AI foundation models more useful and safe”, “OAI-SearchBot is used to surface websites in search results in ChatGPT’s search features”, and ChatGPT-User, which “is not used for crawling the web in an automatic fashion” because it acts when a user asks. Anthropic’s article covers ClaudeBot, Claude-User and Claude-SearchBot and says “Anthropic’s Bots respect ‘do not crawl’ signals by honoring industry standard directives in robots.txt.” Perplexity’s page covers PerplexityBot and Perplexity-User. Neither the OpenAI page nor the Anthropic article asks site owners for an llms.txt. The OpenAI page does mention one, though: its own. “For the complete documentation index, see llms.txt.” That is the use the proposal had in mind, documentation that an agent reads on demand.

What the studies leave open

The studies answer how often the file is fetched across many sites, and whether having it goes with being cited. What an aggregate cannot show is what the same crawlers did instead, on the same site, in the same weeks, and whether they used the other half of the proposal. That is what one site’s log can show, if the requests are checked properly.

How I counted, and what I left out

To answer “does llms.txt work” for this site, I used the access log of its web server: 100,582 lines from August 28, 2026 at 19:38 UTC to November 2, 2026 at 17:19 UTC, the current file plus the one archived on October 2. I counted requests to /llms.txt and to /llms-full.txt, which this site does not publish, and every request from every crawler that names an AI or search company in its user agent.

A user agent is a claim, not an identity. Anyone can send GPTBot in a header, and many do, as I found when I checked the AI crawlers on this site against their owners’ address lists. So a request counts as GPTBot here only if the user agent says GPTBot and the address is in the list OpenAI publishes for GPTBot. I downloaded 13 lists on November 2, 2026: OpenAI’s three (gptbot.json, searchbot.json and chatgpt-user.json), Anthropic’s bots.json, Perplexity’s two, Google’s four (Googlebot, special crawlers and the two lists of user-triggered fetchers), Bing’s, Apple’s and Common Crawl’s.

1,152 requests used one of those crawler names from an address that is not on its operator’s list. I report them apart and attribute them to no one. That includes every Claude-User and Perplexity-User request in the log: none came from a listed address. Anthropic’s wording is that a source address on its list “indicates that the crawler is coming from Anthropic”; it does not say every request from Anthropic comes from one. So those requests are unattributed, not accused. None of them asked for llms.txt either.

My own requests are separated too: anything from this server’s addresses, and anything with the user agent of my checker, which follows the rules on how I measure. That includes the free llms.txt checker on this site, which fetches a file and tests it against the llmstxt.org format whenever someone runs it, including on this domain.

One mistake of mine is worth publishing, because it produces a convincing wrong answer. My first pass verified nothing: zero GPTBot, zero Googlebot, every address “not on the list.” The cause was a literal quotation mark that this server’s log format writes in front of each address, so "66.249.74.169 never parsed as an address. A script that had reported “all AI crawlers here are fake” would have been wrong in a way that sounds like a finding. If a log analysis gives you a dramatic zero, look at one raw line before you believe it.

Who asked for llms.txt

Almost three requests in four were mine: 168 of 229, or 73%. It is the same pattern I found for the six .well-known files this site publishes, where 89% of the traffic was me. So if “does llms.txt work” means “does anyone read it”, the answer here is: mostly the person who wrote it, checking that it is still there.

The 36 browser requests came from 31 addresses. Some are people opening the file to see what one looks like; some may be scripts that send a browser’s user agent. I did not try to tell them apart, because the count is small either way.

The bots are the interesting part. The machines that read llms.txt here are the ones that check whether a site has one. AgentTrustBot announces its own purpose in its user agent as “AI agent directory”; agentcensus and AgentReadinessScanner say what they do in their names. They asked for the file 15 times between them, and one asked for /llms-full.txt as well. These are graders, and a grader reads the file to score it, whether or not any assistant ever will. I have been graded by one of them before: an agent readiness score gave this site Level 5 for failing its only check.

Then come Dataprovider, a web-data crawler, three times, and Meta’s crawler once. Meta publishes no list of crawler addresses that I could check, so that single request stays a claim.

What the AI crawlers fetched instead

The crawlers were not absent. They were busy, on everything except the file written for them.

Put together, the verified crawlers asked for robots.txt 821 times and for sitemaps 657 times. Those are the two files a crawler goes looking for on its own: robots.txt is a standard, and this site’s robots.txt names the sitemap. ClaudeBot is the sharpest case: it requested the sitemap 363 times and robots.txt 360 times, more than any other crawler here, and never the index that was designed for language models. Its last verified visit was on October 18.

OpenAI’s three crawlers made 1,269 verified requests between them and none for llms.txt. That does not contradict Ahrefs, which saw GPTBot as the most frequent named fetcher across 137,210 domains. It means GPTBot fetches the file somewhere, rarely, and in 66 days it did not do it here. Googlebot’s zero is what Google’s own guide says to expect.

ChatGPT-User is the case the proposal describes most closely: an agent that reads a site on demand because a person asked. It came 52 times, each time from a different address, and each time it went straight to an HTML page. What those visits do and do not prove is a separate question, which I took apart in what can be measured about generative engine optimization, and what is guesswork. For this question the answer is plain: the agent that works for a user did not start from the index.

Nor did anyone block them. This site’s robots.txt names 18 AI user agents and gives them all Allow: /, with a Content-Signal line that says search=yes, ai-input=yes, ai-train=yes. I explained that file line by line in the note on robots.txt for AI crawlers. The crawlers were welcome to everything, llms.txt included, and they took everything else.

The half of the proposal that did get used

The Markdown copies got 118 verified requests: 50 from GPTBot, 41 from ClaudeBot, 24 from OAI-SearchBot and 3 from bingbot. Every one of them was answered with a 200. So the crawlers did take the second half of the proposal, the clean text of each page, while ignoring the first.

The log also says how they found the copies. All 50 GPTBot requests and all 24 OAI-SearchBot requests for a Markdown copy carry a referrer, and in every case it is an HTML page of this site. ClaudeBot and bingbot send no referrer at all. Not one Markdown request names llms.txt as the page it came from. The crawlers met the <link rel="alternate" type="text/markdown"> tag in the head of a page they were already reading and followed it.

Two cautions. Fetching a Markdown copy proves that a crawler took it, not that any answer used it. And I can only guess at the reason for the difference. My guess, and it is only that, is that a tag inside a page reaches a crawler while it does what it already does, while llms.txt is a separate file at a separate address that nothing in robots.txt, the sitemap or the pages points to. If that is right, the useful half of llms.txt is the one that does not need llms.txt.

What my log cannot see

Cloudflare sits in front of this site and keeps copies of some responses. On November 2 I requested three kinds of URL twice each, with GET rather than HEAD, because a HEAD request can report a cache status that a real request would not get. /llms.txt came back HIT, the HTML pages came back HIT, and the Markdown copies came back DYNAMIC, which means Cloudflare passed every one of them through to my server. So the 118 Markdown requests are complete, while the zero for llms.txt is a floor. A crawler that asked for the file while Cloudflare held a fresh copy got its answer at the edge, and that request never reached my log. The page counts in the table above are floors for the same reason.

Cloudflare can count what my log cannot. Its AI Crawl Control screen counts AI crawler requests on Cloudflare’s side, and a Path filter narrows them to a single file. On November 2 I set it to the last seven days. First a control: /sitemap_index.xml showed 23 requests, from BingBot 11, GPTBot 7, Googlebot 3, Amazonbot 1 and Bytespider 1, the same 11, 7 and 3 that my log has for the three of them I can verify. Then /llms.txt: zero requests from any AI crawler, with any status.

The weekly totals per crawler agree too: 85 requests from GPTBot with a 2xx answer in Cloudflare’s count and 85 in my log, 93 and 93 for bingbot, 151 and 146 for Googlebot. So for that week, at least, my log saw the same crawler traffic that Cloudflare did, and at the edge nobody asked for llms.txt either. One oddity, if you try this yourself: the same filter on /robots.txt showed zero, while my log has 64 requests for it from AI crawlers that week. The dashboard does not seem to count robots.txt at all, and OAI-SearchBot’s total fits that: 65 in Cloudflare, 87 in my log, and 24 of my 87 were for robots.txt.

The other limit is scale. This is one small site, a consultant’s site rather than developer documentation, measured over 66 days. It says what happened here, not what happens on the web; for that, the studies above are the better evidence, and their numbers point the same way.

So should you publish an llms.txt?

If you already have one, there is little reason to delete it. Google says it will “neither harm nor help” your visibility in Google Search, and the file costs a few kilobytes. If you run developer documentation, the case is stronger: Ahrefs found Claude Code second among named fetchers, and OpenAI points readers of its own docs to its llms.txt. Coding agents are pointed at documentation on purpose, which is the use the index was designed for.

The real cost is maintenance. An llms.txt is a second copy of facts that already live on your pages, and second copies drift. I learned that the hard way when I changed one number on one page and five other places went on saying the old one, llms.txt among them. A stale index read by an agent is worse than no index at all.

If you are deciding where to spend an afternoon, spend it on the things the crawlers in this log actually requested: robots.txt that says what you mean, a sitemap that is current, pages that answer with a 200 to the crawlers you want, and, if you like the idea, a Markdown copy announced in each page’s head. That is the work behind the crawler and AI readiness audit I offer: requesting a site the way each crawler does, one at a time, and showing what comes back. What I will not do is promise that a file will get you cited. Nobody who has read these logs can promise that.

How to check who reads your llms.txt

You do not need a study to answer “does llms.txt work” for your own site. You need your access log, the published address lists, and ten minutes.

1. List every request for the file

On a server with the usual combined log format, this lists the user agents that asked for the file, most frequent first:

zgrep -h -E '"(GET|HEAD) /llms(-full)?\.txt' /path/to/access_log* \
  | awk -F'"' '{print $6}' | sort | uniq -c | sort -rn | head -20

If your log format writes a quotation mark at the start of each line, as mine does, every field moves one place to the right and the user agent is $7 instead of $6. Run it once on a few lines and read the output before trusting the totals.

2. Check the addresses, not the names

For each crawler that matters to you, download its operator’s list and test the addresses that used its name. For GPTBot:

curl -s https://openai.com/gptbot.json -o gptbot.json
python3 - 192.0.2.10 <<'EOF'
import ipaddress, json, sys
ip = ipaddress.ip_address(sys.argv[1])
nets = [ipaddress.ip_network(p.get('ipv4Prefix') or p.get('ipv6Prefix'))
        for p in json.load(open('gptbot.json'))['prefixes']]
print(any(ip in n for n in nets if n.version == ip.version))
EOF

The address in the example is a documentation address, so the answer is False; put in the addresses from your own log. The same pattern works with Anthropic’s https://claude.com/crawling/bots.json, Perplexity’s two lists and Google’s, although some use different field names, so look at the file before parsing it.

3. Find out what your CDN answers from cache

If a CDN sits in front of your server, your log only sees what the CDN passes through. Ask with GET, not HEAD:

curl -s -o /dev/null -D - https://example.com/llms.txt | grep -i -E '^(cf-cache-status|age):'

HIT means the CDN answered that request from its own copy and your server never saw it, so your count is a floor. DYNAMIC on every try means the CDN passes every request through, and your log sees them all.

4. Ask the CDN for its own count

On Cloudflare, the dashboard has a section called AI Crawl Control with a Metrics tab. Choose Add filter, then Path, then equals, and type /llms.txt: the charts show the AI crawlers that requested the file, counted on Cloudflare’s side rather than in your server log. Before you trust a zero, run the same filter on a file you know crawlers request, such as your sitemap. Do not use robots.txt for that test: on this site the dashboard showed zero for it in a week with 64 requests. Cloudflare’s documentation for that screen lists the filters and what each column counts.

Does llms.txt work: frequently asked questions

Does Google use llms.txt?

Not for Google Search. Google’s guide to generative AI features says you don’t need “new machine readable files, AI text files, markup, or Markdown to appear in Google Search (including its generative AI capabilities), as Google Search itself doesn’t use them.” It also says keeping such a file for other services is fine and will neither harm nor help your rankings. In this site’s log, Googlebot made 817 verified requests in 66 days and none was for llms.txt.

Does ChatGPT read llms.txt?

Not on this site, in 66 days. OpenAI’s three crawlers, GPTBot, OAI-SearchBot and ChatGPT-User, made 1,269 requests from addresses on OpenAI’s lists, and none asked for the file. GPTBot did fetch 50 Markdown copies of pages, each time arriving from an HTML page of the site. Across many sites the picture is not zero: Ahrefs saw GPTBot as the most frequent named fetcher of llms.txt files, while 97% of the files it studied got no traffic at all in May 2026.

Is llms.txt the same thing as robots.txt?

No. Robots.txt is an IETF standard, RFC 9309, that tells crawlers what they may request, and the operators’ own documentation says their crawlers follow it. Llms.txt is a proposal for a Markdown index of a site, meant to help a language model find the important pages. It grants or denies nothing. Blocking a crawler in llms.txt does nothing; blocking it in robots.txt does.

Do I need an llms-full.txt as well?

The llmstxt.org page does not define one. This site does not publish one, and in 66 days the requests for it came from my own tests and from one agent directory, all answered with a 404. If you run a small site, a Markdown copy of each page, announced in the page’s head, did more here than any extra file.

Can an llms.txt file hurt my rankings?

According to Google, no. Its guide says keeping one for other services will “neither harm nor help your site’s visibility or rankings in Google Search.” The risk is not in rankings; it is in accuracy. An index that says something your pages no longer say gives any agent that does read it the wrong answer.

How can I tell whether AI crawlers read my llms.txt?

Search your access log for requests to the file, check each requesting address against the operator’s published list, and check whether your CDN answers the file from cache. The four steps above take about ten minutes. If you use Cloudflare, AI Crawl Control can filter its metrics by path; test the filter on your sitemap before you trust a zero.

What llms.txt is, and what it is not

Llms.txt is a reasonable idea: a short, curated map of a site, written for a reader that is a program. On this site, over 66 days, no verified crawler of OpenAI, Anthropic, Google, Perplexity, Microsoft, Apple or Common Crawl asked for it, while the same crawlers made 4,059 requests for everything else, including 118 Markdown copies requested from the pages that announce them. The readers it did have were me, a few dozen browsers, and the bots that grade sites for having one.

So does llms.txt work? It works as a file: it is there, it is valid, and checkers find it. It does not work as a way to be read by the crawlers people usually mean when they recommend it. Keep it accurate if you keep it, and put the effort where this log says the crawlers actually go.

Sources

• llmstxt.org, The /llms.txt file: the proposal by Jeremy Howard, published September 3, 2024 and modified August 10, 2026; the format, the Markdown copies of pages, and the expected use at inference time.
• RFC 9309, Robots Exclusion Protocol: the IETF standard for robots.txt, September 2022.
• Google Search Central, Optimizing your website for generative AI features on Google Search: llms.txt and AI text files are not used by Google Search, and keeping them neither harms nor helps.
• Google Search Central, AI features and your website: no new machine readable files or AI text files are needed.
• OpenAI, Overview of OpenAI crawlers: GPTBot, OAI-SearchBot and ChatGPT-User, and their address lists.
• Anthropic, Does Anthropic crawl data from the web, and how can site owners block the crawler?: ClaudeBot, Claude-User, Claude-SearchBot, robots.txt and the address list.
• Perplexity, Perplexity crawlers: PerplexityBot, Perplexity-User and their address lists.
• Cloudflare, Analyze AI traffic: the Metrics tab of AI Crawl Control and its path filter.
• SE Ranking, LLMs.txt: why brands rely on it and why it doesn’t work: nearly 300,000 domains, October 15, 2026.
• Ahrefs, llms.txt study: 137,210 domains in May 2026, published June 15, 2026.
• Otterly, The llms.txt experiment: one site, 90 days of server logs.
• My own measurements, November 2, 2026: this site’s access log from August 28 to November 2, 2026, checked against 13 address lists downloaded that day from OpenAI, Anthropic, Perplexity, Google, Bing, Apple and Common Crawl, and the cache status of /llms.txt, an HTML page and a Markdown copy, requested through Cloudflare with GET. Cloudflare AI Crawl Control, Metrics, for the seven days to November 2, 2026, filtered by path.