# robots.txt for AI crawlers: every user agent, what it does, and how to verify it

> robots.txt for AI crawlers, agent by agent: what each one does, what its operator says about the file, and who read mine in 55 days of verified logs.

Source: https://visibleorigin.com/robots-txt-ai-crawlers/
Published: 2026-10-26

---

**The short version.** robots.txt asks crawlers to stay out of certain paths; it cannot stop anyone. For AI crawlers, the list of names is the easy part. Their operators sort them into four kinds: crawlers that collect training data, crawlers that build search and answer indexes, fetchers that load a page because a person asked, most of which may skip the file, and two tokens that never visit at all. I checked 55 days of my own server’s log against the addresses each operator publishes. Anthropic read my robots.txt right before 349 of its 350 visits, OpenAI read it under a single name for all its bots, and Amazon read it once. No verified crawler asked for a path the file disallows, and why that zero proves less than it seems is part of the answer.

![robots.txt and AI user agents in four groups: training crawlers, search crawlers, user fetchers and tokens that never visit](https://visibleorigin.com/wp-content/uploads/ai-user-agents-four-kinds.webp)Most lists of AI user agents mix these four kinds of names. A Disallow means something different for each, and for most fetchers that act for a person it may not apply.

Many guides to robots.txt for AI crawlers are a list: copy these user agents, put `Disallow: /` under them, done. Lists go stale, and that is the smaller problem. The bigger one is that a list treats every name as the same kind of thing. Some names belong to crawlers that read the file before every visit. Some belong to fetchers whose own operators say the file may not apply to them. Two are not crawlers at all. And **no list can tell you whether anyone read your rules, let alone followed them**.

This note does both halves. First, every AI user agent I could find documented by its operator, with what the operator says it does and what it says about robots.txt, quoted from the operator’s own page as it read on October 22, 2026. Then a measurement on one real site, this one: which crawlers asked for the file, how often, what they did next, and what my server could not see because my CDN answered first. The log runs from August 28 to October 22, 2026, every crawler in it was checked by address rather than by name, and the commands are included so you can repeat the checks on your own site.

## What robots.txt can and cannot do for an AI crawler

The rules are old; the standard is new. RFC 9309, published in September 2022, “specifies and extends the ‘Robots Exclusion Protocol’ method originally defined by Martijn Koster in 1994,” and it is careful about what that method is. Its rules are ones that crawlers “are requested to honor,” and the next line is blunt: “These rules are not a form of access authorization.” The security section says it again: the protocol “is not a substitute for valid content security measures,” and listing paths in the file “exposes them publicly and thus makes the paths discoverable.”

So **robots.txt is a request addressed to a name**. A crawler looks for the group that names it, reads the rules in that group and decides for itself whether to follow them. Nothing in the protocol checks that the name is honest, and nothing stops a client that never reads the file. Cloudflare’s documentation for its managed robots.txt puts it in four words: “robots.txt compliance is voluntary.” What the file governs, for the crawlers that choose to honor it, is fetching: which paths on which host they may request. It says nothing about what an operator does with a page it already has, and that gap is why two names on most AI lists are not crawlers at all, but switches for how crawled content may be used.

The rules are also per host. The standard puts the file at “scheme:[//authority]/robots.txt,” so `example.com` and `www.example.com` each have their own, and a crawler that visits both asks twice. My `www` host answers with a redirect to the main one, and in 55 days Googlebot asked it for robots.txt 48 times and OpenAI 26 times. Anthropic spells out what that means for an opt-out: add the block “for every subdomain that you wish to opt out from.”

## Every AI user agent, grouped by what it does

The useful way to read a list of AI user agents is by job, because the operators divide their bots that way and **a rule for one job says nothing about another**. OpenAI’s page puts it in one line: “Each setting is independent of the others.” Perplexity’s says “Each setting works independently,” and Amazon’s says the same of its three agents. Below, each operator speaks for itself.

### Crawlers that collect training data

| User agent | What its operator says |
| --- | --- |
| `GPTBot` | OpenAI uses it “to crawl content that may be used in training our generative AI foundation models.” Disallowing it “indicates a site’s content should not be used in training generative AI foundation models.” |
| `ClaudeBot` | Anthropic: it collects “web content that could potentially contribute to their training.” Restricting it “signals that the site’s future materials should be excluded from our AI model training datasets.” |
| `Amazonbot` | Amazon: “used to improve our products and services” and its data “may be used to train Amazon AI models.” |
| `meta-externalagent` | Meta: it “crawls the web for use cases such as training foundation AI models or improving products by indexing content directly.” |
| `CCBot` | Common Crawl, which publishes its crawl as open data. Its page gives `Disallow: /` as the opt-out and adds: “we are aware of crawlers falsely identifying themselves as CCBot.” |

Quoted from each operator’s crawler documentation on October 22, 2026. Applebot also feeds training, through a separate switch covered below.

### Crawlers that build search and answer indexes

| User agent | What its operator says |
| --- | --- |
| `OAI-SearchBot` | OpenAI: “used to surface websites in search results in ChatGPT’s search features.” Sites that opt out “will not be shown in ChatGPT search answers, though can still appear as navigational links.” |
| `Claude-SearchBot` | Anthropic: it “navigates the web to improve search result quality for users.” Disabling it “prevents our system from indexing your content for search optimization.” |
| `PerplexityBot` | Perplexity: “designed to surface and link websites in search results on Perplexity. It is not used to crawl content for AI foundation models.” |
| `DuckAssistBot` | DuckDuckGo: it crawls “in real-time for our AI-assisted answers,” and “This data is not used in any way to train AI models.” A Disallow takes effect “after 72 hours.” |
| `Amzn-SearchBot` | Amazon: for “search experiences in Amazon products and services”; it “does not crawl content for generative AI model training.” |
| `Meta-WebIndexer` | Meta: it “navigates the web to improve Meta AI search result quality for users.” |
| `Googlebot` | Google: its rules “affect Google Search (including Discover and all Google Search features), as well as other products.” |
| `Applebot` | Apple: it powers search “including Spotlight, Siri, and Safari,” and its data “may also be used to help train Apple foundation models.” |

Quoted from each operator’s documentation on October 22, 2026. Googlebot and Applebot are search crawlers first; what they collect also feeds their operators’ AI models, which is what the two tokens below are for.

### Fetchers that load a page because a person asked

**This is the group where a Disallow means least.** These agents are not crawling: a person asked an assistant something, and the assistant went to get one page. Most of their operators say plainly that robots.txt may not bind them.

| User agent | What its operator says about robots.txt |
| --- | --- |
| `ChatGPT-User` | OpenAI: it “is not used for crawling the web in an automatic fashion. Because these actions are initiated by a user, robots.txt rules may not apply.” |
| `Perplexity-User` | Perplexity: “Since a user requested the fetch, this fetcher generally ignores robots.txt rules.” |
| `meta-externalfetcher` | Meta: it fetches “individual links at a user’s request,” and “this crawler may bypass robots.txt rules.” |
| `Amzn-User` | Amazon: “Because actions taken by Amzn-User can be initiated by a user, it may not follow all robots.txt directives.” |
| Google’s user-triggered fetchers | Google: “Because the fetch was requested by a user, these fetchers ignore robots.txt rules.” |
| `Claude-User` | Anthropic, the exception: “Disabling Claude-User on your site prevents our system from retrieving your content in response to a user query.” |

Of the six operators in this table, only Anthropic documents that its user agent follows a Disallow.

### Two tokens that never visit

Google-Extended and Applebot-Extended turn up on lists of AI crawlers, mine included, and neither is one. Google: “Google-Extended doesn’t have a separate HTTP request user agent string. Crawling is done with existing Google user agent strings; the robots.txt user-agent token is used in a control capacity.” It decides whether content Google crawls “may be used for training future generations of Gemini models” and for grounding in Gemini Apps and Vertex AI, and it “does not impact a site’s inclusion in Google Search nor is it used as a ranking signal in Google Search.” Apple is just as direct: “Applebot-Extended does not crawl webpages,” it “is only used to determine how to use the data crawled by the Applebot user agent,” and pages that disallow it “can still be included in search results.”

**A Disallow for these two tokens changes what happens to your pages after a crawler has them**, not whether a crawler comes. It also means any request in a log that carries one of these names was written by someone else. My log has 46 requests calling themselves Google-Extended, all from the same five scanning addresses that appear later in this note.

## How a crawler finds its rules in robots.txt

Once a crawler has the file, RFC 9309 tells it how to find its rules. Three of those rules surprise people who write the file by hand. A fourth comes from the operators rather than the standard, and the fifth problem comes from the tools people use to test the file.

### A named group replaces the * group

A crawler matches its name (the standard calls it a product token) against the user-agent lines: “Crawlers MUST use case-insensitive matching to find the group that matches the product token and then obey the rules of the group.” If several groups name it, their rules “MUST be combined into one group.” Only if no group names it does the crawler fall back to the group for `*`. Google’s documentation states the consequence in one line: “User agent specific groups and global groups (*) are not combined.” **The moment you give GPTBot a group of its own, nothing in your `*` group applies to it.** A group that says only `User-agent: GPTBot` and `Allow: /` opens every path the `*` group was keeping closed. My own file has a comment about exactly this, and repeats its exclusions inside its AI group for that reason.

### The longest rule wins, and Allow wins a tie

**Inside the group, the order of the lines does not matter.** “The most specific match found MUST be used. The most specific match is the match that has the most octets.” When an allow rule and a disallow rule are equivalent, “the ‘allow’ rule SHOULD be used.” Rules may use `*` for any run of characters and `$` for the end of the path, and the file itself is always open: “The /robots.txt URI is implicitly allowed.” Here is what that means for three crawlers reading my file:

| Request | Group, deciding rule and result |
| --- | --- |
| GPTBot asks for `/wp-admin/admin-ajax.php` | The AI group. `Allow: /wp-admin/admin-ajax.php` (24 characters) beats `Disallow: /wp-admin/` (10). Allowed. |
| GPTBot asks for `/?s=robots` | The AI group. `Disallow: /?s=` (4 characters) beats `Allow: /` (1). Disallowed. |
| Applebot asks for `/notes/` | No group names Applebot or Googlebot, so the `*` group. No rule matches. Allowed. |
| AhrefsBot asks for `/notes/` | The group for SEO crawlers. `Disallow: /` matches every path. Disallowed. |

RFC 9309 measures specificity in octets of the rule’s path. The file is shown in full further down.

### Lines a crawler does not know must not break a group

Files now carry records that RFC 9309 does not define, such as `Sitemap` and the newer `Content-Signal`. The standard lets crawlers read them and sets one limit: “Parsing of other records MUST NOT interfere with the parsing of explicitly defined records,” and its example is that “a ‘Sitemaps’ record MUST NOT terminate a group.” My file puts a `Content-Signal` line between the user-agent lines and the rules of two groups, the same place Cloudflare’s managed format puts it. **A parser that treated that line as the end of a group would leave every crawler named above it with no rules at all.**

### Two operators add fallbacks of their own

Apple: “If robots instructions don’t mention Applebot but mention Googlebot, the Apple robot will follow Googlebot instructions.” Amazon says that when a file does not mention Amzn-SearchBot but allows other search bots, it “will crawl in accordance with the robots.txt directives given to other search bots.” Neither rule is in the standard, so a generic parser will not apply it. The README of Google’s open-source robots.txt library, which Google describes as slightly modified production code used by Googlebot, says the library does not handle logic “outside of parsing and matching,” and gives Googlebot-Image following the Googlebot group as the example. **If your file has a Googlebot group and nothing for Applebot, Applebot reads the Googlebot group, not your `*` group.**

### Not every parser follows the standard, and my file shows it

The longest-match rule is the one tools break most easily. Python ships a robots.txt parser in its standard library, and the version on this server, Python 3.9, returns the first rule that matches in the order the lines appear. I gave it my own file:

```
$ python3 -c '
import urllib.robotparser as r
p = r.RobotFileParser(); p.parse(open("robots.txt").read().splitlines())
for ua, path in [("GPTBot", "/wp-admin/"), ("GPTBot", "/?s=test"),
                 ("Googlebot", "/wp-admin/admin-ajax.php")]:
    print(ua, path, p.can_fetch(ua, "https://visibleorigin.com" + path))'
GPTBot /wp-admin/ True
GPTBot /?s=test True
Googlebot /wp-admin/admin-ajax.php False
```

All three answers are wrong under RFC 9309. My AI group starts with `Allow: /`, which matches every path, so a first-match parser stops there and lets GPTBot into `/wp-admin/`. My `*` group lists `Disallow: /wp-admin/` before the exception for `admin-ajax.php`, so the same parser closes that file to Googlebot. In CPython’s source, the 3.9, 3.10 and 3.12 branches return on the first match, while the 3.13, 3.14 and 3.15 branches keep the longest one, as the standard requires. **A robots.txt checker is only as right as the algorithm inside it.** The cheap defense is to write the file so both algorithms agree: the most specific lines first, and no `Allow: /`, which adds nothing under the standard. My file does not do that today; it is correct for any crawler that follows RFC 9309, which is what Google documents for its own.

## When robots.txt itself fails: errors, redirects, size and caching

**What a crawler does when it cannot read the file matters more than most of the rules inside it**, and this is where operators part ways with the standard and with each other.

| Situation | RFC 9309 | What operators document |
| --- | --- | --- |
| The file returns a 4xx, such as 404 | “the crawler MAY access any resources on the server” | Google treats every 4xx except 429 “as if a valid robots.txt file didn’t exist” |
| The file returns a 5xx or times out | “the crawler MUST assume complete disallow” | Google stops crawling for 12 hours, then uses the last good copy for up to 30 days. Amazon: “When a file can’t be fetched, they will behave as if it does not exist” |
| The file redirects | follow “at least five consecutive redirects” and apply the rules “in the context of the initial authority” | Google follows at least five hops. My `www` host redirects to the main one |
| The file is large | the parsing limit “MUST be at least 500 kibibytes” | Google ignores anything after 500 KiB. Mine is 1,610 bytes |
| Caching, and how long a change takes | “SHOULD NOT use the cached version for more than 24 hours” | Google, Perplexity and Meta: up to 24 hours. OpenAI: ~24 hours for search. DuckDuckGo: 72 hours for an opt-out. Amazon: a copy “from the last 30 days” |

RFC 9309 sections 2.3 to 2.5, and each operator’s page as of October 22, 2026.

Two rows deserve a second look. The 5xx row is where a maintenance page can do real damage. Under the standard, a crawler that gets a 503 for robots.txt must keep out of the whole site, and Google does for its first 12 hours. Amazon documents the opposite reaction, so **the same outage reads as “keep out” to one crawler and as “no rules” to another**. The cache row is why a change is never instant. OpenAI warns that “it can take ~24 hours from a site’s robots.txt update for our systems to adjust.” Amazon’s page gives both a figure of “~24 hours for our systems to reflect changes” and a cached copy “from the last 30 days,” and those are not the same promise.

## This site’s robots.txt, and why it names 18 AI agents it does not block

This is the file as my CDN served it on October 22, 2026. WordPress generates it; there is no file on disk. It is served as `text/plain; charset=utf-8`, which is what the standard asks for.

**Update, November 13, 2026.** AhrefsBot and SemrushBot are no longer blocked, so anyone can look this site up in those tools. The file below is the one from October 22, and the counts in this note come from that period.

```
# SEO scrapers: crawl budget spent, nothing returned.
User-agent: AhrefsBot
User-agent: SemrushBot
User-agent: DotBot
User-agent: MJ12bot
User-agent: rogerbot
User-agent: BLEXBot
User-agent: DataForSeoBot
User-agent: Barkrowler
User-agent: serpstatbot
User-agent: SEOkicks
User-agent: sistrix
User-agent: ZoominfoBot
User-agent: Screaming Frog SEO Spider
User-agent: PetalBot
Disallow: /

# AI crawlers, named one by one and welcome. This site sells machine readability;
# blocking the machines would be a strange way to prove it. Named explicitly so
# the welcome is verifiable, not just asserted in a comment.
# Note: robots.txt groups do not inherit, so the exclusions below are repeated here.
User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: Claude-User
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Google-Extended
User-agent: CCBot
User-agent: Applebot-Extended
User-agent: Meta-ExternalAgent
User-agent: Amazonbot
User-agent: Bytespider
User-agent: cohere-ai
User-agent: Diffbot
User-agent: YouBot
User-agent: Timpibot
Content-Signal: search=yes, ai-input=yes, ai-train=yes
Allow: /
Disallow: /wp-admin/
Disallow: /?s=
Disallow: /*?replytocom=
Allow: /wp-admin/admin-ajax.php

# This site exists to show that a website can be readable by agents.
# It blocks none of them, on purpose.
User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=yes
Disallow: /wp-admin/
Disallow: /?s=
Disallow: /*?replytocom=
Allow: /wp-admin/admin-ajax.php

Sitemap: https://visibleorigin.com/sitemap_index.xml
```

It has three groups. The first shuts fourteen SEO crawlers out completely. The second names eighteen AI agents one by one and gives them exactly the rules of the third, the `*` group: stay out of the admin area except its AJAX endpoint, out of internal search results, and out of comment-reply links. So **the AI group changes nothing for the crawlers it names**. It is there because this site sells machine readability, and a welcome stated only in a comment cannot be checked by a machine. The explicit names and the `Content-Signal` line, with search, AI input and AI training all set to yes, are that policy in a form a parser can read. **Not blocking AI crawlers is a decision here, not an oversight.**

Read with the categories above, the eighteen names are less uniform than they look: five training crawlers, three search crawlers, three fetchers whose operators mostly say the file may not bind them, two tokens that never visit, and five names (Bytespider, cohere-ai, Diffbot, YouBot and Timpibot) whose documentation I did not check for this note. The group also misses agents that operators document today, such as Meta-WebIndexer, Amzn-SearchBot and DuckAssistBot. Those fall through to the `*` group, which has the same rules, so nothing changes for them. **On a site that allows everything, a named group is documentation; on a site that blocks, a missing name is a hole.**

Two more details. The `Content-Signal` record is the format Cloudflare documents for its managed robots.txt, with three uses: “search (building a search index), ai-input (feeding content into AI models for real-time answers), and ai-train (training or fine-tuning AI models).” None of the crawler documentation pages quoted in this note mentions it, so **I treat it as a statement of intent rather than a rule anyone has promised to follow**. And one line in the first group, `Screaming Frog SEO Spider`, is not a valid product token under RFC 9309, which allows only letters, underscores and hyphens; whether that tool matches the line is up to the tool.

## Who read it: 55 days of my log, checked by address

The log is my server’s own access log from August 28, 2026 at 19:38 UTC, the day the site was set up, to October 22 at 16:13 UTC: 77,486 lines. I removed my own requests, took the crawler name each request claimed in its User-Agent, and counted a request as verified only if its address was inside the ranges that crawler’s operator publishes, downloaded on October 22. Why that check matters, and how badly names lie without it, is the subject of [the note where 346 requests claimed to be AI crawlers and their owners’ IP ranges disagreed](https://visibleorigin.com/ai-crawlers-verified-by-ip/). Only one result from it matters here: every unverified request carrying the name GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Amazonbot, Applebot, DuckAssistBot or Bingbot came from **five addresses that each wore between 14 and 21 crawler names**. For those eight names, checking against today’s range files lost nothing: every address outside the files was one of the five.

| Operator | Requests for robots.txt (other requests) | Pattern |
| --- | --- | --- |
| Anthropic | 360 (663) | ClaudeBot asks right before 349 of 350 visits |
| OpenAI | 170 (705) | All 170 under the name OAI-SearchBot; GPTBot never asked |
| Google | 162 (448) | Googlebot asks; GoogleOther never does |
| Microsoft | 30 (145) | Bingbot: 51 requests came over 24 hours after the last read my server saw |
| Perplexity | 8 (16) | 7 of 9 visits began with a read |
| DuckDuckGo | 6 (3) | Each visit began with a read |
| Apple | 2 (0) | Read the file twice and crawled nothing |
| Amazon | 1 (787) | One read in 55 days, on October 11 |

Requests that reached my server from addresses each operator publishes, August 28 to October 22, 2026. A visit is a run of requests with no gap longer than an hour.

![Bar chart of how often each crawler operator read this site’s crawler rules in 55 days, beside its other requests](https://visibleorigin.com/wp-content/uploads/who-read-robots-file-55-days.webp)Each operator reads the file its own way: Anthropic before every visit, OpenAI under one name for the whole family, Google once a day for all its crawlers, Amazon once in 55 days.

### Anthropic reads the file right before it crawls

ClaudeBot asked for robots.txt 360 times, on 38 different days, from 27 addresses, and got the file every time. The pattern never changed: a request for robots.txt, then the crawl, seconds later. Of its 350 visits, 349 began within a minute of a ClaudeBot request for the file, and none of its 663 other requests came more than two hours after one. **That is the strictest reading of the standard I saw from anyone.** Claude-SearchBot and Claude-User never arrived from an address Anthropic publishes: all 42 requests calling themselves Claude-SearchBot came from the five scanners, and the 183 calling themselves Claude-User came from 16 addresses, none of them on Anthropic’s list.

### OpenAI reads it once, under one name, for the whole family

OpenAI asked for robots.txt 170 times, from 51 addresses, on 53 different days, and every one of those requests carried the name OAI-SearchBot. Of those, 101 used the user-agent string OpenAI documents for exactly this: “When fetching robots.txt files, we may use a user-agent string with an additional robots.txt marker.” GPTBot made 429 verified requests in the same period, from 25 addresses, and not one was for robots.txt. It did not crawl blind, though. **Of GPTBot’s 49 visits, 40 began within a minute after an OAI-SearchBot request for the file**, and the other nine within 24 hours of one. OpenAI’s page describes the shape: “If your site has allowed both bots, we may use the results from just one crawl for both use cases to avoid duplicative crawling.” On my server, robots.txt looks like the plainest case of that sentence: one read, under the search bot’s name, for the family. The 30 verified ChatGPT-User requests never asked for the file either, which OpenAI’s own description of that agent allows.

### Google shares one copy for a day

Googlebot asked 162 times, from 52 addresses: 113 times on the main host, 48 on `www`, and once on the day the site was set up, before the file existed, when the answer was a 404. GoogleOther, whose rules Google says “don’t affect any specific product,” **made 167 requests and never asked for the file**, and each of its 37 visits began within 24 hours of a Googlebot read. Google documents both halves: it “generally caches the contents of robots.txt file for up to 24 hours,” and “The cached response may be shared by different crawlers.” No Google request in 55 days came more than 23.7 hours after Googlebot had last read the file.

### Amazon read it once in 55 days

**Amazonbot is the outlier.** It made 788 verified requests from 351 addresses, the first on September 2, and asked my server for robots.txt once, on October 11 at 22:17 UTC, after 623 other requests. Amazon documents a long cache: its crawlers “will fetch host-level robots.txt files or use a cached copy from the last 30 days.” A 30-day copy can explain most of that gap. It is still longer than the 24 hours the standard recommends, and none of the 623 requests before October 11 was preceded, as far as my server can tell, by a read of the file. “As far as my server can tell” is doing real work in that sentence, and the next section is about why.

### The rest

PerplexityBot (8 reads, 16 other requests), DuckAssistBot (6 and 3) and Applebot (2 and 0) came too rarely to show more than this: they mostly read the file when they came, and Applebot read it once on each host and left. Bingbot, Microsoft’s crawler and my second search-engine baseline, read it 30 times on 20 days for 145 other requests, and 51 of those came more than 24 hours after the last read my server saw. Whether Bingbot kept a copy longer or read the file through my CDN, my server cannot say.

## Did they obey it? A zero that proves less than it looks

Verified AI crawlers made 2,721 requests to my server in 55 days: 547 for robots.txt and 2,174 for everything else. **None of the 2,174 was for a path the file disallows.** Neither were any of the verified Googlebot, GoogleOther or Bingbot requests. Requests for disallowed paths do reach my server, 2,953 of them from 74 addresses, most from a single probe hammering a WordPress installer and from ordinary browser strings, but none from an address in any of the range files I checked. The only two that carried an AI name, one ChatGPT-User and one meta-externalagent, both asked for `/wp-admin/` three minutes apart on October 21, from addresses their operators do not list.

It would be easy to stop there and write that AI crawlers respect robots.txt. That sentence would not survive a technical reader, because **my file disallows almost nothing these crawlers had a reason to request**. It closes the admin area, internal search and comment-reply links, and no page on this site links to any of them: there are no comments, and the only trace of search on the home page is a template in its structured data. A crawler that never had a reason to ask for a path obeys a rule about it by default. A zero like that is consistent with obedience; it is not evidence of it.

The stronger test is in the group that shuts crawlers out. Six of the fourteen SEO crawlers my file blocks entirely showed up: AhrefsBot, MJ12bot, DotBot, DataForSeoBot, SemrushBot and serpstatbot. Between them they made 16 requests, **all 16 for robots.txt, and none for a page**, although every page on the site was open to everyone else. I did not verify those addresses, so this is a statement about requests carrying those names, but it is the kind of result a disallow-all group can produce and an allow-almost-all group cannot. On the other side, the five addresses that wore 14 to 21 crawler names made 1,029 requests between September 11 and October 21, and asked for robots.txt twice. Ten AI names in my log, among them Claude-SearchBot, Perplexity-User, CCBot and Google-Extended, arrived only from those five addresses, and none of the ten ever asked for the file.

**The way to make the AI result mean something is a canary**: disallow a path nobody else uses, link to it from a page crawlers do read, and watch for verified addresses asking for it over several weeks. I have not run one on this site yet, so the claim I can make is the narrow one: in 55 days, no verified AI crawler requested a disallowed path, on a site that gave them almost no occasion to. Separating what a crawler is told from what it does, and from what the layers in front of the server let through, is the work of the [crawler and AI readiness audit](https://visibleorigin.com/services/#2-crawler-and-ai-readiness-audit), which starts from the same files and goes through the server, CDN and cache layers behind them.

## What the origin log cannot see

Every number above is a request that reached my server. **My CDN keeps a copy of robots.txt at its edge for 30 minutes**: that is the edge cache lifetime in the cache rule on this zone, and the headers agree. On October 22, I asked for the file at 16:13:49 UTC and got `cf-cache-status: MISS`, which my server logged. Later requests came back as `HIT`, with an `age` of 30 to 36 seconds and, at 16:32:18, of 1,109 seconds, and none of them reached my log. At 16:44:30 the answer was `EXPIRED`: the copy was more than 30 minutes old, the edge came back to my server, and the log had a line again.

![A crawler, a CDN edge that keeps a copy for 30 minutes and the origin server, whose log records only what the edge passed on](https://visibleorigin.com/wp-content/uploads/origin-log-cdn-edge-window-test.webp)Requests the edge answers never reach the server’s log. A missing read counts only when the edge had no copy to answer with, and the crawler still did not ask.

So a crawler that asked for robots.txt while the edge held a fresh copy left no trace on my server, and **every count in this note is a floor, not a total**. I measured how large that blind spot is for ordinary pages in [the test where my access log recorded 6 of 60 requests](https://visibleorigin.com/access-log-cannot-see-the-edge/). For robots.txt, the gap matters most for the claims of absence: GPTBot never asking, Amazonbot asking once.

There is one way around it. An edge copy has to come from somewhere, and every copy starts as a request that reached my server and is in the log. If nobody’s request for robots.txt reached my server in the hour before a crawler arrived, no edge anywhere could have held a fresh copy for the 30 minutes before its first request, and a read in that window would have been logged. At the start of 3 of GPTBot’s 49 visits, no request for robots.txt from anyone had reached my server in the previous hour, so GPTBot did not ask for the file in the half hour before it began. For Amazonbot it was 127 visits out of 313, and for Googlebot 60 out of 172, consistent with the cached copies both companies document. **An absence in an origin log means something only when you can show the edge had nothing to answer with.**

The edge can also refuse. A firewall rule or a bot-blocking feature at the CDN answers before the origin, and **a refused crawler leaves no line in the log either**. My zone refuses a handful of login and probe paths at the edge, none of them a page a crawler needs. If yours blocks more, your log will show a crawler that seems to have stopped coming, and as I found when [I made each layer of my site send a 403](https://visibleorigin.com/403-forbidden-which-layer/), the layer that refused is usually readable from the response, not from your server.

## How to verify robots.txt for AI crawlers on your own site

These are the steps behind every number in this note, in the order I would run them on a site I had never seen.

### 1. Fetch the file the way a crawler does

From outside your network, with a GET, not a HEAD, on every host a crawler might use. Check the status code, the content type, the size, and whether a CDN is caching it:

```
$ curl -sS -o robots.txt -D - https://example.com/robots.txt | grep -iE '^(HTTP|content-type|cache-control|age|cf-cache-status)'
$ curl -sS -o /dev/null -w '%{http_code} %{redirect_url}\n' https://www.example.com/robots.txt
$ wc -c robots.txt
```

**A 5xx here is an emergency** for the crawlers that follow the standard; a 4xx means no rules at all; a redirect is fine within five hops.

### 2. Work out which group each agent gets

For each name you care about, find the group that names it, merge duplicates, fall back to `*` only if none does, and apply the longest-match rule to the paths that matter. Remember Apple’s Googlebot fallback, and that the two tokens never visit. The quick way to see this for any domain is the [AI visibility checker](https://visibleorigin.com/ai-visibility-checker/), which reads a site’s robots.txt bot by bot and names the group and rule that decided each one. To test a single URL against a single agent with the code Googlebot uses, Google publishes [its parser](https://github.com/google/robotstxt) with a small command-line tool. Whatever you use, test it once against a file where the order of the lines and the length of the rules disagree, as mine do.

### 3. Check who is asking by address, not by name

Download the range files each operator publishes, such as [OpenAI’s for GPTBot](https://openai.com/gptbot.json) or [Anthropic’s for its bots](https://claude.com/crawling/bots.json), and keep only the requests whose address falls inside the list for the name they carry. Google and Apple also document reverse DNS, and Common Crawl offers both. Counting by name inflates: in my log, 512 requests carry the name GPTBot and 429 come from OpenAI’s addresses. This short script reads a list of addresses and marks each one:

```
$ cat inrange.py
import ipaddress, json, sys
nets = [ipaddress.ip_network(p.get("ipv4Prefix") or p.get("ipv6Prefix"))
        for p in json.load(open(sys.argv[1]))["prefixes"]]
for line in sys.stdin:
    ip = ipaddress.ip_address(line.split()[-1])
    ok = any(ip in n for n in nets if n.version == ip.version)
    print("listed    " if ok else "NOT LISTED", line.rstrip())

$ curl -s https://claude.com/crawling/bots.json -o bots.json
$ grep '"GET /robots.txt' access.log | grep 'ClaudeBot' | awk '{print $1}' | tr -d '"' \
    | sort | uniq -c | python3 inrange.py bots.json
```

On my log, that last command lists 27 addresses, all listed, accounting for all 360 requests.

### 4. Line up each operator’s reads with its crawling

Count the requests for robots.txt by verified name, then group each crawler’s other requests into visits and look at how long before each visit the operator last read the file. That is how the three patterns above showed up: a read seconds before every visit, a read under a sibling’s name, or a read that serves for a day.

```
$ grep '"GET /robots.txt' access.log \
    | grep -oE '(GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-User|PerplexityBot|Googlebot|bingbot|Amazonbot|Applebot|DuckAssistBot)/' \
    | sort | uniq -c | sort -rn
```

The slash after each name is there on purpose. Bing’s user agent carries the word `bingbot` twice, once as the product and once in the address of its help page, so without the slash every Bingbot request counts double. On my log the first version of this command reported 60 reads by Bingbot; there were 30.

### 5. Look for disallowed paths, and ask whether they were reachable

Search the log for requests to the paths your file disallows, keep the ones from verified addresses, and **before reading anything into a zero, check whether your site links to any of those paths**. If it does not, plant a canary and wait.

### 6. Find out what your CDN answers on its own

Ask for robots.txt twice with a GET and read `cf-cache-status` or your CDN’s equivalent, together with `age`. Find the edge cache lifetime in your cache rules. Then treat every count from your origin log as a floor, and every absence as unproven unless the edge had nothing to answer with.

### 7. Know what a borrowed name can tell you

Sending a request with `curl -A "GPTBot"` **shows how your site treats that name**: a rule at the CDN or in a security plugin that matches the string will fire. It does not show what the real GPTBot receives, because the layers that recognize the big crawlers do it by address, and your test comes from somewhere else. A test from a rented server adds a second distortion, which I measured when [my free check turned out to run from a datacenter address that some sites refuse outright](https://visibleorigin.com/free-check-runs-from-a-datacenter/). For what the real crawler got, read the status codes on the log lines from its verified addresses, and remember the edge.

If you would rather have this done on your own domain, with every request and response written down so you can repeat it, the [crawler and AI readiness audit](https://visibleorigin.com/services/#2-crawler-and-ai-readiness-audit) starts from the files in steps 1 and 2 and goes through the server, CDN and cache layers that decide what each crawler receives.

## robots.txt for AI crawlers: frequently asked questions

### Does blocking GPTBot in robots.txt keep my site out of ChatGPT?

Not entirely. GPTBot is OpenAI’s training crawler, and disallowing it tells OpenAI not to use your content for training. Appearing in ChatGPT search is controlled by OAI-SearchBot, and OpenAI says sites that disallow it “will not be shown in ChatGPT search answers, though can still appear as navigational links.” ChatGPT-User, which loads a page when a user asks, is the agent for which OpenAI says “robots.txt rules may not apply.”

### Do AI crawlers respect robots.txt?

The major operators document that their crawlers do, and most say their user-initiated fetchers may not. On this site, in 55 days, no crawler verified by address requested a path the file disallows, but the file gave them almost no reason to. The six SEO crawlers it shuts out completely asked for robots.txt 16 times and for nothing else. Ten AI names in my log arrived only from five scanning addresses that wore other companies’ names, and none of the ten ever asked for the file.

### What is the difference between Google-Extended and Googlebot?

Googlebot is a crawler: it fetches pages for Google Search and its features. Google-Extended is only a name you write in robots.txt. Google says it has no user agent string of its own and is “used in a control capacity” to decide whether crawled content may train Gemini models or ground Gemini Apps and Vertex AI. Disallowing it does not remove a site from Google Search, and Applebot-Extended works the same way for Apple.

### How long does a robots.txt change take to reach AI crawlers?

Up to a day for most. RFC 9309 says a cached copy should not be used for more than 24 hours. OpenAI cites about 24 hours for search, Perplexity and Meta up to 24 hours, and DuckDuckGo 72 hours for an opt-out. Amazon cites about 24 hours for changes but also a cached copy from the last 30 days. On my server, no Google request came more than 24 hours after Googlebot’s last read, and Amazonbot read the file once in 55 days.

### How do I block all AI crawlers with robots.txt?

There is no wildcard for “AI.” You name each agent in a group with `Disallow: /`, including the two tokens Google-Extended and Applebot-Extended, and keep the list current as operators add agents. `User-agent: *` with `Disallow: /` blocks search engines too. Fetchers acting for a user may still come, by their operators’ own account, so enforcement has to happen at the server or CDN, by verified address. Anthropic warns that blocking its addresses instead of using robots.txt can stop it from reading the file at all.

### Can I test robots.txt by sending requests as GPTBot?

Only partly. A request with GPTBot’s name shows how your site and its CDN treat that name. It cannot show what the real GPTBot receives, which is decided by its address, and a test from a datacenter may be refused for reasons unrelated to the name. To see what real crawlers got, read the log lines from their verified addresses.

## The general shape of this

**A list of AI user agents answers the question nobody needs help with**: which names to write down. The useful questions come after it. Which of those names is a crawler, a fetcher acting for a person, or a switch for what happens to content later? Which group does each one read, under an algorithm the file’s author may not share? And did the crawler read the file before it came? On this site the answers were measurable and uneven: one operator reads before every visit, one reads for the whole family under a single name, one shares a daily copy, and one read the file once in eight weeks.

**The one thing the log could not show is obedience, because my file gave the crawlers nothing to disobey.** That is not a reason to skip the check. It is the reason to design it properly: verified addresses, a path the crawlers know about, and a CDN accounted for. Without those three, a robots.txt report tells you what was written, not what happened.

## Sources

- [RFC 9309, Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309.html): group selection, longest match and ties, other records, access results for 4xx, 5xx and redirects, caching and the size limit.
- [OpenAI, Overview of OpenAI Crawlers](https://developers.openai.com/api/docs/bots): GPTBot, OAI-SearchBot, ChatGPT-User, the robots.txt marker and the shared crawl.
- [Anthropic, Does Anthropic crawl data from the web](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler): ClaudeBot, Claude-SearchBot, Claude-User, subdomains and blocking by address.
- [Perplexity, Perplexity Crawlers](https://docs.perplexity.ai/docs/resources/perplexity-crawlers): PerplexityBot and Perplexity-User.
- [Google, Google’s common crawlers](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers): Googlebot, GoogleOther and Google-Extended.
- [Google, How Google interprets the robots.txt specification](https://developers.google.com/crawling/docs/robots-txt/robots-txt-spec): errors, caching shared between crawlers, the size limit and group precedence.
- [Google, Verify requests from Google crawlers and fetchers](https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests): reverse DNS, and user-triggered fetchers and robots.txt.
- [Apple, About Applebot](https://support.apple.com/en-us/119829): Applebot, Applebot-Extended and the Googlebot fallback.
- [Amazon, About AmazonBot](https://developer.amazon.com/amazonbot): Amazonbot, Amzn-SearchBot, Amzn-User, the 30-day cache and unreadable files.
- [Meta, Meta Web Crawlers](https://developers.facebook.com/documentation/sharing/webmasters/web-crawlers): meta-externalagent, Meta-WebIndexer and meta-externalfetcher.
- [DuckDuckGo Help Pages, DuckAssistBot](https://duckduckgo.com/duckduckgo-help-pages/results/duckassistbot): purpose and the 72-hour opt-out.
- [Common Crawl, CCBot](https://commoncrawl.org/ccbot): opt-out, impersonation and verification.
- [Cloudflare, robots.txt setting](https://developers.cloudflare.com/bots/additional-configurations/managed-robots-txt/): voluntary compliance and the Content-Signal format.
- [Google’s robots.txt parser and matcher library](https://github.com/google/robotstxt): the production parser and its limits.
- CPython’s [robotparser in the 3.12 branch](https://github.com/python/cpython/blob/3.12/Lib/urllib/robotparser.py) and [in the main branch](https://github.com/python/cpython/blob/main/Lib/urllib/robotparser.py): first match against longest match.
- My own measurements: this server’s access log from August 28 to October 22, 2026, checked against the range files OpenAI, Anthropic, Perplexity, Google, Microsoft, Apple, Amazon, DuckDuckGo and Common Crawl published on October 22, and requests for robots.txt through my CDN on the same day.