How to read the result
The gauge counts the bots that decide whether a site can appear in AI search and AI answers: the six search crawlers and the three on-demand fetchers. It shows how many of those nine your robots.txt allows. Open means all nine, Partly closed means at least one is disallowed, and Closed means none of them gets in. When robots.txt cannot be read from this server, the gauge says so instead of guessing.
Below it there is one ring per group. Search and answers and On-demand fetches turn green when every bot in the group is allowed, amber when some are not and red when none is. Training is always drawn in blue or gray: allowing or refusing it is a decision about training, and the operators document it apart from search.
Then every bot gets a row with its verdict. Open a row to see what the bot does, a link to its operator’s documentation and the robots.txt rule that decided it. Rows with a problem, a search or on-demand bot that is blocked or only partly allowed, open by themselves.
Training, search and on-demand fetches are three separate decisions
Most robots.txt advice treats “AI bots” as one thing to allow or block. The operators do not. OpenAI, Anthropic and Perplexity each run separate agents for separate jobs, and blocking one says nothing about the others:
- Training. GPTBot collects pages that “may be used to train” OpenAI’s models, and blocking it is OpenAI’s documented way to opt out of that (OpenAI). ClaudeBot does the same job for Anthropic (Anthropic).
- Search and answers. OAI-SearchBot decides whether a site can appear in ChatGPT search, and OpenAI says plainly that sites that block it “will not be shown in ChatGPT search answers.” Claude-SearchBot and PerplexityBot play the same role for Claude and Perplexity. Perplexity states that PerplexityBot is not used to train foundation models (Perplexity).
- On-demand fetches. ChatGPT-User, Claude-User and Perplexity-User load a single page because a person asked for it. OpenAI says robots.txt rules “may not apply” to ChatGPT-User, and Perplexity says its user fetcher “generally ignores robots.txt rules.” Anthropic says blocking Claude-User stops that retrieval.
So a site can refuse training and still be cited, or allow training and vanish from AI search. The checker groups the results the same way, so the trade-off is visible at a glance.
What each bot does, according to its operator
| Bot | Operator | What a Disallow changes |
|---|---|---|
| GPTBot | OpenAI | Pages are not used to train OpenAI models |
| OAI-SearchBot | OpenAI | The site is not shown in ChatGPT search answers |
| ChatGPT-User | OpenAI | May not apply: the fetch is user-initiated |
| ClaudeBot | Anthropic | Future pages are excluded from training data |
| Claude-SearchBot | Anthropic | Pages are not indexed for Claude’s search |
| Claude-User | Anthropic | Claude cannot retrieve the page for a user |
| PerplexityBot | Perplexity | The site is not surfaced in Perplexity results |
| Perplexity-User | Perplexity | Generally ignored: the fetch is user-initiated |
| Google-Extended | No Gemini training or grounding; Search is unaffected | |
| Googlebot | The site leaves Google Search, AI Overviews included | |
| Bingbot | Microsoft | The site leaves Bing, which Copilot draws on |
| Applebot | Apple | Out of Siri, Spotlight and Safari search |
| Applebot-Extended | Apple | No Apple model training; search is unaffected |
| CCBot | Common Crawl | Pages stay out of the open Common Crawl archive |
| meta-externalagent | Meta | Pages are not crawled for Meta AI training or indexing |
Sources: OpenAI, Anthropic, Perplexity, Google, Apple, Common Crawl, Meta and Bing. Two of the rules the checker applies come from these pages: Applebot follows the Googlebot rules when a file has none for Applebot, and a robots.txt that fails with a server error counts as “disallow everything” under RFC 9309.
Google-Extended is not a crawler, and it does not touch Search
Google-Extended is a token you name in robots.txt; no bot with that name ever visits. Google says it controls whether content “may be used for training future generations of Gemini models” and for grounding in Gemini Apps and Vertex AI, and that it “does not impact a site’s inclusion in Google Search.” That second part cuts both ways. Blocking Google-Extended also turns off grounding in the Gemini app, which is more than training. And it does not keep a page out of AI Overviews or AI Mode, because those are part of Search: Google points to Googlebot’s robots.txt rules and to nosnippet, data-nosnippet, max-snippet and noindex as the controls for them (Google). That is why the checker also reads the homepage’s meta robots and X-Robots-Tag.
Applebot-Extended works the same way for Apple: it does not crawl, and disallowing it opts content out of Apple model training while “webpages that disallow Applebot-Extended can still be included in search results.”
Content signals and the block Cloudflare writes
A Content-Signal line in robots.txt states a preference for three uses: search (building a search index and showing results), ai-input (feeding content into an AI model to answer a question) and ai-train (training or fine-tuning models). Cloudflare proposed the format and can write it for sites that turn on Bot Preference Sync (called managed robots.txt until September 2026), together with Disallow rules for a list of AI crawlers, between # BEGIN Cloudflare Managed content and # END Cloudflare Managed Content. Cloudflare’s own documentation is clear that “compliance is voluntary,” and points to its crawler controls for actual enforcement (Cloudflare). The checker reports both: which signals the file carries, and whether that block is present, because a file you did not write yourself is easy to forget.
Why robots.txt is not the whole answer
Everything above is what a site asks. Whether a crawler actually receives a page is decided earlier, by the firewall, CDN or security plugin in front of the site, and those layers recognize the big crawlers by IP address. OpenAI, Anthropic, Perplexity and Common Crawl all publish the ranges their crawlers use, so a firewall can tell the real GPTBot from anyone typing its name.
That is why this checker sends its own User-Agent and never a crawler’s. A borrowed name shows how a site treats the name. It does not show what the real crawler receives, and this server is on nobody’s published list. A site can allow every AI bot in robots.txt and still answer them with a 403 at the edge; four layers can send that 403, and each leaves a fingerprint. The checker also runs from a datacenter, which some sites refuse whatever the request says. When that happens the result says so rather than guessing; the free check has the same limit, for the same reason.
What the checker reads
- robots.txt, parsed by RFC 9309: the group that names each bot, or the
*group when none does; the longest matching rule for the homepage; Allow wins a tie. Each result says which group and which rule decided it. - Content-Signal lines and the Cloudflare managed block.
- The homepage: meta robots for all crawlers, Googlebot and Bingbot, the
X-Robots-Tagheader, the JSON-LD types it declares, and which CDN answers. - llms.txt: whether it exists and is a text file. The llms.txt checker tests it in full. It is optional, and none of the crawler documentation linked on this page mentions it.
- The sitemap declared in robots.txt, or
/sitemap.xml, and how many URLs it lists.
Requests go one at a time, with a pause between them, under the rules in How I measure. The whole check stops after 20 seconds.
What is stored
Nothing about the domain or the result. Your IP address enters only as a keyed hash in a ten-minute counter that limits each visitor to five checks, as the Privacy Policy explains. The checker only fetches public addresses: it refuses private, loopback and cloud metadata addresses, ports other than 80 and 443, and re-checks every redirect.
AI visibility checker: frequently asked questions
How do I check if AI bots can crawl my site?
Start with robots.txt, where a site states what it allows, bot by bot. Enter your domain above: the checker parses the file the way a crawler would and shows, for each bot, the group and the rule that decide your homepage. That answers what your site asks. Whether a bot actually receives your pages is decided by the firewall or CDN in front of the site, and you can see that in your own access log once you verify each crawler by its IP address.
Does blocking GPTBot remove my site from ChatGPT search?
Not by itself. OpenAI uses GPTBot for its models and OAI-SearchBot for ChatGPT search, and says “each setting is independent of the others”: a site can allow OAI-SearchBot “in order to appear in search results while disallowing GPTBot”. Blocking OAI-SearchBot is what keeps a site out: such sites “will not be shown in ChatGPT search answers”, according to OpenAI. That is why the two bots sit in different groups here.
Does a fully open result mean AI answers will cite my site?
No. An open robots.txt removes one reason for a crawler to skip your pages. It does not make any AI system read, cite or rank them, and nobody can honestly promise that it will. It also says nothing about the layers in front of the site, which is the next question.
The checker says a bot is allowed. Why can it still not reach my site?
Because robots.txt is a request, not a gate. A CDN rule, a firewall or a security plugin can refuse a crawler before robots.txt is ever read; on Cloudflare, the AI Crawl Control block does it at the edge. This checker reads files and sends its own User-Agent, so it cannot see those layers, and a refusal can come from any of four of them.
Does Google-Extended control AI Overviews?
No. Google says Google-Extended “does not impact a site’s inclusion in Google Search”, and AI Overviews are part of Search. To limit what Google shows from your pages there, its AI features documentation says to “use nosnippet, data-nosnippet, max-snippet, or noindex”. The checker reads those directives from your homepage.
Can I use it to check my robots.txt?
For AI and search crawlers, yes: it parses the file by RFC 9309 and shows, for each of the 15 bots it knows, which group applied and which rule decided the homepage. It is not a line-by-line syntax validator, and it does not test other user agents or every path on your site.
When the files are not enough
The files are the part anyone can read. The crawler and AI readiness audit goes further: it requests the site under each crawler’s name, one at a time, reports the status code each one gets, and works through the server, CDN and cache layers that decide it. See all services, or send me your domain.