# robots.txt checker for AI crawlers

> Free robots.txt checker and validator: test any URL for GPTBot, ClaudeBot, Googlebot and 12 more bots, with the line that decides each one.

Source: https://visibleorigin.com/robots-txt-checker/
Updated: 2026-11-13

---

Enter a domain or a full URL. This free robots.txt checker fetches the site’s `/robots.txt`, validates every line against RFC 9309 and Google’s open-source parser, and tests the path you entered for 15 AI and search bots, from GPTBot and ClaudeBot to Googlebot and Bingbot. For each crawler it shows the group it follows and the exact line that decides, so you can see why a URL is allowed or blocked, not just that it is.

Domain or URLCheck robots.txt

Fetches /robots.txt only, with its own User-Agent, never a crawler's. Nothing about the domain, the path or the result is stored.

![robots.txt checker result for visibleorigin.com: a valid file, and all 15 AI and search bots allowed at the homepage](https://visibleorigin.com/wp-content/uploads/robots-txt-checker-result.webp)This site's robots.txt in the checker on November 12, 2026: every check passes, and all 15 bots may fetch the homepage. The training ring is blue because training is a separate decision.

## What the robots.txt checker tests

- **The file.** The status code and what crawlers do with it: a 404 means no rules at all, a server error means stop. Also the content type, the size against the 500 KiB that Google reads, UTF-8, and any redirect on the way.
- **Every line.** Fields crawlers do not know, misspellings that only Google forgives, lines with no colon, rules that come before any `User-agent`, paths that can never match, and rules Google stopped reading years ago, such as `Noindex`.
- **The rules as a whole.** How many groups and rules there are, whether the `*` group shuts out the whole site, which sitemaps are declared, and whether Cloudflare writes part of the file.
- **Crawler by crawler.** For the path you entered, each of the 15 bots comes out allowed or blocked, with its group and the line that decided. The file is shown underneath with line numbers, and the lines that matter are marked.

## How a crawler reads robots.txt

The checker applies the rules a compliant crawler applies. They are short, and most surprises come from one of them.

1. **A crawler reads only its own group.** A group is one or more `User-agent` lines followed by rules. If a group names the crawler, it follows that group and ignores the others, the `*` group included. Only a crawler that no group names falls back to `*`. If several groups name the same crawler, their rules are merged.
2. **Names are matched loosely, paths exactly.** A crawler name is compared without regard to case, and only up to the first character that is not a letter, hyphen or underscore, so `Googlebot/2.1` and `googlebot*` both mean `googlebot`. Paths are case-sensitive.
3. **The longest match wins.** Among the rules that match a URL, the one with the most characters decides. When an `Allow` and a `Disallow` are the same length, `Allow` wins.
4. **Two wildcards, nothing more.** `*` matches any run of characters and `$` marks the end of the URL. There are no regular expressions.
5. **No matching rule means allowed.** A path that no rule matches may be fetched.

One exception is built in. Apple says Applebot follows the Googlebot rules when a file has none for Applebot, so the checker does the same.

## What crawlers do when robots.txt is missing or broken

| robots.txt answers | What RFC 9309 and Google do | What the checker shows |
| --- | --- | --- |
| 200, with a text file | The rules apply. Google reads the first 500 KiB and ignores the rest. | Every check, and a verdict for each crawler |
| A redirect | Followed for at least five hops. After five, Google treats the file as missing. | The final address. The checker follows three hops. |
| 404, 410 or another 4xx | No rules: everything may be fetched. Google treats 401 and 403 the same way, but not 429. | “No robots.txt”, every crawler allowed. A 401, 403 or 429 is marked not measurable from here. |
| 5xx, a timeout or a network error | RFC 9309 says to assume everything is disallowed. Google stops crawling for 12 hours, then uses its last good copy for up to 30 days. | A server error, and no verdict per crawler |
| 200, with an HTML page | Usually no valid lines, so nothing is restricted. | An error: the site allows everything by accident |

## Common robots.txt mistakes it catches

- **Giving a crawler its own group and losing the defaults.** Add `User-agent: GPTBot` with a single `Disallow` line, and GPTBot stops reading the `*` group: every rule you wrote there for all crawlers no longer applies to it. The verdict for each crawler names the group it actually follows.
- **`Disallow: /` under `User-agent: *`.** Meant for one bot, it blocks every crawler without a group of its own, search engines included.
- **Misspelled fields.** Google’s parser accepts `Dissallow` and `User agent`, and a space where the colon should be. Other crawlers may skip those lines. A field nobody knows, such as `Dsallow`, does nothing at all.
- **Rules Google no longer reads.** `Noindex` in robots.txt stopped working at Google on September 1, 2019, and Google does not support `Crawl-delay`.
- **A relative sitemap.** `Sitemap: /sitemap.xml` is skipped. The line needs the full URL.
- **An HTML page at /robots.txt.** Common on sites that send every unknown path to the homepage with a 200.
- **Rules that Cloudflare writes.** When Cloudflare manages part of the file, that block can only be changed in the Cloudflare dashboard, not in your own copy.

## What robots.txt cannot do

robots.txt is a request. RFC 9309 says its rules “are not a form of access authorization”: a crawler that follows the protocol obeys them, and anything else can ignore them or borrow another crawler’s name. Three things follow from that.

- **It does not prove who is asking.** Anyone can send `GPTBot` as a user agent, which is why the operators publish their IP ranges. [346 requests said they were AI crawlers](https://visibleorigin.com/ai-crawlers-verified-by-ip/) on this site, and their owners’ IP ranges said otherwise.
- **Allowed is not the same as served.** A firewall, a CDN rule or a bot setting can refuse a crawler that robots.txt allows. [Cloudflare AI Crawl Control](https://visibleorigin.com/cloudflare-ai-crawl-control/) and [the four layers that can send a 403](https://visibleorigin.com/403-forbidden-which-layer/) show where that happens.
- **It does not take pages out of search.** Google says a blocked URL can still be indexed, without being visited, when other pages link to it. Keeping a page out of results takes a `noindex` meta tag or header, on a page crawlers are allowed to fetch.

## How the request is made

From this site’s server, in a datacenter in the United States, with the user agent `visibleorigin-check/0.3` and the rules on [How I measure](https://visibleorigin.com/how-i-measure/). A check is one request for `/robots.txt`, plus up to three redirects. The path you enter is never requested: it is only compared with the rules. The checker never sends a crawler’s user agent, so it reads the file anyone can read.

Some sites refuse datacenter addresses whatever the request says. When robots.txt answers 401, 403 or 429, the result says it is not measurable from here, instead of reporting rules that may only apply to this server. [What running from a datacenter costs a check](https://visibleorigin.com/free-check-runs-from-a-datacenter/).

Nothing is stored: not the domain, not the path and not the result, and nothing is cached. Your IP address only enters a counter, as a keyed hash that expires after ten minutes, to allow ten checks every ten minutes. The [Privacy Policy](https://visibleorigin.com/privacy-policy/#robots-txt-checker) says the same.

## robots.txt checker: frequently asked questions

### What is a robots.txt file?

A plain text file at the root of a site, such as `https://example.com/robots.txt`, that tells crawlers which paths they may fetch. RFC 9309 defines its format. Each crawler finds the group that names it, or the `*` group, and follows its `Allow` and `Disallow` lines.

### How do I test whether a URL is blocked by robots.txt?

Enter the full URL above, path included, for example `example.com/blog/post/`. The checker reads the site’s robots.txt and, for each of the 15 bots, says whether that path is allowed, which group the bot follows and which line decides. With a bare domain it tests the homepage.

### Does blocking GPTBot keep my site out of ChatGPT?

Not on its own. OpenAI runs separate crawlers: GPTBot collects pages that may be used to train its models, and OAI-SearchBot finds pages to show and link in ChatGPT search. OpenAI says sites that block OAI-SearchBot are not shown in ChatGPT search answers, and that robots.txt rules may not apply to ChatGPT-User, which fetches a page when a user asks for it. Each token is a separate decision, and the checker lists them separately.

### Why does my robots.txt work for Google but not for other crawlers?

Google’s parser is forgiving. It accepts a few common misspellings, such as `Dissallow`, and a space where the colon should be. A crawler that follows RFC 9309 to the letter skips those lines. The checker marks each one as a warning, so you can write the file in a way every crawler reads.

### Does robots.txt stop a page from being indexed?

No. It stops compliant crawlers from fetching the page, but Google can still index the URL if other pages link to it. To keep a page out of search, let crawlers fetch it and add a `noindex` robots meta tag or an `X-Robots-Tag` header. `Noindex` inside robots.txt has been ignored by Google since September 1, 2019.

### Why does the checker say robots.txt is not measurable?

The site answered 401, 403 or 429 to this checker, which runs from a datacenter, and many firewalls refuse datacenter addresses. That refusal applies to this checker’s address. What crawlers get depends on how your firewall treats their IP ranges, so check your firewall or CDN log for their requests.

## Further reading

- [robots.txt for AI crawlers](https://visibleorigin.com/robots-txt-ai-crawlers/): every user agent, what it does, and how to verify it.
- [AI visibility checker](https://visibleorigin.com/ai-visibility-checker/): robots.txt plus llms.txt, the sitemap and the homepage directives, bot by bot.
- [llms.txt checker and validator](https://visibleorigin.com/llms-txt-checker/): tests a site’s llms.txt against the format published at llmstxt.org.
- [Agent readiness checker](https://visibleorigin.com/agent-readiness-checker/): robots.txt is one of 23 checks of whether AI agents can find, read and use a site.
- [Cloudflare AI Crawl Control](https://visibleorigin.com/cloudflare-ai-crawl-control/): what blocking at the edge does that robots.txt cannot.

Sources: [RFC 9309, the Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309.html); [How Google interprets the robots.txt specification](https://developers.google.com/crawling/docs/robots-txt/robots-txt-spec); [Google’s open-source robots.txt parser](https://github.com/google/robotstxt); [Google’s note on unsupported rules](https://developers.google.com/search/blog/2019/07/a-note-on-unsupported-rules-in-robotstxt); [Google’s introduction to robots.txt](https://developers.google.com/search/docs/crawling-indexing/robots/intro); and the crawler documentation of [OpenAI](https://developers.openai.com/api/docs/bots), [Anthropic](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) and [Apple](https://support.apple.com/en-us/119829).

If you want the rest of your site read the same way, crawler by crawler, that is what the [free check](https://visibleorigin.com/services/#free-check) is for, and the [crawler and AI readiness audit](https://visibleorigin.com/services/#2-crawler-and-ai-readiness-audit) goes further.