robots.txt checker for AI crawlers

Enter a domain or a full URL. This free robots.txt checker fetches the site’s /robots.txt, validates every line against RFC 9309 and Google’s open-source parser, and tests the path you entered for 15 AI and search bots, from GPTBot and ClaudeBot to Googlebot and Bingbot. For each crawler it shows the group it follows and the exact line that decides, so you can see why a URL is allowed or blocked, not just that it is.

robots.txt checker result for visibleorigin.com: a valid file, and all 15 AI and search bots allowed at the homepage — This site's robots.txt in the checker on November 12, 2026: every check passes, and all 15 bots may fetch the homepage. The training ring is blue because training is a separate decision.

What the robots.txt checker tests

The file. The status code and what crawlers do with it: a 404 means no rules at all, a server error means stop. Also the content type, the size against the 500 KiB that Google reads, UTF-8, and any redirect on the way.
Every line. Fields crawlers do not know, misspellings that only Google forgives, lines with no colon, rules that come before any User-agent, paths that can never match, and rules Google stopped reading years ago, such as Noindex.
The rules as a whole. How many groups and rules there are, whether the * group shuts out the whole site, which sitemaps are declared, and whether Cloudflare writes part of the file.
Crawler by crawler. For the path you entered, each of the 15 bots comes out allowed or blocked, with its group and the line that decided. The file is shown underneath with line numbers, and the lines that matter are marked.

How a crawler reads robots.txt

The checker applies the rules a compliant crawler applies. They are short, and most surprises come from one of them.

A crawler reads only its own group. A group is one or more User-agent lines followed by rules. If a group names the crawler, it follows that group and ignores the others, the * group included. Only a crawler that no group names falls back to *. If several groups name the same crawler, their rules are merged.
Names are matched loosely, paths exactly. A crawler name is compared without regard to case, and only up to the first character that is not a letter, hyphen or underscore, so Googlebot/2.1 and googlebot* both mean googlebot. Paths are case-sensitive.
The longest match wins. Among the rules that match a URL, the one with the most characters decides. When an Allow and a Disallow are the same length, Allow wins.
Two wildcards, nothing more. * matches any run of characters and $ marks the end of the URL. There are no regular expressions.
No matching rule means allowed. A path that no rule matches may be fetched.

One exception is built in. Apple says Applebot follows the Googlebot rules when a file has none for Applebot, so the checker does the same.

What crawlers do when robots.txt is missing or broken

robots.txt answers	What RFC 9309 and Google do	What the checker shows
200, with a text file	The rules apply. Google reads the first 500 KiB and ignores the rest.	Every check, and a verdict for each crawler
A redirect	Followed for at least five hops. After five, Google treats the file as missing.	The final address. The checker follows three hops.
404, 410 or another 4xx	No rules: everything may be fetched. Google treats 401 and 403 the same way, but not 429.	“No robots.txt”, every crawler allowed. A 401, 403 or 429 is marked not measurable from here.
5xx, a timeout or a network error	RFC 9309 says to assume everything is disallowed. Google stops crawling for 12 hours, then uses its last good copy for up to 30 days.	A server error, and no verdict per crawler
200, with an HTML page	Usually no valid lines, so nothing is restricted.	An error: the site allows everything by accident

Common robots.txt mistakes it catches

Giving a crawler its own group and losing the defaults. Add User-agent: GPTBot with a single Disallow line, and GPTBot stops reading the * group: every rule you wrote there for all crawlers no longer applies to it. The verdict for each crawler names the group it actually follows.
Disallow: / under User-agent: *. Meant for one bot, it blocks every crawler without a group of its own, search engines included.
Misspelled fields. Google’s parser accepts Dissallow and User agent, and a space where the colon should be. Other crawlers may skip those lines. A field nobody knows, such as Dsallow, does nothing at all.
Rules Google no longer reads. Noindex in robots.txt stopped working at Google on September 1, 2019, and Google does not support Crawl-delay.
A relative sitemap. Sitemap: /sitemap.xml is skipped. The line needs the full URL.
An HTML page at /robots.txt. Common on sites that send every unknown path to the homepage with a 200.
Rules that Cloudflare writes. When Cloudflare manages part of the file, that block can only be changed in the Cloudflare dashboard, not in your own copy.

What robots.txt cannot do

robots.txt is a request. RFC 9309 says its rules “are not a form of access authorization”: a crawler that follows the protocol obeys them, and anything else can ignore them or borrow another crawler’s name. Three things follow from that.

It does not prove who is asking. Anyone can send GPTBot as a user agent, which is why the operators publish their IP ranges. 346 requests said they were AI crawlers on this site, and their owners’ IP ranges said otherwise.
Allowed is not the same as served. A firewall, a CDN rule or a bot setting can refuse a crawler that robots.txt allows. Cloudflare AI Crawl Control and the four layers that can send a 403 show where that happens.
It does not take pages out of search. Google says a blocked URL can still be indexed, without being visited, when other pages link to it. Keeping a page out of results takes a noindex meta tag or header, on a page crawlers are allowed to fetch.

How the request is made

From this site’s server, in a datacenter in the United States, with the user agent visibleorigin-check/0.3 and the rules on How I measure. A check is one request for /robots.txt, plus up to three redirects. The path you enter is never requested: it is only compared with the rules. The checker never sends a crawler’s user agent, so it reads the file anyone can read.

Some sites refuse datacenter addresses whatever the request says. When robots.txt answers 401, 403 or 429, the result says it is not measurable from here, instead of reporting rules that may only apply to this server. What running from a datacenter costs a check.

Nothing is stored: not the domain, not the path and not the result, and nothing is cached. Your IP address only enters a counter, as a keyed hash that expires after ten minutes, to allow ten checks every ten minutes. The Privacy Policy says the same.

robots.txt checker: frequently asked questions

What is a robots.txt file?

A plain text file at the root of a site, such as https://example.com/robots.txt, that tells crawlers which paths they may fetch. RFC 9309 defines its format. Each crawler finds the group that names it, or the * group, and follows its Allow and Disallow lines.

How do I test whether a URL is blocked by robots.txt?

Enter the full URL above, path included, for example example.com/blog/post/. The checker reads the site’s robots.txt and, for each of the 15 bots, says whether that path is allowed, which group the bot follows and which line decides. With a bare domain it tests the homepage.

Does blocking GPTBot keep my site out of ChatGPT?

Not on its own. OpenAI runs separate crawlers: GPTBot collects pages that may be used to train its models, and OAI-SearchBot finds pages to show and link in ChatGPT search. OpenAI says sites that block OAI-SearchBot are not shown in ChatGPT search answers, and that robots.txt rules may not apply to ChatGPT-User, which fetches a page when a user asks for it. Each token is a separate decision, and the checker lists them separately.

Why does my robots.txt work for Google but not for other crawlers?

Google’s parser is forgiving. It accepts a few common misspellings, such as Dissallow, and a space where the colon should be. A crawler that follows RFC 9309 to the letter skips those lines. The checker marks each one as a warning, so you can write the file in a way every crawler reads.

Does robots.txt stop a page from being indexed?

No. It stops compliant crawlers from fetching the page, but Google can still index the URL if other pages link to it. To keep a page out of search, let crawlers fetch it and add a noindex robots meta tag or an X-Robots-Tag header. Noindex inside robots.txt has been ignored by Google since September 1, 2019.

Why does the checker say robots.txt is not measurable?

The site answered 401, 403 or 429 to this checker, which runs from a datacenter, and many firewalls refuse datacenter addresses. That refusal applies to this checker’s address. What crawlers get depends on how your firewall treats their IP ranges, so check your firewall or CDN log for their requests.