The short version. I was about to publish that AI crawlers read this site almost seven times more than Google does. The number came from counting user-agent strings. That is the exact mistake the post was supposed to be about.
So I cross-checked every request against the IP ranges each operator publishes for its own crawler. Of 850 requests claiming to be an AI crawler, 504 verified and 346 did not. Googlebot, checked by the method Google documents, verified 113 out of 113 — from 22 distinct addresses, 22 of which passed.
The gap is not that AI companies are careless. It is that their verification surface is younger, thinner, and in one important case does not exist at all.
The number I nearly published
Eighteen days of access log. I grouped every request by the crawler name in its User-Agent header, added them up, and got a headline: 746 requests from AI crawlers against 111 from Googlebot. Six point seven to one. For a site that sells crawler accessibility work, that is a tempting sentence to write.
It is also worthless, and I knew why before I wrote it down. A user agent is a string that the client chooses. It is not an identity, it is not a credential, and nothing in HTTP asks the sender to prove it. I had spent the same week explaining that to a colleague about a different log. Then I built a headline on top of it.
The interesting part of this note is not that the number was wrong. It is how far it moved once I checked it, and which direction it moved in — because it did not move the way I expected.
What a crawler operator actually gives you
There are two ways to check whether a request that says it is a crawler really is one, and the operator has to give you one of them.
Reverse DNS, verified in both directions
Google’s method is the older one and it needs no file. You take the IP, ask for its reverse DNS name, check the name ends in a domain the operator controls, and then resolve that name forward again and check it comes back to the same IP. Both directions, or it does not count. Common Crawl recommends the same approach for its own bot, and says so in the file it publishes: “For verification of IPv4 addresses, FCrDNS is also recommended.”
A published list of address ranges
The newer method is a JSON file listing the prefixes the crawler will come from. OpenAI publishes one per agent. Anthropic publishes one at claude.com/crawling/bots.json. Perplexity publishes two. Common Crawl publishes one. They are small: 21 prefixes for GPTBot, 39 for OAI-SearchBot, 26 for Anthropic, 8 for PerplexityBot, 5 for CCBot. Downloading all of them takes about four seconds.
Perplexity is the only operator I found that states the combination explicitly in its own words: “When configuring WAF rules for Perplexity bots, combine both User-Agent string matching and IP address verification for enhanced security while ensuring legitimate bot traffic can access your content.” OpenAI recommends allowing its ranges; that is a different sentence, and I am not going to stretch it into a verification instruction it does not contain.
What the cross-check returned
I downloaded all seven range files, parsed the prefixes, and walked the log once. For every request carrying a crawler name, I asked one question: is this address inside the range that that crawler’s own owner publishes?
| Claims to be | Verified | Not verified |
|---|---|---|
| ClaudeBot | 284 | 75 |
| GPTBot | 132 | 104 |
| OAI-SearchBot | 81 | 96 |
| PerplexityBot | 7 | 64 |
| CCBot | 0 | 7 |
| Total | 504 | 346 |
Crawlers only. Requests from user-initiated agents are counted separately further down, and for a reason.
Forty-one per cent of the requests claiming to be an AI crawler did not come from the addresses their owner publishes. Not a rounding error. Not a handful of stragglers. Four in ten.
And the shape inside the table matters more than the total. ClaudeBot is the most verifiable thing in it: 284 of 359. PerplexityBot is the least: 7 of 71, which means nine out of ten requests wearing that name came from somewhere Perplexity does not claim. CCBot verified zero times out of seven.
Two addresses wearing fifteen names
Before any of that counting is allowed to mean anything, one thing has to come out of the data. Two addresses in this log were presenting themselves as many different crawlers.
| Address | Distinct bot identities used | Requests |
|---|---|---|
| 35.236.182.226 | 12 | 342 |
| 34.61.189.20 | 15 | 93 |
The fifteen include GPTBot, ClaudeBot, PerplexityBot, Amazonbot, Applebot, CCBot, Bytespider, Google-Extended, YouBot, cohere-ai and DuckAssistBot — from one machine.
A single host does not run fifteen companies’ crawlers. The rule I now use is blunt and it works: an address presenting three or more distinct bot identities is not a crawler, it is something scanning you.
There is an obvious objection and it is a good one: a site behind a CDN can log the edge’s address instead of the visitor’s, and one edge address legitimately carries every crawler on earth. I checked before trusting my own filter. This log holds 1,516 distinct addresses, real crawlers arrive on their own well-known ranges, and Cloudflare publishes no range beginning 34 or 35. Both addresses reverse to googleusercontent.com. They are rented cloud machines, not my CDN.
What that filter removes is the part I did not expect. Excluding those two addresses makes Google-Extended, Applebot, CCBot, Bytespider, DuckAssistBot, cohere-ai and YouBot disappear from the log entirely. Not reduced. Gone. Every request I had that claimed to be one of those seven came from the two scanners.
Google-Extended is the one worth pausing on, because it could never have been real. Google documents that “Google-Extended doesn’t have a separate HTTP request user agent string. Crawling is done with existing Google user agent strings; the robots.txt user-agent token is used in a control capacity.” It is a name you write in robots.txt — a file whose format is now standardised as RFC 9309 — not a name that arrives in a request. Any log line containing it as a user agent is fabricated by definition, whatever address it came from.
If you have ever read a report telling you which AI companies crawl your site, and that report counted user-agent strings, this is what was in it.
If you want this done against your own log rather than read about mine, that is exactly what the free check starts with — the requests written out so you can repeat them without me. The full version, including the range files and the reverse-DNS pass, is part of the crawler and AI readiness audit.
The 308 requests I refuse to call fake
The same cross-check produced a second pile, and this is where a tempting number had to be thrown away.
| Claims to be | Inside published range | Outside |
|---|---|---|
| ChatGPT-User | 6 | 68 |
| Claude-User | 0 | 155 |
| Perplexity-User | 0 | 85 |
Adding these to the failure column would have produced a much better headline and a false one.
Three hundred and eight requests, almost none inside the published ranges. Dropping them into the “did not verify” column would have taken my failure rate from 41% to 56%, and it would have been wrong.
These are not crawlers. Anthropic documents what Claude-User is: “Claude-User supports Claude AI users. When individuals ask questions to Claude, it may access websites using a Claude-User agent. Claude-User allows site owners to control which sites can be accessed through these user-initiated requests.”
A person asked a question and something fetched a page on their behalf. That is a different kind of traffic from a crawler working through a site on a schedule, and there is no reason it must originate from the crawler’s fleet. The file Anthropic publishes is named /crawling/bots.json, and it does not say it covers that agent.
So the honest sentence is narrow and boring: these requests did not come from the range their owner publishes for crawling. Whether they are genuine, I cannot tell you from this log, and neither can anyone else who only has a log. Knowing which number you are not allowed to use is most of this job.
Google is the boring, verifiable one
Here is the comparison that survived, and it is not the one I set out to make.
| Method | Verified | Failed | |
|---|---|---|---|
| Googlebot/2.1 | Reverse DNS, both directions | 113 of 113 | 0 |
| AI crawlers | Published IP ranges | 504 of 850 | 346 |
Googlebot arrived from 22 distinct addresses. All 22 passed the forward-confirmed reverse DNS check.
Twenty-two addresses, twenty-two verified, not a single impostor in eighteen days. Nobody bothers to fake Googlebot here, and everybody fakes the AI crawlers — which makes sense the moment you think about who is checking. Blocking rules have watched for forged Googlebot for twenty years. The AI names are new, and almost nothing is looking.
The industry knows. There is an active IETF draft, draft-ietf-webbotauth-httpsig-protocol, whose authors are from Cloudflare and Google, proposing that bots sign their requests cryptographically. Its existence is the admission: the current answer to “is this really GPTBot” is a text file and a hope.
Until that ships, the operator files are what there is, and the gap between them is large. OpenAI publishes one file per agent. Anthropic publishes one file for everything and names it after crawling. There is no shared schema, no shared location, and no agreed answer for user-initiated fetches.
How to run this on your own log
Four steps, no tooling required beyond what you have.
1. Count by address, never by name
Extract the client IP, not the user agent, as your unit. If your access log shows only a handful of distinct addresses, stop: you are logging your CDN’s edge and every number after this is about your CDN. This is the same failure as reading a configuration from the wrong process, which I have also done to myself.
2. Drop the addresses wearing several names
Group identities per address. Three or more distinct bot names from one address means the address is scanning you, and every request from it must come out before you count anything.
3. Fetch the range files and check membership
Seven files, four seconds. Then ask, per request, whether the address is inside the prefixes published by the operator whose name that request is wearing. Not any operator: that one.
4. Use reverse DNS where that is the documented method
For Googlebot and for CCBot, the operator’s own instruction is forward-confirmed reverse DNS. Resolve the name, then resolve it back, and require the same address. One direction proves nothing.
The whole pass took me under an hour on eighteen days of log, and it changed the headline. If you have never done it, assume your bot numbers are wrong in the direction that flatters you — mine were.
The method above is written out in full, with its limits stated, in how I measure. If you would rather have the result than the afternoon, send me a domain.
Verifying AI crawlers: frequently asked questions
Can I trust the user-agent string of an AI crawler?
No, and not as a matter of degree. The user agent is a value the client writes; nothing verifies it. In this log, 41% of requests carrying a crawler name did not come from that crawler’s published addresses, and seven bot names existed only as forgeries from two scanning machines.
How do I verify Googlebot?
Forward-confirmed reverse DNS. Resolve the IP to a name, require it to end in googlebot.com or google.com, then resolve that name back to an address and require a match. Checking only the reverse name is not enough, because reverse records can be set by whoever controls the address block.
Where are the AI crawler IP ranges published?
Each operator publishes its own. OpenAI has a file per agent for GPTBot, OAI-SearchBot and ChatGPT-User. Anthropic publishes one at claude.com/crawling/bots.json. Perplexity publishes files for its bot and its user agent. Common Crawl publishes one for CCBot. There is no common registry and no common format.
Why does Google-Extended never appear as a real user agent?
Because it is not one. Google states that Google-Extended has no separate user-agent string and that it exists as a robots.txt token used in a control capacity. Every request I saw carrying it was fabricated.
Are requests from ChatGPT-User or Claude-User fake if they fall outside the ranges?
Not necessarily, and I would not publish them as fake. Those agents fetch pages because a person asked a question, and the published range files are documented for crawling. Absence from a crawling range is not evidence of forgery for a non-crawling agent.
Does blocking unverified bot traffic help my site?
It reduces load and it cleans your analytics, which is worth something on its own. What it does not do is change whether a real crawler can reach you — that is a separate question, and the one worth answering first.
The general shape of this
The mistake was not carelessness. I know what a user agent is. The mistake was that the unverified number said something I wanted to be true, and a number that flatters you does not get audited as hard as one that does not.
That is the whole reason the cross-check exists as a habit rather than a decision. If I only verified the figures that looked wrong, I would publish every figure that looked right, and the ones that look right are exactly the ones that end up in a headline.
The corrected finding is smaller than the one I started with and considerably more useful: Google can be verified and is not being impersonated; the AI ecosystem can be verified less well and is being impersonated heavily; and a third category exists that I cannot verify at all and have to say so. That last part is not a gap in the work. It is the work.
Verified or not, much of what arrives is not about your site at all: five machines asked mine for the same 272 paths, and the list turned out to be a catalogue of other people’s deploy mistakes.
Verifying who is really crawling is the same skill this site sells as a service: a free check that requests your homepage as five different crawlers, and is honest that it does so from a datacenter.
