My free check runs from a datacenter, and today I measured what that costs it

Where the check actually stands

The method page for this service has always said the requests come from a VPS. What it did not say, because I had not measured it, is what that means in practice.

curl -s https://api.ipify.org
  2.24.98.5

curl -s 'http://ip-api.com/json/?fields=country,city,isp,hosting'
  country  United States
  city     Boston
  isp      Hostinger US
  hosting  true

curl -s https://www.cloudflare.com/cdn-cgi/trace | grep -E '^(loc|colo)='
  colo=EWR
  loc=US

The check does not run from Chile, where I do, and it does not run from a residential connection. It runs from an address that commercial databases classify as hosting, in New Jersey, and a good deal of the internet has opinions about addresses like that.

Five sites, one machine

I requested the homepage of five well-known sites — all of which load perfectly in a browser — three times each, changing only the User-Agent: an ordinary browser string, a crawler string, and my own probe’s identifier. Redirects followed, bodies read.

Five for five. Not one of those refusals carries information about how those sites treat crawlers, because the browser string was refused exactly the same way. A check that only sent crawler strings would have recorded five sites blocking five crawlers and been wrong twenty-five times.

What my tool was doing with that

This is the part I would rather not write. The relevant line looked like this:

bloqueados = [k for k, v in R["bots"].items() if v in (403, 401, 429) or v == 0]
if bloqueados:
    avisos.append("BLOQUEA a: " + ", ".join(bloqueados) + "  ← el gancho de venta")

Any 403 became “this site blocks X”, annotated, in my own source, as the sales hook. No control, no comparison, nothing that could distinguish a site refusing crawlers from a site refusing my address.

What a control costs

One extra request. The check now asks for the same page with an ordinary browser User-Agent, from the same address, in the same batch as the crawler requests, and reports the comparison rather than the crawler result alone.

── ACCESO DE BOTS ──
  (control nav.)  403  ← el control TAMBIEN cae: lo de abajo no se puede atribuir al UA
  Googlebot       403  ← BLOQUEADO
  Bingbot         403  ← BLOQUEADO
  GPTBot          403  ← BLOQUEADO

When the control is refused too, the report says the site is not measurable from here instead of accusing it of anything. And when the control gets through and the crawlers do not, the finding stands and now carries the evidence that makes it a finding.

The one where the User-Agent really does decide

A sixth site, measured separately, produced the result I did not expect and would not have predicted in either direction:

So the User-Agent can be decisive, and the direction is not the one anybody assumes. Here a browser is challenged and two AI crawlers walk in. Neither “it is always the address” nor “it is always the User-Agent” is true, which is exactly why the comparison has to be made rather than guessed.

What a status code is entitled to say

The standard is unusually clear about how little a refusal tells you, and reading it would have saved me the whole exercise.

RFC 9110, on 403: “The 403 (Forbidden) status code indicates that the server understood the request but refuses to fulfill it. A server that wishes to make public why the request has been forbidden can describe that reason in the response content (if any).”

The same document on redirects: “The 3xx (Redirection) class of status code indicates that further action needs to be taken by the user agent in order to fulfill the request.” Further action needs to be taken. The request is not yet fulfilled. Which is why recording a 3xx as access was wrong twice over: not only had the probe failed to follow it, the standard says a 3xx is by definition not the end of the story.

And on the thing my check will never be able to do: Google’s own guidance on verifying Googlebot tells site owners to “run a reverse DNS lookup on the accessing IP address” and confirm the name is googlebot.com, google.com or googleusercontent.com, or else to “match the crawler’s IP address to the lists of Google crawlers’ and fetchers’ IP ranges”. Both tests are about the address, which is precisely the thing a User-Agent string cannot imitate. That is the ceiling on this kind of check, and it is structural rather than a matter of effort.

Three things I got wrong today

All three were caught before this was published, by a pass whose only job is to attack what I have written. None of them were caught by me feeling careful.

I published a count that was wrong

The method page briefly said three of the five refusals were identical and one ran the other way. That exception came from a redirect I had recorded without following it: a 302 at the first hop, which I read as the site letting a crawler through. Following it lands on the same refusal as everyone else. The page now says all five, with the old figure quoted rather than quietly swapped.

I introduced a bug into the tool while fixing it

The probe follows redirects, so a 3xx in its output cannot be a redirect the site performed — it can only be a redirect the probe failed to complete. The classic automation wall does exactly this: set a cookie, bounce back to the same URL. A client with no cookie jar loops and gives up.

My new code put 3xx in the “got through” bucket. A crawler stuck in a wall would have been reported as having access, which is the same class of error as the one I was fixing, introduced in the act of fixing it. Any 3xx is now labelled as a redirect that did not complete, and is not counted as access.

The compiler approved code that could not run

Moving that logic left the function defined in one scope and called from another. python3 -m py_compile reported no errors. Running the tool once raised NameError immediately. Syntax checking is not execution, and a tool installed on the strength of a clean compile would have been broken the next time anyone used it.

What the check can and cannot tell you

Measuring from a datacenter IP: frequently asked questions

Why do scanners get 403 from sites that work in my browser?

Commercial IP databases classify hosting ranges, and a great many sites refuse or challenge them by default. It usually has nothing to do with the User-Agent you send, which is why a comparison against an ordinary browser string from the same address is the only way to tell the two apart.

Does a 403 mean a site blocks crawlers?

Only if a control request from the same place, at the same time, with a non-crawler User-Agent, is not also refused. Without that control the 403 is a fact about your vantage point rather than about the site.

Is a 200 proof that the request got through?

No. Challenge screens are served with success codes often enough that the status line alone is not evidence. Cloudflare describes a challenge as a mechanism “used to verify whether a visitor to your site is a real human and not a bot or automated script” — and its own reference page does not tie that to any particular status code, which is the problem in one sentence. Read the body: a few kilobytes titled with an interstitial phrase is a refusal that answered politely, not access.

Why does a redirect show up as not completed?

Because a probe that already follows redirects can only report a 3xx when it could not follow one. The usual cause is a wall that sets a cookie and redirects to the same URL: a client without a cookie jar loops until it gives up. It is a refusal, not a destination.

How many vantage points do you need?

More than one, and this check has one. That is a real limit and not one I can engineer away by being careful. What I can do is say which question the single vantage point answers, and refuse to answer the others.

Should a check like this name the sites it measures?

Mine does not without writing to them first and waiting seven days, and never for anything measured about a prospect. That rule cost me the most vivid part of this note, which is the correct trade.

The general shape of this

An instrument that cannot fail in a particular direction will never tell you it is failing in that direction. Mine could only see 403s, and had no way to distinguish “refused you” from “refused everyone from where you are standing” — so every one of those became a finding about the customer’s site, because that was the only shape a finding could take.

The repair is not care, which I already had. It is a control: one request that is supposed to succeed, made at the same time and from the same place as the ones that might fail. A measurement with nothing to compare against is a reading, and a reading is not a finding.

Simulating crawlers is one way to learn what reaches you; asking a search engine to crawl on demand is another, and IndexNow woke Bing in thirteen seconds while Google ignored it.

Refusals run in the other direction too: one of my own firewall rules would have blocked my own article about firewall rules.

The tool, the method page and this note were all changed today, in that order, because a page that promises what the code does not do is the failure I write about in other people’s systems. If you want the check run against a domain of yours, with its limits stated before the results rather than after, that is the free check, and the rules it follows are on the method page including the corrections made to it. The paid work starts from the same place.