What the crawler does
It sends unauthenticated GET requests to paths your own site publishes — your homepage, your robots.txt, your sitemap, your policy files. Requests are serialized, with a pause between them. It identifies itself:
visibleorigin-check/0.x (+https://visibleorigin.com/how-i-measure/)
It does not log in, does not submit forms, does not follow links into checkout or account areas, and does not test for vulnerabilities.
The rules
On security
A security finding is never published. If I find one while measuring something else, it goes to the site owner privately, and it stays there.
On naming a site publicly
I name a site in public only when all three are true: the finding is about a public policy artifact the site publishes itself; the exact command that reproduces it is included; and the site belongs to a well-known organization, or to a company that sells the very thing it is failing at.
I never name a local business, a professional practice, a freelancer or a small company without a technical team. The asymmetry is the point: a company that sells performance tooling can answer a public measurement. A dentist cannot.
Before publishing anything by name I write to the site’s public contact with the finding and the command, and wait seven days. If they show me the measurement is wrong, or that it is already fixed, I correct the page within 24 hours.
On prospects and clients
Nothing measured about a prospect is ever published — not by name, and not inside an aggregate where they could be identified. That includes anyone who asked for the free check and did not become a client.
When an agency asks me to measure the sites it manages, the result goes to that agency and nowhere else. Those sites did not ask to be measured; publishing anything about them would be using someone else’s property as my advertising.
On the numbers
Every published measurement carries the date it was taken and the date it was last re-checked. A measurement without a date is an opinion.
If the instrument fails to complete a check, that check does not become a zero and does not enter any published figure. A failed measurement is a failed measurement, not a finding.
The first row of anything I publish is this site, with its own failures named.
The limit of this method — read this part
The crawler tests bot access by sending requests with the User-Agent string of the crawler in question, from a VPS. That is not the same as being that crawler.
Corrected 12 September 2026. This paragraph used to say: “A 403 proves a block exists for that User-Agent.” That was wrong, and the correction is left visible here rather than edited away. A 403 proves a block exists for that request. Attributing it to the User-Agent requires a control, because the block may be on the address the request came from.
Measured on 12 September 2026 against five well-known sites that work normally in a browser: all five refused this machine. On all five the refusal was identical for a browser User-Agent, a crawler User-Agent and my own — so the User-Agent was not what was being judged. (This count was first published here as “three of them, and one the other way round”. That was wrong: the exception came from a redirect I had recorded without following it, and following it leads to the same refusal. Re-measured the same day.) On a sixth site, measured separately, the User-Agent was decisive and in an unexpected direction: a browser User-Agent and one crawler were sent a challenge screen while two other crawlers were served the real page.
So the crawler now sends a control request with an ordinary browser User-Agent alongside the crawler ones, and reports on the comparison rather than on the crawler result alone. If the control is refused too, the report says the site is not measurable from here instead of saying it blocks anything.
And a 200 is no longer taken at face value. Challenge screens are served with a success status often enough that the status line alone is not evidence: the crawler now reads the body it received and, if it recognises a challenge or interstitial, records it as a refusal that answered rather than as access. The report shows the status code it actually got alongside that judgement, so you can see both. A 200 still does not prove that the real Googlebot or the real GPTBot get through — those are verified by IP range and by reverse DNS, which I cannot fake and do not claim to.
So: when a report says your site blocks a crawler, that is a measurement and it now carries the control that makes it one. When it says a crawler gets through, that is an absence of evidence of a block, which is a weaker statement. The reports say which is which, because the difference is the whole distance between a measurement and a headline.
If you would rather I did not
Add visibleorigin-check to your robots.txt and the crawler stops. Or write to john@visibleorigin.com and I will remove your domain from anything I run. No explanation needed.