# How I measure > The crawler, the rules, and the limit of the method: a 403 proves a block exists; a 200 does not prove the real crawler gets through. Source: https://visibleorigin.com/how-i-measure/ Updated: 2026-09-12 --- *Last updated: 12 September 2026.* This work involves measuring websites I do not own. That is easy to do badly, so the rules I follow are written down here rather than left to judgment. If you have seen my crawler in your logs, this page is why. ## What the crawler does It sends unauthenticated GET requests to paths your own site publishes — your homepage, your `robots.txt`, your sitemap, your policy files. Requests are serialized, with a pause between them. It identifies itself: `visibleorigin-check/0.x (+https://visibleorigin.com/how-i-measure/)` It does not log in, does not submit forms, does not follow links into checkout or account areas, and does not test for vulnerabilities. ## The rules ### On security **A security finding is never published.** If I find one while measuring something else, it goes to the site owner privately, and it stays there. ### On naming a site publicly I name a site in public only when all three are true: the finding is about a public policy artifact the site publishes itself; the exact command that reproduces it is included; and the site belongs to a well-known organization, or to a company that sells the very thing it is failing at. **I never name a local business, a professional practice, a freelancer or a small company without a technical team.** The asymmetry is the point: a company that sells performance tooling can answer a public measurement. A dentist cannot. Before publishing anything by name I write to the site's public contact with the finding and the command, and wait **seven days**. If they show me the measurement is wrong, or that it is already fixed, I correct the page within 24 hours. ### On prospects and clients **Nothing measured about a prospect is ever published** — not by name, and not inside an aggregate where they could be identified. That includes anyone who asked for the free check and did not become a client. When an agency asks me to measure the sites it manages, the result goes to that agency and nowhere else. Those sites did not ask to be measured; publishing anything about them would be using someone else's property as my advertising. ### On the numbers Every published measurement carries the date it was taken and the date it was last re-checked. A measurement without a date is an opinion. If the instrument fails to complete a check, that check does not become a zero and does not enter any published figure. A failed measurement is a failed measurement, not a finding. The first row of anything I publish is this site, with its own failures named. ## The limit of this method — read this part The crawler tests bot access by sending requests with the User-Agent string of the crawler in question, from a VPS. That is not the same as being that crawler. **Corrected 12 September 2026.** This paragraph used to say: “A 403 proves a block exists for that User-Agent.” **That was wrong, and the correction is left visible here rather than edited away.** A 403 proves a block exists for *that request*. Attributing it to the User-Agent requires a control, because the block may be on the address the request came from. Measured on 12 September 2026 against five well-known sites that work normally in a browser: all five refused this machine. On **all five** the refusal was identical for a browser User-Agent, a crawler User-Agent and my own — so the User-Agent was not what was being judged. (This count was first published here as “three of them, and one the other way round”. That was wrong: the exception came from a redirect I had recorded without following it, and following it leads to the same refusal. Re-measured the same day.) On a sixth site, measured separately, the User-Agent *was* decisive and in an unexpected direction: a browser User-Agent and one crawler were sent a challenge screen while two other crawlers were served the real page. **So the crawler now sends a control request with an ordinary browser User-Agent alongside the crawler ones, and reports on the comparison rather than on the crawler result alone.** If the control is refused too, the report says the site is not measurable from here instead of saying it blocks anything. **And a 200 is no longer taken at face value.** Challenge screens are served with a success status often enough that the status line alone is not evidence: the crawler now reads the body it received and, if it recognises a challenge or interstitial, records it as a refusal that answered rather than as access. The report shows the status code it actually got alongside that judgement, so you can see both. **A 200 still does not prove that the real Googlebot or the real GPTBot get through** — those are verified by IP range and by reverse DNS, which I cannot fake and do not claim to. So: when a report says your site blocks a crawler, that is a measurement *and it now carries the control that makes it one*. When it says a crawler gets through, that is an absence of evidence of a block, which is a weaker statement. The reports say which is which, because the difference is the whole distance between a measurement and a headline. ## If you would rather I did not Add `visibleorigin-check` to your `robots.txt` and the crawler stops. Or write to [john@visibleorigin.com](mailto:john@visibleorigin.com) and I will remove your domain from anything I run. No explanation needed.