The short version. I asked my own site for a path I invented on the spot — twenty random characters, never published, never linked. Over HTTP it answered with a 301 redirect. Over HTTPS the same path answered 404.
The redirect was not about the path. It was about the scheme. A route that does not exist redirects exactly as enthusiastically as one that does, which makes the 301 useless as evidence of anything.
And then the standard took away the other half of what I thought I knew: a 404 does not prove absence either.
The measurement that started it
I was validating a firewall rule. The job was to check that none of its clauses collided with a live route, so I needed a list of which routes on this site are live. The obvious shortcut is to treat anything that does not 404 as real.
Running that over a set of 273 paths — a scanner’s probe list, harvested from my own access log — produced a result that looked alarming: 1,264 responses in the 301 family. On the shortcut’s logic, that is a lot of live routes for a rule to trip over.
Not one of them corresponded to an existing page. The only path in the whole set that returned 200 was the site root.
What the 301 redirect is actually answering
Cloudflare’s Always Use HTTPS does what its name says and the documentation is not coy about the scope: it “redirects all your visitor requests from http to https, for all subdomains and hosts in your application”.
All requests. The redirect is a function of the scheme in the URL, not of whether anything lives at the path. So the test is trivial to run on your own site:
| Request | Response | What it tells you |
|---|---|---|
http://site/a-path-i-invented-12345 | 301 | Nothing about the path |
https://site/a-path-i-invented-12345 | 404 | Something, but less than you think |
https://site/a-real-page/ | 200 | It exists |
Measured on this site with a path that has never existed. Your own result may differ in the code — a different redirect configuration can answer 302 — but not in the substance.
I should be precise about one thing I cannot tell you: where that redirect sits relative to other edge products in the order of evaluation. I looked for documentation stating it and did not find any. What I did observe on my own zone is that a path matching my firewall rule answered 403 over plain HTTP rather than redirecting — so at least there, something else fired first. That is one measurement on one zone, not a rule.
The half I did not expect: 404 is not proof either
Having established that the 301 says nothing, I assumed the 404 was solid. RFC 9110, the current HTTP semantics specification, disagrees in its own definition:
“The 404 (Not Found) status code indicates that the origin server did not find a current representation for the target resource or is not willing to disclose that one exists.”
Two meanings in one sentence, and the sender chooses which. It gets more explicit a few sections later, where the standard describes hiding a forbidden resource: an origin server “that wishes to ‘hide’ the current existence of a forbidden target resource MAY instead respond with a status code of 404”.
So a 404 is a claim, not a fact. It may mean nothing is there. It may mean something is there and you are not being told. Both are correct HTTP, and nothing in the response distinguishes them.
Which leaves exactly one code that proves existence
| Code | What the specification says it means | Proves the resource exists? |
|---|---|---|
200 | The request has succeeded; a representation was sent | Yes |
301 | The target has been assigned a new permanent URI | No — says where to look, not what is there |
302 | The target resides temporarily under a different URI | No |
404 | Not found or not willing to disclose that one exists | No — and does not prove absence either |
Four codes, one conclusion. Everything except the first row is a statement about the response rather than about the resource.
The same document defines the one that does: “The 200 (OK) status code indicates that the request has succeeded.” A representation was found and sent. That is the only one of the four codes in this note that establishes a resource is there.
The 3xx family is defined as “further action needs to be taken by the user agent in order to fulfill the request”. Not fulfilled — pending. And a 301 specifically says “the target resource has been assigned a new permanent URI”, which is a statement about where to look next, not about what you would find there.
Distinguishing “this route is live” from “this route answered something” is most of what the free check does before it says anything about your site, with the requests written out so you can repeat them. The same discipline applied across a stack — edge rules, cache, redirects and what each crawler actually receives — is the crawler and AI readiness audit.
Why the heuristic feels safe
It is worth asking why “not a 404 means live” survives as long as it does, because the answer is not that people are careless.
On a server with no CDN and no forced HTTPS, it is very nearly true. Request a path that does not exist and you get a 404; request one that does and you get a 200. The heuristic was built in that world and it worked there. What changed is that an edge now answers before the origin does, and it answers questions about the request — its scheme, its method, its headers — rather than about the resource.
So the rule did not become wrong because anybody made a mistake. It became wrong because a layer was inserted underneath it that has its own opinions, and the rule never knew the layer existed. That is a common shape for a heuristic to fail in, and it fails silently: the responses still arrive, still carry plausible codes, and still tabulate into a number.
What it costs to get this wrong
In my case the consequence was specific and it ran in the safe direction, which is luck rather than design.
I was checking whether a firewall rule would block anything real. Under the loose test, 1,264 redirects looked like live routes, and every one of them was a potential collision. Faced with that many apparent conflicts, the rational move is to narrow the rule until the conflicts go away — and I would have ended up with a weaker rule protecting against fewer probes, in order to avoid breaking routes that were never there.
That is the shape of the damage: a bad definition of “live” does not usually announce itself as a wrong answer. It quietly biases the next decision. The same mistake in the opposite direction — treating a hidden-but-real resource as absent because it 404s — would have produced a rule with a hole in it instead.
The related trap, which I have written about separately, is that validating a rule against today’s routes cannot see a conflict with a page you have not published yet. Between the two, most of what a rule review feels confident about turns out to rest on assumptions worth stating out loud.
How to do this properly
Ask over HTTPS, and ask for the code you mean
If the question is “does this exist”, request the scheme your site actually serves and require a 200. Anything else is a different question with a different answer.
Do not count 3xx as anything
Redirects in a bulk scan are noise from the transport layer. Filter them out before counting, or your totals describe your TLS configuration rather than your site.
Treat 404 as “no answer”, not as “no”
For most purposes this does not matter. It matters when you are enumerating what exists on a host you do not control, because the specification permits the host to lie to you by omission — deliberately and correctly.
Follow the redirect before you conclude
A 301 is an instruction to look elsewhere. Following it turns a non-answer into an answer. I have made the opposite mistake in the other direction and published from it: recording a 3xx as though the request had arrived, when the standard says plainly that it had not.
The full method, including the places where it stops being able to tell you anything, is in how I measure. If you would rather have the result than the afternoon, send me a domain.
HTTP status codes as evidence: frequently asked questions
Does a 301 mean the page exists?
No. With a scheme-level redirect such as Always Use HTTPS, every request over plain HTTP is redirected regardless of whether anything lives at that path. A path invented at random redirects identically to your home page.
Does a 404 mean the page does not exist?
Not necessarily. RFC 9110 defines it as the server either not finding a representation or not being willing to disclose that one exists, and explicitly allows a server to answer 404 in order to hide a forbidden resource.
Which status code actually proves a resource is there?
200. The specification defines it as the request having succeeded, with a representation sent. It is the only one of these codes that establishes existence.
Why does my bulk scan show thousands of redirects?
Almost certainly because it requested http:// and your edge redirects every such request to https://. Re-run against HTTPS and the number will collapse to whatever is actually there.
Is the redirect always a 301?
Not necessarily. Always Use HTTPS is documented as a 301, but other redirect configurations can answer 302, and a 302 carries different caching and semantics. Check what your zone actually returns rather than assuming.
Where can I read the definitions themselves?
RFC 9110 for the semantics, and the IANA status code registry for the authoritative list of which code is defined where. Both are short enough to read directly instead of trusting a summary.
The general shape of this
The shortcut was not lazy. “Anything that is not a 404 is live” is a reasonable heuristic that happens to be wrong on a site behind a CDN, which is most sites. It survived because the number it produced was large and alarming, and a large alarming number does not feel like something to double-check — it feels like a finding.
What broke it was asking the smallest possible question: what does this site do with a path that certainly does not exist? Twenty random characters, one request, ten seconds. That control should have been the first thing I ran and it was the last.
The rest of the note is the standard telling me that the other half of my instinct was wrong too, which is the usual outcome of actually reading the specification rather than remembering it.
