I made 60 requests to my own site and the access log recorded 6

What an access log actually records

An access log is a list of requests that arrived at a particular process. That is all it claims to be, and it is honest about it. The trouble is what people — including me — read it as: a record of what clients did.

Those two things are the same only when nothing sits in front of the server. The moment a CDN answers a request on the server’s behalf, the request happened, the client got a response, and the log has nothing to say about it. There is no error, no gap marker, no line reading 54 requests served elsewhere. The log is simply shorter, and it looks complete, because a log always looks complete.

This matters for a specific kind of claim. “My server received N requests” is safe. “Googlebot visited this page N times” is not, and “Googlebot has never visited this page” is a claim an origin log is structurally incapable of supporting. I wrote the second kind anyway.

What the specification and the vendor documentation say

A cache is not supposed to store a no-store response

RFC 9111, section 5.2.2.5 is unambiguous: “The no-store response directive indicates that a cache MUST NOT store any part of either the immediate request or the response and MUST NOT use the response to satisfy any other request.” The same document, in section 1.3, defines a shared cache as “a cache that stores responses for reuse by more than one user” — which is what a CDN edge is.

My sitemap is served with no-cache, no-store, must-revalidate, max-age=0. Four consecutive requests for it came back cf-cache-status: HIT with age: 1548, while the origin was still sending that header on every one of them. The specification says the cache must not store it. Mine stores it, because a rule of mine tells it to, and a rule beats a header.

The edge does not cache pages unless you ask it to

Cloudflare’s own documentation on default cache behavior states plainly that “the Cloudflare CDN does not cache HTML or JSON by default”, and lists the file extensions it does cache — images, stylesheets, fonts, archives. HTML is not among them.

So the blind spot in my measurement is not something a vendor did to me. I built it. I wrote a cache rule to make pages fast, it worked, and then I spent weeks reading a log that the rule had quietly emptied.

Where the real number lives

Google publishes the figure from its own side of the connection. The Crawl stats report counts “the total number of crawl requests issued for URLs on your site, whether successful or not” — issued, not received. That distinction is the whole subject of this note. A crawler’s own count and your server’s count are different measurements, and when a cache sits between them they are supposed to disagree.

The test, with the ground truth known in advance

The problem with auditing a log is that you are using the log to check the log. So I made the traffic myself, which means I knew the true number before I started.

Six URLs, chosen to cover the kinds of thing a crawler asks for: the home page, the blog index, one post, robots.txt, the sitemap, and an image. Ten requests each, from outside the network, each carrying a unique user agent so I could find them again:

IP=$(dig +short @1.1.1.1 A visibleorigin.com | head -1)
M="vo-cache-probe-$(date +%s)"
curl -s -o /dev/null --resolve visibleorigin.com:443:$IP -A "$M" \
     https://visibleorigin.com/robots.txt -D - | grep -i "cf-cache-status"
grep -c "$M" /home/visibleorigin.com/logs/visibleorigin.com.access_log

The --resolve matters. This server’s own /etc/hosts points the domain at 127.0.0.1 so that WordPress can call itself, which means an ordinary curl run on the box talks to the origin and never touches the edge at all. A test run that way would have shown 60 requests and 60 log lines, and confirmed the wrong answer perfectly.

The result was uniform to the point of being boring, which is the useful part:

Ten percent of what happened reached the machine that keeps the records. And note what the table does not let me say: that a real crawler sees the same ratio. I hammered one edge location ten times in a row. Real crawler traffic arrives spread across time and across data centers, so its mix of hits and misses is different, and I cannot derive it from the origin. The honest statement is that the undercount is real, it applies to every cached resource, and its size is not visible from here.

The part of this that is about me

I found all of the above while verifying the material for a different note. In the queue at the time was a post about IndexNow containing this sentence: “Googlebot fetched my sitemaps thirty-three times and entered exactly one of my eleven posts.” It was four days from publishing.

The number came from the origin access log. So I corrected it, and wrote that the comparison had been unfair because sitemaps are served no-store and therefore always reach the origin, while posts are cached and sometimes do not.

That correction was also false, and I had already published the reason three weeks earlier. In a note about nine files and one cache rule I wrote the sentence “the file I marked no-store is stored at the edge right now” — and then, on a different afternoon, reasoned as though no-store meant what the specification says it means. Both the posts and the sitemaps are cached. Which of the two is undercounted more is not something I can determine from my own server, and I had asserted it in the direction that suited my argument.

Two overstatements on the same subject in one afternoon, the second one inside the correction of the first. The pattern is the same both times: the sentence was mechanical, plausible, and unmeasured, and it favored the point I was making. A correction is a new claim. It needs the same evidence as the claim it replaces, and I gave it none.

What you can and cannot claim from an origin access log

How to run this on your own site

Four steps, and none of them need tooling beyond what you already have.

1. Resolve the public address explicitly. Use --resolve or run the test from a machine that is not the server. A request made on the origin box may never reach your own edge, and will tell you everything is visible.

2. Measure with GET, not HEAD. A HEAD request can be handled differently from the GET for the same URL, so the status it reports is not necessarily the one your visitors get.

3. Ask each resource separately. Do not assume the header you send decides the outcome. Request the same URL four times in a row and watch cf-cache-status and age. If it says HIT, that request does not exist as far as your log is concerned — whatever Cache-Control your origin is sending.

4. Get the crawler’s own count. For Google it is Search Console, under Crawl stats. For your CDN it is the cache analytics. If the two numbers differ by a lot, the difference is the part of your traffic you have been unable to see, and it is usually the part you were drawing conclusions about.

The same discipline applies one layer down. I have written before about asking the wrong process what your configuration is: the answer comes back confident, well formatted, and about a different program. A log read through a cache is the same failure with a different instrument.

Frequently asked questions

Does this mean my access log is useless?

No. It is the best record you have of what your server did, and it is the only place some things appear at all — origin errors, slow responses, request bodies, internal traffic. What it is not is a census of client behavior. Use it for questions about the server and a second source for questions about clients.

Why did my sitemap get cached when it says no-store?

Because a cache rule can be configured to ignore origin cache headers, and when it is, the rule wins. The header is per response; the rule is a blanket instruction matched on the shape of the request. Nothing warns you when they disagree. Check each file individually rather than reasoning from the header you configured.

Would turning off page caching fix the measurement?

It would make the log more complete and the site slower, which is a bad trade for a reporting problem. The fix is to get the number from the side that has it. Caching is not the error; treating a partial record as a complete one is.

Does this affect crawler verification by IP?

No, and this is worth separating. Checking that an address really belongs to the crawler it claims to be — forward-confirmed reverse DNS for Googlebot, published IP ranges for the AI crawlers — is a test applied to requests you already have. Caching changes how many requests you have. It does not make the ones you do have less trustworthy.

How much of my traffic is invisible?

From the origin alone, you cannot tell, and anyone who gives you a percentage from an access log alone is guessing. My controlled test hid 54 of 60, but that number is an artifact of requesting the same URLs ten times in a row from one location. Compare against Search Console or your CDN analytics if you need the real figure.

Is a low origin count evidence that my pages are not being crawled?

It is weak evidence at best, and it is the specific mistake this note exists to correct. A low count is consistent with a page being crawled often and served from cache every time. Before concluding anything from the absence of lines, confirm whether that resource is cacheable at your edge.

The verdict

An access log answers one question well: what reached this server. It was never designed to answer the question I kept asking it, which was what crawlers do with my site. Between those two questions sits a cache that I configured, that works exactly as instructed, and that is silent about everything it absorbs.

The measurement I trust most was the one where I already knew the answer. Sixty requests, six lines. Everything else in this note is downstream of that, including both of the retractions — and the reason I am confident in the correction is not that it sounds more careful than the sentences it replaces. It is that this time I generated the traffic before I counted it.

The log misleads in more than one way: besides what the edge hides from it, a large share of what it does record is the site calling itself.

Reading the log has its own failure modes, too: my log parser once read a Chrome version number as an IP address.

Sources

• RFC 9111, HTTP Caching — section 5.2.2.5, the no-store response directive, and section 1.3 for the definition of a shared cache.
• Cloudflare, Default cache behavior — the statement that HTML is not cached by default, and the list of extensions that are.
• Google Search Console, Crawl stats report — the definition of a crawl request as issued by Google rather than received by the site.
• Measurements taken against visibleorigin.com on 17 September 2026, with the commands reproduced above.