Notes from the server
Write-ups of failures that leave no trace in any log: a rule that turns a crawler away, a setting the server silently ignores, a check that rewards the wrong answer. Every claim here carries the command that produced it, so you can run it against your own domain and see for yourself. If you would rather I ran it for you, that is the free check.
By John Patzi 29 notes
Case notes My log parser read a Chrome version number as an IP addressA dotted-quad pattern skipped every IPv6 client in my access log, then filled the gap with the browser version string. 3,565 lines affected, and the output looked like ordinary data.Read the note
Crawler access I publish six .well-known files. 89% of the traffic to them was meTwenty-one days of requests for this site's machine-readable files, with my own traffic removed and every remaining identity checked against what its operator publishes. 365 requests, 274 of them mine, 33 proved.
Server and CDN The PHP syntax check on this server crashes, and a pipe turns that into a passphp -l segfaults on a one-line file here. The manual documents two exit codes and this is neither, and piping the output converts the crash into the value that means success.
Server and CDN I made 60 requests to my own site and the access log recorded 6Sixty requests left my laptop and six reached my server. The rest were answered at the CDN edge. What that does to every crawler conclusion drawn from an origin log, including two of mine I had to retract.
Server and CDN A quarter of my server log is the site calling itself, and I caused half of itBetween a sixth and two fifths of the requests reaching my server are WordPress asking WordPress whether anything is due. The busiest days were the days I spent measuring the site.
Case notes I built a three-step workaround for something with a one-word fixI scheduled a post four days out and it published immediately. Then I did it again an hour later. WordPress has a flag for exactly this case, and the comment above the code that caused it names the escape hatch.
Server and CDN Five machines asked my site for the same 272 paths, and none of them are about my siteFive rented machines requested an identical list of 272 paths. Sorted by what they are looking for, the list stops being an attack and becomes a catalogue of the same deployment mistake, written down forever.
Server and CDN My probe returned the same code whether the rule was alive or deadI asked for a file that did not exist to find out whether a block rule was working, got 404, and concluded it was dead. It was alive. The condition my probe relied on is the one that guarantees the rule gets skipped.
Case notes The rule in my own documentation was backwards, and following it never hurtFor months my notes said WP-CLI strips style and link tags unless you name a user. I measured it. Nothing is stripped without the flag, and naming a user is the only thing that could turn the filtering on.
Server and CDN A path I invented returned 301, and the one that exists returned 200I asked my own site for twenty random characters that have never been published. Over HTTP it answered 301. The redirect was about the scheme, not the path, and then the standard took away my 404 too.
Server and CDN My firewall rule would block my own article about firewall rulesI deployed a rule that blocks probes for phpinfo files. It also blocks any article whose URL contains that word, and this site writes about those probes. No measurement taken today could have found it.
Server and CDN php -i answers for a different program than the one serving your siteI diagnosed OPcache on this site with php -i, wrote it up, and sent it to a colleague. Every word was about a different binary from the one the virtual host names.
Crawler access I turned on IndexNow to fix a Google problem. Google is not in the protocolGooglebot was crawling almost none of my posts, so I enabled IndexNow and watched Bing fetch my key file thirteen seconds later. It worked exactly as documented, and it cannot touch the problem I turned it on for.
Crawler access 346 requests said they were AI crawlers. Their owners’ IP ranges said otherwiseI counted AI crawler traffic the way everyone counts it, by user-agent string, and was about to publish the number. Then I checked every address against the ranges each operator publishes for its own crawler.
Crawler access My free check runs from a datacenter, and today I measured what that costs itFive well-known sites refused this machine identically whether I claimed to be a browser, a crawler or my own probe. My tool was reporting exactly that as evidence the site blocks crawlers, and calling it the sales hook. Fixed, then written up.
Server and CDN My cache rule asks a different question from the one it looks like it asksIt misses TEXT/MARKDOWN, which the HTTP standard says is the same media type, and it fires on a vendor type that is not Markdown at all. Both errors come from one substitution, and I published the wrong explanation first.
Server and CDN The two tools that ship with OpenDKIM both misreported my keys, in opposite directionsOne corrupts the signature it has just computed, but only in its default invocation. The other prints a DNSSEC warning about every key it is ever given, because the build has no DNSSEC support compiled in. Both measured and reproducible.
Crawler access I changed one number on one page, and five other places went on saying the old oneTaking a catalogue from seven items to eight took four days and five hand-edits, and two of them were found by an adversarial review rather than by me. The only surfaces that stayed correct were the two nobody had to remember.
Case notes I deleted the plugin. Its settings stayed behind and switched the tracker back onRemoving an analytics plugin deleted its files, not its configuration. Four days later somebody activated it again and it armed itself in seven seconds — no login, no consent screen — against a page that promises nothing loads until you choose.
Server and CDN The cache was never ignoring the header. I had never told it the page variedI spent two days proving my server refused to vary its cache by Accept, and wrote three explanations for it. The first two were wrong. The cache keys on request headers perfectly well — it just has to be told by the response, and my site never told it.
Server and CDN A cache-buster gave me my own copy of the page, and hid what everyone else was servedA subdomain of mine served a hosting panel's splash screen to every visitor while a cache-busted request returned the real site. Both readings were correct. Only one was the URL anybody else would type — and the fix is three seconds of work.
Crawler access An agent readiness score gave my site Level 5 for failing its only checkThree requests to the same scanner in one minute, for the same site, returned Level 5, Level 4, and Level 5 with a score of zero. A preset from its own dropdown returns 100. Here is where each number comes from.
Case notes PHP strip_tags deleted three quarters of my post over one characterA word count said 1,111 for an article of 3,742. The cause was one ASCII arrow in a code block: PHP's strip_tags took it for a tag and discarded everything after it. Escaping that single character recovered 15,466 bytes.
Server and CDN Cloudflare stores the file I marked no-store, and my own rule is whyNine files tell agents what this site is. I measured what the edge did with each against what the origin asked for, and they disagree about three of them, in both directions.
Server and CDN Three ways to be told a cache purge worked, and none of them looked at the cacheI ran the command that purges one URL. It printed Success, the edge kept serving the old copy, and the origin never dropped its own. That message is not a report about a cache: it is a report that an HTTP request came back 200.
Server and CDN Four ways to ask PHP what it is configured to do, and three of them are not your siteI wanted to confirm that one PHP setting was active on this site. There are four ways to ask, all of them one line long, and three of them answer about something that is not my website. Two of those three read the same php.ini as the real one and still disagree with it.
Crawler access Two ways to serve Markdown to agents, and each one fails quietlyThis site publishes every page twice: once as HTML, once as Markdown. Choosing between content negotiation and a parallel .md route is not a style question — each one fails silently, in a different way. Here is what I measured on my own server and on Cloudflare's.
Case notes Every pretty URL returned 404, and the line I blamed was innocentEvery pretty URL on this site returned 404 and nothing was written to any log. I blamed one unescaped quote in the virtual host's log format line, fixed it, and wrote that lesson down three times. Another site on the same server runs that line today and works fine.
Server and CDN Passing the DNS-AID check with an SVCB record, and the two records I rejectedA public scanner's DNS-AID check fails Infoblox, Cloudflare and the scanner's own site. The one domain I found passing was passing with a record type the draft does not define. Here is what the IETF draft actually specifies, the single SVCB record I published, and the two I rejected because they would have been false.