Two ways to serve Markdown to agents, and each one fails quietly

This site publishes every page and post twice: once as HTML, and once as Markdown at /slug.md. An agent reading the Markdown gets the text without the theme, the CSS, the scripts or the page builder’s scaffolding — fewer tokens, and no chance of it quoting a piece of template as if it were my writing. When I built that, I had to pick between the two mechanisms. This note is what I found out, including the part where my own setup is not safe to switch on.

Why the choice is not cosmetic

Both mechanisms deliver the same bytes to a client that asks correctly. They differ entirely in what happens to everyone who did not ask.

Content negotiation puts two representations behind one address. That address is also the address a browser uses, the address Googlebot uses, and the address every cache between you and the reader stores under. The mechanism that decides which representation a visitor receives is a request header — and a shared cache, by default, does not store a request header alongside the response.

Parallel routes put the two representations at two addresses. Nothing can be confused about which one a client asked for, because the URL is the whole question. The cost moves somewhere else: now there are two documents, and something has to keep them the same.

So the question is not which is more elegant. It is which failure you would rather have, because you are going to get one of them.

The specification defines both, and names the second one

This is the part I expected to be a judgment call and was not. RFC 9110, the current HTTP semantics specification, describes both mechanisms in the same chapter, and it is not neutral between them for this situation.

Proactive negotiation, and the four disadvantages the spec lists

Deciding at the server based on the request’s headers is called proactive negotiation (§12.1). The specification introduces its drawbacks with a sentence that does not hedge: “Proactive negotiation has serious disadvantages”. It then lists four. The fourth is the one that matters here:

*  It limits the reusability of responses for shared caching.

And it closes the section by telling the client not to count on any of it: “A user agent cannot rely on proactive negotiation preferences being consistently honored”.

Reactive negotiation, which is the boring one

The parallel-URL approach also has a name: reactive negotiation, sometimes agent-driven negotiation (§12.2). The client gets an initial response, sees a list of alternatives, and fetches the one it wants. The specification says when that is the right choice, and it describes this exact deployment:

Reactive negotiation is advantageous when the response would vary
over commonly used dimensions (such as type, language, or encoding),
when the origin server is unable to determine a user agent's
capabilities from examining the request, and generally when public
caches are used to distribute server load and reduce network usage.

A WordPress site behind a CDN is public caches distributing server load. The unglamorous option is the specified one for this shape of deployment, and I had it backwards before I read the section.

It is fair about the cost too: reactive negotiation “suffers from the disadvantages of transmitting a list of alternatives… and needing a second request”. And there is a footnote worth knowing about. HTTP once had a header for advertising those alternatives, and §15.4.1 records what became of it:

lack of deployment and disagreement over syntax led to both URI and
Alternates (a subsequent proposal) being dropped from this
specification.  It is possible to communicate the list as a Link
header field value [RFC8288] whose members have a relationship of
"alternate", though deployment is a chicken-and-egg problem.

That is the specification describing the exact mechanism this site uses, and naming its exact weakness. RFC 8288 is where the Link header is defined, and rel="alternate" is the relation that means “the same thing, in another form.” I send that header on every page. Whether anything reads it is not something I can measure from here, and I would rather say so than invent a number.

Vary is an instruction to a cache, not a promise from one

The header that is supposed to make negotiation safe is Vary. §12.5.5 is unambiguous about what it obliges:

To inform cache recipients that they MUST NOT use this response to
satisfy a later request unless the later request has the same values
for the listed header fields as the original request [...] In other
words, Vary expands the cache key required to match a new request to
the stored cache entry.

RFC 9111 §4.1 repeats it with the same force. Read carefully, though, and the obligation runs one way. It binds the cache when the origin sends Vary. It does not oblige the origin to send it, and it does not make any particular cache implement it. The origin’s side is a conditional SHOULD — send it “when it wishes that response to be selectively reused” — and the spec even blesses leaving it out: “Vary might be elided when an origin server considers variance in content selection to be less significant than Vary’s performance impact on caching”.

So Vary is correctness expressed as a request. Whether you get correctness depends on what is actually running between you and the reader.

What my two cache layers do with it

There are two caches in front of this site: LiteSpeed’s own cache at the origin, and Cloudflare at the edge. I measured both rather than trusting either.

LiteSpeed first, straight at the origin, four different Accept values against one URL:

for a in 'text/html' 'text/markdown' 'application/json' '*/*'; do
  curl -s -4 -o /dev/null -D - -H "Host: visibleorigin.com" -H "Accept: $a" \
    http://127.0.0.1/services/ | grep -i '^x-litespeed-cache'
done
# x-litespeed-cache: miss     <- the first request stores the entry
# x-litespeed-cache: hit
# x-litespeed-cache: hit
# x-litespeed-cache: hit

One entry, four different requests, three hits. Accept is not part of the cache key. That is not a bug — nothing told it to be — but it is the whole risk in one command.

Cloudflare documents its own behavior in one entry of its Cache-Control reference, and the whole story is in three sentences:

vary — By default, Cloudflare does not consider vary values in
caching decisions. Vary values are respected when you configure the
Cache Rules Vary setting, when Vary for images is configured, and
when the vary header is vary: accept-encoding.

So it is a setting, not a limitation. Accept-Encoding works out of the box because compression would be unusable otherwise; anything else is opt-in. A default is what almost every site is running, which is why the mechanism has the reputation it has.

And here is the part I got wrong until I measured it, so I will say it in full. I had written in my own notes that these two layers “do not respect Vary: Accept,” as though it were a property of the software. It is a property of the configuration. Cloudflare will vary on Accept if you ask it to, and on its own site it does.

One letter of difference between my edge and my origin

To serve Markdown on the HTML URL safely behind Cloudflare, the usual pattern is a Cache Rule that bypasses the cache when the request asks for Markdown. Then the negotiated response is never stored, and nothing can be handed to the wrong client. That is the pattern I set up here.

It matches on a substring of the Accept header. Here is what it actually does, measured on this site, changing nothing but that one header:

IP=$(dig +short @1.1.1.1 A visibleorigin.com | head -1)
for a in 'text/markdown' 'TEXT/MARKDOWN' 'Text/Markdown' 'text/x-markdown' '*/*'; do
  curl -s -o /dev/null -D - --resolve visibleorigin.com:443:$IP \
    -H "Accept: $a" https://visibleorigin.com/services/ | grep -i '^cf-cache-status'
done

The rule matches case-sensitively. HTTP header field names are case-insensitive, and media types are too: TEXT/MARKDOWN is a correct way to ask. A client that asks that way does not get the bypass.

On its own that is harmless. It becomes something else when the application half disagrees. In PHP the obvious way to test the header is stripos(), which is case-insensitive. The equivalent mistake exists in every language: a lowercased comparison at the origin, an exact substring match at the edge. If both halves are live, a request with different capitalization gets Markdown from the application through the ordinary cacheable path — and the edge stores it under the HTML URL and serves it to whoever comes next, including a browser, including */*, which is what a lot of crawlers send. That is the MUST NOT in RFC 9111 §4.1, happening for a whole TTL.

I found this while auditing a setup where both halves were present. I reported it there and it is being fixed; I am not naming it or publishing the request until it is. What I can publish is why negotiation is switched off here: the rule above has the gap, so turning the application half on would import the problem. The general rule I took away is worth more than the incident anyway — when a decision is made in two layers, the lower layer must be at least as strict as the upper one. If the edge skips fewer cases than the origin answers, the difference gets cached.

The other failure: the alternate that stopped being the same page

Parallel routes have none of that. There is no header to disagree about, and a cache keyed on the URL is keyed on exactly the right thing. Their failure is slower and, I think, harder to notice.

Cloudflare’s own site is a good place to see it, because they ship both mechanisms and they ship them well. Their pages negotiate correctly on the same URL, and they publish .md routes beside them. Here are the two representations of their homepage, fetched a second apart:

CFIP=$(dig +short @1.1.1.1 A www.cloudflare.com | head -1)
curl -s --resolve www.cloudflare.com:443:$CFIP https://www.cloudflare.com/ \
  -o home.html -w "HTML %{http_code} %{size_download}B %{content_type}\n"
curl -s --resolve www.cloudflare.com:443:$CFIP https://www.cloudflare.com/.md \
  -o home.md  -w ".md  %{http_code} %{size_download}B %{content_type}\n"
# HTML 200 1317097B text/html; charset=utf-8
# .md  200 2998B text/markdown; charset=utf-8

The HTML page’s <title> is “Cloudflare: Build for the agent era”. The Markdown document’s front matter reads title: "Cloudflare - Global Serverless Platform", and its body is about Cloudflare Workers. They are not the same page in two formats. They are two different pages, and the HTML advertises the other one itself:

grep -o '<link[^>]*text/markdown[^>]*>' home.html
# <link rel="alternate" type="text/markdown" href="/.md">

The response headers explain how they drifted, and it is not carelessness so much as arithmetic:

Nothing on either document says they disagree. An agent that follows the advertised rel="alternate" gets a coherent, well-formed Markdown file about a different subject, with no signal that it is looking at the wrong thing. That is the failure mode of parallel routes, and it is the one I have to defend against on this site, because parallel routes are what I chose.

What I ship, and the four defects in it

This site uses parallel routes, and negotiation is implemented but switched off. Every page and post is available at /slug.md, announced with a Link header on the HTML response:

curl -s -o /dev/null -D - --resolve visibleorigin.com:443:$IP \
  https://visibleorigin.com/services/ | grep -i '^link:.*markdown'
# link: <https://visibleorigin.com/services.md>; rel="alternate"; type="text/markdown"

Four things about that are worse than I would like, and all four are measurable today.

One: the HTML declares no Vary at all. Not Vary: Accept, not anything — the origin sends none, and the only one that reaches you is Accept-Encoding, added at the edge. Since this site does not negotiate, no cache can currently get it wrong; the day negotiation is switched on, that omission becomes the thing RFC 9111 §4.1 calls out when a response is stored under a key that does not describe it.

Two: my .md routes are not being cached, and I asked for them to be. They go out with cache-control: public, max-age=3600 and the edge answers DYNAMIC, which means it is not storing them at all. The cacheability advantage that made me choose parallel routes in the first place is, right now, not being collected.

That one is worth sitting with, because it generalizes. The origin’s Cache-Control is a request, not an instruction, and an edge with its own rules can overrule it in either direction — here it declined to cache something I asked it to cache. Any design whose safety rests on the origin sending the right cache headers is resting on the layer that does not have the final say. That is the same lesson as Vary, arriving from the opposite side.

Three: the bypass rule distinguishes upper case from lower case and the standard does not, as the table above shows. It is inert here only because the other half is switched off.

Four: my own written procedure for switching negotiation on tells me to verify it with curl -sI — a HEAD request. I have already published a whole note about a diagnosis I got wrong, and this is the same species: HEAD returns DYNAMIC where GET returns HIT, so that verification step would have handed me a false all-clear. It is fixed in the file now; it was written down wrong for a week.

Serving Markdown to agents: frequently asked questions

Should I use Accept negotiation or a .md route?

If your site sits behind a shared cache and you are not going to configure that cache deliberately, use the .md route. That is not a shortcut; RFC 9110 §12.2 says reactive negotiation is advantageous “generally when public caches are used to distribute server load.” Use negotiation when you control the cache key and have verified it, and then keep verifying it, because the failure is silent and it is served to humans.

Does Vary: Accept make negotiation safe?

Only if something between you and the reader implements it. Vary obliges a cache that honors it; it does not oblige a cache to honor it, and it does not oblige your origin to send it. Cloudflare’s documentation states that by default it does not consider vary in caching decisions. Measured on my own origin, LiteSpeed’s cache returned one stored entry to four different Accept values.

How do I tell whether my edge is caching a negotiated response?

Request the same URL twice with different Accept headers and compare cf-cache-status and content-type — using GET, never HEAD. A HEAD request can report DYNAMIC for a URL whose GET is HIT, which reads as “not cached” when the opposite is true. Use a throwaway query string so you are testing a cold URL rather than poisoning a real one.

What should the .md document contain?

The same page. That sounds too obvious to write down until you compare a real pair: on the day I wrote this, one large site’s homepage and the Markdown alternate it advertises carried different titles and covered different products. If the two are generated by different pipelines or cached for different lengths of time, assume they will drift, and check them against each other rather than checking that each one exists.

Do agents actually send Accept: text/markdown?

Some do, and many send */*, which asks for nothing in particular. RFC 9110 §12.5.1 is blunt about how much weight a stated preference carries at all: wildcards have “limited practical value,” and a client “still need[s] to be able to handle a different response since the server is allowed to ignore their preference.” That cuts both ways here, and it is part of why a .md route is easier to reason about: a URL is not a preference, it is a request for one specific document.

I have no query data on which clients fetch the .md routes on this site, and I am not going to guess. Publishing them is a true statement about the site, not a traffic strategy.

Is text/markdown a real media type?

Yes. It is defined in RFC 7763 and listed in the IANA media types registry, and it carries an optional variant parameter so a sender can say which flavor of Markdown it means. Almost nobody sends it, this site included: what goes out is a bare text/markdown; charset=utf-8, which names the syntax family and says nothing about which extensions are in play.

What I would actually do

Publish the .md route. Announce it with a Link header and a <link> in the head, because that is the mechanism RFC 9110 points at even while admitting its adoption is a chicken-and-egg problem. Then write the check that compares the two documents, because that is the failure you have just chosen, and it is the one nobody builds an alarm for.

If you want negotiation as well, configure the cache first and verify it with GET before you touch the application, in that order. And make the origin stricter than the edge, never the reverse: whatever set of requests the edge agrees to skip, the origin should answer for a subset of them, so that a request the edge did not recognize gets ordinary HTML rather than a surprise that outlives the request.

The general shape of this

Both of these mechanisms are correct. Both of them are in the same chapter of the same specification. What decides between them is not which is more standards-compliant but which failure your infrastructure is more likely to hide from you — and the answer to that question is not in any specification, because it depends on a cache configuration that the specification cannot see.

That is the recurring shape of this whole category of work. The standard tells you what a correct answer looks like. It cannot tell you what your third layer of caching is doing to it on a Tuesday. The only thing that closes that gap is sending the request and reading what comes back, which is cheap, and doing it again after you change something, which is the part everybody skips.

Serving a file to agents is the easy half; finding out who actually asks for it is the other. When I measured that for this site’s .well-known files, 89% of the traffic turned out to be me.

Every command above runs against a public address, and the ones aimed at this domain you can run yourself. If one returns something other than what I said it does, that is worth an email — including the Cloudflare comparison, which may well read differently by the time you try it, since one of those two documents is allowed to be a day old. And if it is your own site you want read the way a machine reads it, that is what the free check is: you send the domain, I make the requests one crawler at a time, and you get back what each one received.