I changed one number on one page, and five other places went on saying the old one

Where a number actually lives

You edit a page. The page is right. You are done, and you are not, because a modern site restates its own facts in places that are not pages and that nobody visits.

Here is the full inventory for one fact on this site — how many services it offers — found by grepping for the count rather than by remembering where I had put it:

The pattern is not subtle once the table exists. It is also not the pattern I expected to find: I went looking for “static files rot, generated ones do not” and that is wrong, because robots.txt here is also typed by hand, into a plugin’s settings box, and it was fine. Where the bytes live — a file on disk, a database field, a string inside a plugin — turns out to predict nothing at all. The variable is whether the surface restates a fact it does not own, and a case further down sharpens even that.

The one that was wrong twice

The worst of these is /llms.txt, and it is worth being specific because it is the one this site has least excuse for.

Before the change, its services section listed two of the eight. Its notes section listed one of the seven articles then published. And the line pointing at the catalogue read “the eight services, with prices” — except with “seven”, and pointing at a page that states, in its own pricing section, “Nothing here carries a list price.” Two documents on the same domain, contradicting each other about the same page.

I fixed it on a Tuesday. When I checked it again on the Saturday, in order to write this, it listed four of the eight services and seven of the nine notes. It had gone stale again in three days, from the same cause, in a file I had personally corrected while thinking about exactly this problem.

Four days, in order

The page itself was correct on the first attempt, on 9 September. Here is what it took for the rest of the site to agree with it, and what prompted each step, because “I checked” is doing no work in that sentence.

The card in the last row is the one I find least comfortable. It was not a machine-readable file that nobody opens: it was a service description on the home page, under the right title, containing the entire body text of a different service. I had verified that card’s title, its image and its animation delay, and not its text. The check I ran was the check I had thought of.

What llms.txt is, and what it is not

Worth stating plainly, because the file gets treated as both more and less official than it is. The proposal is by Jeremy Howard, first published in September 2024, and it describes itself as “a proposal to standardise on using an /llms.txt file”. It is not an RFC and not a standard. Its stated purpose is to “provide information to help agents use a website”, on the reasoning that web pages are built for people.

It is also not ignored: the proposal notes that Chrome’s Lighthouse audits for one, and several model providers publish one for their own documentation. So the honest position is that a wrong llms.txt is not a catastrophe, and is not nothing either. What makes mine embarrassing is not its readership. It is that this site sells machine readability and its own machine-readable index described a different site.

The neighbouring surfaces are better defined. The /.well-known/ prefix those others live under is RFC 8615, which exists to give origin-wide metadata a reserved location: it “defines a path prefix for ‘well-known locations’… in selected Uniform Resource Identifier (URI) schemes”. And the sitemap protocol is the oldest member of this family. None of that helps with the problem in this note, because a reserved location says nothing about whether what you put there is true.

The check that finds this on any site

Pick a fact about your site that is a number or a list: how many products, how many locations, how many people on the team. Then ask every surface that might repeat it, rather than the ones you remember writing.

D=yourdomain.com
for u in / /llms.txt /sitemap.xml /.well-known/ai-catalog.json; do
  printf '%-34s ' "$u"
  curl -s "https://$D$u" | grep -oiE '.{0,40}(seven|eight|[0-9]+) (services|products|locations).{0,30}' | head -2
  echo
done

Then do the same against your meta descriptions, which do not appear in the body and are the single most commonly forgotten copy of any fact:

curl -s "https://$D/yourpage/" | grep -oE '<meta name="description" content="[^"]*"'

On this site that last command was how the first stale copy turned up: the page said eight, and the description that search engines actually display began with the word “Seven”. It had been that way for as long as the page had.

Two of the five were found by someone else

I want to be exact about how the remaining two surfaced, because “I checked carefully” would be false.

I fixed the page, then the home page, then the meta description, then llms.txt, and I believed I was finished. The skill document and the second rot of llms.txt were both found by running an adversarial review over the work — a set of independent passes whose only instruction is to attack what I have written and try to prove it wrong.

It has now killed or rewritten the premise of three notes in a row, including this one: the version of this article I set out to write claimed nine of ten machine-readable surfaces were generated and one was typed. That was false, and the file inventory above is what replaced it after the claim was checked file by file. The same pass also found a home-page card whose title said one service and whose body text was copied verbatim from another, which had been live and visible to human readers for a day.

The control case, which sharpens this

One surface spoils the tidy version of the story, and it is the one that makes the rule usable.

This site also publishes /auth.md, a short document telling agents there is nothing to authenticate against. It is produced by a plugin that returns a nowdoc — a block of literal text with no substitution of any kind — and it makes zero calls to any site content. By construction it is exactly as hand-typed as the skill document that was wrong.

It restates a countable fact too: “Five read-only tools”. And it is correct. The site’s MCP server answers tools/list with exactly five.

Which means I can tell you where the next failure is, because it is already written down. The day a sixth tool is added to that server, /auth.md will say five, and nothing will warn me, and I will find out when somebody reviews it or when an agent acts on the wrong number. Writing that sentence does not fix it. It just means I no longer get to be surprised.

The fix that is not discipline

The two surfaces that were correct were correct because they are rendered from the page at request time. Nobody remembered them because there is nothing to remember: change the page and they change.

So the repair for the rest is not a checklist. A checklist is the same memory problem wearing a hat, and this site already had one. The repair is to reduce the number of places a fact is stated to one, and derive the rest, and where that is genuinely impractical, to make the copy testable: one assertion, run on a schedule, that fetches both surfaces and compares the count.

Both of those are more work than editing a file. That is the actual trade, and it is worth naming honestly rather than pretending the discipline option works. It does not work. I am the control group.

Machine-readable surfaces going stale: frequently asked questions

How many places restate a fact on a typical site?

More than you will guess before counting. On this one, a single number lived in seven: the page, the home page, the meta description, two hand-written machine-readable documents and two generated ones. A site with a shop, a footer and an about page will usually have more.

Does anything actually read llms.txt?

Its author’s own page notes that Chrome’s Lighthouse audits for one and that several model providers publish them. It is a proposal rather than a standard, so treat a wrong one as a correctness problem rather than an emergency — but if you publish one, it is a claim about your site like any other.

Why did my meta description keep the old text after I edited the page?

Because it is not part of the page body. SEO plugins store the title and description as separate fields, and a search-and-replace over your content will not touch them. They are also what a search engine displays, which makes them a worse place than most to leave an old fact.

Is a generated document always better than a written one?

No. A generated document cannot say anything the source does not already contain, so a hand-written summary is often better writing. The point is narrower: any surface that restates a fact it does not own will eventually disagree with the source, and the more of them there are, the sooner.

How do I find the copies I have forgotten about?

Grep for the fact rather than for the file. Request every surface your site publishes and search the responses for the number or the phrase. You are looking for the copies you do not remember making, so any method that starts from your memory of them will miss the ones that matter.

What should I check after changing a published fact?

The page, the meta description, any summary page that lists it, and every machine-readable file you publish. Then request them from outside the server, because what is in your database and what a visitor receives are different questions.

The general shape of this

The failure here was not carelessness with the page, which I edited correctly the first time. It was that the page is only one of the places the site makes that claim, and the other six were invisible in the workflow that changed it. Nothing warned me. Nothing could: a stale copy is a perfectly valid file that happens to describe a site that no longer exists.

Each of these files was written by someone being careful, including the ones written by me while thinking about this exact failure mode. That is what makes it worth a note rather than an apology. The surfaces that stayed true are the ones nobody had to remember.

External scores have the same problem from the other side: an agent readiness score gave this site Level 5 for failing its only check.

If you only want to know whether a site’s llms.txt is well formed and its links still work, the llms.txt checker tests that for any domain.

Everything above was measured against this site while writing, and the commands are printed so you can run the same inventory against your own. If you would rather have that done for you, with the requests written out so you can repeat them without me, that is what the free check is for — and keeping these surfaces honest as a site changes is part of the work I do for money. The two notes either side of this one, on measuring a cache with a URL nobody else requests and on settings that outlived the plugin that wrote them, are the same shape of problem in different layers.