# I publish six .well-known files. 89% of the traffic to them was me

> I logged 1,067 requests for this site's .well-known files in three weeks. 952 were me. The most-requested path by strangers returns 404.

Source: https://visibleorigin.com/who-asks-for-well-known-files/
Published: 2026-10-13

---

**The short version.** I publish six machine-readable files under **/.well-known/** so that automated clients can find out what this site is. Over twenty-one days my server logged 1,067 requests for them.

**952 of those were me — eighty-nine per cent.** Two hundred and seventy-seven were my own direct checks. The other 675 came from a readiness scanner I was pointing at my own site, from a single address, always asking for the same eight paths.

That leaves 102 requests from genuine third parties, across 44 addresses. I can prove the identity of 33. And the single most-requested path among them is one I do not serve.

![Funnel diagram of 1,067 requests to well-known files, of which 952 were the site owner](https://visibleorigin.com/wp-content/uploads/wellknown-funnel.webp)
The largest source of traffic to my machine-readable surfaces was the person measuring whether they had any traffic, and the second largest was a tool that person was holding.

## The six .well-known files I publish, and why

Six files, none of them a page a person visits. They exist so that a crawler, an agent or a language model can discover what this domain offers without guessing: an ARD manifest, its predecessor format, a server card, and a handful of smaller descriptors. This site sells making sites legible to machines, so publishing them is not a demonstration — it is the product, pointed at itself.

Which makes the obvious question uncomfortable to ask and worth asking anyway: **does anything actually fetch them?**

## What the log says once I remove myself

The raw answer was 1,067 requests in twenty-one days, which sounds like a real little audience. Then I took out the traffic that exists because of me, and there are two kinds of it. The first is obvious: the two addresses that are this machine — `127.0.0.1`, which is the site calling itself, and the public address this server goes out through whenever I test anything.

The second kind took longer to see, because it does not look like me at all. It arrives from an address on someone else's network with a user agent naming a third-party product: `AgentReadinessScanner/1.0 (+https://isitagentready.com)`. That is an agent-readiness checker — and it is a checker *I have been running against this site*, with my own saved scan results sitting in a folder on this server. Its busiest days are the days I was building and measuring these files. **Six hundred and seventy-five requests, one address, the same eight paths every time.** It is not visiting me. I am fetching myself through it.

| Source | Requests | Share |
| --- | --- | --- |
| Me, directly — my own checks | 277 | 26% |
| Me, through a scanner I was running | **675** | **63%** |
| Certificate renewal challenges | 13 | 1% |
| Genuine third parties | 102 | 10% |
| **Total logged** | **1,067** | |

28 August to 17 September 2026. The first row is excluded by address; the second by user agent, and then confirmed against my own scan records.

This is not a small correction. It is the difference between "my agent surfaces get fifty requests a day" and "they get five". And the direction of the error is the one that flatters the product, which is the direction that should always make you look twice.

There is a general form of this worth naming, because a scanner is the easy case. **Any tool you point at your own site becomes traffic to your own site.** Uptime monitors, link checkers, preview fetchers, the validator you ran once and forgot, the CI job that curls your home page. None of them carry your address, and all of them are you.

## Who was actually there

One hundred and two requests from forty-four addresses. A user agent is a string the client chooses, so the only honest next step is to check each one against evidence its operator publishes. That gives three groups, and the middle one is the interesting one.

| Claims to be | Requests | Addresses | Can it be checked? | Proved |
| --- | --- | --- | --- | --- |
| A browser, or unnamed | 50 | 32 | No method exists | — |
| AgentTrustBot | 32 | 4 | Yes — list and reverse DNS | **27** |
| Claude-User | 9 | 2 | Only one-directionally | 0 |
| GPTBot | 4 | 2 | Yes — published ranges | **4** |
| python-requests | 3 | 3 | No method exists | — |
| ClaudeBot | 2 | 1 | Yes — published ranges | **2** |
| Dataprovider | 2 | 1 | No method exists | — |

"Proved" means the address was checked against evidence the operator publishes and passed. Blank means there is nothing to check against. Addresses sum to more than 44 because one appears under two names.

### The two that verify cleanly

OpenAI publishes [a file of address ranges for GPTBot](https://openai.com/gptbot.json). Both addresses claiming to be GPTBot fall inside them. Anthropic publishes [an equivalent file](https://claude.com/crawling/bots.json), and the single ClaudeBot address is in it. Four requests and two requests respectively — small numbers, but they are numbers I can defend.

### The one whose own two methods disagree

AgentTrustBot was the largest genuine source, and it publishes more than most: [a list of current addresses and a reverse-DNS procedure](https://agenttru.st/crawler), with an explicit preference between them — *"the fleet changes hosts. The reverse-DNS check above is true whenever it is true, which is why it is the one to rely on."*

Four addresses reached me. **All four are on the published list. Only three pass the reverse-DNS check.** The fourth, responsible for five requests, resolves to a name that has nothing to do with the operator's domain, so by the operator's own preferred method it is not their crawler — while their own list says it is.

I am not calling that dishonest; a fleet that changes hosts will do this, and they say so themselves. But it is a concrete instance of something this site keeps running into: **publishing a way to be verified is not the same as being verifiable**, and the gap only shows up when someone actually runs both checks.

### The ones with nothing to check against

Sixty-nine of the hundred and two requests cannot be resolved either way. Thirty-two addresses arrived wearing browser user agents, which for a JSON descriptor nobody links to is odd but proves nothing. `Claude-User` is a special case worth stating precisely, because it is easy to get wrong: Anthropic's documentation says that *"if a crawler has a source IP address on this list, it indicates that the crawler is coming from Anthropic"*. That is an implication in one direction. It does not claim the list is complete, and user-initiated fetches are a different path from the training crawler. So the correct verdict for those nine requests is **unverifiable, not false** — and treating the second as if it were the first is the error, not the caution.

Separating your own traffic from real traffic — including the traffic your own tools generate — is the first thing [the free check](https://visibleorigin.com/services/#free-check) does before it reports anything about crawlers, with every request written out so you can repeat it. Getting a site's discovery surfaces to exist, respond and agree with each other is [the technical SEO and server work](https://visibleorigin.com/services/).

## The most-requested path is one I do not serve

With my own traffic and the scanner removed, the ranking changes shape completely. The file most third parties asked for is not one of the six I publish:

| Path | Requests | Addresses | Response |
| --- | --- | --- | --- |
| `/.well-known/security.txt` | **21** | **14** | **404** |
| `/.well-known/ai-catalog.json` | 21 | 17 | 200 |
| `/.well-known/ard.json` | 14 | 14 | 200 |
| `/.well-known/agent-card.json` | 10 | 4 | **404** |
| `/.well-known/agent.json` | 10 | 4 | **404** |
| `/.well-known/mcp/server-card.json` | 8 | 4 | 200 |
| jwks, webmcp, mcp.json, ucp, others | 13 | 6 | mostly 404 |

Requests from third parties only. The widest spread of distinct addresses belongs to the ARD manifest and its predecessor — seventeen and fourteen different clients, mostly one request each.

**Twenty-one requests from fourteen different addresses for `security.txt`, and every one of them got a 404.** That is a published standard, it is the most-asked-for path on the list, and the reason it is missing is not a decision I made — it is a file I never thought about. I spent three weeks building descriptors for a future that may or may not arrive, while the thing people were actually asking for had been specified years ago.

## What this does and does not prove

Two limits, both of which cut against the story being tidy.

**These counts are a floor, not a total.** I checked, and these files are cached at my CDN edge: request one three times and the first is a miss, the next two are hits that never reach my server. So an unknown number of fetches are missing from every figure above — and worse for the comparison, a client that asks once is more likely to be logged than one that asks repeatedly. I wrote about [sending sixty requests and finding six in my log](https://visibleorigin.com/access-log-cannot-see-the-edge/) after discovering this; the same caveat applies here and I cannot quantify it from the origin.

**Thirty-three verified requests is not a market.** It is evidence that identifiable operators fetch these files, at small volume, in three weeks, on a site with almost no inbound links. It is not evidence that publishing them produces traffic, or referrals, or anything else. Anyone telling you these surfaces are busy should show you their log with their own requests removed first — and then removed again for the tools they were running, which on the evidence of mine is where most of the number lives.

If you have published discovery files and do not know whether anything reads them, that is exactly what [the free check](https://visibleorigin.com/services/#free-check) answers, with your own traffic separated out and each identity checked against what its operator publishes. If you want the surfaces built and then measured rather than assumed, [tell me what you have published](https://visibleorigin.com/contact/).

## How to run this on your own site

**1. Exclude yourself by address, then exclude your tools by name.** The first is easy and the second is the one that gets you. Anything you pointed at your own site — scanner, monitor, validator, CI job — arrives from somebody else's network with somebody else's user agent, and is still you. Go through the top user agents one by one and ask whether you caused each of them.

**2. Count distinct addresses per path, not just requests.** Six hundred requests from one address and six hundred from four hundred are completely different findings, and the request count alone hides which one you have.

**3. Check every identity against something the operator published.** A range file, a reverse-DNS procedure, anything. Where no method exists, write "unverifiable" and leave it there. Do not upgrade it to "verified" because the name looks plausible, and do not downgrade it to "fake" because you could not check.

**4. Run both methods when both exist**, and notice when they disagree. That disagreement is a real finding about the operator, not noise in your data.

**5. Check whether your CDN is answering these files** before you present any count as a total. This is the step I skipped for three weeks.

## Frequently asked questions

### Why not just count user agents?

Because the user agent is chosen by the client and nothing in HTTP asks it to prove anything. I wrote a whole note about [the number I nearly published from user-agent counts](https://visibleorigin.com/ai-crawlers-verified-by-ip/), which was wrong by a factor that mattered. Counting by address and verifying against published evidence is more work and it is the only version that survives being questioned. Note the direction of this note's own error, though: user agents are what let me *find* the scanner. Addresses alone would have left 675 requests looking like a real visitor.

### Is 89% self-traffic unusual?

On a low-traffic site that is being actively worked on, I would expect it to be common and almost never measured. Every time I checked a file responded, that check became a line in the log I was about to draw conclusions from. The fix is mechanical: exclude your own addresses first, then work down the user agents asking which ones you caused.

### Does a 404 on these paths hurt me?

Not directly — a 404 is a correct answer to "do you have this". What it tells you is what clients expect, which is information you cannot get any other way. Whether to serve any given file is a separate decision about whether you can honour what it claims.

### Should I serve every path something asked for?

No. Serving a descriptor you cannot back is worse than not serving it, because it makes a promise a machine will act on. The only one from my list I intend to add is `security.txt`, because it is a published standard and the thing it promises — that someone reads security reports sent to that address — is a promise I can actually keep.

### Why is unverifiable not the same as fake?

Because they call for different actions. A verified fake is something to block. An unverifiable request is something to leave alone while noting that the evidence to decide does not exist. Collapsing the second into the first produces confident numbers about impostors that are really numbers about missing documentation.

### What would change your mind about publishing these files?

Sustained zero. Thirty-three verified requests in three weeks is small, but it is not zero, and the verified operators are ones whose fetches plausibly end up in front of a person. If a year of measurement produced nothing verifiable at all, the honest move would be to say so here and stop claiming the surfaces do anything.

## The verdict

I built six files so machines could find out what this site is. Then I generated eighty-nine per cent of the traffic to them myself — a quarter by hand, and nearly two thirds through a tool I was holding and did not think of as me. That is not a scandal; it is what happens when you measure something you also maintain. It only becomes a problem at the moment you publish the total.

**The useful number was never the traffic.** It was the path that fourteen strangers asked for and I did not have, and the discovery that one operator's two published verification methods disagree about one of its own addresses. Neither of those needed a big audience to find. They needed the small one to be counted honestly, which took two passes and both of them were mine to get wrong.

Being found is a separate problem from being asked for: [I turned on IndexNow to fix a Google problem and learned that Google is not in the protocol](https://visibleorigin.com/indexnow-does-not-reach-google/).

## Sources

- [OpenAI, published IP ranges for GPTBot](https://openai.com/gptbot.json) — checked against every address claiming that agent.
- [Anthropic, published crawler IP ranges](https://claude.com/crawling/bots.json), and [the accompanying documentation](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) for the one-directional wording about what a match means.
- [AgentTrustBot's crawler page](https://agenttru.st/crawler) — its address list, its reverse-DNS procedure, and its own statement that the reverse-DNS check is the one to rely on.
- Measurements taken against visibleorigin.com over 28 August to 17 September 2026, from the origin access log, with the site's own two addresses and its own scanner traffic separated out.