Generative engine optimization and AEO: what can be measured, and what is guesswork

By Jhon Patzi · Published · 31 min read

A wooden pier lit in blue in the dark, its far end disappearing into fog - generative engine optimization

The short version. In generative engine optimization (GEO, also marketed as LLM SEO), as in answer engine optimization (AEO), a site can measure what happens around an AI answer, not inside it: which AI crawlers fetched its pages, whether Google and Bing index them, when an assistant opened a page for someone, and, in one Google report and one Bing report, whether a link to it was shown or cited. Why an engine picked a page, and whether your change caused it, is guesswork. The paper that named generative engine optimization measured its “up to 40%” in an engine its authors built, with the search results fixed and, in their code, quotes, sources and statistics the model was allowed to invent. On this site, in 55 days, ChatGPT fetched pages for its users 30 times from OpenAI’s addresses, Claude and Perplexity zero times that I can verify, and Bing’s citation report had no data on October 20.

Generative engine optimization in six layers, from crawler access to the answer, each marked measured, a floor or guesswork
Each layer with the instrument that reads it and what it read for this site. The sixth is where the GEO paper measured, inside an engine its authors built.

The phrase generative engine optimization comes from a research paper, and so does the number that travels with it. I read the paper, its appendix and the code its authors published, then measured the one site I can measure from the inside: this one, whose business is being read correctly by machines. The two readings disagree in a useful way. The paper measured the one step a site owner never sees: how an answer is put together from its sources. Everything a site owner can see happens before that step or after it.

This note goes layer by layer: what the paper did, what Google, Bing, OpenAI, Anthropic, Perplexity and DuckDuckGo document, and what 55 days of this site’s server log and the two search engines’ own tools show. Every number about this site was read between October 19 and 22, 2026, and the commands are near the end, so you can run them against your own log.

Where the term generative engine optimization comes from

Generative engine optimization was named in “GEO: Generative Engine Optimization,” by Pranjal Aggarwal of IIT Delhi and five co-authors: three at Princeton University and two independent researchers. It was first posted to arXiv on November 16, 2023. The third version, from June 28, 2024, carries the header of KDD 2024, the ACM conference on knowledge discovery and data mining, where it was published. The paper calls systems such as Bing Chat, Google’s SGE and Perplexity “generative engines,” because they search for information and write an answer from several sources, and it proposes GEO as “a flexible black-box optimization framework” that helps content creators improve “their content visibility in generative engine responses.”

Its starting point is a fair description of the problem, and it is worth quoting because the rest of this note depends on it: “Given the black-box and fast-moving nature of generative engines, content creators have little to no control over when and how their content is displayed.” To test what creators could still do, the authors built GEO-bench, a set of 10,000 queries from nine sources, among them real search queries from datasets such as MS MARCO and Natural Questions, essay and debate questions, questions from Reddit’s ELI5, trending queries from Perplexity’s Discover page, and queries written by GPT-4. Then they tried nine ways of rewriting a web page: making it sound authoritative, adding statistics, stuffing it with keywords, citing sources, adding quotations, simplifying the language, improving fluency, and adding unique words or technical terms.

The abstract states the result: “we demonstrate that GEO can boost visibility by up to 40% in generative engine responses.” The introduction adds that citations, quotations and statistics did best, and that on Perplexity, “a real-world generative engine,” the improvement reached “up to 37%.” Both numbers are real measurements, and both measure something narrower than the words suggest.

The term has since reached Bing’s documentation. Its webmaster guidelines say that “Generative Engine Optimization (GEO) focuses on content eligibility for grounding and reference in AI responses,” and in the next sentence: “SEO does not guarantee rankings or traffic, and GEO does not guarantee grounding or citations in AI experiences.” Answer engine optimization, the name this site uses for its service, aims at the same thing: being the source an AI answer draws on. Everything below applies to both names.

What the GEO paper measured, exactly

Everything in the paper happens inside an engine the authors assembled themselves, and its details decide what the numbers mean. For each query they took “only the top 5 sources” that Google returned and gave their text to gpt-3.5-turbo with one fixed prompt. That prompt orders: “Every sentence in the answer should be immediately followed by an in-line citation to the search result(s).” They sampled five answers per query, at a temperature of 0.7, to smooth out the randomness.

Then, for each query, they picked one of the five sources at random, rewrote it with each method in turn, and generated the answers again with the other four sources untouched. A method scores by how much more of the answer the rewritten source received. The main metric, position-adjusted word count, starts from “the normalized word count of sentences related to a citation” and weights sentences near the top of the answer more heavily. And because “all impression metrics are normalized by multiplying them with a constant factor so that the sum of the impressions of all citations in a response equals 1,” the five sources split each answer between them. An untouched source scored 19.3 in the paper’s main table, about a fifth.

Rewrite applied to one of the five sourcesPosition-adjusted share of the answerChange against the untouched source
None (the untouched source)19.3—
Quotation Addition27.2+41%
Statistics Addition25.2+31%
Fluency Optimization24.7+28%
Cite Sources24.6+27%
Authoritative21.3+10%
Keyword Stuffing17.7−8%

From Table 1 of the paper (arXiv version 3), the “Overall” column of position-adjusted word count; the percentages are my arithmetic on those values. The text puts Cite Sources, Quotation Addition and Statistics Addition at “30-40%”; in the table, Cite Sources is at 27%.

So the “up to 40%” is a relative change in a share. A source that held about a fifth of a GPT-3.5 answer held a little over a quarter after the best rewrite. Inside that engine the effect is real, and the paper reports it with five random seeds and small deviations. It is not a traffic figure, a ranking or a count of citations, and no engine reports that share to the people who own the pages.

Diagram of the GEO paper’s test, one of five fixed sources rewritten, and bars of its share of the answer by method
The “up to 40%” is a change in one source’s share of a GPT-3.5 answer while the other four stay fixed; the Perplexity test used uploaded files.

How the three best methods were written

The paper says the methods were applied “using a large language model, prompted to perform specific stylistic and content changes,” that “GPT-3.5 turbo was used for all experiments,” and that all the prompts are in the authors’ public repository, whose README says that running one script will “replicate results in paper.” I read the prompts there, in src/geo_functions.py, mapped to method names in src/run_geo.py. These are the instructions behind the three methods the paper ranks highest:

MethodWhat its prompt tells the model
Quotation Addition“Add more quotes in the source, even though fake and artificial.”
Cite Sources“You may invent these sources but ensure they sound plausible and do not mislead the reader.”
Statistics Addition“Add positive, compelling statistics (even if hypothetical) at multiple relevant places in the text.”

From the repository GEO-optim/GEO on GitHub, read on October 22, 2026; the prompts file has not changed since November 26, 2023. Each prompt also asks the model to keep the source’s core content and structure.

The paper’s own example of Cite Sources, in its table of representative rewrites, adds to a page about Swiss chocolate the phrase “According to a survey conducted by The International Chocolate Consumption Research Group” and credits the rewrite with a 132.4% gain. The paper does not say whether that group exists, and the prompt behind the method did not require it to. What the experiment measured is how much one 2023 model favors text that looks supported, whether or not the support is real. That is worth knowing. It is not evidence that adding real sources and real statistics to a page gets it cited by a live engine, and it was never tested as that.

The test on Perplexity, and the appendix table

The paper’s evidence that the methods work on a deployed engine comes from Perplexity, and the appendix explains how the test was run: “Since perplexity.ai does not allow the user to specify source URLs, we instead provide source text as file uploads to perplexity.ai while ensuring all answers are generated only using the file sources provided.” It used “a subset of 200 samples” of the test queries. Perplexity did not search the web in that test; it summarized five uploaded files.

The body of the paper shows four rows for Perplexity. The full table, with eight, is in the appendix, and two of its rows run against the text. Cite Sources, one of the three methods the paper recommends, scored 19.0 on the second metric, subjective impression, against 24.7 for the untouched source: about 23% lower. That metric is a score GPT-3.5 itself assigns to each citation, following a method called G-Eval. Keyword stuffing, which the text says “performs 10% worse than the baseline,” does so on position-adjusted word count, but it scored 28.1 on subjective impression, about 14% higher than the untouched source. With 200 queries and two metrics that disagree, “up to 37%” says less than it seems to.

What the authors say does not carry over

The limitations section is short and candid. The methods “may need to adapt over time as GEs evolve, mirroring the evolution of SEO,” queries change, and: “owing to the black-box nature of search engine algorithms, we didn’t evaluate how GEO methods affect search rankings.” That last sentence matters most. In the experiment, the five sources were chosen before any rewrite. On a live engine, whether a page is among the sources at all is decided first, by retrieval, and retrieval is the step the paper did not test.

One more result follows from the metric being a share. In its section on optimizing several websites, the paper reports that visibility moved between sources rather than growing: with Cite Sources, “a substantial 115.1% increase in visibility for websites ranked fifth in SERP, while on average, the visibility of the top-ranked website decreased by 30.3%.” Shares that add up to one can only grow at another source’s expense, so a method that every source adopts moves visibility around instead of adding to it.

What carries over to a real site, and what does not

None of this makes the paper wrong. It makes it a controlled experiment, and a controlled experiment answers a narrower question than the one a site owner asks. The paper asked how a fixed set of sources shares one engine’s answer. A site owner asks whether their pages will be used at all, by engines they cannot see into, for questions they cannot list.

What decides the resultIn the paperOn a live engine
Which pages are the sourcesThe top five Google results, fixed in advanceDecided by retrieval before any answer is written; the paper did not test it
The enginegpt-3.5-turbo and one prompt, in 2023Models that change; Google says AI Overviews and AI Mode “may use different models and techniques”
What gets citedEvery sentence, by instructionWhatever the engine decides to show
The scoreShare of the answer’s words, weighted by positionNot reported by any engine; Google reports impressions, and Bing reports citations and a share of citations per query
The added contentQuotes, sources and statistics the model could inventBing: content “should be created for users, not to manipulate ranking systems or trigger citations or AI responses”
The deployed testPerplexity, given five uploaded files, 200 queriesPerplexity visits and links web pages itself, through its own crawlers

From the paper (arXiv version 3), Google’s page on AI features, Bing’s webmaster guidelines and Perplexity’s crawler documentation, all read on October 22, 2026.

What carries over is modest and familiar. In that engine, text that answered the query with specifics did better than text stuffed with the query’s words, and a persuasive tone alone did little: the paper found “no significant improvement” from it. A page that answers plainly and shows its evidence is good practice for any reader, human or model. What does not carry over is the arithmetic: no percentage from that table applies to your pages in a live engine.

What Google, Bing and the AI companies document

The operators’ own documentation is the closest thing to ground truth a site gets, so I read it the same day as the paper, October 22, 2026.

Google is the most direct. Its page on AI features says: “There are no additional requirements to appear in AI Overviews or AI Mode, nor other special optimizations necessary.” To be eligible as a supporting link, “a page must be indexed and eligible to be shown in Google Search with a snippet,” and “You don’t need to create new machine readable files, AI text files, or markup to appear in these features.” The first fundamental on its list is “Ensuring that crawling is allowed in robots.txt, and by any CDN or hosting infrastructure.” The page also gives the usual caveat: “Indexing and serving isn’t guaranteed.”

Bing says much the same in more words. Its guidelines state that “Bing and Copilot search experiences rely on the same core crawling, indexing, and ranking foundation as traditional search.” They go further than Google on what helps, without a number attached: pages are “more likely to be selected for grounding queries and citations when content stands on its own,” and “URLs focused on a primary topic are more likely to be selected for grounding results.” Microsoft’s advertising blog, in its own guide to appearing in AI answers, opens the checklist with: “there’s no secret sauce that guarantees selection in AI answers.”

The AI companies document crawlers, and their only lever is permission. OpenAI: “Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though can still appear as navigational links.” Anthropic: disabling Claude-SearchBot “may reduce your site’s visibility and accuracy in user search results.” Perplexity: “To ensure your site appears in search results, we recommend allowing PerplexityBot” in robots.txt. DuckDuckGo says DuckAssistBot “crawls pages in real-time for our AI-assisted answers, which prominently cite their sources.”

Read together, these pages describe two kinds of lever and only two: the conditions a page must meet to be eligible, and switches that take it out. The switches are precise. Bing’s noarchive tag “prevents content from being used in Copilot responses and grounding results.” Google’s Search generative AI control, open to every site since August 31, 2026, means “links to your site and your site’s content won’t appear in Search generative AI features,” which I compared with Cloudflare’s switches in what Cloudflare’s AI settings stop and what they only ask. A robots.txt group for OAI-SearchBot keeps a site out of ChatGPT search answers. Every operator documents how to leave its AI answers; none says, with a number, how to get into them.

That asymmetry is why this site’s Terms list “appearing in ChatGPT, Perplexity, AI Overviews or any AI assistant” among the things not delivered and not promised, and why the first item under what I will not promise is exactly that. Nobody outside those companies controls what their systems surface. A site owner controls whether the pages can be fetched, read and indexed, and that much can be measured.

Measured on this site: 55 days, layer by layer

My server logged 78,036 requests between August 28, 2026 at 19:38 UTC and October 22 at 20:00 UTC. I counted a crawler’s request only when it came from an address on the list its operator publishes, downloaded on October 22, the method I described in verifying AI crawlers by IP instead of by name. Five addresses that used three or more crawler names each, 1,029 requests between them, count as impostors, and the server’s requests to itself are left out.

One limit applies to every number from the log. Cloudflare answers part of the traffic from its cache, and those requests never reach my server: when I tested it, my access log recorded 6 of the 60 requests I sent. Every count below that comes from the log is a floor, not a total.

1. Can AI crawlers fetch the pages?

Crawler, and what its operator says it doesVerified requestsSame name, other addresses
OAI-SearchBot, ChatGPT search416104
Claude-SearchBot, Claude search042
PerplexityBot, Perplexity search2657
Googlebot, Google Search and its AI features44540
Bingbot, Bing and Copilot1779
GPTBot, OpenAI model training43083
ClaudeBot, Anthropic model training1,02393

This server’s access log, August 28 to October 22, 2026. Of the 3,531 verified requests from every crawler whose operator publishes a list, 3,268 got a 200, 226 a redirect and 37 a 400, 404 or 405 for paths that do not exist or methods the site does not accept. None got a 401, 403, 429 or 5xx.

This is the layer a site controls completely, and the only one where my answer is a plain yes: every verified crawler that asked for a page got it. The gap is Claude-SearchBot, Anthropic’s search crawler, which never reached my server from Anthropic’s addresses; its name arrived 42 times, all from the impostors. GPTBot and ClaudeBot collect training data, not answers; they are in the table to compare with the search crawlers of the same two companies. How often each one reads robots.txt before it crawls is in robots.txt for AI crawlers, agent by agent.

A clean log on your own server is the easy half. The hard half is whether a crawler gets the same answer through your CDN, firewall and cache, which an origin log cannot show, because a refusal at the edge never reaches it. Testing that from outside, crawler by crawler and layer by layer, is the core of the crawler and AI readiness audit, and it is the measurable part of AEO on a site I do not own.

The same log answers a smaller question. This site publishes an llms.txt, a proposed file that summarizes a site for language models. In 55 days, 48 requests for it reached my server, and none came from an address on any AI operator’s published list; the ones I could identify were browsers, directories and checkers. That fits Google’s documentation, which says AI text files are not needed for its AI features. The file costs nothing to keep, but on this evidence it is not how my pages reach an answer.

2. Are the pages in the indexes the answers draw on?

Google says a page needs to be indexed, with a snippet, to be a supporting link in AI Overviews or AI Mode, and Bing says Copilot rests on the same index as its search. Both indexes can be read. Google’s URL Inspection API, queried on October 22 between 07:10 and 07:13 UTC for the 42 posts and pages published at that moment, called 40 of them “Submitted and indexed.” The other two were “URL is unknown to Google,” and one of them was my own Services page. Bing’s Webmaster API counted 11 pages of this site in its index on October 20, up from 0 thirty days earlier; it says how many, not which. Why Bing’s count is so much lower, and what each way of submitting a site actually did, is in submit your site to the Bing index, measured.

This layer is a necessary condition, and the documentation says so in plain terms. A page that is not in the index is not eligible to be a supporting link in AI Overviews, and Bing’s guidelines tie Copilot citations to the same crawling and indexing. Being indexed makes a page eligible; it does not make it chosen.

3. Did an assistant open a page for someone?

OpenAI, Anthropic and Perplexity each run a separate agent for fetches a user triggers. OpenAI: “When users ask ChatGPT or a CustomGPT a question, it may visit a web page with a ChatGPT-User agent.” Anthropic: “When individuals ask questions to Claude, it may access websites using a Claude-User agent.” Perplexity: “When users ask Perplexity a question, it might visit a web page to help provide an accurate answer and include a link to the page in its response.” These requests are the nearest thing in a server log to an answer being written, so I counted them one by one.

ChatGPT-User made 30 requests from OpenAI’s published addresses, each from a different address, between September 7 and October 22, 17 in September and 13 in October. All 30 got a 200, and they asked for 9 URLs: the homepage 16 times, my note on the DNS-AID record 5 times, two other notes twice each, and five other pages once. Another 101 requests carried the ChatGPT-User name from addresses that are not on OpenAI’s list, 81 of them from the five impostors.

The others verify to almost nothing. Of 183 requests carrying the Claude-User name, none came from an address on Anthropic’s list: 115 came from the impostors and 68 from 11 addresses I cannot attribute. All 97 requests named Perplexity-User came from the impostors. DuckAssistBot came three times from DuckDuckGo’s addresses, on October 15, 20 and 22, and each visit followed the same pattern within about a minute: robots.txt, one page, robots.txt again. Those are 9 verified requests, against 14 that used its name from the impostors.

Bar chart of requests named ChatGPT-User, Claude-User, Perplexity-User and DuckAssistBot, split by operator address
Only ChatGPT-User and DuckAssistBot came from their operators’ published addresses. A verified fetch shows a page was opened, not that it was cited.

A verified fetch proves that an assistant opened the page while doing something for a person. It does not say what the person asked, whether the page ended up in the answer, whether a link was shown, or whether anyone clicked. The homepage’s 16 fetches carry no hint of the question, and some may come from people looking up the site itself. The fetches do not follow OpenAI’s search crawler either. For five of the nine URLs, ChatGPT-User reached my server before OAI-SearchBot ever did, and three of those OAI-SearchBot has never requested from OpenAI’s addresses. For one more, the two arrived in the same second. A fetch is evidence of use, not of citation, and the log cannot tell you why it happened.

One coincidence is worth recording and not worth more. The DNS-AID note, the page ChatGPT-User fetched most after the homepage, is also the page with the most Google impressions in Search Console for the 28 days to October 16, with 78; the whole site had 273. With 30 fetches, I cannot tell whether that is a pattern.

4. Was a link shown or cited?

Two reports answer this directly, one from each search engine, and neither from the AI companies.

ReportWhat it showsWhat it does not
Google Search Console, generative AI performance reportImpressions of links to your pages in AI Overviews and AI Mode, by page, country, device and dateQueries or clicks, which its help page does not list; the Search Console API offers no search type for it
Bing Webmaster Tools, AI PerformanceCitations in Copilot, Bing’s AI-generated summaries and some partners, by page and by grouped “grounding queries”Clicks, full prompts, and why a page was cited; it is a sample, and sparse activity “may not surface”
OpenAI, Anthropic, PerplexityTheir crawlers’ names, purposes and address listsAny report of where a site was cited, on any of the pages I read

From Google’s help pages for the generative AI performance report and the Search Analytics API, Bing’s AI Performance help article and the three companies’ crawler documentation, all read on October 22, 2026.

Google’s report is new. Its help page says that “As of August 31, 2026, we’ve rolled out these insights to all websites worldwide,” and defines its unit plainly: “Impressions are how many times links to your site were shown to a user in a generative AI feature on Google Search.” In the general performance report, the same impressions are part of the Web totals, where “An AI Overview occupies a single position in search results, and all links in the AI Overview are assigned that same position.” If the report does not appear, Google says this “usually happens when your site hasn’t received enough impressions in generative AI features.”

For this site, Search Console’s general performance report, which still counts AI features inside its Web totals, showed 273 impressions and 1 click for the 28 days to October 16. I have not read the generative AI report for this note: it lives in the dashboard, and the Search Console API that my scripts use, in its reference last updated on August 11, 2026, lists no search type for it. That is the one number in this note that has to be read by hand.

Bing’s report counts citations, and its documentation is careful about what that means. AI Performance covers Microsoft Copilot, AI-generated summaries in Bing and select partner AI integrations. Its grounding queries “are not full user questions or prompts,” and its data “represents a sample of overall citation activity,” so “Very low or infrequent citation activity may not surface.” When this site’s Bing Webmaster Tools account was checked on October 20, AI Performance had no data. Bing’s API reported 0 impressions and 0 clicks in ordinary Bing search from September 18 to October 19, and of the 62 methods Microsoft documents for that API, none is for AI Performance; the report is read in the dashboard or exported from it as CSV or Excel. On a site this young, a blank report is what the documentation predicts: when sparse citations do not surface, Bing says, that “does not indicate a penalty or exclusion.”

5. Did anyone click?

Bing answers the question its own report raises: “Do citations represent clicks or traffic? No.” Clicks from AI answers arrive at a site like any other visit, usually with a referrer, and an analytics tool can count them by source. On this site I do not count them, by choice. The server log records every request, but the Privacy Policy says those logs are used “to diagnose problems and detect attacks,” so I use them to count crawlers and not people, and Analytics only runs for visitors who accept it. Clicks are measurable; this site measures them only for visitors who agree to it.

6. Did the page shape the answer?

No engine reports this, and it is the quantity the GEO paper measured: how much of an answer came from a source. Outside an experiment like the paper’s, where the researchers generate every answer themselves, the only way to estimate it is to ask an assistant questions and record what comes back. A tracker does exactly that, so it can only count the answers it asked for: its own prompts, its own account, its own moment.

How much those answers vary has been measured. In a study published by SparkToro in January 2026, 600 volunteers ran 12 prompts through ChatGPT, Claude and Google’s AI answers 2,961 times, and the authors concluded: “there’s a <1 in 100 chance that ChatGPT or Google’s AI, if asked 100X, will give you the same list of brands in any two responses.” They also concluded, against their own starting hypothesis, that “visibility % across dozens to hundreds of prompts run multiple times is a reasonable metric.” The study measured brand recommendations, not page citations. Its authors say they are not professional researchers and invite others to repeat the work, and Fishkin’s research partner works for a company in AI tracking, which the post raises itself as a possible conflict of interest. A single answer is an anecdote; a percentage needs many runs and the prompts behind it.

What is guesswork, and why it stays that way

Put the layers side by side and the line between measurement and guesswork is easy to draw.

QuestionCan a site measure it?With what
Can each AI crawler fetch my pages?YesThe server log, checked against each operator’s address list, and requests from outside
Are my pages in Google’s and Bing’s indexes?YesURL Inspection in Search Console; Bing Webmaster Tools
Did an assistant open a page for someone?As a floorThe server log: ChatGPT-User, Claude-User, Perplexity-User, DuckAssistBot
Was a link shown in AI Overviews or AI Mode?YesSearch Console’s generative AI performance report
Was a page cited in Copilot?As a sampleBing Webmaster Tools, AI Performance
Was a page cited in ChatGPT, Claude or Perplexity?No report exists—
Did a visit come from an AI answer?YesReferrals in an analytics tool
Why was a page cited, and did my change cause it?No—
How much will a tactic raise my citations?No—

As documented by Google, Bing, OpenAI, Anthropic, Perplexity and DuckDuckGo on October 22, 2026. “As a floor”: requests answered from a CDN’s cache never reach the server’s log.

The last two rows are guesswork for a reason that no tool can fix, and Bing’s documentation says so about its own report. Its trends “reflect aggregated citation activity only and cannot be attributed to a specific cause or event,” and its changes “do not indicate the impact of any single update, model change, or content modification.” On its new Citation Share metric, Bing answers the obvious question directly: it “can help you observe whether your share changes over time after content updates, but it does not establish a causal link.” Its Compare view “helps you observe patterns, not establish causation.” The only search engine that reports citations says, in writing, that its numbers cannot tell you whether your change caused them.

The reasons are the same ones the paper controlled away. The engine changes models. The query mix changes; Bing lists “user demand, content freshness, model updates, partner refresh cycles, and shifts in the topics users ask about” among the causes of change. Other pages change. A before-and-after comparison on one live site cannot hold those still, so it cannot separate your edit from everything else that moved in the same weeks.

Scores that combine these signals into one number inherit the problem, and add their own: they report on what they were able to check. I found the same gap in a public readiness checker, whose agent readiness score measured what a site declares, not what it does. An AI visibility score built from sampled prompts measures those prompts. Neither is useless, and neither is a measurement of how often real people see your page in real answers.

That leaves the promise. This site’s Terms close the list of what is not promised with one sentence: “Any offer that promises them is guessing.” Nothing in the paper, the documentation or my log changes that sentence. What can be offered honestly is the measurable part: pages that every crawler can fetch, that the indexes hold, and that say plainly what they know, measured before and after a change.

How to measure your own site, in order of what you control

1. Count verified fetches, by agent

Download each operator’s list of addresses from its crawler page, then split the requests that carry the name into those that come from the list and those that do not. This is the script I used to check my own count, run against a standard access log whose first field is the client address:

shell
$ curl -sL https://openai.com/chatgpt-user.json -o chatgpt-user.json
$ cat verify.py
import ipaddress, json, sys
log, agent, listfile = sys.argv[1:4]
nets = [ipaddress.ip_network(p.get('ipv4Prefix') or p.get('ipv6Prefix'))
        for p in json.load(open(listfile))['prefixes']]
on = off = 0
for line in open(log, errors='replace'):
    if agent not in line:
        continue
    ip = ipaddress.ip_address(line.split()[0].strip('"'))
    if any(ip in n for n in nets):
        on += 1
    else:
        off += 1
print(agent, 'from the published list:', on, '| from other addresses:', off)
$ python3 verify.py access.log ChatGPT-User chatgpt-user.json
ChatGPT-User from the published list: 30 | from other addresses: 101

The same script works for OAI-SearchBot with searchbot.json, for Anthropic’s agents with the list at claude.com/crawling/bots.json, and for Perplexity’s two lists, whose published addresses redirect, hence the -L. Count by address and by day, and treat any address that presents three or more crawler names as an impostor.

2. Check what each crawler is allowed, and what it receives

Read robots.txt crawler by crawler; the AI visibility checker does it for any domain and sorts each AI crawler by what the rules let it do, which is permission, not appearance. Then remember that a request with a borrowed User-Agent shows how your server and CDN treat that name, not what the real crawler receives, and that from a rented server the picture is distorted again, as I found when my free check turned out to run from an address some sites refuse outright.

3. Read the indexes

Use URL Inspection in Search Console for the pages that matter, and the index count in Bing Webmaster Tools. A page that is “unknown to Google” is not in the running for AI Overviews, whatever its content says.

4. Read the two reports, and export them

Open the generative AI performance report in Search Console and AI Performance in Bing Webmaster Tools. Export both regularly: Bing’s report offers ranges of 7 days, 30 days and 3 months, and custom ranges only “within available historical data.” On a small site, a blank report is expected, not a verdict.

5. Count AI referrals in analytics, if you run analytics

Visits from AI answers arrive with a referrer like any other. Count them by source, and keep in mind that consent banners and tracker blocking take some of them out of any count.

6. If you track prompts, publish the method

Run each prompt many times, report the share of answers that mention you rather than a position, and keep the prompts with the numbers. If a vendor gives you a score, ask for the same three things.

Generative engine optimization: frequently asked questions

What is generative engine optimization?

Generative engine optimization (GEO) is the practice of making content more likely to be used and cited in AI-generated answers. The term comes from a 2023 research paper by Aggarwal and co-authors, published at KDD 2024, and Bing’s webmaster guidelines now define it as focusing “on content eligibility for grounding and reference in AI responses.”

Is generative engine optimization different from SEO?

Less than the name suggests. Google says “The best practices for SEO remain relevant for AI features in Google Search” and that there are no special optimizations for AI Overviews or AI Mode. Bing says Copilot relies on the same crawling, indexing and ranking foundation as its search. What differs is the measurement: impressions in AI features and citations, instead of rankings and clicks.

What is the difference between AEO and GEO?

Both name the same aim: being the source an AI answer draws on. GEO comes from the 2023 paper and now appears in Bing’s documentation; answer engine optimization, AEO, is the name this site uses for its service. The difference in name changes nothing about what a site can measure: crawler access, indexing, assistant fetches, and the two search engines’ reports.

Can anyone guarantee that ChatGPT or Google AI Overviews will cite my site?

No. Google says indexing and serving “isn’t guaranteed,” and Bing says “GEO does not guarantee grounding or citations in AI experiences.” The operators document how to keep a site out of their AI answers, not how to put it in. A site can make its pages fetchable, indexed and clear, and measure that; whether an engine cites them is its decision.

How can I tell if ChatGPT used my content?

Look for ChatGPT-User in your server log and check each address against OpenAI’s published list: a verified request means ChatGPT opened the page while answering someone. It does not prove a citation, and a CDN cache hides some of them. OpenAI’s crawler documentation offers no citation report; visits that come from ChatGPT appear as referrals in analytics.

Do I need llms.txt or special schema to appear in AI Overviews?

Google says no: “You don’t need to create new machine readable files, AI text files, or markup to appear in these features,” and there is “no special schema.org structured data” to add. On this site, no request from an AI operator’s published addresses asked for llms.txt in 55 days of logs.

The general shape of this

The paper that named generative engine optimization measured something real, in the one place it could: inside an engine it controlled. There, a 2023 model gave more of its answer to sources that looked better supported, including support the model had been allowed to invent. What the experiment held still is everything a real site depends on: whether its pages are fetched, indexed and retrieved at all. Those are the measurable layers, and on this site they are where the numbers are: every verified crawler served, 40 of 42 pages in Google’s index, 11 in Bing’s, and 30 fetches by ChatGPT for its users.

Past that point, the honest instruments are two dashboards that count what was shown or cited and say, in their own documentation, that they cannot tell you why. Anyone who can show you a log line, an index status or a row in one of those reports is measuring. Anyone who tells you which change will make an engine cite you is guessing, and the authors of the paper that started the term call the engines black boxes in their first paragraph.

Sources