LLM SEO: The Practical Guide

Published

· Updated

Cover image for a Zaprev practical guide to LLM SEO and the AI retrieval pipeline

LLM SEO is the work of getting your pages retrieved, read and cited when a language model answers a question. Six stages sit between the question and the citation, and a page can drop out at any of them. The most common silent failure is crawler access: blocking OAI-SearchBot removes you from ChatGPT search answers entirely.

What LLM SEO means, and the naming problem

The same work goes by at least five names. LLM SEO, GEO, AEO, LLMO, AI search optimization. There is no agreed distinction, and vendors pick whichever term suits their positioning. We use LLM SEO here for the technical, mechanism-first version of the work, and cover the strategic framing in GEO vs SEO.

Do not let the vocabulary decide your evaluation of a provider. A team that says GEO and can show you a documented prompt set with citation share over time is worth more than one that says LLM SEO and shows you a screenshot.

The one thing worth being precise about: this is not model training. You are not trying to get into the weights. You are trying to be retrieved at answer time. That distinction determines almost every technical decision below.

The pipeline, stage by stage

Diagram of the six stages between a question and an AI citation: expansion, retrieval, shortlist, fetching, citation and synthesis.

When a user asks a question with a retrieval-backed assistant, roughly this happens -

#

Stage

What happens

How you fail it

1

Query expansion

The question is split into multiple sub-questions run in parallel, a technique Google named query fan-out publicly at I/O 2025

Your content covers the head term but not the surrounding cluster of sub-questions

2

Retrieval

Candidate passages are pulled per sub-question from an index and live fetches

You are not in the index, or your content is not in the HTML

3

Shortlisting

Candidates are filtered on relevance, authority, recency, and diversity of perspective

Stale page, weak entity, or a title and description that do not signal a match

4

Fetching

Survivors are fetched and parsed in full

Slow response, redirect chains, blocked user agent, JavaScript-dependent content

5

Selective citation

Sources are attached to the specific claims they support

Nothing on your page is a clean, attributable claim

6

Synthesis

The answer is composed, conflicts resolved towards fresher and more authoritative sources

A more current or better-corroborated source wins the contested point

Two things follow from this. First, retrieval is passage-level, not page-level, so the unit you are optimising is a paragraph. Second, being retrieved is not being cited. Most candidates are rejected before the answer is written, which is why several visibility tools now expose a “found but not cited” view. That view is the single most useful diagnostic in the category.

Diagnosing which stage you fail

Work through this before changing anything.

Symptom

Likely stage

First check

Never appears for any prompt in the category

2 or 4

Server logs for retrieval bots, robots.txt, CDN rules

Appears in Perplexity but never ChatGPT

4

OAI-SearchBot specifically, per engine

Retrieved but not cited

5

Answer placement and evidence density on the page

Cited for brand queries, absent for category queries

1

Sub-question coverage, topical depth

Was cited, then stopped

3 or 6

Page age since substantive update, competitor freshness

Ranks first on Google, absent from AI Overviews

5

Where the answer sits on the page

Layer 1: crawler access

This is where the quiet failures live.

Three families of bot reach your site and they do completely different jobs.

Family

User agents

What it does

Blocking it means

Training

GPTBot, ClaudeBot, Google-Extended, CCBot, meta-externalagent, Applebot-Extended, Bytespider

Bulk collection for model training

Your content is not used in training. No effect on live citations

Retrieval and search indexing

OAI-SearchBot, Claude-SearchBot, PerplexityBot

Builds and refreshes the index behind AI search

You become ineligible for citations on that engine

User-initiated fetch

ChatGPT-User, Claude-User, Perplexity-User

Fetches a page because a user asked the assistant to read it

The assistant cannot read pages a user points it at

The two columns that matter are the middle and the right. OpenAI’s own documentation tells publishers that sites blocking OAI-SearchBot will not appear in ChatGPT search answers. Blocking training crawlers costs you nothing in citations. Blocking retrieval crawlers costs you everything.

Three practical points that catch teams out:

  • Google-Extended is not Googlebot. Disallowing Google-Extended opts your content out of Gemini training and grounding. It does not affect how Googlebot crawls, indexes, or ranks you, and it does not remove you from AI Overviews, which run on the core Search index.

  • Check the CDN, not just robots.txt. Cloudflare and similar layers now ship AI-bot blocking controls, and several hosting platforms and CMS templates block AI crawlers by default. Your robots.txt can say allow while your edge says no.

  • Verify with logs, not intentions. Filter server logs by user agent and confirm the retrieval bots are actually arriving and receiving 200s.

A reasonable default for a brand that wants AI visibility is to allow everything, keep the option to disallow training crawlers as an editorial decision, and hold a server-level rule in reserve if a bot misbehaves. Our robots.txt generator covers the AI directives.

Note that robots.txt is a request, not enforcement. It was standardized as RFC 9309, and compliance is voluntary. Real blocking happens at the server or CDN.

Layer 2: rendering

If your content is injected by client-side JavaScript, assume it is not there.

Googlebot renders JavaScript, eventually, and Google’s AI surfaces run on that index, so you have some protection there. The retrieval crawlers behind ChatGPT, Claude, and Perplexity are far less forgiving, and a live fetch at answer time has a very short patience budget. One redirect hop too many and the page is dropped from the answer.

Three checks:

  1. Fetch your key pages with JavaScript disabled and confirm the substantive content is in the raw HTML.

  2. Keep redirect chains to a single hop. Retrieval bots tolerate roughly three to five, and every hop is latency you may not have.

  3. Watch time to first byte on the pages you most want cited.

Layer 3: passage structure

Now the content work starts, and it is narrower than most guides suggest.

The goal is one thing: a self-contained paragraph that answers a specific sub-question and survives being lifted out of the page without losing its meaning.

  • Open every section with a direct answer of roughly 40 to 60 words, then supply the detail. An analysis of 100 Google AI Overview citations found 55% came from the first 30% of the source page.

  • Do not use pronouns that reach backwards across headings. A passage beginning “This means that” is unusable in isolation.

  • Use tables for comparisons. They extract cleanly and carry relationships that prose loses.

  • Match headings to real questions, in the language buyers actually use.

What you do not need, per Google’s May 2026 guidance: content chunked into tiny pieces, a machine-flavoured writing style, prompt-shaped headings, or any particular page length. Google states its systems handle multiple topics on one page and that there is no ideal length. Write for readers, and put the useful part where it can be found.

Layer 4: evidence and citability

A retrieval system attaches sources to claims it can support. Give it supportable claims.

The founding research on this, the GEO paper presented at ACM SIGKDD in 2024 by Aggarwal and colleagues, tested content interventions across roughly 10,000 queries. The three strongest were adding citations to credible sources, adding relevant quotations, and adding statistics, each worth roughly 25% to 40% on the paper’s visibility metric. Keyword stuffing did nothing.

The paper also found an equalizer effect: lower-ranked pages gained the most, with the citation method lifting visibility by over 100% for pages ranked around fifth. Evidence density partially substitutes for domain authority, which is the most useful finding in the literature for a challenger brand.

One correction is worth carrying. A later benchmark, C-SEO Bench, tested conversational SEO tactics systematically and found most do not help and several hurt, while plain source relevance keeps working. The safe reading: interventions that also make the page better for a human reader are the ones that hold up. Anything that only makes sense as a trick to a machine probably is one.

In practice this means specific figures rather than adjectives, named and dated sources, and original data or genuine practitioner detail wherever you have it.

Layer 5: brand mentions off-site

The strongest off-page correlate is not links. Ahrefs, across 75,000 brands in 2026, found branded web mentions correlated with AI visibility at 0.664, roughly three times the strength of backlinks.

Corroboration is what a retrieval system is implicitly testing. A brand named consistently across review platforms, industry press, community threads, and comparison content looks like a real entity. One that exists only on its own domain does not.

Practically: consistent naming of your brand, products, and people everywhere; presence in the places your category is genuinely discussed; and a realistic timeline. This accrues over years. Nobody moves it in a quarter, and providers who promise to are describing something else.

Layer 6: freshness

Retrieval systems weight recency more heavily than classic ranking does. Ahrefs analysed 17 million citations and found AI-cited URLs were 25.7% fresher on average than URLs ranking organically. AirOps reported that on commercial and evaluation queries, 83% of AI citations came from pages updated within twelve months, with more than 60% inside six.

Set a cadence on your commercially important pages and make substantive changes. Updating the date is not updating the page.

What to measure

Rank tracking does not transfer, because generative answers are not stable results. Citation rates for the same fixed prompt set can move 40% to 60% month over month without any change on your side.

So measure by sampling:

  1. Build a prompt set. Twenty to fifty real buyer questions across the funnel.

  2. Run it per engine, on a schedule. Coverage differs sharply between engines and one blended number hides that.

  3. Track citation share and share of voice against named competitors.

  4. Use first-party data where it exists. Search Console added generative AI performance reporting in June 2026 for Google surfaces. Bing gives site owners useful visibility data too.

  5. Segment AI referrals in analytics. They are routinely misattributed as direct. Volume is small, typically around 1% of site traffic, and the conversion rate is usually several times organic, because the visitor arrives after the comparison rather than before it.

Tooling for this is covered in best AI SEO tools 2026. For a first read on where you stand, the free AI visibility checker will give you a baseline.

Important notes

  • This article reflects the position as of September 2026. Crawler names, provider behaviour, and platform reporting change frequently, so verify user agent strings against each provider’s current documentation before editing robots.txt.

  • Research findings cited here come from published benchmarks and observational studies. They describe correlations and controlled test results rather than guaranteed outcomes.

  • Statements about Google’s systems come from Google Search Central and named Google staff, and describe Google’s AI surfaces only.

Read more from the team: browse all our posts.

Frequently asked questions

What is LLM SEO?

The practice of making your pages retrievable, readable, and citable by language models answering user questions. It sits on top of technical SEO rather than replacing it.

How long does LLM SEO take?

Access and structure fixes can show up within weeks. Authority and brand-mention effects move over quarters and years. Expect the fast wins on page structure and the slow grind on entity strength.

Why does my page get retrieved but never cited?

Usually because there is no clean, self-contained, attributable passage on it. The answer is buried, or the claims carry no evidence a model can stand behind.

Do I need llms.txt for LLM SEO?

No. Google says it does not use it, and crawler log studies show AI bots rarely request the file. It has real value for developer documentation consumed by coding agents. See our llms.txt guide.

See more

Want this run on your site?

We do the sitewide version, with the fixes prioritised and shipped.

Zaprev

Zaprev

Zaprev