Guide

Before you write a single AEO page: the 20 minute check that tells you if ChatGPT, Claude, and Google AI are even allowed to cite you

For owners and founders running a site with no dedicated SEO team. All vendor documentation below was verified on August 16, 2026.

Published 2026-08-16 · CitationLoop Research

A page cannot be used reliably as a fresh source in a search-based AI answer if the relevant search system cannot access or index it. Those access failures are often accidental. Before you rewrite a single paragraph for "AI readability," check four things: whether your robots.txt allows the search-specific crawlers you want to reach you (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot), whether your firewall is quietly returning 403s or 429s to those crawlers, whether your pages carry a nosnippet or max-snippet directive that limits eligibility in Google AI Overviews and AI Mode, and whether the important information on your pages is reliably available as textual HTML. This takes about twenty minutes. It is the technical eligibility work to do before content-level AEO optimization.

Google states this directly: to be eligible as a supporting link in AI Overviews or AI Mode, a page must be indexed and eligible to appear in Search with a snippet, and there are no additional technical requirements beyond that (Google Search Central, AI features and your website).

The four engines this covers are ChatGPT, Claude, Perplexity, and Google's AI Overviews and AI Mode. Each one reaches your site differently, and the differences are the whole point of this article.

This is a technical eligibility check, not a visibility strategy. Passing it does not get you cited. Failing a check can make the affected page ineligible or materially harder to retrieve in the engine that depends on that access path. If you want the strategy layer, the next step is deciding which buyer questions and source gaps are worth targeting after you finish this audit.

Who this is for

Owners and founders with a site under roughly 500 pages, no dedicated SEO or infrastructure person, and a security or hosting setup they did not personally configure. If you run an enterprise site with a CDN team and a crawl budget dashboard, you already have this covered.

Why this comes first

Almost every AEO article starts with content tactics: answer up front, add statistics, structure your headings. That advice assumes the engine can read your page. Frequently it cannot, for reasons the owner never chose.

The pattern is the same in every case. Sometime in 2023 or 2024, a developer, an agency, or a security vendor added a rule to block "AI bots" from scraping the site for training. Reasonable at the time. But the AI companies have since split their crawlers into separate agents with separate jobs, and a blanket block now removes you from the search products too.

The scale of that split is visible in public data. Cloudflare's AI Insights dashboard on Radar breaks AI crawler traffic down by purpose, training versus search, and shows the distribution of HTTP status codes that sites return to those crawlers. A large share of 4xx responses in that data is the industry-wide version of the problem this article is about.

How each engine actually accesses your site

For the shorthand version of this section, jump straight to OAI-SearchBot vs. GPTBot, Claude-SearchBot, or Google-Extended below.

The audit, in order

Work through these in sequence. Fix anything that fails before moving on. If you would rather have this run for you, the free CitationLoop scan performs steps 1 through 4 automatically and returns the results by email.

Read your own robots.txt

Open yourdomain.com/robots.txt. Look for any line mentioning GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Google-Extended, or CCBot. A blanket Disallow: / under any of the search-specific agents is your problem. Cloudflare maintains a plain-language reference of current AI crawler user agents if you find a string you do not recognize.

A common configuration for a business that wants AI search visibility but not training use:

Whether you block the training bots is a business decision, not a visibility one. Some owners allow everything on the theory that being in the training data helps them be described accurately. Both positions are defensible. Blocking the search bots by accident is not.

Check that your firewall agrees with your robots.txt

This is the failure most owners never find, because robots.txt looks perfect while requests are rejected one layer up. If you use Cloudflare, Sucuri, Wordfence, or any managed WAF, look for bot-blocking or "AI scraper" rules and confirm the search agents are not being challenged, rate limited, or served a 403. A crawler that receives an interstitial, a CAPTCHA, or a 429 cannot treat your page as a clean source, and it has no way to tell you. Check your access logs: if you see the agents above but every response is a 403 or 429, robots.txt is not your problem. Cloudflare's public AI Insights dashboard on Radar shows ecosystem-level crawler activity. If you use Cloudflare, inspect your own site's bot and security analytics, including AI Crawl Control where available, to see whether AI crawlers are being allowed, challenged, rate limited, or blocked.

Look for snippet suppression

View source on your most important page and search for nosnippet, max-snippet, and data-nosnippet. Also check your X-Robots-Tag HTTP headers, since these can be set server-side and never appear in your HTML. If max-snippet:0 or nosnippet is on pages you want in AI answers, remove it, then confirm the change with the URL Inspection tool rather than waiting and guessing. Google notes that recrawling after a preview-control change can take anywhere from days to months.

Confirm your important content is reliably available as text

Disable JavaScript and reload a key page as a quick diagnostic. If your headings, body copy, prices, service areas, or other critical facts disappear, you have created an extra rendering dependency. Google can render JavaScript, but not every retrieval system documents equivalent rendering behavior, and even Google recommends making important content available in textual form. Its JavaScript SEO basics guide covers common implementation issues. For information you want search and answer systems to retrieve reliably, prefer server-rendered or otherwise crawlable textual HTML rather than requiring client-side JavaScript to reveal the core answer.

Set up measurement before you change anything else

OpenAI's publisher FAQ states that ChatGPT automatically includes utm_source=chatgpt.com in referral URLs, which makes that traffic identifiable in analytics instead of being dumped into direct (Publishers and Developers FAQ). On the Google side, AI Overviews and AI Mode impressions and clicks are reported inside the normal Search Console Performance report under the Web search type. Finally, record on a fixed date how many of your target buyer questions currently name you at all. Without a before, there is no after — see how to check if your business appears in AI search for exactly how to run that baseline and why a single check isn't enough.

What this audit will not do

It will not get you cited. Eligibility is a floor, not a lever. Being blocked, being retrieved but not cited, and being cited but not converting are three separate problems with three separate fixes — see how to check if your business appears in AI search for how to tell retrieved-but-not-cited apart from never-retrieved-at-all.

Once a crawler can reach you, the question becomes whether your page is the most relevant answer to the specific question asked, and whether it gets retrieved at all against competitors. Two controlled studies are worth knowing about here.

C-SEO Bench (Puerto et al., NeurIPS 2025 Datasets and Benchmarks Track, arXiv:2506.11097, code) tested nine conversational-SEO methods across six domains and, importantly, under competition where multiple documents adopt the same tactic. It compared them against a plain retrieval baseline of moving the source higher in the model's context, and found that position advantage held up better than most content rewrites.

"What Gets Cited: Competitive GEO in AI Answer Engines" (Vishwakarma, Kumar, and Jamidar, SIGIR 2026, arXiv:2605.25517, DOI 10.1145/3805712.3808445) ran 252,000 paired trials across six models, varying exactly one of eighteen content factors at a time with brands anonymized and source order counterbalanced. Topical relevance and list position came out as the primary determinants of which source got cited first. The authors' practical conclusion is that formatting changes showed minimal return and can be deprioritized once the relevance baseline is met.

Both are controlled RAG testbeds, not live production pipelines, so do not read them as universal ranking rules. The earlier and more widely quoted GEO paper (Aggarwal et al., KDD 2024) is where the "add statistics and citations" advice originates, and it is worth reading alongside the two above rather than instead of them. All three point the same direction as Google's plainer statement that ordinary SEO fundamentals still apply and no special AI markup is needed.

So the order of operations is: be reachable, then be relevant, then be better evidenced than the alternative, then measure whether any of it changed the answers. This audit is step one only.

Frequently asked

If you would rather not do this by hand

The checks above are free and manual, and doing them yourself is the fastest path if you have twenty minutes and access to your hosting panel.

What manual checking does not give you is the second half of the picture: whether the engines currently name your business for the questions your buyers actually ask, which competitors they name instead, and which third-party sources those answers keep drawing from. That requires running a fixed prompt set repeatedly across engines and recording the results, because a single run of a single prompt is one sample from a nondeterministic system, not a ranking — see how to check if your business appears in AI search for exactly how we run that and what we count.

CitationLoop runs the crawler and snippet checks above automatically and pairs them with that prompt-level baseline, so you can tell whether an access problem or a relevance problem is keeping you out of the answer.