You rank on Google: so why isn’t AI search showing your website?

You rank on Google. Your competitors do too. But when a potential customer asks ChatGPT, Perplexity or Google’s AI features for a recommendation, your brand is nowhere to be found.

That does not necessarily mean your SEO is failing. Ranking in traditional search and being cited in an AI-generated answer are related, but they are not the same outcome. A page can rank well on Google and still be difficult for an AI system to access, understand or use as a source.

Three useful areas to investigate are access, clarity and the value of the information on the page. A system may be unable to retrieve or use your content, the page may not answer the question clearly, or other sources may offer more useful detail. These are starting points for an investigation, not an exhaustive explanation of how sources are selected. Google’s AI-feature guidance also stresses eligibility, accessible text and helpful, reliable content.

The answer in brief

• A Google ranking shows that your page is appearing for a particular search. AI search visibility concerns whether your business or content appears in an AI answer. One does not guarantee the other.
• Other providers use crawlers and access controls that differ from Google’s. A site can allow Googlebot while blocking OAI-SearchBot, for example. Search access and model-training access are separate decisions.
• AI answers can draw on information within relevant pages. Clear explanations, descriptive headings and nearby supporting evidence help readers understand what a page says and where its claims come from.
• Google does not require special AI markup or an llms.txt file for its AI search features. Accurate structured data still has uses in SEO, but neither markup nor a file guarantees an AI citation.
• First-hand experience, original data and specific examples can give a page useful information that generic competitors do not provide.

The key takeaway

A strong ranking is valuable, but it does not settle whether your page will be selected for a particular AI answer. Check the relevant platform’s access and eligibility requirements, then assess how well the page answers the customer’s question.

Ranking and citation are not the same job

Traditional ranking concerns which pages match a search query. An AI answer may draw on several sources to address a more detailed question. Google’s documentation explains that AI Overviews and AI Mode can use query fan-out: multiple related searches across subtopics and data sources. The features can also use different models and techniques, so the links they show can differ. An AI Overview does not appear for every query. Google explains how its AI features work here.

A page can rank well for a broad search term while another source provides a clearer, more specific answer to the question an AI system is addressing. Answer engine optimisation, or AEO, describes work intended to improve visibility in these answer-based search experiences.

The SEO work that earned the ranking still matters. Google’s generative AI optimisation guide confirms that its AI search features rely on its Search index and core ranking and quality systems. Other platforms have their own retrieval arrangements, so Google’s guidance should not be read as a specification for every AI product.

Reason one: the AI system cannot access or use your content

One of the most straightforward things to check has nothing to do with the quality of your writing. A robots.txt file, a content delivery network (CDN) or a web application firewall (WAF) may be blocking the crawler you want to reach the page.

For Google, check eligibility too. A page needs to be indexed and eligible to appear in Search with a snippet. Google’s Search generative AI control became available worldwide on 31 August 2026. In Search Console, go to Settings → Search generative AI and check the effective setting. Inclusion is the default, but a property can inherit an exclusion from a parent. The control covers AI Overviews, AI Mode and generative AI features in Discover. Google’s control documentation explains the scope.

Excluding your site affects its own links and content in those features; Google says the choice is not used as a ranking or inclusion signal elsewhere in Search. Review page-level controls as well: noindex prevents Search inclusion, nosnippet prevents content being used as direct input for AI Overviews and AI Mode, and max-snippet can limit the amount used. A crawl block can prevent Google discovering a noindex instruction. Google’s robots meta documentation explains these different effects.

Googlebot is not the only crawler to consider. The following names have different roles, and should not all be treated as interchangeable ‘AI bots’:

For ChatGPT, the important distinction is between GPTBot and OAI-SearchBot. GPTBot relates to potential model training, while OAI-SearchBot supports search. OpenAI documents independent settings: blocking GPTBot does not, by itself, prevent OAI-SearchBot from accessing a site. Anthropic similarly separates ClaudeBot from Claude-SearchBot and Claude-User. Its guidance says blocking the latter agents can reduce visibility. See the documentation from OpenAI, Anthropic and Perplexity.

Google-Extended is a separate content-use control. It does not determine Google Search inclusion or rankings, and it has no separate HTTP user-agent string. Use the Search Console control above to check participation in the covered Search AI features. Google’s crawler documentation explains the distinction. User-requested fetchers also have their own rules; do not assume that every provider handles them in the same way.

A site can allow Googlebot while restricting other crawlers, but a blanket rule blocking every user-agent containing ‘bot’ also matches Googlebot. Keeping Googlebot accessible would require an effective exception or a more specific rule. Its name does not exempt it.

Check the site’s robots.txt file and its hosting, CDN and firewall settings separately. An allowance in robots.txt cannot override a firewall block. Review logs for blocked requests, rate limits and challenge pages, and check that a successful response contains the useful page content. Follow the operator’s verification guidance: a user-agent name can be impersonated. Access makes retrieval possible; it does not guarantee indexing or citation.

Reason two: the answer is difficult to find

Even when a crawler can reach a page, the page may not answer the customer’s question clearly. A long introduction, vague headings or important qualifications scattered across several sections can make useful information harder for a reader to find. Before rewriting everything, check whether the page explains the service, who it suits and what a buyer needs to consider.

It is sensible to state the core answer plainly, then build out the supporting detail. Descriptive headings help readers navigate the page. Specific claims are easier to assess when the source, date and relevant limitations are close by.

None of this means forcing every section into short, robotic paragraphs. Give the explanation enough room to be accurate, and keep qualifications beside the claims they limit. Google says special chunking and an AI-specific writing style are unnecessary in its optimisation guide. The aim is a clear answer, not a fixed number of sentences.

Reason three: competing content is more distinctive and useful

Even a page that is accessible and clearly written may offer little beyond information already available elsewhere. Compare it with the sources being cited for the questions that matter to your customers. Do those sources explain a practical limitation, provide original data or answer a buying question that your page leaves open? Those are useful gaps to investigate, even though a comparison alone cannot reveal exactly why a system chose a source.

Google’s people-first content guidance asks publishers to consider original information, first-hand experience and clear sourcing. In practice, that means demonstrating expertise rather than simply describing it. A specialist’s explanation, a measured case study or a detailed example gives a reader something to assess. State what the evidence covers, how any results were measured and where the limits lie.

Schema and llms.txt are not the visibility switch

A specific piece of structured data, or a file such as llms.txt, should not be sold as a guaranteed route into AI Overviews or AI Mode.

Google says it ignores llms.txt for Search visibility and rankings, and there is no special schema.org markup required for its generative AI features. Other services may have their own uses for such files, so assess a proposed implementation against a documented purpose. Google’s optimisation guide sets out what is and is not needed.

Structured data still has a role in SEO. Google’s general guidance describes its use in understanding content and enabling eligible search features. However, Google stopped showing FAQ rich results on 7 May 2026, as confirmed in its documentation changelog. FAQPage remains a Schema.org type, but that does not establish that Google still uses this particular markup for comprehension. Keep useful questions and answers for readers, without promising a citation benefit from their markup.

So, what should businesses actually do?

Start with the pages and customer questions that matter to your business. Three actions provide a practical way to organise the work.

1. Protect your technical accessibility. Check indexing, participation and preview controls alongside robots.txt, CDN and WAF rules. Confirm that your intended search access is available without assuming training access must also be allowed.
2. Make your expertise easy to understand. Answer important questions directly, use descriptive headings and make claims clear and attributable. Keep essential qualifications with the answer.
3. Build content worth citing. Add first-hand experience, original findings, expert explanation and specific evidence where they help a customer make a decision. For local or retail businesses, keep relevant Business Profile or Merchant Center details consistent with the website.

Then measure what changes. Record the questions tested, the platform, date and relevant location, and whether web search was used. Separate a mention of your business from a citation linking to your website, and track visits and enquiries alongside both. Repeat the checks rather than treating one answer as a definitive result.

Google’s Generative AI performance report for Search rolled out worldwide on 31 August 2026. It provides impression data for links shown in AI Overviews and AI Mode; the documented report does not provide a separate click or query breakdown. Low impression volume can mean the report is unavailable. These data are already included in overall Web performance, so do not add them to those totals. Google’s report documentation explains how to read it.

As checked on 6 September 2026, Google’s latest named ranking update was the August spam update, completed on 21 August; the latest named core update was completed on 2 June. These are separate from the new Search Console tools. Google’s spam policies also cover manipulation of generative AI responses, so publishing near-duplicate pages purely to target variations of a question is not a sound strategy. Ranking update history, spam policies.

A practical audit checklist

Use this checklist before committing to a wider rewrite. It will help you separate confirmed technical problems from content improvements worth considering.

• Check eligibility and crawler rules. Confirm the important pages’ Google indexing and snippet eligibility, then review the effective Search generative AI setting. Inspect robots.txt for rules affecting the search crawlers relevant to your goals. Keep training preferences separate.
• Check the CDN or WAF separately. Review actual requests and responses, including blocks, rate limits and challenges. A robots.txt allowance does not prove that a crawler can retrieve the page.
• Read the page as a prospective customer. Can you find the answer, understand its limits and reach supporting detail? Improve unclear wording without imposing an arbitrary sentence count.
• Check whether the page says anything a dozen competitors do not. Original data, first-hand experience and clearly sourced explanations can make it more useful. Compare the answers and evidence, not just the keywords.
• Review markup and measurement. Make structured data match the visible content, without relying on FAQPage or llms.txt to secure citations. Record mentions, links, visits and enquiries as different outcomes.

Common mistakes to avoid

Assuming a ranking problem and an AI visibility problem have the same cause. They may not. A site can allow Googlebot while restricting another provider’s crawler. Google’s own AI features also have eligibility and participation controls to check.

Chasing a schema or llms.txt fix before checking access and eligibility. These additions will not remove a firewall block or change an exclusion setting.

Rewriting content to sound ‘AI-friendly’ instead of making it more useful. Short, choppy paragraphs and forced question-and-answer blocks are not a documented universal route to citations. Clear, specific, well-supported answers are a better editorial priority.