How to Check if ChatGPT and AI Search Can See Your Website

If ChatGPT search, Perplexity, Claude, or other AI assistants never cite your pages, the first place to look is access—not content quality. AI products use different crawlers (and sometimes user-triggered fetchers) with separate robots.txt rules. Blocking a training bot is not the same as blocking a search bot. This guide walks through the official agents, how to test them, and what else can hide your site from AI systems.

Scope note: This article is about crawl and fetch access. It does not claim that allowing a bot guarantees citations. For a free, no-signup check of SEO signals, AI-crawler/robots.txt readiness, optional competitor comparison (up to three, per the live site), and a prioritized fix list, you can run SiteScope (limited free reports per day). Some reports may also include an experimental AI-knowledge check: it runs on Cloudflare Workers AI (Llama 3.1 8B), uses the model’s learned knowledge—not a live ChatGPT, Perplexity, or AI-search crawl—and does not verify brand relevance or show what those products actually answer.

Training bots vs search bots vs user fetchers

Treat each product’s agents as independent switches.

OpenAI (ChatGPT)

OpenAI’s crawler overview states that OAI-SearchBot and GPTBot are controlled separately in robots.txt:

• OAI-SearchBot — used to surface websites in ChatGPT search. Sites opted out “will not be shown in ChatGPT search answers, though can still appear as navigational links.” OpenAI recommends allowing OAI-SearchBot (and its published IP ranges) if you want search visibility. Changes can take about 24 hours to apply. (OpenAI: Overview of OpenAI Crawlers)
• GPTBot — crawls content that may be used in training generative foundation models. Disallowing GPTBot signals that content should not be used for that training. (same source)
• ChatGPT-User — used for certain user-initiated actions in ChatGPT and Custom GPTs. It is “not used for crawling the web in an automatic fashion.” OpenAI states that because these actions are user-initiated, “robots.txt rules may not apply,” and that ChatGPT-User is “not used to determine whether content may appear in Search”—use OAI-SearchBot for Search opt-outs. (same source)

Many sites allow OAI-SearchBot, set GPTBot by training preference, and do not rely on robots.txt alone against ChatGPT-User.

Perplexity

Perplexity documents two agents (Perplexity Crawlers):

• PerplexityBot — surfaces and links sites in Perplexity search results; “not used to crawl content for AI foundation models.” Allow it in robots.txt (and published IPs) if you want search visibility. Changes may take up to 24 hours.
• Perplexity-User — supports user actions; may visit a page to help answer a question and link it. “Since a user requested the fetch, this fetcher generally ignores robots.txt rules.”

If a WAF blocks them, Perplexity documents allowlisting by User-Agent and published IPs.

Anthropic (Claude)

Anthropic’s help center describes three robots (Does Anthropic crawl data from the web…):

• ClaudeBot — may contribute to generative model training; restricting it signals future materials should be excluded from training datasets.
• Claude-User — user-initiated retrieval when people ask Claude questions.
• Claude-SearchBot — crawls to improve search result quality for users.

Anthropic states its bots honor standard robots.txt directives and supports a non-standard Crawl-delay. Example site-wide block:

User-agent: ClaudeBot Disallow: /

Google-Extended (Gemini / Vertex—not Search ranking)

Google documents Google-Extended as a robots.txt control token (not a separate HTTP User-Agent string). It lets publishers manage whether crawled content may be used for training future Gemini models that power Gemini Apps and Vertex AI API for Gemini, and for grounding in those products. Google states explicitly: Google-Extended does not impact a site’s inclusion in Google Search and is not used as a ranking signal in Google Search. (Google’s common crawlers)

So: disallowing Google-Extended is a Gemini/training/grounding preference—not a Google Search opt-out.

Example robots.txt groups (copy carefully)

Place robots.txt at the site root for that host (for example https://example.com/robots.txt). Google documents that subdirectory robots files are not valid for crawlers. (Google robots.txt specification)

# ChatGPT search on; training off (example—choose your own policy)
User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

# Perplexity search
User-agent: PerplexityBot
Allow: /

# Anthropic training crawler (example block)
User-agent: ClaudeBot
Disallow: /

# Gemini Apps / Vertex grounding & training control (does not affect Google Search ranking)
User-agent: Google-Extended
Disallow: /

Re-check each vendor’s docs and IP JSON before deploying—strings and ranges change.

How to test whether AI bots can see you

1. Read your live robots.txt

Open https://yourdomain.com/robots.txt and confirm the groups match your intent. User-agent: * can interact with more specific groups—see Google’s intro if unsure. (Intro to robots.txt)

2. Curl with the bot’s User-Agent

Simulate a fetch (status codes and body length matter more than “looks fine in Chrome”):

curl -sI -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot" \
  "https://yourdomain.com/"

curl -sI -A "Mozilla/5.0 (compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)" \
  "https://yourdomain.com/"

Use vendor example strings (versions change). A WAF 403/challenge means the bot never sees your HTML—even if robots.txt allows it.

3. Check server or CDN logs

Filter logs for those bot names and compare status codes. Vendors publish IP JSON files linked from their crawler docs—use them when verifying authenticity.

4. Review Cloudflare (or other) AI bot policies

If you use Cloudflare, AI Crawler and Agent access policies let you set actions for Search, Agent, and Training behaviors (Block, Allow, and for Training a Disallow via robots.txt). Cloudflare notes that robots.txt directives are voluntary and crawlers can ignore them; with Bot Preference Sync, Disallow can also block AI training crawlers at the edge. (Cloudflare: AI Crawler and Agent access policies)

A global AI-bot block can remove you from AI search even when robots.txt looks open.

Beyond robots.txt: noindex, JavaScript, schema, llms.txt

noindex and X-Robots-Tag

Google’s robots meta documentation: <meta name="robots" content="noindex"> (in the HTML head) or an X-Robots-Tag: noindex HTTP header tells crawlers not to show the page in search results. Rules can also target specific tokens. (Robots meta tag specifications)

If a page is noindex, fixing AI bots in robots.txt will not make that URL a normal search result.

JavaScript rendering

If critical copy only appears after client-side rendering, test the raw HTML with curl and ensure important text is present before relying on a browser engine. Pair that with sound structured data.

Structured data (schema.org)

JSON-LD (FAQPage, Organization, Product, etc.) helps systems understand entities. Follow schema.org and Google’s structured-data intro and policies. Structured data does not guarantee AI citations or Google FAQ rich results (eligibility is restricted). Markup does not replace allowing the right crawlers.

llms.txt — proposed, not confirmed by major engines

llmstxt.org proposes a /llms.txt Markdown file (and optional .md alternates) so agents can find concise, LLM-friendly entry points. It is explicitly a proposal. Some documentation platforms and sites publish llms.txt, and tooling may audit for its presence—but no major AI search engine has confirmed that it uses llms.txt as a ranking or inclusion signal. Treat it as an optional convenience for agents, not a replacement for robots.txt, crawl access, or clear on-page content.

FAQ

If I block GPTBot, will I disappear from ChatGPT search?
Not according to OpenAI’s split: Search visibility is tied to OAI-SearchBot; GPTBot is the training-oriented crawl. (OpenAI crawlers)

Does Google-Extended hide me from Google Search?
No. Google states it does not impact Search inclusion and is not a ranking signal. (Google common crawlers)

Why can Perplexity still fetch a page I disallowed?
Perplexity-User generally ignores robots.txt because the fetch is user-requested. Use WAF/IP controls if you need a hard block. (Perplexity Crawlers)

Soft next step

After you fix robots.txt and WAF rules, re-test with curl and logs. For a free, no-signup pass focused on SEO checks, AI-crawler / robots readiness, and a prioritized fix list (plus optional comparison against up to three competitors)—with a few free reports a day—run your URL at https://sitescope.pages.dev. An experimental AI-knowledge check may appear on some reports; it is not guaranteed and is not a live ChatGPT or Perplexity answer readout.

Sources

• OpenAI — Overview of OpenAI Crawlers (HTTP 200)
• Perplexity — Perplexity Crawlers (HTTP 200)
• Anthropic Help Center — Does Anthropic crawl data from the web… (HTTP 200)
• Google — Google’s common crawlers (Google-Extended; HTTP 200)
• Google Search Central — robots.txt specification, Intro to robots.txt, Robots meta tags (HTTP 200)
• Cloudflare — AI Crawler and Agent access policies (HTTP 200)
• llmstxt.org (HTTP 200); schema.org (HTTP 200)
• SiteScope product page — sitescope.pages.dev (HTTP 200; 18+ SEO checks and up to 3 competitors verified on the live page)