robots.txt Rules for AI Crawlers (Examples)

AI products rarely share one crawler. Training crawls, search indexing, and user-triggered fetches often use different User-agent tokens—and sometimes different rules about whether robots.txt applies at all. Below: copy-paste groups for agents verified in official vendor docs, how named groups override *, testing tips, and Cloudflare’s managed robots.txt / AI bot policies.

Cloudflare documents that robots.txt directives are voluntary—crawlers can ignore them. Edge blocking is a separate control. (AI Crawler and Agent access policies)

How groups work (and why * is not enough)

Put the file at the host root (for example https://example.com/robots.txt). A robots file in a subdirectory is not valid for crawlers. (Google robots.txt specification)

Google’s matching rules:

• Only one group applies to a crawler: the most specific matching User-agent.
• Specific groups and * are not combined. If User-agent: GPTBot exists, GPTBot uses that group alone.
• Group order in the file does not change which group wins.

(Order of precedence)

Name each AI token you care about. Robots.txt mainly manages crawl traffic; it is not a reliable way to hide a URL from Google Search (use noindex or access control). (Intro to robots.txt)

Verified AI-related tokens (vendor docs only)

Official docs linked in Sources.

Example 1 — Allow everyone (default-open)

User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

This does not force user-fetch agents (ChatGPT-User, Perplexity-User, Meta-ExternalFetcher) to obey the file when vendors say they may bypass it.

Example 2 — Block training, allow search

# --- OpenAI: search on, training off ---
User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

# ChatGPT-User may ignore robots.txt; do not rely on this alone
User-agent: ChatGPT-User
Disallow: /

# --- Perplexity: search on ---
User-agent: PerplexityBot
Allow: /

# Perplexity-User generally ignores robots.txt
User-agent: Perplexity-User
Disallow: /

# --- Anthropic: search/user preference vs training ---
User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: ClaudeBot
Disallow: /

# --- Google Gemini training/grounding control (not Google Search) ---
User-agent: Google-Extended
Disallow: /

# --- Apple foundation-model training opt-out (does not stop Applebot search crawl) ---
User-agent: Applebot-Extended
Disallow: /

# --- Open corpus / Meta training-or-indexing crawl ---
User-agent: CCBot
Disallow: /

User-agent: meta-externalagent
Disallow: /

User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

• Google-Extended does not change Google Search inclusion or ranking. (Google common crawlers)
• Applebot-Extended does not crawl; it controls training use of Applebot-collected data. (About Applebot)
• Meta-ExternalFetcher may bypass robots.txt; allow up to 24 hours for robots changes. (Meta web crawlers)

Example 3 — Block named AI tokens

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Disallow: /

User-agent: ChatGPT-User
Disallow: /

User-agent: PerplexityBot
Disallow: /

User-agent: Perplexity-User
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Claude-SearchBot
Disallow: /

User-agent: Claude-User
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: meta-externalagent
Disallow: /

# Keep Google Search / classic search bots separate unless you intend otherwise
User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

Disallowing OAI-SearchBot or PerplexityBot is the Search opt-out those vendors document; GPTBot alone is not a ChatGPT Search opt-out. (OpenAI, Perplexity)

Testing

1. Fetch https://example.com/robots.txt with curl.
2. Request a page with a bot User-Agent string from vendor docs (spot-checks access, not a full robots parser).
3. Verify published IP ranges (OpenAI, Perplexity, Anthropic; Apple reverse DNS). Common Crawl warns about fake CCBot strings. (CCBot)
4. Check CDN/WAF AI policies—they can block even when robots.txt allows.

Cloudflare managed robots.txt and AI policies

Cloudflare can inject managed Disallow rules for known AI training crawlers (documented examples include GPTBot, Google-Extended, ClaudeBot, Applebot-Extended, CCBot, meta-externalagent) and can create a robots file if you lack one. (Managed robots.txt)

AI Crawler and Agent access policies set Search / Agent / Training actions. Robots directives remain voluntary; with Bot Preference Sync, Training Disallow also blocks matching training crawlers at the edge (mixed-purpose Search+Training bots are excluded from that edge block). (AI policies)

FAQ

Does a named User-agent group override *?
Yes, for crawlers that follow Google’s / REP-style matching: the most specific matching group applies; it is not merged with *. (Google robots.txt specification)

Is robots.txt enough to stop ChatGPT or Perplexity from fetching a page?
Not always. OpenAI says ChatGPT-User may not apply robots rules; Perplexity says Perplexity-User generally ignores them. Use WAF/IP controls for hard blocks. (OpenAI, Perplexity)

Will blocking Google-Extended hurt Google Search rankings?
Google says no—Google-Extended does not impact Search inclusion and is not a ranking signal. (Google common crawlers)

Soft next step

After you publish robots groups, re-fetch /robots.txt and spot-check logs. For a free, no-signup report focused on SEO checks, AI-crawler / robots readiness, and a prioritized fix list (plus optional comparison against up to three competitors), run your URL at https://sitescope.pages.dev. An experimental AI-knowledge check may appear on some reports; it is not guaranteed and is not a live ChatGPT or Perplexity answer readout.

Sources

• OpenAI — Overview of OpenAI Crawlers (HTTP 200)
• Perplexity — Perplexity Crawlers (HTTP 200)
• Anthropic — Does Anthropic crawl data from the web… (HTTP 200)
• Google — Google’s common crawlers (Google-Extended; HTTP 200)
• Google Search Central — robots.txt specification, Intro to robots.txt (HTTP 200)
• Apple Support — About Applebot (Applebot-Extended; HTTP 200)
• Common Crawl — CCBot (HTTP 200)
• Meta — Meta Web Crawlers (meta-externalagent; HTTP 200)
• Cloudflare — Managed robots.txt, AI Crawler and Agent access policies (HTTP 200)
• SiteScope — sitescope.pages.dev (HTTP 200; 18+ checks and up to 3 competitors verified on live page)