AI crawlers and robots.txt: which AI bots to allow and which to block

For most businesses the answer is: allow the AI search crawlers, keep classic search crawlers open, and make a separate, deliberate decision about training crawlers. Search crawlers are what let ChatGPT, Claude and Perplexity cite and link to your pages. OpenAI says sites that opt out of OAI-SearchBot will not be shown in ChatGPT search answers^[1], and Anthropic says blocking Claude-SearchBot reduces your visibility in its search results^[2].

Training crawlers are different. OpenAI describes disallowing GPTBot as a signal that your content should not be used for training, and states that each of its settings is independent of the others^[1]. So a publisher can say no to training and yes to being found. Whether to allow training is a content-licensing question, not a visibility tactic, and reasonable sites choose differently.

Before you change anything, see what your site currently allows. The free AI crawler checker reads your robots.txt and response headers for 21 AI and search bots and flags Cloudflare's AI-bot blocking.

Each operator publishes its own list of tokens. The table groups the ones most sites meet by purpose, using each operator's own description. The purpose matters more than the company name, because one company often runs several bots with very different effects.

The split matters because each kind of bot has a different cost and benefit for you. Treat them as three separate decisions.

• Search and indexing bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot) feed the index an assistant searches when it answers. Blocking them is the most direct way to disappear from AI answers that cite sources.
• Training bots (GPTBot, ClaudeBot, CCBot, Meta-ExternalAgent) collect pages for future models. Blocking them affects what later models learn, not whether today's assistants can cite you.
• User-triggered fetchers (ChatGPT-User, Claude-User, Perplexity-User, Meta-ExternalFetcher) load one page because a person asked. OpenAI, Perplexity and Meta say these may not follow robots.txt the way crawlers do^[1][3][7]; Anthropic says its bots honor robots.txt and the non-standard Crawl-delay rule^[2].

A typical choice for an online store in Mexico or a B2B software company in Germany is to welcome search and user fetches and to block training. A news publisher with licensing deals might block training and also limit search bots to some sections. A local clinic in Australia probably wants everything open. There is no single right answer, only a deliberate one.

Pick the pattern that matches your intent, then adapt the paths. Each group starts with one or more User-agent lines; a bot follows the most specific group that names it and ignores the * group in that case. Keep your existing rules for other bots and your Sitemap line.

1. Allow AI search, block AI training (the common choice)

# AI search and user fetches: allowed
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
Allow: /

# Model training: blocked
# (Google-Extended also covers Gemini grounding; see the note below)
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: Meta-ExternalAgent
User-agent: Bytespider
Disallow: /

# Everyone else, including Googlebot and Bingbot
User-agent: *
Allow: /

Sitemap: https://www.example.com/sitemap.xml

One judgment call in this pattern: Google says Google-Extended controls both Gemini training and grounding, meaning content from the Search index given to the model at prompt time^[4]. If being used in Gemini answers matters more to you than opting out of training, leave Google-Extended out of the blocked group.

2. Allow everything

User-agent: *
Allow: /

Sitemap: https://www.example.com/sitemap.xml

3. Keep AI bots out of one section only

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Disallow: /members/
Disallow: /research/paid/

User-agent: *
Disallow: /members/

robots.txt has been an IETF standard since September 2022, published as RFC 9309^[8]. A few of its rules explain most of the surprises people hit with AI bots:

• The file must live at /robots.txt on each host, so shop.example.com needs its own file^[8].
• User-agent matching is case-insensitive. If several groups name the same bot, their rules are combined; if none names it, the * group applies^[8].
• When Allow and Disallow both match a path, the longest (most specific) rule wins, and an equal tie goes to Allow^[8].
• Crawlers should not use a cached copy for more than 24 hours, and parsers must handle at least 500 KiB^[8]. OpenAI says its systems can take about 24 hours to reflect a change^[1].

The standard also says plainly that its rules are not a form of access authorization^[8]. Compliance is voluntary. The operators above say their crawlers honor it, but anyone can send a request that claims to be GPTBot. If you need a hard block, enforce it at your server or CDN, and verify bots by IP where the operator publishes ranges, as Common Crawl does^[6]. Anthropic advises against IP blocking alone, because it can stop its bots reading robots.txt at all^[2].

If your site runs on Cloudflare, your robots.txt may not be the whole story. On July 1, 2025 Cloudflare announced that new domains would block AI crawlers by default, with owners asked at sign-up whether to allow them^[9]. That means a robots.txt that welcomes OAI-SearchBot can sit behind an edge rule that refuses it.

The setting has since changed shape. Cloudflare's documentation says the legacy Block AI bots toggle is deprecated from September 15, 2026, replaced by AI bot policies with separate choices for Search, Training and Agent bots. Under the new defaults for new domains, search stays allowed while training and agent bots are blocked on pages that show ads^[10]. Each policy can block on all pages, block only on pages with ads, or allow^[10].

1. In the Cloudflare dashboard, open your domain, then Security Settings and Configure AI bot policies (older accounts may still show Block AI bots)^[10].
2. Set Search to allow if you want to be cited. Set Training and Agent to match your licensing choice^[10].
3. Check whether managed robots.txt is on. Cloudflare prepends its own directives, including Content Signals and Disallow lines for known training crawlers, to your existing file^[11].
4. Open AI Crawl Control to see which AI services actually visit, allow or block individual crawlers, and track which ones follow your robots.txt. Cloudflare says it is available on all plans^[12].
5. Run the AI crawler checker again to confirm the served robots.txt and headers now match your intent.

Most crawler problems come from a few predictable places. Work through these in order:

• Fetch https://yourdomain/robots.txt directly and read what is actually served, not what is in your repo or CMS.
• Look for a blanket User-agent: * plus Disallow: / left over from a staging site.
• Confirm OAI-SearchBot, Claude-SearchBot and PerplexityBot are allowed if you want citations.
• Check every subdomain that serves content, such as help. or docs..
• Check X-Robots-Tag headers and meta robots tags: noindex keeps pages out of search indexes regardless of robots.txt.
• On Cloudflare or another CDN, check AI-bot and bot-challenge settings, which can return a 403 to bots robots.txt allows.

Access is only the entry ticket. Being crawlable does not mean an assistant will recommend you; that depends on what it finds about you across the web. To see whether ChatGPT, Google AI Overviews, Google AI Mode, Gemini, Perplexity and Claude actually mention you, run the free AI visibility checker, and read how to get recommended by ChatGPT for the content side. On paid plans, the Source Map shows which sites engines cite for your questions.