Free tool
AI crawler checker
Is your site blocking GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot or Google's AI? Enter your domain and we read your robots.txt and homepage the way 21 AI and search crawlers do, then flag Cloudflare's default AI blocking.
Search, training and user agents: what each AI crawler does
Not every AI bot is the same, and blocking the wrong one is the most common reason a site disappears from AI answers. There are three kinds that matter, plus classic search crawlers:
- AI search index
- Builds the index an AI assistant searches when it answers. Block it and that assistant usually can't cite or recommend your pages.
- OAI-SearchBot, Claude-SearchBot, PerplexityBot, DuckAssistBot
- Search engine
- Classic search crawler. Google AI Overviews and AI Mode are built on Google's search index, and Copilot on Bing's.
- Googlebot, Bingbot, Applebot
- User-triggered fetch
- Fetches a page live because someone asked the assistant to read it. Operators say these may not follow robots.txt like crawlers do.
- ChatGPT-User, Claude-User, Perplexity-User, Meta-ExternalFetcher, MistralAI-User
- Model training
- Collects pages to train future models. Blocking it is a content-licensing choice; it doesn't stop assistants recommending you today.
- GPTBot, ClaudeBot, CCBot, Meta-ExternalAgent, Bytespider, Amazonbot, anthropic-ai
- Usage control token
- Not a separate crawler. A robots.txt token that controls whether content already crawled for search may be used for AI.
- Google-Extended, Applebot-Extended
Cloudflare blocks AI crawlers by default
Since July 2025, Cloudflare blocks known AI crawlers by default on newly added sites and offers a Block AI bots switch, AI Crawl Control and a managed robots.txt. Those blocks run at the firewall, before robots.txt is ever read, so a crawler can be refused with a 403 while your robots.txt says Allow: /.
If this checker says your site is on Cloudflare, open the dashboard, go to Security → Bots (or AI Crawl Control) and allow the search crawlers you want to be found by: OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot and Bingbot. You can keep training bots blocked.
AI crawler FAQ
Should I block GPTBot?
GPTBot collects pages to train OpenAI's models. Blocking it is a licensing decision and doesn't remove you from ChatGPT answers today. If you want ChatGPT search to cite you, keep OAI-SearchBot allowed: it's the crawler that builds ChatGPT's search index.
Which AI crawlers should I allow to be recommended by AI?
The search crawlers: OAI-SearchBot (ChatGPT search), Claude-SearchBot (Claude), PerplexityBot (Perplexity), Googlebot (Google Search, AI Overviews and AI Mode) and Bingbot (Bing and Copilot). Blocking these is what usually costs AI visibility.
Does blocking Google-Extended remove me from AI Overviews?
No. Google-Extended is a control token, not a crawler. It decides whether content Google already crawled may be used for Gemini training and grounding. AI Overviews and AI Mode use Googlebot's search index, so only blocking Googlebot (or noindex/nosnippet) affects them.
Why does my site block AI bots when robots.txt allows them?
Firewalls and CDNs act before robots.txt. Cloudflare, for example, has blocked known AI crawlers by default on new sites since July 2025 and offers a one-click 'Block AI bots' setting. A blocked bot gets a 403 whatever robots.txt says, so check your CDN's bot settings too.
Do user-triggered agents like ChatGPT-User follow robots.txt?
Not always. These agents fetch a page because a person asked the assistant to. OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores robots.txt. Blocking them mostly breaks the experience for people already asking about you.
How does this checker read robots.txt?
It follows RFC 9309, the robots.txt standard: each bot uses the group naming its own token (groups for the same token are merged), otherwise the * group; the longest matching rule wins and Allow wins ties. A missing robots.txt (4xx) means everything is allowed; a server error (5xx) means crawlers assume everything is blocked.
Crawlers can reach you. Does AI actually recommend you?
Access is step one. Enter your website and verify your email for a limited report, with question previews and a short answer excerpt when available. Paid plans unlock full collected insights and ongoing tracking.
Free website check · No card needed · Full AI tracking on paid plans
More free tools
- AI visibility checkerPaste your website. See whether ChatGPT, Gemini, Perplexity and Google's AI recommend you, with a limited answer preview.Open
- ChatGPT visibility checkerIs your brand in ChatGPT? See whether ChatGPT names you for your category, who it recommends instead and what it cites.Open
- Brand mention checkerPreview how AI mentions your brand. Upgrade to explore full answers, competitors and cited sources.Open
- llms.txt generatorDraft an llms.txt from your homepage in seconds, or check an existing file against the llmstxt.org format.Open
- AI Overview checkerType one Google search. See whether AI Overviews and AI Mode answer it, which sites they cite, and whether you're one of them.Open
Crawler list reviewed 2026-10. Operators change their bots often; check their documentation for the latest.