Which AI crawlers does a website block?
GPTBot, ClaudeBot, Google-Extended and the rest, with the site’s content signals, llms.txt and WebMCP.
Enter a website above to see what its robots.txt says to each of these nine AI crawlers:
- GPTBotOpenAI · AI training
- OAI-SearchBotOpenAI · AI search
- ClaudeBotAnthropic · AI training
- Google-ExtendedGoogle · AI training
- PerplexityBotPerplexity · AI search
- CCBotCommon Crawl · AI training
- Applebot-ExtendedApple · AI training
- Meta-ExternalAgentMeta · AI training
- BytespiderByteDance · AI training
With its content signals, TDMRep, noai, llms.txt, WebMCP and MCP server.
Free for everyone, more with a free account
Without an account
- Each AI crawler, blocked or allowed today
- Content signals, llms.txt and WebMCP
With a free account
- Everything without an account
- Since when each block has stood
- Whether the website or its CDN set it
- The share of its industry blocking each crawler
About the AI crawler checker
Which AI crawlers does the checker cover?
Nine: GPTBot and OAI-SearchBot (OpenAI), ClaudeBot (Anthropic), Google-Extended (Google), PerplexityBot (Perplexity), CCBot (Common Crawl), Applebot-Extended (Apple), Meta-ExternalAgent (Meta) and Bytespider (ByteDance).
What counts as blocked?
A crawler is blocked when the website’s robots.txt keeps it out of the home page, by a group that names it or by its rules for every crawler. A crawler the file lets in is allowed. Not blocked by name means the file does not name that crawler.
How do I block GPTBot or ClaudeBot?
Add a group for the crawler to the robots.txt at the root of the website: a User-agent line with its name, such as GPTBot or ClaudeBot, followed by Disallow: /. Each crawler is named in a group of its own, or listed together in one.
What are content signals?
A Content-Signal line in robots.txt states what a website’s pages may be used for: search, AI input and AI training, each yes or no. The checker shows whether a website has one, and whether it refuses AI training.
What are llms.txt, WebMCP and an MCP server?
llms.txt is a Markdown file at the root of a website that summarises it for AI assistants. WebMCP and an MCP server go further: they give AI agents tools to act on the website.
Why is a block set by its CDN?
Some CDNs write a managed section into the robots.txt of the websites they serve. A free account shows when a block comes from that section rather than from the website’s own rules, and since when each block has stood.