A company's robots.txt states in a few public lines whether it lets AI crawlers read its website, and the stance changes when the company changes its mind. This post shows how to read that stance and use it as company data. After it you can name the AI crawler tokens and what each operator says they are for, tell a training crawler from a search crawler and a user-triggered agent, work out which group of a file applies to a crawler, set aside rules that say nothing about AI, read Content-Signal lines, and follow a company's stance over time through the dated rows of Technology Stack and the events of Technology Changes.
The AI crawler list: each token and what its operator says it is for
A robots.txt file addresses a web crawler by its product token, the name on a User-agent line. The table lists the nine tokens the data records a block for, with what the operators' own pages said on 21 October 2026, sorted into kinds that the next section explains.
| Token | Operator | Kind | What the operator's page says |
|---|---|---|---|
| GPTBot | OpenAI | Training | Gathers pages that may go into training OpenAI's generative foundation models; a disallow asks that the site's content be kept out of that training |
| OAI-SearchBot | OpenAI | Search | Finds websites for ChatGPT's search features; a site that blocks it is dropped from ChatGPT's search answers, but may still show as a navigational link |
| ClaudeBot | Anthropic | Training | Gathers public pages that may help train Anthropic's generative models; a site that restricts it asks that its future material be kept out of training data |
| Google-Extended | Usage token | A product token with no user agent string of its own; it decides whether pages Google crawls may train later Gemini models or ground their answers, and leaves inclusion and ranking in Google Search untouched | |
| PerplexityBot | Perplexity | Search | Finds and links websites for Perplexity's search results, and gathers nothing for AI foundation models |
| CCBot | Common Crawl | Open archive | Crawls for a non-profit whose copy of the web is free to researchers, companies and individuals; its FAQ says it checks robots.txt before fetching a page |
| Applebot-Extended | Apple | Usage token | Fetches no pages; governs whether what Applebot gathers may train the Apple foundation models behind features such as Apple Intelligence, and a disallow leaves pages eligible for search results |
| Meta-ExternalAgent | Meta | Training and indexing | Gathers pages for uses that include model training and direct indexing to improve Meta's products |
| Bytespider | ByteDance | Unclassified | Listed here by its token and operator only |
Anthropic's page names three bots: ClaudeBot, Claude-SearchBot, which indexes content for its search results, and Claude-User, which fetches pages when a person asks Claude a question. The token anthropic-ai is absent from that page, and the data records a block on it as a block on ClaudeBot. OpenAI also documents ChatGPT-User, which visits a page when a person asks ChatGPT a question, and Meta documents Meta-WebIndexer for Meta AI search results and Meta-ExternalFetcher for links a user asks it to fetch.
Training crawlers, search crawlers and user-triggered agents
The tokens fall into five kinds, and the kind decides what a block means.
- Training crawlers: GPTBot, ClaudeBot and Meta-ExternalAgent collect pages that may go into a model's training data. A block refuses that use of the site's text.
- Usage tokens: Google-Extended and Applebot-Extended fetch nothing. Google and Apple crawl with their ordinary crawlers, and the token tells them how the content may be used. By Google's and Apple's own statements, a block on either leaves the site's place in their search untouched.
- Search crawlers: OAI-SearchBot and PerplexityBot, with Claude-SearchBot beside them, index pages so that an AI search product can surface and link them. A block asks to be left out of those results.
- User-triggered agents: ChatGPT-User, Claude-User, Perplexity-User and Meta-ExternalFetcher fetch a page because a person asked. OpenAI says robots.txt rules may not apply to ChatGPT-User, Perplexity says Perplexity-User generally ignores them, and Meta says Meta-ExternalFetcher may bypass them, each because a user made the request. Anthropic's page says its bots, Claude-User among them, honour robots.txt directives.
- The open archive: a block on CCBot keeps the site's pages out of an archive anyone may download, whatever they then build from it.
The same word, block, therefore stands for different decisions. A company that keeps GPTBot out and lets OAI-SearchBot in refuses training while staying visible in ChatGPT search; OpenAI states that each of its settings is independent. A company that keeps OAI-SearchBot or PerplexityBot out has chosen to stay out of those answers. Counting both as blocking AI merges two opposite strategies.
How RFC 9309 decides which group of a robots.txt applies
RFC 9309, the IETF standard for the Robots Exclusion Protocol published in September 2022, fixes how a crawler finds its rules. Reading a stance correctly depends on four of them.
- A group is one or more User-agent lines followed by rules, so one group can address several crawlers at once.
- A crawler matches its product token against the User-agent lines without regard to case, and if several groups name it, their rules are combined into one.
- The
*group applies only to a crawler that no group names. For a crawler with a group of its own, none of the*rules carry over. - Within the group that applies, the most specific rule, the one matching the longest stretch of the path, wins, and an Allow is preferred in a tie with a Disallow.
Take this file.
User-agent: *
Disallow: /account/
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
Disallow: /
User-agent: OAI-SearchBot
User-agent: PerplexityBot
Allow: /Read it crawler by crawler. GPTBot, ClaudeBot and CCBot share the second group, which disallows the root, so all three are kept off the whole site: three blocks. OAI-SearchBot and PerplexityBot share the third group, which allows everything, so they may read every page, the account folder included, because the * rules no longer reach them. Google-Extended is named nowhere, so the * group decides for it and the site objects to Gemini training in the account folder only. In one line, the stance is training crawlers and the archive out, search crawlers in.
The third rule has a consequence worth knowing: a friendly group such as Allow: / for a search crawler opens to it the folders the * group closes, whether the author meant to or not.
What is not a stance on AI
Some files keep crawlers out without saying anything about AI, and reading them as an AI stance inflates any count built on them. Two patterns look like a stance and are not.
The first is a file that keeps everyone out.
User-agent: *
Disallow: /It names no crawler. It can be the file of a staging site, an internal tool or a website closed to all automated reading, and it refuses search engines as firmly as AI. The crawler blocks in Technology Stack, part of the marketing stack dataset, are recorded when a group names the crawler and keeps it off the site's home page, so this file records none.
The second is a block on a few folders.
User-agent: GPTBot
Disallow: /checkout/
Disallow: /account/It names GPTBot but keeps it out of two folders only, and the home page and the content behind it stay open. It reads as housekeeping and is not counted as a block. Nor is a group that names a crawler and allows everything, or holds no rules: under RFC 9309 it leaves the whole site open.
Content-Signal lines and the ai-train=no refusal
Rules in robots.txt say whether a crawler may read a page. Content signals say what the content may be used for once it is read. Cloudflare published the Content Signals Policy under a CC0 licence in September 2025, in its announcement, and the Content Signals site sets out the syntax: a Content-Signal line inside a group, carrying signals separated by commas, each set to yes or no.
User-Agent: *
Content-Signal: search=yes, ai-train=no
Allow: /The site defines three signals.
- search: building a search index and returning results such as links and short excerpts. It leaves out AI-generated search summaries.
- ai-input: feeding content into AI models while they answer, as in retrieval-augmented generation and grounding.
- ai-train: using the content to train or fine-tune AI models.
Yes grants the use and no refuses it. A signal left out neither grants nor refuses, so the example above says nothing about ai-input. The policy text in Cloudflare's announcement describes any restriction as an express reservation of rights under Article 4 of the EU directive on copyright in the Digital Single Market (2019/790), so a refusal interests anyone recording the terms under which a corpus was gathered. RFC 9309 lets a crawler interpret records outside the protocol, provided they never disturb the groups. In the data a Content-Signal line is recorded as Content signals in robots.txt, and a line that states ai-train=no adds a second entry, Refuses AI training (content signal).
One fact about origin belongs in the reading. Cloudflare's announcement said it would add search=yes, ai-train=no to the robots.txt it serves for customers who have turned on its managed robots.txt feature. A refusal of training can therefore come from a setting at the company's network provider as well as from a line written by hand. Either way it is the preference the website publishes.
A robots.txt rule states a preference; it does not show compliance
RFC 9309 says its rules do not amount to access authorisation, and Cloudflare says content signals state preferences and offer no technical defence against scraping. A block is what a site asks of a crawler, with the date it was asked. Whether a crawler obeyed is a question for the site's server logs.
Three statements from operators refine the reading.
- Three operators say robots.txt may not govern their user-triggered agents, as set out above, so a file that names one states a wish with less force.
- Anthropic says a ClaudeBot restriction concerns the site's future material, to be kept out of its training data. A block looks forward from the day it appears.
- OpenAI says its search systems can take about 24 hours to adjust to a robots.txt change, and Perplexity and Meta each say up to 24 hours. RFC 9309 asks crawlers not to use a cached copy of the file for more than 24 hours. Allow a day between a dated change and its effect.
Using a company's AI crawler stance as dated data
Read once, a stance is a fact about a website. Read over time, it is a record of dated decisions, in four patterns.
- A block added. A company that adds GPTBot and ClaudeBot to its file has stated, from that date, that its text should not train those models. Set the date beside its announcements: Company News, in the announcements dataset, dates each one, and partnership is among its 13 event types. A content licensing agreement with one AI developer may arrive with blocks on the others.
- A block lifted. A removal is as dated as an addition. A crawler let back in within days of an announced agreement is consistent with terms that cover crawling, which the announcement may not spell out.
- Training out, search in. A file that keeps the training crawlers and usage tokens out and leaves OAI-SearchBot and PerplexityBot open, perhaps with ai-train=no beside them, suggests a publisher or brand that wants to be cited in AI answers without supplying training text.
- Everything out. Blocks on the search crawlers as well as the training ones suggest a company that prefers not to appear in AI search products at all.
Each reader takes something different from the same rows.
| Reader | What to read | What it says |
|---|---|---|
| Analyst or fund | Blocks added and lifted across a sector or a portfolio, and the weekly Market Series of technology added, removed and prevalence per 100 websites of a same-store cohort | How a group of companies is positioning its content towards AI developers, week by week |
| Revenue team selling AI search visibility | Blocks on search crawlers against blocks on training crawlers, and content signals | A training-only stance suggests a company that wants citations; a new block on a search crawler is a dated decision to open a conversation about |
| Revenue team selling bot management | Blocks added, especially several on one day, and ai-train=no | A preference the file states and cannot enforce, with the date the company acted on it |
| AI data team | Each source domain's blocks and signals as of the day a page was collected | The preference in force at collection, point-in-time, for data provenance records and for deciding where to ask for a licence |
The overview of AI readiness signals from company websites combines the crawler stance with the other signals of its group, and the posts on what llms.txt is and what WebMCP is cover what a company publishes for AI systems rather than against them.
Example: Acme Robotics changes its stance
Acme Robotics is an invented company and this example is illustrative. Its file changes twice after its website is first read.
| Reading | What the file says | Technology Stack | Technology Changes |
|---|---|---|---|
| First reading | A group naming CCBot with Disallow: / | Blocks CCBot (Common Crawl), dated by the first reading | None: the first reading is the baseline |
| Six weeks later | A group naming GPTBot, ClaudeBot and Google-Extended with Disallow: /, and Content-Signal: search=yes, ai-train=no | Five new rows: three blocks, Content signals in robots.txt, Refuses AI training (content signal) | Five added events on the same day |
| Five months later | GPTBot taken out of that group | Blocks GPTBot keeps its last-seen date | One removed event |
The first reading shows a company that had already kept the open archive out, with no date for that decision. Six weeks later it refused training on three fronts in one change, and its search crawlers stayed open. Five months later it let one developer's training crawler back while keeping the other two out: a change about one relationship, worth setting beside Acme's announcements of that week.
How to read the rows in Technology Stack and the events in Technology Changes
Technology Stack lists each technology seen on each company website, with its first-seen and last-seen dates and the kind of evidence that showed it, robots.txt for every entry in this post. A block appears as one row per crawler in the category AI crawler access, named for the crawler and its operator: Blocks GPTBot (OpenAI), Blocks ClaudeBot (Anthropic), Blocks Google-Extended (Google), and so on through Blocks Bytespider (ByteDance). The content signal entries sit beside them. The catalogue recognises 6,298 technologies in 69 categories, and the post on what a Fokals technology row tells you sets out how to read a row.
Technology Changes holds a dated event for every addition and removal. The first reading of a website, or of a kind of evidence, sets a baseline and is never counted as a change, so a block already in place carries the baseline's first-seen date and writes no event. An added event is a block observed to appear, and a removed event is a block observed to go, with the last-seen date in Technology Stack giving the latest reading that showed it. The query below lists the events of the AI crawler access category from a time you set; add a filter on company ID for the companies you track.
select observed_at, company, technology_name, change
from company_tech_events
where category = 'technology'
and technology_category = 'ai_crawlers'
and change in ('added', 'removed')
and observed_at >= timestamp '2026-10-01 00:00:00'
order by company, observed_at;The data dictionary defines each field, and section 5.1 of the methodology states the rules for this group of entries.
Frequently asked questions
How do you block AI crawlers in robots.txt?
Give each crawler you want kept out a group of its own, or list several on consecutive User-agent lines, and follow them with Disallow: /. Use the token each operator publishes, such as GPTBot, ClaudeBot, Google-Extended or CCBot. If the new group allows any path, repeat in it the folder rules of the * group, because under RFC 9309 a crawler with a named group no longer follows the * rules. Operators say a change can take about a day to reach their systems.
Does blocking GPTBot remove a site from ChatGPT search?
No. OpenAI documents GPTBot as its training crawler and OAI-SearchBot as the crawler that surfaces websites in ChatGPT's search features, and it says the two settings work independently. A site that blocks GPTBot and leaves OAI-SearchBot open refuses training while staying eligible for ChatGPT search answers. To leave those answers, a site blocks OAI-SearchBot, and OpenAI says navigational links can still appear.
Does Google-Extended affect Google Search rankings?
Google says it does not. Its crawler documentation presents Google-Extended as a product token that no crawler sends: Google fetches pages with its usual crawlers, and the token tells it whether those pages may train later Gemini models and ground their answers. The same page says the token changes neither whether a site appears in Google Search nor how it ranks there.
Do AI crawlers have to obey robots.txt?
Nothing in the protocol forces them. RFC 9309 says robots.txt rules do not amount to access authorisation, so a crawler that ignores the file meets no technical barrier. Operators state their own practice: Anthropic says its bots honour robots.txt, while OpenAI, Perplexity and Meta say their user-triggered fetchers may not follow it, because a person asked for the page. Server logs, not the file, show what a crawler did.
What is a Content-Signal line in robots.txt?
It is a line from the Content Signals Policy that Cloudflare published in September 2025, stating how a site's content may be used once read. It carries three signals, search, ai-input and ai-train, each set to yes or no, such as Content-Signal: search=yes, ai-train=no. A signal left out neither grants nor refuses that use. Like any robots.txt rule, it expresses a preference and does not stop a crawler technically.
How can you find out which companies block AI crawlers?
Read each company's robots.txt group by group, or use a technographic dataset that does it across the companies you track. In Fokals data, Technology Stack holds one row per blocked crawler, such as Blocks GPTBot (OpenAI), with first-seen and last-seen dates, and Technology Changes dates each block added or lifted after a website's first reading. Content signals, including a refusal of AI training, are recorded beside them.
Which AI crawlers are used for training and which for search?
By their operators' pages, GPTBot and ClaudeBot crawl for model training and Meta-ExternalAgent for uses that include it, while OAI-SearchBot, PerplexityBot and Claude-SearchBot index pages for AI search answers. Google-Extended and Applebot-Extended crawl nothing and govern how content their companies already crawl may be used for AI. ChatGPT-User, Claude-User and Perplexity-User fetch pages when a person asks a question. CCBot builds Common Crawl's open archive of the web.
The queries and code on this page are examples to adapt. Test them in your own environment before you rely on them.