AI readiness signals: how to read a company website's stance on AI

A company website states a position on AI in files and tags that few visitors ever see: which AI crawlers it turns away, which uses of its content it refuses, and what it publishes for AI assistants and agents. This guide sets out every signal of that group in one table, with where each sits, who defined it and what it says about the company. After it you can sort the signals into the two directions they point, combine them with hiring and intent into a readiness profile for each company you track, and read a change in any of them as a dated event rather than a first sighting.

The AI readiness signals a company website publishes

Seven kinds of signal make up the group. Four guard content from AI systems, three open the site to them, and a content signal can point either way. Each was defined by a different body, which matters when you weigh what it commits the company to.

Technology Stack, in the marketing stack dataset, records each signal as an entry with its first-seen and last-seen dates. The category AI crawler access holds the guarding entries: one per crawler blocked, nine crawlers in all, from Blocks GPTBot (OpenAI) to Blocks Bytespider (ByteDance), then Content signals in robots.txt, Refuses AI training (content signal), Reserves text and data mining (TDMRep) and Opts out of AI use (noai). The category Artificial intelligence holds llms.txt for AI assistants, WebMCP tools for AI agents, and AI assistants and AI search tools sold as such, among them Chatbase, Kapa.ai and yellow.ai, and domain verifications with AI vendors. The post on what a Fokals technology row tells you explains how each row is dated.

Guarding content: crawler rules, content signals, TDMRep and noai

Crawler blocks and content signals

Under RFC 9309 a crawler obeys the group of robots.txt that names its product token, and the RFC states that these rules are not a form of access authorisation. A block is recorded when the group naming an AI crawler keeps it off the site's home page. A file that keeps every crawler out is not read as a stance on AI, and neither is a rule that keeps a crawler out of a few folders.

Which crawler is named tells you which use was refused. OpenAI's crawler documentation says that disallowing GPTBot signals content should stay out of the training of its foundation models, that OAI-SearchBot surfaces sites in ChatGPT's search features, and that each setting is independent. Google's list of common crawlers describes Google-Extended as a token governing use in training Gemini models and in grounding, with no effect on inclusion in Google Search. The post on blocking AI crawlers in robots.txt covers every token and how a group is matched.

The Content Signals policy, published by Cloudflare, adds a line that states uses rather than access. Search means building an index and returning links and short excerpts, AI summaries excluded; ai-input means feeding content to a model as it answers, as in retrieval or grounding; ai-train means training or fine-tuning a model.

User-Agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=no
Allow: /

The policy presents any restriction as an express reservation of rights under Article 4 of the EU copyright directive, and a use left without a signal is neither granted nor restricted. A line with ai-train=no is recorded twice: as Content signals in robots.txt and as Refuses AI training (content signal).

TDMRep: reserving text and data mining rights

Directive (EU) 2019/790, on copyright in the Digital Single Market, provides for an exception for text and data mining of lawfully accessible content unless the rightholder has expressly reserved that use. Its recitals say that for content published online the reservation should be made by machine-readable means, such as metadata. TDMRep is a protocol for that reservation.

The TDM Reservation Protocol Community Group published it as a Final Community Group Report on 10 May 2024, which states that it is not a W3C Standard. It defines two properties.

• tdm-reservation. A value of 1 means mining rights are reserved. A value of 0 means they are not, so a miner may proceed without contacting the rightholder. Any other value is treated as unset.
• tdm-policy. The address of a policy that tells a miner how to obtain permission, readable by a person as HTML or by a machine as JSON.

A rightholder can state them in a JSON file at /.well-known/tdmrep.json covering paths of a site, in HTTP response headers, in meta tags in an HTML page, or in the metadata of EPUB and PDF files.

tdm-reservation: 1
tdm-policy: https://www.example.com/policies/tdm.json

A page tag or response header with a reservation of 1 is recorded as Reserves text and data mining (TDMRep). Read it as a legal statement before a technical one: it addresses anyone who mines content, AI developers among them, and it can point to the terms of a licence. A company that sets it belongs with the guarded even when its robots.txt names no AI crawler.

noai and noimageai

DeviantArt introduced the two directives on 11 November 2022. For an artist who declines AI use, the page carries noimageai in an HTML tag, and a direct download of the image returns noai in its header. Its terms require third parties that train models on DeviantArt content to exclude anything carrying either directive, and it opened noai to any site.

X-Robots-Tag: noai
X-Robots-Tag: noimageai

The same values can sit in a page's robots meta tag, and the announcement presents both as a statement that the content is not authorised for AI datasets. Where TDMRep comes from a group's report, noai comes from one platform's announcement. On a company website, either value is recorded as Opts out of AI use (noai).

Opening the site: llms.txt, WebMCP and AI assistants

llms.txt. Jeremy Howard proposed the file on 3 September 2024. Its only required part is a top-level heading naming the site, which may be followed by a summary and lists of links, and the proposal expects it to serve mainly when an assistant answers rather than in training. It is recorded when the site's root holds a Markdown file that opens with a heading; an address that answers with the home page or another web page is not one. The post on what llms.txt is covers the format.

WebMCP. The draft of the W3C Web Machine Learning Community Group, edited by engineers at Microsoft and Google, lets a page give AI agents tools: JavaScript functions with a description and an input schema, registered through document.modelContext.registerTool(). A declarative form, in which an HTML form carrying a toolname attribute becomes a tool, is set out in a separate explainer. The draft states that it is not a W3C Standard. The post on what WebMCP is covers both forms.

AI assistants and AI search. A chat assistant or answer engine on the site, sold as AI by its vendor. A detection shows that the tool runs on the website; how many visitors use it is a separate question.

Two directions, and why one company shows both

The guarding and opening signals are not two ends of one scale, because they address different uses. Content Signals separates ai-input from ai-train, OpenAI keeps its training and search crawlers on independent settings, and the llms.txt proposal places its use at the moment an assistant answers. So a publisher that states ai-train=no and blocks GPTBot, while it leaves OAI-SearchBot free and publishes llms.txt, is consistent: it wants assistants to cite it accurately and its archive kept out of model training.

Measure the two directions on two axes and the profiles fall out.

A website whose robots.txt names no AI crawler leaves each one to the rules for every crawler, but it has made no statement about AI, so never count an absence as openness.

Building a readiness profile for each company

A profile keeps every part visible, with its date, so a reader can see why a company sits where it does.

1. Take the current guarding and opening entries of each website from Technology Stack, leaving out any whose removal awaits confirmation.
2. Sort them onto the two axes, and separate the crawlers an operator documents for training from those it documents for search.
3. Add hiring. Job Postings, in the hiring dataset, carries an AI role flag on each posting, and Hiring Activity counts the open postings with it each day. The share of AI roles among open postings over 28 days shows whether the company builds AI itself.
4. Add intent. Intent Scores, in the intent dataset, give a weekly score from 0 to 100 for the topic AI and machine learning, with a surge flag and the dated evidence behind it. Keep the score in a column of its own beside the parts rather than adding it to them, and read the guarding entries as a stance, not a purchase.
5. Write one row per company: guarding entries, opening entries, AI role share and topic score, each with its date.

The query below produces the website part of the row; join its result to Hiring Activity and Intent Scores on the company ID.

select company_id, domain,
       count(*) filter (where technology like 'ai_block_%') as ai_crawlers_blocked,
       bool_or(technology = 'ai_training_refused')         as refuses_ai_training,
       bool_or(technology = 'tdm_reservation')             as reserves_tdm,
       bool_or(technology = 'noai_directive')              as opts_out_noai,
       bool_or(technology = 'llms_txt')                    as publishes_llms_txt,
       bool_or(technology = 'webmcp')                      as offers_webmcp,
       count(*) filter (where technology_category = 'artificial_intelligence'
                          and technology not in ('llms_txt', 'webmcp')) as other_ai_entries
from company_technologies
where missing_since is null
  and technology_category in ('ai_crawlers', 'artificial_intelligence')
group by company_id, domain
order by domain;

Example: Acme Robotics

Acme Robotics is an invented company and this example is illustrative: the inputs are assumptions chosen to show the logic, not findings.

The profile is selective: training refused, answers and assistants welcome, AI built in house and rising. Each part can be opened and checked against its entry and date.

Reading change over time

Technology Changes holds a dated event for every addition and removal of these entries. The first reading of a website, or of a kind of evidence, sets a baseline and is never counted as a change. A GPTBot block or an llms.txt present at that reading carries the baseline's first-seen date and writes no event, though the company may have published it years before. A first-seen date is therefore not an adoption date. An adoption you can date has an added event, which places the decision between two readings, and a removal event marks a stance given up.

The order of events carries the story.

• A training crawler blocked, then ai-train=no, then a TDM reservation: a company hardening its position.
• A crawler block removed in the weeks an llms.txt is added: a turn from guarding towards courting assistants.
• WebMCP added after an AI assistant: a move from answering visitors to letting agents act.

select domain, company, technology_name, change, observed_at
from company_tech_events
where category = 'technology'
  and technology_category in ('ai_crawlers', 'artificial_intelligence')
  and change in ('added', 'removed')
order by domain, observed_at;

The weekly Market Series carry additions, removals and prevalence per 100 websites of a same-store cohort for the crawler, content signal, TDMRep, noai, llms.txt and WebMCP entries, so you can tell whether a company moved with its market or ahead of it. Two rules keep a reading honest. A robots.txt rule or a content signal states a preference; it does not show whether any crawler obeys it. And a detection shows presence on the website between two dates. The methodology defines each entry in section 5.1.

Uses by reader

A fund tracking AI adoption across listed companies

The opening entries extend the tools leg in measuring AI adoption across listed companies: an llms.txt, WebMCP or AI assistant added is a dated, checkable act, where a mention in an announcement is a claim. The guarding entries answer a separate question: how a company treats its content as an asset. Aggregate brands to the listed parent with the ISIN or FIGI each listed company carries, and back-test on the observation dates.

A revenue team selling AI search visibility, agent tooling or bot management

• AI search visibility. Companies that added llms.txt or an AI assistant, with content signals that allow search and ai-input, are working on how assistants present them. One that blocks OAI-SearchBot has asked to be left out of ChatGPT's search answers.
• Agent tooling. Companies that added WebMCP, or that publish llms.txt and hire AI roles, are preparing for agents to act on their sites.
• Bot management. Companies that added several crawler blocks in a short span are policing AI traffic with a file that RFC 9309 says is no access control, the gap a bot management product addresses.

Time the approach from the date of the added event in Technology Changes.

A data team building an AI training corpus that respects opt-outs

Treat the guarding entries as a per-domain exclusion list: Refuses AI training (content signal), Reserves text and data mining (TDMRep), Opts out of AI use (noai), and blocks of crawlers that operators document for training. Because Technology Changes dates each opt-out, the list is point-in-time: you can test every document against the stance its website held on the day you collected it, and drop a domain from the next training run when an opt-out arrives later. Keep the entry and its date beside each exclusion: Content Signals and TDMRep both frame a refusal as a reservation under Article 4 of the EU directive, and a legal review will ask which reservation applied and when.

Frequently asked questions

The queries and code on this page are examples to adapt. Test them in your own environment before you rely on them.