llms.txt is a Markdown file a website publishes to give AI agents a short guide to its content. This post sets out who proposed it and what the proposal asks for, shows the format with a worked example, compares the file with robots.txt and sitemap.xml, and gives a three-part test that separates a real file from a server that answers every address with its home page. After it you can check any company's website for the file, and read its presence and its dates as evidence of how that company is preparing for AI agents.
What is llms.txt?
llms.txt is a proposal for a plain Markdown file that tells AI agents what a website is about and where its most useful content sits. Jeremy Howard published it on 3 September 2024, in a post on the Answer.AI blog and on the proposal site llmstxt.org. The file gives a little background and advice, then points to the pages that hold the detail. It is kept short enough for a model to take in whole, and an agent opens the pages behind its links only when it needs them.
The proposal starts from a practical problem. A web page surrounds its text with menus, scripts and advertising that a model must remove before it can work with the content, and most websites are far too large to read whole. The author of a site knows which pages matter, so the file gives that author a fixed place to say so.
In August 2026 Howard revised the proposal as version 2. Its list of changes presents the revision as a response to two years of use: agents get a standard way to discover the file, Markdown pages may take either of two address forms, and a file placed in a subfolder now has a defined scope. The text is kept in a public GitHub repository, and the proposal site describes it as open for community input.
The llms.txt format, with an example
The proposal fixes the order of the parts of the file, and only the first heading is required.
- An optional byte-order mark.
- A first-level heading with the name of the project or site. This is the one required part.
- A blockquote with a short summary, holding the facts a reader needs to make sense of the rest of the file.
- Any number of paragraphs or lists, with no headings, giving more detail and explaining how to read the linked files.
- Any number of sections under second-level headings, each a list of links. Every item is a Markdown link, optionally followed by a colon and a note on the page behind it.
A section headed Optional is where a file puts links of lesser importance, which an agent can pass over when it is short of context. In the first version that heading carried a set meaning for the tools that expanded a file into one long context; version 2 keeps it as a convention and nothing more.
The file sits at the root of a site, at /llms.txt. It can also sit under a path, such as /docs/llms.txt, and version 2 defines what such a file covers: the pages beneath that path, with agents asked to use the narrower file where two apply.
Here is an illustrative llms.txt for Acme Robotics, an invented company.
# Acme Robotics
> Acme Robotics builds autonomous picking arms for warehouses, sold with a fleet management service and a REST API for integrators.
The arms ship in two payload classes. Every page linked below has a Markdown version at the same address with .md added.
## Products
- [Picking arms](https://acme-robotics.example/products/arms.html.md): Payload classes, reach, grippers and safety ratings
- [Fleet manager](https://acme-robotics.example/products/fleet.html.md): Monitoring, alerts and software updates for deployed arms
## Developers
- [API reference](https://acme-robotics.example/docs/api.html.md): Endpoints, authentication and rate limits
- [Quickstart](https://acme-robotics.example/docs/quickstart.html.md): Connect a first arm and read its status
## Support
- [Service and warranty](https://acme-robotics.example/support.html.md): Service levels and how to open a case
## Optional
- [Company history](https://acme-robotics.example/about/history.html.md)Read from the top, the file is the company's own account of itself. The blockquote is a one-line description written for machines, and the section names show which questions the company expects agents to bring. A Developers section, as here, says the product is used through code.
Markdown versions of pages and llms-full.txt
The proposal makes a second request: that a page holding information an agent may want is also available as plain Markdown, at an address derived from its own. Version 2 accepts two forms, .md added to the full address (page.html.md) or the extension replaced by .md (page.md), with index.html.md or index.md for an address that ends in a folder.
So that an agent can find these files without guessing, version 2 recommends two standard link relations: rel="alternate" with type="text/markdown" points to the Markdown version of a page, and rel="describedby" points to the llms.txt that covers it. Either can be written as a <link> element in the page or sent as an HTTP Link header, and the header form can also be sent with the Markdown files, which have no HTML of their own.
The proposal defines no file called llms-full.txt. Its first version described tooling that expanded an llms.txt into larger context files, and version 2 takes that tooling out. Some documentation platforms generate an llms-full.txt. Mintlify's documentation says it generates and serves an llms.txt at the root of each customer's documentation, listing every page and kept up to date without maintenance, and an llms-full.txt holding the full Markdown text of every page in one file.
How llms.txt differs from robots.txt and sitemap.xml
All three are plain files at the root of a site, and all three are read by software rather than by people. They answer different questions.
| File | What it is for | Written for | Defined by | What is enforced |
|---|---|---|---|---|
| robots.txt | Says which paths a crawler may request | Crawlers | RFC 9309, a Proposed Standard of the IETF, September 2022 | Nothing: the RFC states that its rules are not a form of access authorisation |
| sitemap.xml | Lists the pages a site offers for crawling, with optional hints | Search engines | The Sitemaps protocol | Nothing: the protocol site says using it does not guarantee that pages are included in search engines |
| llms.txt | Gives a short guide to the site and links to Markdown pages | AI agents working for a user | The proposal at llmstxt.org | Nothing: an agent chooses whether to fetch the file and follow its links |
The proposal draws the line between the first and the third itself. robots.txt states which automated visits a site accepts, while an agent turns to llms.txt only at the moment a task for a user calls for it. A sitemap is no substitute in the proposal's account, because it often leaves out the Markdown versions of pages, excludes links to other sites, and covers far more than fits in a context window.
For a reader of company data the difference matters. A robots.txt rule states a stance on who may crawl a site: the post on blocking AI crawlers in robots.txt reads that stance, and the robots.txt glossary entry covers how the file is matched. An llms.txt is an invitation instead, naming the pages a company wants an agent to read first.
Which AI systems read llms.txt
Publishing a file shows what a company intends. Whether a given assistant reads it is a question for that assistant's operator, and the statements to rely on are the operators' own.
- The author of the proposal. In version 2, Howard names software documentation as the main use, with coding agents reading the files to find their way to API references and tutorials. He adds that they have served mainly at inference, when an agent answers a question, rather than in training.
- Google Search. Google's page on AI features and your website says a site needs no new machine-readable files or AI text files to appear in AI Overviews or AI Mode.
- Google Chrome. Lighthouse includes an llms.txt audit among its agentic browsing audits. It flags a server error when the file is requested, and marks a missing file, a 404, as not applicable because the file is optional for now.
Read an llms.txt, then, as a statement of what the company wants agents to find. For a claim that a named assistant fetches the file, look for that operator's own documentation.
Which companies publish llms.txt
The proposal cites, as examples, the files the AI labs keep for their developer documentation. Each answers today as a Markdown file that opens with a heading: OpenAI's, Anthropic's and Google's for the Gemini API. In the version 2 text of August 2026, Howard writes that thousands of sites publish one.
The publishers fall into three groups, and each says something different about the company behind the file.
- Software sold through code. Companies with an API, an SDK or a developer platform have the most direct reason to publish, because coding agents read their documentation on behalf of their customers. A Developers or API section in the file confirms the reading.
- Companies working on how AI answers describe them. A business that writes a one-line summary of itself for agents and chooses the pages behind it is doing for AI assistants what search optimisation does for search engines. Read the file beside the content platform and the search optimisation tools in the same stack.
- Sites whose platform writes the file. Wix's help centre says Wix generates and maintains a site's llms.txt automatically, lists an upgraded site, a connected custom domain and search engine indexing as prerequisites, and lets the owner edit the file or opt out. Yoast's page says its WordPress plugin generates and refreshes the file once the site owner enables the feature. On the first, a file can exist without a decision at the company; on the second, someone switched it on.
Technology Stack records the content platform a website runs on beside its llms.txt, so the deliberate file and the platform default can be told apart. Opening the file helps too: one that lists every page in order with no summary of its own looks generated, and one with a written summary and a few chosen sections looks written.
The file also leaves questions open. It shows an intention, and on its own it says nothing about whether an assistant has read it or how the company uses AI in its own work. The other entries of the same group widen the reading: WebMCP tools on a page offer agents actions to take on the site, covered in what is WebMCP, and crawler rules show which AI crawlers a site keeps out. A company can keep training crawlers away and still invite agents that work for a user, because the two files address different readers. The overview of AI readiness signals from company websites combines them per company.
How to tell a real llms.txt from a page that answers every address
A request for /llms.txt on a site without the file should end in a 404. Many sites answer otherwise. A web application with a catch-all route, or a server set to send unknown addresses to the home page, returns status 200 and an HTML page, and a check that looks at the status alone counts every such site as a publisher.
Three tests separate a file from a page.
- Status. The final answer, after any redirect, is 200.
- Content type. The answer is not HTML. A real file is typically served as text/plain or text/markdown, and its text does not open with
<as an HTML document does. - Opening heading. The text opens with a first-level Markdown heading,
#followed by a name, which is the one part the proposal requires.
Two requests run the tests on any site.
curl -sL -o /dev/null -w "%{http_code} %{content_type}\n" https://acme-robotics.example/llms.txt
curl -sL https://acme-robotics.example/llms.txt | head -n 3The replies fall into a few patterns.
| What the site sends | Status | Content type | Opens with | Reading |
|---|---|---|---|---|
| A real file | 200 | text/plain or text/markdown | # Acme Robotics | Published |
| The home page for every address | 200 | text/html | <!doctype html> | No file |
| A redirect to the home page | 301, then 200 | text/html | <!doctype html> | No file |
| A plain not found | 404 | Any | An error page | No file |
| Plain text with no heading | 200 | text/plain | Free text | Outside the proposal's format |
This is the reading behind the llms.txt entry in Fokals data. Technology Stack records llms.txt for AI assistants when the root of a company website answers with a Markdown file that opens with a heading, and an address that answers with the home page or any other web page is not counted as one.
Reading llms.txt as dated company data
In the marketing stack dataset, llms.txt is read like any other technology. Technology Stack holds a row for llms.txt for AI assistants, in the category Artificial intelligence, with its first-seen and last-seen dates and the kind of evidence that showed it, here the file at the root of the site. Technology Changes holds a dated event when the file is added and when it is removed. The catalogue recognises 6,298 technologies in 69 categories, so the file sits on the same company ID as the content platform, the analytics and the rest of the technographic data for that company.
Two dates answer two questions.
- The first-seen date is the first observation that showed the file. The first reading of a website, or of a kind of evidence on it, sets a baseline and is never counted as a change, so a file already in place at that reading carries the baseline date: the file was there, and its arrival was not observed.
- The addition event in Technology Changes marks a file observed to arrive, dated by the time of the observation. Use it when the question is when a company started, and the first-seen date when the question is whether it has a file.
A removal event is read the same way, with one check. Technology Changes records a platform migration as its own event, and a file that disappears in the same week as one may have gone with the old platform rather than by a decision about AI agents.
For the account view, the intent dataset gives a weekly score on topics such as AI and machine learning, and SEO, content and CMS, each with the dated evidence behind it. For adoption across an industry or a country, the weekly Market Series carry llms.txt like the other entries of its group: the websites that added it, those that removed it, and its prevalence per 100 websites of a same-store cohort, so a change in the rate reflects adoption rather than a change in which websites were read. Section 5.1 of the methodology describes the group.
An illustrative case shows the reading. Acme Robotics is invented, and the dates below are not findings.
| Date | Record | Reading |
|---|---|---|
| 9 September 2026 | First reading of the website, with no llms.txt | Baseline |
| 14 October 2026 | Technology Changes: llms.txt for AI assistants added | Arrival observed and dated |
| The same week | No platform migration; the file opens with its own summary and a Developers section | A decision by the company, aimed at developers |
Read together, the rows say that Acme Robotics chose in October to guide AI agents to its API documentation, which places it among the software companies preparing for coding agents. The post on what a Fokals technology row tells you sets out the dating rules behind every row.
Frequently asked questions
What is llms.txt used for?
llms.txt gives AI agents a short guide to a website: a heading with the site's name, a brief summary and lists of links to the pages that matter, ideally in Markdown. The author of the proposal names software documentation as the main use, with coding agents reading the files to find their way to API references and tutorials, and says they serve mainly when an agent is answering a question for a user rather than in model training.
Who created llms.txt?
Jeremy Howard proposed llms.txt on 3 September 2024, in a post on the Answer.AI blog and on the proposal site llmstxt.org. He revised it as version 2 in August 2026, adding link relations that let agents find a page's Markdown version and the file that covers it, and defining what a file placed under a subfolder covers. The text is kept in a public GitHub repository open for community input.
Is llms.txt an official standard?
Not in the way robots.txt is. robots.txt was published by the IETF as RFC 9309 in September 2022. llms.txt is a proposal published at llmstxt.org, revised as version 2 in August 2026, with its text in a public repository open for community input. Nothing obliges an agent to fetch it, so treat it as a convention that a website and an agent may each choose to follow.
What is the difference between llms.txt and llms-full.txt?
llms.txt is a short index: a name, a summary and links to the pages that matter. llms-full.txt, where a site has one, holds the full text of its documentation in a single file. The proposal defines llms.txt and the Markdown versions of pages; documentation platforms such as Mintlify generate llms-full.txt, and Mintlify's documentation describes it as every page of a site, in full, in one file.
Does Google use llms.txt for AI Overviews?
Google's Search Central page on AI features says a site needs no new machine-readable files or AI text files to appear in AI Overviews or AI Mode, and no special structured data either. Separately, Google's Chrome team includes an llms.txt audit in Lighthouse's agentic browsing checks, which marks a missing file as not applicable because the file is optional for now. For any other assistant, rely on its operator's own documentation.
How can I check whether a website has an llms.txt file?
Request /llms.txt at the root of the site and apply three tests. The final status should be 200, after any redirect. The content type should not be HTML, and the text should not open with an HTML tag. The text should open with a Markdown first-level heading naming the site. A site that returns its home page for every address passes the first test and fails the other two, so it has no file.
How do you know when a company added llms.txt?
Use the dated addition in Technology Changes, in the Fokals marketing stack dataset. It marks a file observed to arrive, with the time of the observation. The first-seen date in Technology Stack answers a different question: a file already in place when the website was first read carries that reading's date as a baseline and writes no event, so it shows presence rather than the day of adoption.
Does llms.txt replace robots.txt?
No, the two do different jobs. robots.txt says which paths crawlers may request, and it is where a website states its stance on AI crawlers. llms.txt points agents to the content a site wants read. A site can keep training crawlers out in robots.txt and still publish an llms.txt for agents that fetch pages for a user, since the proposal places llms.txt in that on-demand reading.
The queries and code on this page are examples to adapt. Test them in your own environment before you rely on them.
What this page says about the products it names was checked against their public documentation on 21 October 2026. Product and company names are trademarks of their owners. Fokals is not affiliated with them or endorsed by them.