robots.txt is a plain-text file at the root of a website that tells automated crawlers which paths they may and may not request. Its rules are grouped by crawler name and written as Allow and Disallow lines. Well-behaved crawlers follow it voluntarily: it is a published convention, not an access control.
How the file works
The Robots Exclusion Protocol was first defined by Martijn Koster in 1994 and specified as a Proposed Standard, RFC 9309, by the IETF in September 2022. The file must be served as /robots.txt at the top of the host. A crawler follows the group that names it, or the * group when none does. Where an Allow and a Disallow both match a path, the longer rule wins, and Allow wins a tie. The standard asks for case-sensitive path matching.
User-agent: *
Disallow: /private/
Allow: /private/press/
User-agent: ExampleBot
Disallow: /In this file every crawler except ExampleBot may read the site apart from /private/, whose press folder stays open, and ExampleBot may read nothing. If the file is missing and the server answers with a 4xx status such as 404, no rules exist and a crawler may fetch any public page. If the server errors or cannot be reached, the crawler must assume everything is disallowed. It should not rely on a cached copy older than 24 hours.
What it does not do
It is not security. The file is public, so listing a path advertises it, and a crawler that ignores it meets no technical barrier; the RFC states that its rules are not a form of access authorisation. It also says nothing about reuse: it neither grants nor refuses a licence for what a crawler has read. It controls crawling, not search listings, so a blocked page that other sites link to can still appear in a search engine's results, and a page meant to stay out of them needs a noindex directive or a login. Crawl-delay, which some crawlers honour to space their requests, is not part of the standard.
In Fokals data
Fokals data is collected from first-party company sources and public records, and processed in-house, with every observation dated. The sourcing statement sets out the collection rules.
Related terms
robots.txt governs a web crawler. How a vendor treats it is a standard question in a due diligence questionnaire and part of its data provenance, alongside the question of whether its sources are first-party data.
Frequently asked questions
Does robots.txt stop a crawler from reading a page?
Only if the crawler chooses to obey it. The file is a request with no technical enforcement, and RFC 9309 says it is not a substitute for content security measures. A page that must stay private needs a login or another access control. Because the file is public, listing a path in it also tells anyone who looks where that path is.
What happens if a site has no robots.txt file?
A missing file, one that returns a 4xx status such as 404, means the site states no rules, and the standard lets a crawler fetch any public page. A file that cannot be reached because of a server error or a network failure is treated differently: the crawler must assume everything is disallowed until it can read the file. Good conduct still means limiting the request rate and reading public pages only.
Where does the robots.txt file go?
At the top level of each host, such as https://example.com/robots.txt, named in lower case. A subdomain such as blog.example.com is its own host, so it has its own file at its own top level. The file is UTF-8 plain text, and a crawler must read at least its first 500 kibibytes, so keep the rules inside that size.
The queries and code on this page are examples to adapt. Test them in your own environment before you rely on them.