FokalsBot
What the Fokals crawler reads, how it behaves, and how to stop it.
User-Agent: Mozilla/5.0 (compatible; FokalsBot/1.0; +https://www.fokals.com/bot)
What it is
FokalsBot is the crawler of Fokals, a business information service. It reads public company homepages and the job listings companies publish on their job boards, to describe which marketing and business technologies companies use and how their hiring changes over time.
What it reads
- A company's homepage, and its careers page when the homepage links to one.
- A company's newsroom or press page when the homepage links to one, and the RSS or Atom feeds the homepage declares, to record the company's own announcements (title, date and a short excerpt).
- Websites linked from those pages, read once to see whether they belong to a company.
- The public tag-manager containers a homepage loads.
- Public job-board listings a company publishes for distribution.
- Public DNS records (mail servers and TXT records) of the company's domain.
- For listed companies, public SEC EDGAR filings (Form D, Form 8-K and the annual report), within SEC's fair-access rules.
It does not log in, fill in forms, read pages behind a sign-in, or collect personal data. Contact details found in job descriptions are removed before anything is stored.
How it behaves
- It reads and obeys robots.txt, including a Crawl-delay.
- It makes one request at a time to a host, at least 1.5 seconds apart.
- It reads most homepages once a day to once a week.
- It never changes its identity. When a site's network protection refuses the direct read, the page may be fetched through a proxy service; robots.txt is read and obeyed on that path too, and a robots.txt Disallow always stops it. A site that refuses on every path is asked again only a month later.
How to stop it
Add this to your robots.txt. FokalsBot reads robots.txt again at least once a day.
User-agent: FokalsBot Disallow: /
To have information about your company removed, or for any other question, use the contact form.
How the data is sourced, and what is never collected: sourcing and compliance.