Adding technographic filters to a prospecting tool

A technology filter looks like a dropdown over a column of tool names. The work is in the definitions behind it: what "runs it now" means, what "first seen" and "recently added" mean, and how to answer three filters at once without a full scan. This guide gives product and data engineers a definition for each filter and an index to serve it, built on the three datasets of the marketing stack: Technology Stack, Technology Changes and Website Profile.

Technographic data records which technologies a company runs on its own website, and Fokals dates every change to it. The filters return companies, so a prospecting tool built on them lists accounts, each with its technologies, its dated changes and its markets, ready to be taken into a CRM.

What the three datasets give a filter

The datasets sit at three grains, all described in the data dictionary. The table gives each by name, with the identifier you type in a query.

The vocabulary is 6,283 recognised technologies in 68 categories. The technology ID is stable and is the value a filter should store, while the technology name is for display, so a saved search survives a change of label. One company can have several websites, and Technology Stack and Website Profile are per website. Decide for each filter whether it answers for the site or for the company, and collapse on the company ID for the company view. A brand is a company in the index that carries the identifiers of its listed parent, so group on ISIN when a user wants the whole group and keep the brands apart when they want one company.

The filters, defined exactly

In the definitions below, the identifiers are the ones the queries use.

Runs it now. technology equals the chosen ID and missing_since is empty. A row with missing_since set is absent from recent readings and not yet counted as removed. Serve it as a second state, "not seen recently", and let the user decide whether to include it.

In a category. Filter on technology_category. The harder form is negative: companies that run something in one category and nothing in another. Do not compute that as an anti-join at query time. Precompute each company's categories as a list and test against the list.

First seen between two dates. Filter on first_seen_at, and call it "First seen" in the interface, never "Adopted". The first observation of a website sets a baseline and writes no changes, so every technology already on a newly observed site carries the date of that first observation, which is a record of when Fokals first saw it.

Recently added or removed. Filter Technology Changes on change being added or removed, on category being technology (or dns for tools recognised from public DNS records) and on observed_at inside the window. Only a change observed between two readings writes an event, so this is the filter that means a company adopted or dropped something. A removal is confirmed before it is written, so a tool that comes and goes with a consent banner or a test does not flicker in your results.

Has a public account ID. account_ids holds the public IDs found with a technology: advertising pixel IDs, analytics properties and tag-manager containers. Filter on presence for "has an advertising account", and read the number of IDs as a rough sign of how many accounts a company runs. A new ID under a technology already in place is its own event, with category set to technology_id, which lets a product tell a user that a company has added a new account ID to a tool it already used.

-- companies that added a tool in a chosen category in the last 30 days
select e.company_id, e.technology_name, e.observed_at
from company_tech_events e
where e.category = 'technology'
  and e.change = 'added'
  and e.technology_category = :category
  and e.observed_at >= now() - interval '30 days'
order by e.observed_at desc;

A serving table and its indexes

Do not run filters against the raw datasets. Build one serving table for companies and one for events, and index them for the way filters combine, which is set logic: has all of these, has none of those, has anything in this category.

create table company_stack (
  company_id   text primary key,
  domains      text[],
  running      text[],      -- technology ids with missing_since empty
  unconfirmed  text[],      -- technology ids with missing_since set
  categories   text[],
  last_read_at timestamptz
);
create index on company_stack using gin (running);
create index on company_stack using gin (categories);

create table stack_event (
  company_id          text,
  technology          text,
  technology_category text,
  change              text,
  observed_at         timestamptz
);
create index on stack_event (technology, change, observed_at desc);
create index on stack_event (technology_category, change, observed_at desc);

A request for companies that run A and B but not C becomes running @> array['a','b'] and not running && array['c'], answered from the inverted index without touching the events. If your store is a search engine instead of a SQL database, the design is the same: one document per company with the running technologies and categories as keyword lists, and the events in a second index.

Facet counts, the numbers beside each option, change only when a load lands. Recompute them at the end of each load and serve them from a table, because counting live under every filter is avoidable work. An option can also carry its own trend. Market Series include technology added and technology removed families with a technology as the subject, per 100 websites on same-store cohorts over 7, 30 and 180 days, so a dropdown can show how fast a technology is spreading without counting your own events.

Keeping the index current

The marketing stack is refreshed daily to weekly, depending on the company, as the methodology sets out. Show the last-read date of the website beside every row and let a user filter on it, so a result carries its own freshness.

Apply each load as a change set. Follow the events since your bookmark, as the guide to incremental API sync describes, and rebuild only the serving rows of companies whose state or events changed. When a company joins your list for the first time, load its state before its events: its first observation wrote no events, so a new company has a stack and begins its history of changes from that point.

A saved search is a filter definition plus a bookmark. To tell a user what is new, run the definition over the events since the bookmark instead of repeating the whole search and comparing the two result lists.

What the interface should say

Three things make a result credible: how a technology was detected, when, and what a detection means.

• How. seen_via lists the detection views: page content, script and network requests, response headers or public DNS records. Show it. A tool seen only in DNS records is evidence of a relationship with a software vendor, such as a domain verification or a mail provider, and the user should know that is what they are looking at.
• When. Show first seen, last seen and, for a change, the observation that recorded it.
• What a detection means. A detection shows presence on the company website, so "detected" reads as "the company runs it on its site". Pair it with the date and the detection view and a user can judge it at a glance.

For finding accounts from the same datasets outside a product, see the guide to finding accounts by the technology they run.

Testing the definitions before launch

Four checks on a sample for the companies you track settle most disputes about a definition before users find them.

1. Baseline. Take a website that has no added events. It should appear under "First seen" and never under "Recently added".
2. Removal. Take a technology with a removed event and confirm that your serving table does not list it as running. Check how a removed technology appears in your copy of Technology Stack, and where its row is still there, let the later removal event win.
3. Provenance. Sample twenty rows, open each website, and record how many show the technology in the page source or the network panel. Use seen_via to explain the differences: a technology seen in DNS records appears in neither place.
4. Freshness. Plot last_read_at across the sample. The spread follows the refresh cadence of each company, from daily to weekly.

Licensing the data into a product

Showing Fokals data to your own customers inside a product is embedding. Fokals licenses internal use, embedding in a product and redistribution by written agreement, and the data licensing page describes the three. Settle which applies before the filter ships, because the choice bears on what users may export from the results.

Frequently asked questions

The queries and code on this page are examples to adapt. Test them in your own environment before you rely on them.