Enriching company records in your data product

Your product holds company records: a name, a domain, perhaps a ticker. Enrichment attaches to each record what a user wants to know without leaving your screens, such as the stack a company runs, whether it is hiring and what it announced last week. This guide is for the product lead and the engineer who build that. It covers how to match records on domain, which fields to add first, how to keep them fresh, how to show them honestly and what to settle in a licence that allows embedding.

Match first: domain, then identifiers

Fokals keys every company by one stable Fokals company ID, company_id, and every dataset carries it. Your records hold a domain and sometimes a ticker or an ISIN. The domain column appears in Technology Stack (company_technologies) and Web Traffic (traffic_ranks), each beside company_id, so a domain map is a distinct list of those pairs. For listed companies, listed_securities holds isin, lei, ticker, mic and figi with the company_id of the company each listing belongs to, matched by ISIN, then LEI, then ticker on the same market, then name and country.

Your own domain matching should follow a similar order: exact domain after normalisation, then identifiers for listed companies, then a review queue for what is left. Measure the match rate on your own records before you rely on the map: it tells you how much of your book each panel will fill.

with domain_map as (
  select domain, company_id from company_technologies
  union
  select domain, company_id from traffic_ranks
)
select
  r.record_id,
  d.company_id
from my_company_records r
left join domain_map d
  on d.domain = lower(regexp_replace(r.website, '^(https?://)?(www\.)?([^/]+).*$', '\3'));

Normalise before you match: lower-case, strip the scheme, path and a leading www., and convert internationalised names to one form. Store the key and how you got it: company_id, the basis (domain, identifier or manual review), the date and the version of the file you matched against. Once matched, company_id is the join key. It is stable, so store it and not the domain.

Three cases need rules. A brand site matches the brand's own company_id and carries the identifiers of its listed parent, so group on isin or figi if your users think in groups. A domain that belongs to no company in the index returns no match, and your interface should say so plainly instead of showing empty panels. A record whose domain is a redirect, a link shortener or a free mailbox domain needs a person.

Measure the match rate twice, as a share of all records and as a share of your active accounts, since the two differ. Review a sample of matched pairs by hand against company and the site title in Website Profile (company_site_facts), and keep the records with no match in a queue instead of dropping them.

Which fields to add

Start with the groups that answer a question your users already ask. Announcements and hiring need the least explanation, and intent needs careful wording, set out below. Decide early how a parent account shows its brands: a user looking at a listed group expects the postings and announcements of its brand sites too, so roll the rows up on isin, and show the brand's name beside each item so that a user can see where it came from.

Every field above is company-level: what a company runs, hires for, announces and scores on. A leadership change in Company News is delivered as the role concerned and the direction. A panel that also shows other kinds of data takes them from the providers of that data, and the Fokals company ID joins them to the same account.

The website labels, which cover industry, business model, growth stage and more, are listed in the data dictionary as labels carried in the dashboard and the data browser. Confirm what your delivery includes before you design a screen around them.

Keeping the records fresh

Load the history once from bulk files, then follow the feeds. Feeds run oldest first from a time you set, so the last cursor you stored is your bookmark, and the next call starts where the last one ended. Daily and weekly rows are written once after the period closes and never changed, so you have no revisions to chase, and an upsert on a row's natural key is safe to repeat. A period written more than seven days after it closed carries reconstructed=true, and you should keep the flag. The guide to keeping a warehouse in step with incremental API sync works through the pattern.

Keep a copy of the fields you show, refreshed by sync, instead of calling the API from a user's page. Each key has scopes and rate limits by minute and by day, which a busy screen can reach, and a stored copy keeps your product working while a call is retried.

Plan for versions. Labels and scores are produced under named, frozen versions, among them jobs-v2, news-v1 and intent-v2, and a breaking change ships as a new version with at least 90 days' notice. Postings keep the version they were labelled under until they close, so one table can hold two versions at once. Store label_version with every value you copy, show or filter on it, and treat a new version as a migration with a date and not as a silent change.

Let refresh follow the source, not the screen. A technology panel that says a tool was last seen three days ago is honest about a refresh that runs daily to weekly, and a headcount shown with its date is honest about a yearly one. A weekly intent score should not be drawn as a live meter.

Showing the data honestly

• Provenance. Each record carries its source and the time it was observed. Show both where a user will ask where a fact came from: the job board, the newsroom, a regulatory disclosure.
• Attribution. Each Web Traffic row carries its attribution text in attribution. Show it wherever the monthly traffic tier is shown.
• Intent wording. A score is evidence-backed: every score carries the dated signals behind it. Show evidence, the five strongest signals, beside the score so that a user can see why it moved. The surge flag marks a score of at least 50 that is at least double the company's own average for the topic over the previous twelve weeks. The guide to powering account scores in a sales platform covers scoring in depth.
• Missing is not zero. A record with no match has no data yet, and a company that publishes no roles shows no hiring. Say that there is no data, not that the value is 0.

The licence for embedding

Fokals licenses data by written agreement for internal use, for embedding in a product or for redistribution. A feature that shows a value inside your product and a feature that hands rows to your customers are different uses, and the agreement says which of the three applies to each. The points below are for you to settle before you build, because each changes what you can store and show. The terms themselves are in the agreement and not on this page.

• Scope. Which datasets and fields are covered, and whether datasets added later are in scope.
• Display and export. Whether users see rows, derived scores or both, and whether they may export them.
• Retention. How long you may keep data in your product, and what happens to stored copies when the agreement ends.
• Attribution. Where the attribution text and any source notes appear.
• Change notice. Label versions are frozen and a breaking change carries at least 90 days' notice: confirm how the notice reaches you and who acts on it.
• Manifest. Each bulk export carries a manifest naming sources, period, label versions and licence. Keep it with the files as your record of what you received.

The page on data licensing for platforms is written for products that embed or redistribute company data. The guides to redistributing company data under licence and what a data licence covers go through the three uses, and the data dictionary defines every column named above.

Frequently asked questions

The queries and code on this page are examples to adapt. Test them in your own environment before you rely on them.