Data provenance is the record of where data came from, when and how it was collected, and what has been done to it since. A record with provenance answers three questions: what was the source, when was it observed, and which process produced each value you are reading.
What a provenance record holds
- Source. The page, filing or system the value came from.
- Observation time. When it was seen, which is not the same as when it was published.
- Method. The collector, the rules and any model used, with their version.
- Transformations. Conversions, matching and aggregation applied since.
- Terms. The licence or attribution the source requires.
Provenance has to travel with the record. A statement at dataset level, such as sourced from public websites, answers a questionnaire but not the question an analyst asks about one value on one date.
Why it matters
Due diligence needs it to show that data was public when collected, gathered within the source's rules and free of personal data. Debugging needs it to find which source or label version produced a surprising value. Open licences need it: a licence such as CC BY requires the credit to travel with the data. It also keeps what a source stated apart from what a model inferred from it, and the two should never share a column unlabelled.
Provenance in Fokals data
Fokals data is collected from first-party company sources and public records and processed in-house. Each record carries the time it was observed, and every observation is written once and never revised. Company News keeps the link to the original announcement or disclosure. Company Signals states the kind of each signal, and Technology Stack gives the first-seen and last-seen dates of each technology.
Derived values carry the frozen version that produced them: labels and scores are produced under named versions, and Job Postings records the label version of each posting. Where a source requires a credit, the credit travels in every row, and a bulk export arrives with a manifest naming sources, period, label versions and licence. See the sourcing statement and the data dictionary.
Related terms
Provenance for model-made values rests on a label version, and a bulk export states it in a data manifest. The two sides of sourcing are first-party data and third-party data.
Frequently asked questions
What is the difference between data provenance and data lineage?
Lineage usually describes how data moves and is transformed through your own systems, from source tables to the outputs built on them. Provenance starts earlier, at the original source, and adds when and how the data was obtained. In practice the terms overlap, and what matters is that the information is recorded for each record, not once for a whole file.
How do you check the provenance of a dataset you buy?
Ask the vendor for a source statement and a description of the method. Then look in the data: does each record name its source and show the time it was observed, and does a derived value name the version that made it? Trace a sample of values back to the original page or filing and see whether they match. Ask what the licence says about attribution, because some sources require it.
Does provenance tell you the data is accurate?
No. Provenance says where the data came from and how it was made, so that you can judge it. It does not show accuracy by itself: a value copied faithfully from a wrong source has perfect provenance. Accuracy needs a check against a reference, such as a hand-graded sample, and a measure that is stated per version of the method.