To document data lineage for a licensed external dataset, keep one record for every delivery and tie it to the agreement that governs the data. The record names the licence and its permitted use, the delivery batch and the period it covers, a checksum for each file, the schema version and every job that transforms the data after it lands, with the tables each job writes.
This post sets out what each part of the record holds and maps it to two published standards: the OpenLineage specification (documentation version 1.55.0) and the Data & Trusted AI Alliance Data Provenance Standards (version 1.0.0). It then fills in a lineage record for an illustrative delivery and ends with a checklist a governance team can run on any third-party table. Every source was read on 15 November 2026.
How do you document data lineage for a licensed external dataset?
Document lineage for a licensed external dataset in five layers, recorded per delivery: the licence and permitted use, the delivery batch, the file checksums, the schema version and the transformations that follow. The first four are fixed when the files arrive. The fifth grows each time a job reads the data, so the jobs must write it.
| Layer | What to record | What it lets you prove |
|---|---|---|
| Licence | Agreement ID, licensor, permitted use, attribution owed, term end, deletion deadline | Each copy sits inside the terms |
| Delivery | Provider's batch ID, period covered, issue date, previous issue date, file list, row counts, landing path | A row in your warehouse came from one shipment |
| Integrity | A hash digest of each file and of the manifest, checked on arrival | The files you loaded are the files that were sent |
| Schema | Provider's schema version, columns and types, any definition versions carried in the rows | What each column meant on the day it was loaded |
| Transformations | Job name, run ID, start and end time, code version, tables read and written, columns used | Where every copy and derivative went |
The licence and permitted use
Start with the agreement, because it decides what every later table may be used for. A data licence can grant internal use, embedding or redistribution, and one agreement can grant more than one of them. What a data licence covers sets out how each scope works. Store the licence ID on the dataset itself, as a table property or a column, so the terms travel with every copy.
The delivery batch
Each delivery gets its own entry with the fields in the table above. If the provider sends a manifest, keep it next to the files and record its location. A manifest is the provider's own statement of what it sent, so it is the reference your checks run against.
Checksums for each file
Compute a hash digest of every file when it lands, for example SHA-256, and compare it with the digest in the provider's manifest when the manifest carries one. NIST's Secure Hash Standard, FIPS 180-4, describes message digests as a way to detect whether a message has changed since the digest was generated.
A mismatch stops the load. Record both values and the result of the comparison, so the record shows that the check ran and what it found.
The schema version
Record the provider's schema version and the columns and types as delivered. Some providers also version single definitions, such as a score or a label, inside the rows, and those versions belong in the record too.
Compare each delivery's columns with the last one and log any difference before the load, since a column can keep its name after its definition changes.
Every transformation after it lands
Every job that reads the landed files adds an entry with the fields in the table above. Where your tooling can see it, record which output columns come from the licensed input.
This layer answers the questions a licence raises later. It shows which dashboards display the values, which customers received them and which models were trained on them.
Why licensed data needs more than ordinary lineage
A table built from a licensed dataset inherits the licence's limits on use, attribution and retention, so the lineage record is how a team shows that each copy sits inside the permitted use.
When a contract ends, the same record becomes the list of places to clear. What happens to licensed data when a contract ends covers the destruction clause, derived data and certificates of destruction.
Regulation can ask for the same record. For high-risk AI systems trained on data, Article 10(2) of the EU AI Act, Regulation (EU) 2024/1689, requires data governance practices that concern "data collection processes and the origin of data" and data-preparation operations such as cleaning, enrichment and aggregation. A per-delivery record answers both points for an external dataset. This post is general information only, and the open points on any agreement or regulation belong with your counsel.
How the record maps to the OpenLineage specification
OpenLineage is an open specification for lineage metadata. Its object model defines three kinds of event: a RunEvent sent while a job runs, a JobEvent for design-time job metadata and a DatasetEvent for dataset metadata such as schema or ownership. A dataset is named by a namespace and a name derived from its physical location, and extra metadata travels in facets.
A run moves through the states START, RUNNING, COMPLETE, ABORT, FAIL and OTHER, and the run cycle page says no events should follow a terminal state. The object model puts static metadata in dataset facets and per-run detail, such as the partition a job read, in input or output facets. That split suits licensed data well: the licence and the owner are static, while the batch and its checksums belong to the run that loaded them.
| Layer | Where it goes in OpenLineage | Note |
|---|---|---|
| Licence | A custom dataset facet | We found no licence facet in the dataset facet list for 1.55.0 |
| Provider | Datasource facet, dataSource | Carries a name and a uri in its JSON schema |
| Steward | Ownership facet, ownership | A list of owners, each with a name and a type |
| Delivery | Version facet, version, plus a custom input facet | datasetVersion holds the batch or issue date |
| Integrity | A custom input facet | Digests and the result of the check |
| Schema | Schema facet, schema | Fields with a name, a type and a description |
| Transformations | One RunEvent per job, with inputs and outputs | The Column Level Lineage facet adds column detail |
| Deletion | Lifecycle State Change facet | Records a change in a dataset's lifecycle state |
A custom facet must carry _producer and _schemaURL. To use your own structure, point _schemaURL at a JSON schema you define and make accessible.
What the Data & Trusted AI Alliance provenance standards add
The Data Provenance Standards from the Data & Trusted AI Alliance (D&TA, formerly the Data & Trust Alliance), version 1.0.0, set out 22 metadata fields in three standards: Source (6 fields), Provenance (8) and Use (8). The page was last updated on 9 July 2024 and says further updates will come through a practitioner community managed by the standards body OASIS.
OpenLineage records how data moved between jobs and tables, while the D&TA fields record where a dataset came from and the terms on its use. That makes the D&TA fields a good template for the licence facet. We found no statement on the standards page of which fields are required, so the table lists the ones that matter for a licensed delivery.
| D&TA standard | Field | What to put for a licensed delivery |
|---|---|---|
| Source | Standards version used | 1.0.0 |
| Source | Unique metadata identifier | The ID of this lineage record |
| Source | Metadata location | The record's address in your catalogue |
| Source | Dataset issuer | The provider |
| Provenance | Dataset issue date | The delivery date |
| Provenance | Date of previously issued version of the dataset | The previous delivery date |
| Provenance | Range of dates for data generation | The period the delivery covers |
| Use | License to use | The agreement ID and its scope |
| Use | Intended data use | The use your team has approved |
| Use | Proprietary data presence | Yes, for licensed commercial data |
A worked example: one delivery of a licensed dataset
Assumptions: Acme Robotics licenses a weekly company dataset from an outside provider for internal use only. The provider delivers each week as CSV files with a manifest. Acme lands the files in object storage, checks them and loads a warehouse table, then a second job joins that table to its CRM accounts. Every name, identifier, date, count and digest below is illustrative, and the digests are shortened.
The load job sends this COMPLETE event. The event's own producer and schemaURL fields, and those of the standard facets, are left out for space.
{
"eventType": "COMPLETE",
"eventTime": "2026-11-09T06:42:10Z",
"run": { "runId": "01937c4e-5a2b-7f10-9c3d-2e8b6a4f1d07" },
"job": { "namespace": "acme-warehouse", "name": "load_vendor_company_weekly" },
"inputs": [
{
"namespace": "s3://acme-landing",
"name": "vendor/company_weekly/issue=2026-11-09",
"facets": {
"dataSource": { "name": "vendor-bulk-export", "uri": "https://vendor.example/exports" },
"version": { "datasetVersion": "2026-11-09" },
"schema": { "fields": [
{ "name": "company_id", "type": "string", "description": "Provider's stable company ID" },
{ "name": "score", "type": "int", "description": "0 to 100, definition v3" }
] },
"acme_licence": {
"_producer": "https://acme.example/lineage/loader/1.4",
"_schemaURL": "https://acme.example/schemas/licence-facet-1-0.json",
"agreement_id": "LIC-2026-014",
"permitted_use": "internal",
"term_end": "2027-10-31",
"deletion_deadline_days": 90,
"dta_standards_version": "1.0.0",
"dta_dataset_issuer": "Vendor Ltd",
"dta_issue_date": "2026-11-09",
"dta_previous_issue_date": "2026-11-02",
"dta_data_generation_range": "2026-11-02/2026-11-08",
"dta_intended_data_use": "account scoring for internal sales planning"
}
},
"inputFacets": {
"acme_delivery": {
"_producer": "https://acme.example/lineage/loader/1.4",
"_schemaURL": "https://acme.example/schemas/delivery-facet-1-0.json",
"batch_id": "B-2026-45",
"provider_schema_version": "4.1",
"manifest_sha256": "7c0e...91ab",
"files": [
{ "name": "part-0001.csv", "rows": 48210, "sha256": "4f1a...c9e2", "manifest_match": true }
]
}
}
}
],
"outputs": [
{ "namespace": "warehouse.acme.example", "name": "raw.vendor_company_weekly" }
]
}The licence sits in a dataset facet because it holds for every delivery under the agreement. The batch, the provider's schema version and the digests sit in an input facet because they describe what this run read.
The join job then sends its own event. Its input is raw.vendor_company_weekly, its output is crm.accounts_enriched and Acme's join code writes the same acme_licence facet on the output. When the agreement ends, a search of the lineage store for LIC-2026-014 returns the landing path, the raw table and the enriched table, which is the starting list for the deletion.
On the provider side, Fokals bulk exports come with a manifest naming the period, the data versions and the licence, and each file that names a company carries the stable company ID set out in the data dictionary. Daily and weekly datasets are written once after the period closes and never revised, as set out in why we write each table once, so the rows for a closed day or week match in every later delivery of that period. If you load into Apache Iceberg, tag the snapshot each delivery creates so it is kept for audit, as described in Apache Iceberg tables for licensed external data.
What are the best practices for documenting lineage of third-party data?
The best practices for documenting lineage of third-party data are to record at the point of delivery, keep the licence attached to the data and let jobs write lineage as they run. A record built by hand after the fact misses copies, and a record without the licence cannot say what each copy may be used for.
- Record every delivery as its own entry. A dataset-level entry cannot say which batch a row came from.
- Put the agreement ID on the data as a table property or a column, and carry it to every derived table.
- Check before you load: compare digests and row counts with the manifest, diff the schema against the last delivery and record the result.
- Keep the landed files read-only and transform copies in separate jobs, so the files you received remain evidence of what was sent.
- Emit a lineage event from every job that reads the licensed data instead of documenting by hand.
- Record derived outputs. Scores, aggregates and trained models built from licensed rows are derived data, and the agreement decides who may keep them after the term.
- Log deletions as lineage events, with the date and the system cleared.
Lineage has limits worth stating. It sees only systems that emit events, so a file exported to a laptop or values pasted into a spreadsheet leave no trace in it. Column-level detail depends on the integration that parses each job. A lineage record also says nothing about whether the provider had the right to collect the data, which belongs in the due diligence on the provider.
Governance best practices for the provenance of licensed external data
Good governance of provenance for licensed external data gives the record an owner, a home and a review cycle. One named steward answers for each licensed source, the provenance record lives in the catalogue next to the table and the record is checked against the agreement at each renewal and when the term ends.
The steward approves new uses. Before a team uses the data for a new purpose, such as a customer-facing feature or model training, the steward compares it with the permitted use in the record and logs the approval with a date.
At each renewal, compare the signed agreement with the record field by field, and keep the dates each version of the terms applied.
At the end of the term, search the lineage store for the agreement ID, clear each system it returns and record each deletion with its date. Keep the provenance record afterwards as evidence of what was delivered and where it went.
A checklist for any third-party table
Run these tests on any table that holds third-party data. Every no is a gap to close.
- The table carries an agreement ID that resolves to a signed licence in your contract register.
- The permitted use on that licence covers every current consumer of the table.
- Every attribution the licence requires appears where the values are shown.
- Each row can be traced to one delivery batch with its period and issue date.
- Each delivered file has a recorded digest that matched the provider's manifest on arrival.
- The schema version of each delivery is recorded, and the last schema change was reviewed before load.
- Every job that reads the table emits lineage events with a run ID, its inputs and its outputs.
- Every derived table, score and model carries the same agreement ID.
- The table has a named steward, and its provenance record has an address in the catalogue.
- The term end and the deletion deadline are recorded, with an owner for the deletion.
- A search of the lineage store for the agreement ID returns every copy, checked once a year against a manual search of shared drives and BI tools.
Frequently asked questions
What is the difference between data lineage and data provenance?
Lineage traces how data moves and changes inside your systems: which jobs read it and which tables they wrote. Provenance describes where a dataset came from and on what terms, such as the issuer, the collection method and the licence. Licensed external data needs both, which is why the worked example puts D&TA provenance fields inside an OpenLineage facet.
Does OpenLineage have a standard field for a data licence?
We found no licence facet among the dataset facets listed in OpenLineage documentation version 1.55.0, read on 15 November 2026. The specification allows custom facets, which must carry a _producer and a _schemaURL. A custom licence facet with a JSON schema you publish can carry the agreement ID, the permitted use and the term.
Can a provider's manifest replace your own lineage record?
A manifest describes what the provider sent, such as the period, the versions and the licence. It says nothing about what happened after the files landed. Keep the manifest as one input to the record, check your digests against it and add the transformation layer yourself.
Should lineage cover models trained on licensed data?
Yes. A trained model is derived data, and the licence decides whether it may be built and kept after the term. Record the training run as a job with the licensed tables as inputs and the model version as the output, so a change in the licence can be traced to every model it touches.
How long should you keep the lineage record after the data is deleted?
The agreement and any record-keeping rules that apply to your organisation set the period, so ask your counsel. Keep the record long enough to show what was delivered, where it went and when each copy was deleted, since that is the evidence a provider may ask for at the end of the term.
The queries and code on this page are examples to adapt. Test them in your own environment before you rely on them.
What this page says about the products it names was checked against their public documentation on 15 November 2026. Product and company names are trademarks of their owners. Fokals is not affiliated with them or endorsed by them.