Topic
Data engineering
Loading, modelling and maintaining external data in a pipeline.
- The identifiers that make company data joinableCompany data is only useful once it joins to what you already hold. Here are seven identifiers, what each one names, the join each one serves and how a brand reaches its listed parent.
- Building a company knowledge graph from identifiers and eventsA company graph answers questions that span datasets. This guide gives the node and edge tables, the join that finds a brand's listed parent and the way to link your own records.
- Keeping a warehouse in step with incremental API syncA daily sync that resumes after a failure, runs twice without harm and notices a period that arrives late. The design, table by table, with the keys and the checks.
- Retrieval-augmented generation over structured company signalsMost questions about companies ask for a count, a date or a list, which a database answers exactly. This guide shows what to embed, what to filter and what to leave to SQL.
- Apache Iceberg tables for licensed external dataA table format decides how licensed files are stored, partitioned, loaded again and read back. This guide works through those choices for Fokals bulk exports.
- AWS Glue pipelines for licensed external dataLand the files, catalogue them without letting a crawler rewrite your schema, process only new days with job bookmarks, and check every load. Worked on the Fokals tables.
- Bringing external company signals into HubSpotFokals is delivered direct, and your pipeline writes it to HubSpot through the CRM API. Here is the property design, the domain match, the batch update and the traps, with SQL and the HubSpot calls.
- Exploring bulk company data with DuckDBRead a sample file with read_csv or read_ndjson, then run seven checks on grain, dates, nulls, JSON cells, labels and baselines. The queries are written for the Fokals tables.
- Feature engineering on company signals in DatabricksWhich timestamp to key a feature table on, how to join labels as of a date, and the leakage traps left in daily and weekly company data, with worked SQL and Python.
- Governing licensed data with Unity CatalogA worked Unity Catalog design for licensed data: a catalog per agreement, group grants, licence tags, lineage and share checks, with the limits of each control stated.
- Ingesting a vendor API with Fivetran or AirbyteFokals is delivered direct by REST API and bulk files. This guide maps a cursor API onto the Fivetran Connector SDK and the Airbyte Connector Builder, setting by setting.
- Joining licensed company data to CRM tables in SnowflakeA worked join model in Snowflake SQL: a domain normaliser, a bridge to the company ID, subdomain matching, a match-rate report and a view for sales teams, with the pitfalls named.
- Loading company data into Amazon RedshiftWorked SQL for loading company data into Amazon Redshift: a table definition, COPY from S3, the SUPER type for JSON cells, and a daily incremental load that can run twice without duplicates.
- Loading company data into BigQueryA worked path from files in a bucket to partitioned BigQuery tables: LOAD DATA with an explicit schema, JSON columns for the object cells, MERGE for daily files, and checks after each load.
- Loading company data into Databricks with Auto LoaderA worked load of Fokals files into Databricks: landing in a volume, bronze with Auto Loader, typed silver tables kept free of duplicates with MERGE, and a gold join to your identifiers.
- Loading company data into Oracle Autonomous DatabaseA worked load of company CSV and JSON files into Oracle Autonomous Database with DBMS_CLOUD, with the format options that decide the result, the log tables, a load pipeline and JSON queries.
- Loading company data into Snowflake from files and an APIA worked pipeline from delivered files and API pages to history tables: stages, loading CSV by header name, JSON Lines into VARIANT, and an idempotent daily MERGE.
- Microsoft Fabric: bringing external company data into OneLakeWhere external company data goes in Fabric and how it gets there: lakehouse or warehouse, files kept as they arrive, loaded into Delta tables on a daily schedule.
- Modelling licensed company data with dbtA working dbt layout for licensed company data: sources, staging models, tests on keys and versions, freshness set from each table's closing rule, and the one table worth a snapshot.
- Querying company data on Amazon S3 with AthenaHow to query bulk company files with Athena: JSON Lines external tables over your own S3 prefixes, daily partitions with partition projection, and converting to Parquet with CTAS.
- Serving company signals from ClickHouseDesign ClickHouse tables for company change events, signals and weekly scores: which engine, which ordering key, and how a second access path is added when a second query needs one.
- API vs bulk files for company data deliveryMost teams need both: files to build the copy, an API to keep it current and to answer look-ups. How to assign each job to a route, with the HTTP and file-format rules that decide it.
- Building crawlers vs licensing company dataA crawler framework fetches pages, and company data needs more work after the fetch. What building takes, what a licence delivers in its place, and how the two combine.
- CSV vs JSON Lines vs Parquet for bulk data deliveryThree file formats, three sets of trade-offs. What each keeps and loses when a vendor hands you a dataset as files, and a tested way to turn a CSV export into typed Parquet.
- Databricks vs Microsoft Fabric for external company dataTwo lakehouse platforms set side by side for a vendor's daily files: Delta Lake with Unity Catalog beside OneLake, OpenSharing beside external data sharing, and the tools each gives you to fence licensed data.
- Fivetran vs Airbyte for ingesting a vendor APIA vendor's REST API reaches your warehouse through a connector you build. How the two tools differ, as each documents itself: where the code runs, what you write, where the bookmark lives and how rows are counted.
- Microsoft Fabric vs Snowflake for external company dataOne daily vendor file taken into a Fabric warehouse and into Snowflake: the two COPY INTO dialects, a JSON cell on each, what each bills for, and how the platforms read each other's tables.
- Snowflake vs BigQuery for external company dataOne vendor file loaded into both warehouses: COPY INTO beside LOAD DATA, VARIANT beside JSON, shares beside linked datasets, and what each documented pricing model means for a daily feed.
- Snowflake vs Databricks for licensed company dataOne daily vendor file taken through both platforms: stages and COPY INTO beside volumes and Auto Loader, VARIANT on each, then sharing, governance and AI functions as each vendor documents them.
- Alternatives to building your own company data crawlersFour routes to rows of company data from the web: your own crawler, a managed scraping service, open datasets or a licensed feed. What each takes off your hands, and what stays with you.
- Applicant tracking system (ATS)An applicant tracking system is where employers publish jobs and manage candidates. This entry covers what it means for hiring data, how it shows in hiring data and a common trap.
- Bulk exportA bulk export hands over a dataset as files, not as individual calls. This entry shows when to use one, what to check when it arrives and how Fokals delivers bulk files.
- Cursor paginationCursor pagination marks where a page ended so the next call can resume from there. This entry compares it with offset paging and shows how to use it safely.
- Data lakehouseA data lakehouse puts a table layer over open files in object storage. This entry explains the parts, how vendor files are landed, and why time travel is not point-in-time data.
- Data warehouseA data warehouse stores integrated, historical data modelled for analysis. This entry explains grain and keys and shows how vendor files fit a warehouse model.
- Entity resolutionEntity resolution decides which records describe the same company and links them under one key. This entry covers the order of matching, how to measure it and the rules Fokals publishes.
- Incremental syncIncremental sync fetches only the records added since the last run. This entry shows the bookmark, the load and the failure modes, with an example on daily company data.
- JSON LinesJSON Lines holds one JSON record per line, so large exports can be streamed and appended to. This entry gives the rules, a worked example and the traps.
- Rate limitA rate limit caps the requests a client may send in a period. This entry shows how to plan a backfill against minute and daily limits and how to act after a refusal.
- robots.txtrobots.txt is the file at the root of a website that tells crawlers which paths to avoid. How its rules are matched, what happens when it is missing, and why it is not security.
- Web crawlerA web crawler visits pages automatically and records what it reads. How crawlers work, how they differ from scrapers and what separates responsible conduct from harmful.