Alternative data delivery formats hedge funds expect from a vendor

For hedge funds, there is more to check in an alternative data vendor than a choice of delivery formats. Before any research starts, a fund's data engineering team checks six parts of a delivery: the route it travels by, the file format, the manifest, the schema and its versions, the timestamps, and the way a correction arrives. This post sets out each check, with the standard behind it where one applies, gives you a checklist to put to any vendor, and validates one day's delivery step by step.

Which delivery formats do hedge funds expect from an alternative data vendor?

A hedge fund's data team checks that an alternative data vendor delivers by a route a scheduled job can read: an API, object storage such as Amazon S3, or SFTP. It checks for files in CSV, JSON Lines or Parquet, and with them a manifest, a versioned schema and timestamps for when each record was known.

FISD's Alternative Data Council, a set of working groups founded in January 2019, publishes documents whose adoption it calls entirely voluntary. Three are closest to a delivery by title or description: a vendor tear sheet that lets a buyer assess a dataset quickly, guidelines for describing data, and best practice recommendations on service levels, response times and communications procedures. Among the documents listed on that page on 7 October 2026, we found none whose title or description sets a route or a file format.

Each row below is a question for the vendor and a test to run on sample data. Completion, part of the route, has a row of its own.

The data feed requirements hedge funds set for delivery by API, S3 and SFTP

Whatever carries an alternative data feed, a hedge fund can set the same four requirements on the delivery: a credential issued to one client, a job that resumes where it stopped, a signal that a period is complete, and proof that the bytes received are the bytes sent. An API, Amazon S3 and SFTP meet them differently.

API

With an API the state sits on your side: the job holds a cursor, and you ask the vendor how the last page of a period is marked. The comparison of API and bulk files covers the HTTP rules a client relies on. Test one thing more: that a cursor saved today resumes tomorrow, which makes the run an incremental sync.

Object storage: Amazon S3 and other cloud storage

Amazon S3 settles two things for a reader. The PutObject reference says Amazon S3 never adds partial objects: a success response means the entire object was added. The S3 consistency model gives strong read-after-write consistency for writes and deletes in all AWS Regions, so a read or a listing started after a successful write returns what was written. A job that lists a prefix sees whole files only.

The same page states the limit: updates are key-based, and there is no way to make an atomic update across keys. A delivery of twenty files is twenty writes, and a listing taken after the tenth is accurate and incomplete. That is the case for a manifest written last, and for starting the load from it.

Amazon S3 and Google Cloud Storage both store a checksum with the object. S3 validates a checksum sent with an upload against its own calculation before it stores the object, under a choice of algorithms that includes CRC-64/NVME, its default, and SHA-256. Cloud Storage supports CRC32C and MD5, recommends CRC32C, and gives no MD5 validation for objects uploaded in chunks. A digest in a manifest therefore has to name its algorithm.

SFTP

SFTP is file transfer over SSH. The SSH transport layer, RFC 4253 of January 2006, provides encryption, server authentication and integrity protection, and RFC 4252 makes public key authentication the one method every implementation must support. The file transfer protocol itself was written as an IETF Internet-Draft, which the IETF's document tracker shows as expired. The draft says it assumes a secure channel whose ends are already authenticated.

The channel protects the bytes in transit, and the state of a file on the server is for the two sides to agree. The draft defines a rename message, so a vendor can upload under a temporary name and rename the file when it is whole. Ask which signal the vendor uses, a rename, a marker file or a manifest written last, and poll for that.

Can the file show that it is whole?

The comparison of CSV, JSON Lines and Parquet sets out types, nesting and size. A delivery check asks a narrower question of each format.

• A gzip file can. RFC 1952 ends each member with a CRC-32 of the uncompressed data and its size modulo 2 to the power 32, and says the check value is there for detecting data corruption. Test every compressed file before you parse it.
• A Parquet file can. The file format starts and ends with the four-byte magic number PAR1, and the file metadata is written after the data. Readers are expected to read that metadata first, so a file that lost its tail gives a reader nothing to start from.
• A text file leaves it to a count. RFC 4180 for CSV and the JSON Lines page define no trailer, record count or checksum. The total has to come from outside the file.

The manifest: counts and checksums

A manifest describes one delivery. Ask the vendor what its manifest carries: the period covered, the schema and label versions, the licence and the time it was written, and what it states for each file, such as a name, a size in bytes, a row count or a digest with its algorithm.

A digest and a count answer different questions. NIST's Secure Hash Standard, FIPS 180-4, specifies hash algorithms whose digests are used to detect whether a message has changed since the digest was generated. Its seven algorithms include SHA-256, and NIST deprecated SHA-1 in 2011, so ask for SHA-256 rather than SHA-1. A matching digest shows that the file you hold is the file the vendor wrote. A matching row count shows that your parser read all of it. Whether the vendor wrote the whole period is a third question, and a comparison with earlier periods is the test for it.

Count the rows and compute a digest of every file on receipt, whatever the manifest carries. Where it states a count or a digest for a file, compare yours with it. Where it states neither, keep yours beside the file as your own receipt, as the worked example does.

A stable schema, with versions

A schema is stable when a load written this quarter runs unchanged next quarter. Ask where the schema is published, which changes can happen inside a version, and how much notice a breaking change gets. An added column is safe. A column renamed or removed stops the load, and a column given a new meaning damages a model while the load still succeeds.

On every delivery, compare the columns with the data dictionary: log one you did not expect and stop on one that is missing. Model output needs a version of its own: a label or a score should carry the name of the label version that produced it.

Timestamps that say when each record was known

A record can carry three times: when the event happened, when the vendor observed it, and when the delivery reached you. A back-test may use a value only from the moment it could be known, so the second and third are the ones that count, and a dataset dated by the first alone invites look-ahead bias. Ask for the observation time on every record, and whether the history was observed on its dates or assembled later, which is the test of point-in-time data.

Then check the form. RFC 3339, the internet profile of ISO 8601, puts an offset on every timestamp, either Z for UTC or a number of hours and minutes, and calls the interoperability problems of unqualified local time unacceptable. Then stamp every file with your own arrival time and keep it on the rows you load. For live data it is the time you can show the value was known to you.

How a correction arrives

A correction can arrive four ways: a row changed in place, a file sent again under the same name, a new file marked as a restatement, or a dated correction record beside the original. The first two leave no trace unless you kept the first copy. The last two are the ones a point-in-time store can hold. Our note on why we write each table once sets out the design from the vendor's side.

On your side, keep every file as it arrived. The S3 guide says Versioning lets you preserve every version of every object in a bucket. With it on your landing bucket, a file sent again under an old name becomes a new version beside the first, and two digests show whether anything changed.

Worked example: validating one day's delivery

This example is illustrative, and its row counts, landing time, band and company row are invented. A fund takes a vendor's daily file of company hiring counts, open, new and closed postings per company, through the export endpoint of the vendor's API. Acme Robotics is an invented company among the rows.

Assumptions:

• The period is one closed UTC day, 5 October 2026, requested as JSON Lines at 1,000 rows a page. The vendor's API names a next cursor on every page but the last.
• The export returns 18,640 rows. The previous day's file held 18,412.
• The fund's job stores the last page at 06:14 UTC on 6 October 2026.
• The fund lets a day's row count move 5 percent either way before a person looks.

The working:

1. Completion. 18,640 rows at 1,000 a page is 19 requests: 18 full pages and one of 640. The first 18 responses name a next cursor and the 19th names none, so the period is complete.
2. Parse. Every line is read as JSON: 18,640 objects and no failures.
3. Period. The earliest and the latest day in the file both equal the day requested.
4. Keys. The pair of day and company ID has 18,640 distinct values, the same as the row count, so no row arrived twice.
5. Schema. Every column in the data dictionary is present. An unknown key would be logged and kept.
6. Volume. 18,640 less 18,412 is 228 rows, and 228 divided by 18,412 is 1.24 percent, inside the 5 percent band.
7. Known-at. The day closed at 00:00 UTC on 6 October and the file landed 6 hours 14 minutes later. A back-test may use these rows from 06:14 UTC on 6 October, never on 5 October itself.

The row for Acme Robotics, trimmed to five keys:

{"company": "Acme Robotics", "day": "2026-10-05", "open_postings": 44, "new_postings": 2, "closed_postings": 1}

Steps 2 to 4 and the digest fit in a few lines of Python.

import hashlib
import json

rows, keys, days = 0, set(), set()
digest = hashlib.sha256()
with open("hiring_counts_2026-10-05.jsonl", "rb") as f:
    for line in f:
        digest.update(line)
        row = json.loads(line)  # a broken line stops the load here
        rows += 1
        keys.add((row["day"], row["company_id"]))
        days.add(row["day"])

assert days == {"2026-10-05"}, "a row outside the period"
assert len(keys) == rows, "a key arrived twice"
print(rows, digest.hexdigest())

Store the period, the page count, the row count, the digest and the arrival time beside the file as its receipt. Some weeks later, request the same day again and compare the rows that appear in both loads. Where the vendor writes each day once, after it closes, and never revises it, every value should match.

How Fokals delivers against the checklist

Fokals delivers firmographic, technographic, hiring and intent data on public and private companies direct, by REST API and as bulk exports, and you land it where your pipeline reads it. The API reference, the data dictionary and the methodology hold the detail behind each row below, and the delivery page gives the summary.

Every row that names a company also carries the stable company ID and, where the company or its parent is listed, its ticker, MIC, ISIN, LEI and share-class FIGI, the keys for mapping alternative data to a security master. Sample data for the companies you track is sent on request with the data dictionary and the methodology, so tests from the first table can run on your own list.

Delivery is one of the ten questions for a company data vendor, and the compliance review beside it is in the guide to alternative data due diligence questionnaires.

Frequently asked questions

The queries and code on this page are examples to adapt. Test them in your own environment before you rely on them.

What this page says about the products it names was checked against their public documentation on 7 October 2026. Product and company names are trademarks of their owners. Fokals is not affiliated with them or endorsed by them.