Enterprise: daily changefiles replaced by the daily snapshot

[Casey Meyer]

Hi All,

Today we stopped supporting daily changefiles. The replacement is the daily snapshot: a complete OpenAlex dataset rebuilt every day under s3://openalex-snapshots/full/<date>/, in both JSON Lines and Parquet. 

Docs: https://developers.openalex.org/download/download-to-machine#download-with-an-enterprise-api-key

Migrating from changefiles

Each entity is partitioned by updated_date, so an incremental pull means syncing only the partitions newer than your last run rather than the whole folder:

  aws s3 sync s3://openalex-snapshots/full/2026-08-03/jsonl/works/ ./works/ \
    --exclude "*" --include "updated_date=2026-08-02*" --profile openalex

One thing to watch: when an entity changes, it's re-emitted under a new updated_date partition, and the old copy stays where it was. If you're merging incremental pulls into an existing table, upsert on ID and keep the row with the latest updated_date, otherwise you'll accumulate duplicates.

Why we made the change

We expected a single file of daily deltas to be the simpler option, but the snapshot turned out better on three counts:

1. Flexibility:  pull one day, several days, or a full rebuild from the same source, depending on whether you're doing a routine update or standing up a new copy.
2. One access path:  same bucket, credentials, and tooling as the rest of the snapshot. No separate endpoint or code path to maintain.
3. Faster to download and ingest:  the snapshot is split into many part files that transfer and load in parallel. A single gzipped changefile can't be split, so it processes serially.

Hopefully this change isn't too disruptive. If you have problems, email sup...@openalex.org and we'll help you out!

Thanks,
Casey