New OpenAlex snapshot — now in Parquet (+ updated layout)

[Casey Meyer]

Hi All,

The latest OpenAlex snapshot is posted, and it now ships in Parquet alongside the existing JSON Lines format.

Parquet is a columnar, typed, compressed format that's the standard for analytical data. Compared to gzipped JSONL, it lets you:

- Query it in place once downloaded:  point DuckDB, Polars, Spark, pandas, or BigQuery straight at the files; types and nested fields (authorships, locations, etc.) come through natively, no JSON parsing step.

- Move and store less:  columnar layout compresses better than row-based JSONL, so smaller downloads and lower storage.

- Read only the columns you need:  pull ID, DOI, and cited_by_count, etc without scanning the other ~40 fields, so most queries touch a fraction of the data.
- Skip data you don't need:  use built-in column statistics to skip whole chunks of files when filtering (e.g. by publication_year), instead of reading everything.

To support both formats cleanly, we changed the snapshot layout. Data now lives under format-specific paths:

- s3://openalex/data/parquet/<entity>/…
- s3://openalex/data/jsonl/<entity>/…

If you have scripts pointing at the old s3://openalex/data/<entity>/ paths, you'll need to update them. JSONL hasn't gone away — it's just under data/jsonl/ now. We also moved the per-entity/per-format manifests to JSON (manifest.json), matching the structure they already had.

Full details are in the updated docs:
https://developers.openalex.org/download/snapshot-format

Let us know if you run into any issues!

Thanks,
Casey