Mismatch between Authors collection size in manifest and records contained in the latest Parquet files

[Antonio Jose Molina Gutierrez]

Hi everyone,

Thank you for your continued efforts to improve OpenAlex and make the data easier to use.

We are currently importing the latest OpenAlex snapshot at Scitility, using the new Parquet files. While validating the import, we noticed a significant mismatch between the number of author records we loaded and the number reported in the manifest.

The manifest reports the Authors collection size as:

119,129,660

However, after importing the latest Authors Parquet files, we only obtained:

70,648,580

The import completed successfully, and we have not detected load errors. We also ran several validation checks to confirm that the import process did not silently fail or skip files.

Could someone clarify whether this difference is expected?

Any guidance would be appreciated. We want to make sure we are interpreting the manifest and Parquet snapshot structure correctly before treating this as an import issue.

Thanks,

Antonio
Scitility

[Rainer M Krug]

Hi

I checked the overnight downloaded parquets, and there are 119’129'660 rows in the authors parquet dataset.

Cheers

Rainer

--
You received this message because you are subscribed to the Google Groups "OpenAlex Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openalex-commun...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/openalex-community/8bcd9373-c96d-4de1-931f-6cf1654609e2n%40googlegroups.com.

[Antonio Jose Molina Gutierrez]

Hi Rainer,

Thank you for checking this so quickly.

We have found the issue on our side. The Parquet dataset is correct, and the mismatch was caused by our import/validation process rather than the OpenAlex snapshot.

We appreciate your help and the quick confirmation.

Best,

Antonio

[Rainer M Krug]

Glad that you found the bug.

Cheers

Rainer

To view this discussion visit https://groups.google.com/d/msgid/openalex-community/5e50c1c2-ad34-4d04-9511-a1e96f847cd9n%40googlegroups.com.