HTTP 500 during Dataverse multipart upload finalization (~25 GB file)

[Manaswini Ganjam]

Hi everyone,

I am uploading a large dataset to Harvard Dataverse using DVUploader v1.4.0 in direct-upload mode from a university HPC data transfer node.

I am encountering an HTTP 500 error during the finalization of a multipart upload, and I am hoping someone familiar with the Dataverse/S3 upload pipeline can help determine whether this is a known issue.

• Total dataset size: ~957 GB
• DVUploader: v1.4.0
• Upload mode: Direct upload
• Failed file: zip file (~25 GB)
• Multipart parts: 25

The upload proceeds normally through the S3 multipart stage:

• All 25 parts uploaded successfully to the Dataverse S3 bucket.

• Each part reached 100% and returned a valid ETag.

• The subsequent Dataverse request to complete/finalize the multipart upload returned HTTP 500.

• The file was consequently not registered in the dataset.

The DVUploader log reports:

“Partial upload of [...]/beam_rad_annual_conus_tiles.zip, complete upload failed with status: 500”

followed by:

“Not uploaded due to error during processing.”

Unfortunately, DVUploader does not appear to log the response body from the HTTP 500, so I do not have the server-side exception.

I previously encountered a different error (SignatureDoesNotMatch) during another large-file upload, but the current failure is specifically an HTTP 500 after all multipart parts have successfully uploaded.

I am particularly interested in whether anyone has encountered this sequence:

S3 multipart parts successfully uploaded → Dataverse multipart completion/finalization → HTTP 500 → file not registered

and whether there are known limitations related to file size (~25 GB), number of multipart parts (25), or the Dataverse finalization/registration step.

I have approximately 39 archives, averaging ~24 GB each to upload. If anyone has suggestions for additional logging, relevant Dataverse configuration, known issues, or a GitHub issue that matches this behavior, I would greatly appreciate the guidance.

Thank you,
Manaswini

[Jacek Chudzik]

Hi,

from our experience this might be the problem with timeouts during checksum validation.

I would set low number on dataset/datafile checksum validation limit:

https://guides.dataverse.org/en/6.9/installation/config.html#datafilechecksumvalidationsizelimit

And just to be extra safe I would alse go with this option:

curl -X PUT -d 'false' http://localhost:8080/api/admin/settings/:FileValidationOnPublishEnabled

https://guides.dataverse.org/en/6.9/installation/config.html#filevalidationonpublishenabled

Hope this will help.

Regards

Jacek

[Björn Selent]

Hi all,

we also encounter issues with uploads of larger files (>50Gb). By trial and error we managed to upload (direct upload into on-premise NetApp S3 via DVUploader) a file of 4.5 TB by setting the min-part-size to 512GB. We also experience issues with file sizes that haven't been a problem in the years (ca. 30-50GB) with regular "non-direct" upload. As we haven't properly investigated it yet, I cannot tell at which part of the stack the problem occurs. I just mention it in case anybody else has similar issues.

Regards

Björn

--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-commu...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/dataverse-community/f0452afb-ae76-42df-8e80-156516fb38a2n%40googlegroups.com.