HTTP 500 during Dataverse multipart upload finalization (~25 GB file)
Manaswini Ganjam
Hi everyone,
I am uploading a large dataset to Harvard Dataverse using DVUploader v1.4.0 in direct-upload mode from a university HPC data transfer node.
I am encountering an HTTP 500 error during the finalization of a multipart upload, and I am hoping someone familiar with the Dataverse/S3 upload pipeline can help determine whether this is a known issue.
- Total dataset size: ~957 GB
- DVUploader: v1.4.0
- Upload mode: Direct upload
- Failed file: zip file (~25 GB)
- Multipart parts: 25
The upload proceeds normally through the S3 multipart stage:
All 25 parts uploaded successfully to the Dataverse S3 bucket.
Each part reached 100% and returned a valid ETag.
The subsequent Dataverse request to complete/finalize the multipart upload returned HTTP 500.
The file was consequently not registered in the dataset.
The DVUploader log reports:
“Partial upload of [...]/beam_rad_annual_conus_tiles.zip, complete upload failed with status: 500”
followed by:
“Not uploaded due to error during processing.”
Unfortunately, DVUploader does not appear to log the response body from the HTTP 500, so I do not have the server-side exception.
I previously encountered a different error (SignatureDoesNotMatch) during another large-file upload, but the current failure is specifically an HTTP 500 after all multipart parts have successfully uploaded.
I am particularly interested in whether anyone has encountered this sequence:
S3 multipart parts successfully uploaded → Dataverse multipart completion/finalization → HTTP 500 → file not registered
and whether there are known limitations related to file size (~25 GB), number of multipart parts (25), or the Dataverse finalization/registration step.
I have approximately 39 archives, averaging ~24 GB each to upload. If anyone has suggestions for additional logging, relevant Dataverse configuration, known issues, or a GitHub issue that matches this behavior, I would greatly appreciate the guidance.
Thank you,
Manaswini
Jacek Chudzik
Björn Selent
--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-commu...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/dataverse-community/f0452afb-ae76-42df-8e80-156516fb38a2n%40googlegroups.com.
James Myers
I don’t have any solid ideas for why this occurred but I can say that
- In general, there’s no limit on number of parts and we’ve tested with thousands of small parts on test machines,
- That call to Dataverse results in the Dataverse server calling the S3 store to complete the multipart transfer, so the error could potentially have been with the S3 store or Dataverse
- DVUploader does do retries, but for the calls to Dataverse, it only retries for 429 responses due to throttling, on the assumption that 500 errors aren’t going to change. If anyone finds they occur intermittently, we could change that and also retry after a 500 error.
- In general 500 errors don’t provide much info in the response to the client (which could give malicious user more info), but there should be details in the Dataverse log – if you have the time the error occurred or can reproduce the problem, I’d suggest sending an email to sup...@daverse.org so someone at Harvard can see what was happening on the server.
- Signature does not match is possibly due to a timeout – I don’t know what the timeout is set at for Harvard, but if all calls to the S3 store to upload parts of a given file don’t start within that timeout, they’ll be rejected by the S3 store. Using a faster network or asking Harvard to increase the timeout would be the two ways to fix it.
- I don’t think it would be at this step, but a file over the size limit for the dataset/over your quota, etc. will definitely fail at some point.
W.r.t. the other responses:
- Non-direct upload is much less scalable in many ways – files are uploaded as one part, timeouts on the Dataverse server and any proxy/load balancer in front of it apply (the upload must complete in that time), and the Dataverse server must have sufficient temp space for the file (along with any other uploads from other people), etc. In the new https://guides.dataverse.org/en/latest/admin/big-data-administration.html#scaling-dataverse-with-data-size guide, we highly recommend S3 and direct upload for large data.
- The checksum validation settings shouldn’t matter during upload, but setting those size limits and/or disabling validation during publication are best practices – again now noted in the Scaling guide linked above.
The only other thought I have – single/small numbers of large files can also be uploaded through the Dataverse UI – either the basic upload a file or the upload a folder option. Both will use direct multipart uploads and should nominally function the same as DVUploader. If there’s some unknown issue in DVUploader, that may be a way around it, or you may just see similar failures (though in these cases the 500 response would be from a Javascript call in the browser so the full response should be available in the browser console.)
I hope something in all that helps,
-- Jim
To view this discussion visit https://groups.google.com/d/msgid/dataverse-community/CAFMNE%3DQCyzP9wcE86KdB5FqmyAcy_wbMSDPWjciWBLwT8M95vQ%40mail.gmail.com.