uploading large files

[Data CUHK]

Hello all,

Do you have experience uploading large files to Dataverse? Our users are having issues with uploading large files (15GB- 30GB). They tried to upload one file at a time, with each file sized at least 15GB, and the interface just timed out on them. It doesn't sound like they can break down the files further and upload smaller size files instead. 

Is there an API we could use to assist with the large file upload? 

Thanks!

Qinqin

[Philip Durbin]

Hi Qinqin,

Yes, for large file uploads, the API is recommended. Even better might be to use a tool like DVUploader, which we mention in the guides: https://guides.dataverse.org/en/6.7.1/user/dataset-management.html#command-line-dvuploader

There's even a new video from TDL about how to use it: https://www.youtube.com/watch?v=7qINkx38mNg

There's also a Python equivalent: https://github.com/gdcc/python-dvuploader

I hope this helps!

Phil

--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-commu...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/dataverse-community/3d93f06d-1cfc-44ad-afdd-76c0fba5569fn%40googlegroups.com.

--

Philip Durbin
Software Developer for http://dataverse.org
http://www.iq.harvard.edu/people/philip-durbin

[Stefano Bolelli Gallevi]

Hi there, is it Dataverse able to manage such big files (15Gb, 30Gb each) after uploaded, without problems in vieweing the dataset and downloading them?

Thanks and best regards
Stefano

[Philip Durbin]

Hi Stefano,

I don't have a lot of real-world experience with how Dataverse handles 30GB files after they're uploaded but for files of this size I would suggest storing them on S3 (or compatible) and using the direct download feature. That way, the bytes are streamed to the client directly from S3 rather than passing through Dataverse. Please see https://guides.dataverse.org/en/6.8/installation/config.html#file-storage

Also, when files get truly big, I'd suggest Globus, which we use in production for Harvard Dataverse. Please see https://guides.dataverse.org/en/6.8/developers/big-data-support.html#globus-file-transfer

I'd also love to hear from others in the community about what their experience is for files in the range you mentioned: 15 GB to 30 GB.

I hope this helps,

Phil

To view this discussion visit https://groups.google.com/d/msgid/dataverse-community/a4c8d820-4a5c-4bff-b8e5-f17b59a1bb18n%40googlegroups.com.

[Kirill Batyuk]

We have files larger than 30GB. It is not a problem for the Dataverse to list them in the dataset.

We use Globus for uploading and downloading the files that big.

Of course, it all depends on the installation and resources allocated to the machines running the Dataverse.

-Kirill.

 

 

Kirill Batyuk

Systems Librarian

MBLWHOI Library

Data Library and Archives

Woods Hole Oceanographic Institution

508-289-2850

kba...@whoi.edu

mblwhoilibrary.org -- whoi.edu

 

 

 

From: 'Philip Durbin' via Dataverse Users Community <dataverse...@googlegroups.com>
Sent: Friday, October 10, 2025 11:49 AM
To: dataverse...@googlegroups.com
Subject: [EXTERNAL] Re: [Dataverse-Users] uploading large files

 

This email originated outside of WHOI. Please use caution if clicking on links or opening attachments.

To view this discussion visit https://groups.google.com/d/msgid/dataverse-community/CABbxx8HHv1whS7-wmp2uwHeVLh9CjRny27R38Zf2%3DdxFRk4jFQ%40mail.gmail.com.