Direct download and multi-file download behavior in Dataverse

[Laura Lo Gerfo]

Hi everyone,

I’m relatively new to Dataverse, so apologies in advance if some of my questions are basic. I’m looking for some clarification on how direct download works. Specifically, I would like to understand whether, when enabling direct download:

• You can confirm that the direct download is actually active only for single files, whereas when selecting multiple files, a ZIP file is generated instead, and therefore no separate URLs are created for downloading each selected file individually;
• Is there a specific configuration that allows downloading files one by one using their respective S3 URLs, similar to how Dataverse handles file uploads, even when multiple files are selected, instead of generating a ZIP file?

In case there is no alternative configuration, I would really appreciate understanding how the download logic works in detail, in order to better understand the limitation of the :ZipDownloadLimit.

Additionally, I’m wondering about best practices for large datasets in Dataverse:

Would it make sense, or is it recommended, to upload both the original dataset (with all the standard Dataverse features such as metadata, versioning, and searchability) and a ZIP bundle containing the same files for easier bulk download?

Thanks in advance for any clarification or pointers to documentation.

Best regards,
Laura

[Philip Durbin]

Hi Laura,

Welcome. Good questions.

Direct download is only for single files.

To download many single files at once, you could try the "download files from a dataset" script* at https://github.com/gdcc/dataverse-recipes

As for duplicating the files as a zip file, I don't love the idea, especially for large files, but, yes, it would be a workaround. When the user clicks the "download all (or some) files as a zip" button, the zip file is created on-the-fly (which can take a while) on the Dataverse server and it isn't served over direct download. That's why we have this :ZipDownloadLimit threshold you mentioned to prevent a poor user experience when many large files would be zipped up.

Perhaps Dataverse should work like Google Takeout, which generates a zip for you and then emails you when it's ready. 

I hope this helps! Please keep the questions coming!

Thanks,

Phil

* The download script was originally added in this pull request: https://github.com/gdcc/dataverse-recipes/pull/17

--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-commu...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/dataverse-community/39fcc689-4262-4234-82e1-dae312589ebbn%40googlegroups.com.

--

Philip Durbin
Software Developer for http://dataverse.org
http://www.iq.harvard.edu/people/philip-durbin

[James Myers]

One related option: if you upload a zip only, the Zip Previewer can be used to let people download individual files without doubling the storage. That’s also a recommended option for reducing the file count in general.

-- Jim

To view this discussion visit https://groups.google.com/d/msgid/dataverse-community/CABbxx8E4aLvQasUGU5%3DCHbcr79C5PEiGEsTRduJntPUgdfUnxA%40mail.gmail.com.