Bulk import of datasets

[Daniela Serrano]

Hello everyone,

I've been reading the documentation about creating new datasets via the migration and native api, however I want to know if there is any supported way of importing datasets in bulk or if we'll need to create our own scripts to do so. We would also need to upload files for each resource. We'll be starting a new 6.10 docker instance. 

Thanks in advance

[Zeynep Cevik]

Hello Daniela, 

As far as I know, there's no built-in feature for that out of the box, so you'll likely need to script it via the native API. If there is, please let me know. 

Some time ago, I put together a quick repo that might help: https://github.com/zbcevik/BulkLoadData_bash

It's a bash script that handles bulk dataset creation and file uploads for Dataverse. I tested it on Borealis - borealisdata.ca (Dataverse 6.8), but it should work on 6.10 too. A few things to note:

- It's for fresh uploads, not migration (won't preserve DOIs or version history)

- Needs a specific folder structure: metadata.json + files.zip per dataset

- please give it a good look before using it in production.

Hope it helps, let us know how it goes!

Best,

Zeynep

--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-commu...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/dataverse-community/48a3d873-176a-4567-bebd-995dda5bd4d1n%40googlegroups.com.

[Philip Durbin]

Zeynep, your scripts look great. Would you like to have them listed at a future version of https://guides.dataverse.org/en/6.10.1/api/apps.html ? If so, please feel free to create an issue or pull request at https://github.com/IQSS/dataverse !

Daniela, I think you should consider using Zeynep's scripts as a starting point. Dataverse doesn't have any sort of bulk import feature. It requires scripting. Here's some other code you might want to look at:

- https://github.com/IQSS/dataverse-sample-data - similar to Zeynep's solution

- https://github.com/gdcc/dataverse-recipes - specifically, the "Create datasets from Excel files" script, if you happen to be using Excel

There might be other solutions out there as well!

I hope this helps,

Phil

To view this discussion visit https://groups.google.com/d/msgid/dataverse-community/CAPHxwJRsDjQPRM0wgi6NrxqKhuumOKWYeaD34ahCpLt19ptcXg%40mail.gmail.com.

--

Philip Durbin
Software Developer for http://dataverse.org
http://www.iq.harvard.edu/people/philip-durbin