Dataverse API file download restrictions

[Shantanu Modak]

Hi,

I am trying to download files from the Harvard Dataverse data access api by making the following call  :

/api/access/datafile/$id

As of today, there are about 364,390 files listed on Harvard Dataverse. Ideally, I would like to download all of these files by making multiple consecutive calls to the api.

Are there any restrictions on downloading such large amounts data from the api  ?

Does Harvard Dataverse restrict/block an IP address if it is trying to access the API through multiple consecutive calls and downloading huge amounts of data?

Regards,

Shantanu 

[Philip Durbin]

Hi! This mailing list is for all users of Dataverse, not just Harvard's installation of it.

Questions about the Harvard Dataverse should be directed to sup...@dataverse.org or entered via the support button at the top of https://dataverse.harvard.edu

I will say that you should familiarize yourself with The Harvard Dataverse API Terms of Use at http://dataverse.org/best-practices/harvard-api-tou before embarking on such an extensive download operation. This item jumps out at me: "In using the Dataverse APIs, you shall not... 4. use Dataverse APIs in a manner that may impair the functionality, stability, or operation of Harvard Dataverse servers or adversely impact the behavior of other users or applications using the Dataverse APIs;"

From the Dataverse software/product perspective, we've talked about restricting APIs in a few GitHub issues:

- API: Create mechanism to shut off API access selectively and globally. - https://github.com/IQSS/dataverse/issues/1103
- API: Add rate limiting logic - https://github.com/IQSS/dataverse/issues/1339

I hope this helps! I'm glad you're so interested in the files hosted by Harvard Dataverse! :)

Phil

p.s. Some of those files are restricted so you won't be able to download them all. If you click "Files" under "Access" you'll see "Public" vs. "Restricted".

--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsub...@googlegroups.com.
To post to this group, send email to dataverse-community@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/dataverse-community/e4cc22c9-da40-445e-bd1d-379655ef3cc3%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Philip Durbin
Software Developer for http://dataverse.org
http://www.iq.harvard.edu/people/philip-durbin

[danny...@g.harvard.edu]

Hey Shantanu - it looks like Phil gave you some feedback, but I'd love to hear more about what your needs are here regarding the files in Harvard Dataverse. If it's something that's of use to the greater community we can discuss further on this thread. Feel free to reach out to the Harvard Dataverse Team at sup...@dataverse.org and I'll make sure that I handle the ticket. 

Thanks,

Danny