Getting error during downloading the image

[Naitik Agrawal]

I have written a Python code to download images. The image URLs are stored in a JSON file. I have used multithreadin to download images, with max worker = 10, but after downloading some images I am getting error like too many request. so I want to know what is the limit on the API for downloading images?

[Jolan Wuyts]

Hi there! thank you for your question. I have forwarded this query to some of my colleagues so we can give you advice on how to use our thumbnail API without overloading it, I'll get back to you ASAP! You seem to have asked this question in the Google Group twice so I'll delete your first question. We can continue the conversation in this thread. 

Best

Jolan

[Jolan Wuyts]

Hiya,

I've been informed by our technicall staff that we recently saw a spike in our Thumbnail API usage on 26/06 and 27/06 from a user with user agent  python-requests/2.32.3  , making up to 43K requests per minute. Our thumbnail API was failing because of this spike in requests so we temporarily rate-limited the IP address of this user. If this was you, apologies for having to rate-limit your usage, we did so to protect our own infrastructure which was failing under the heavy load. 

We will remove the rate limit we imposed after this spike, for now. The current set rate limit for external requests is set at a maximum of 120 requests per 10 seconds to keep our infrastructure healthy and prevent resource exhaustion. Please feel free to run your code as long as it does not go over this rate limit. 

These rates may be reviewed by our technical staff next week, and I will let you know of these rates change in future. Additionally, we'd love to hear about what you're working on! If you want to share more about your project using Europeana's data, feel free to send an email at a...@europeana.eu or tell us about your creative project using this form.

Best

Jolan

[Naitik Agrawal]

Actually in my project I am downloading approx 10mn images from europeana free api, but due to limit of 120 request per 10 seconds it will take weeks to do this task.Is there any other way I can download all 10mn images in minimum time

[Hugo Manguinhas]

Dear Naitik Agrawal,

Thank you for your interest in the Europeana APIs.

Let me introduce myself first, I am the product manager responsible for the development and operations of the Platform Services at Europeana, where APIs are the primary product we offer.

About your question, we dont offer any means to bulk download thumbnails we keep for the content we make available on the platform. One of the reasons being that we wouldnt know in what way we would cluster them since that would very much depend on the use case which is, in fact, what I would like to understand from you... 

... Could you tell us more about your project/work/intention?

How did you make the selection of thumbnails you would like to download? Did you use the Search API and what criteria are you using?

How frequently would you like to do this? and do you always need a fresh copy of the image?

If you feel more comfortable speaking directly with us, you can reach us via the API mailbox at: a...@europeana.eu or directly to me if you prefer.

Another area that I was hoping to understand is about your script/process and ways to fine-tuning it. For example, the Thumbnail API supports HTTP caching [1] so you can use conditional headers [2] to check if images have changed to avoid downloading unnecessarily an image twice. It also supports HEAD requests in case you want to check if an image exists at all. Improving your code to use these features could greatly reduce your demand in terms of image downloads and make your work finish faster.

Finally, about the rate limiting, we have recently enabled this for the Europeana APIs so that we are able to offer the same level of service to all our customers. We are still understanding what limits could be found reasonable and that is also the reason why my interest in knowing more about your project.

Looking forward to your email.

[1] https://developer.mozilla.org/en-US/docs/Web/HTTP/Caching

[2] https://developer.mozilla.org/en-US/docs/Web/HTTP/Conditional_requests#conditional_headers

With kind regards,

Hugo Manguinhas

--
Visit Europeana Labs for API Documentation, Open Datasets, and our Apps Showcase - http://labs.europeana.eu
---
You received this message because you are subscribed to the Google Groups "Europeana API forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to europeanaAPI...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/europeanaAPI/7c7dc42f-b83c-4b54-86b3-b371d1e1308dn%40googlegroups.com.

--

Hugo Manguinhas | Head of Engineering
T: +31 (0)70 3140657
E: hugo.ma...@europeana.eu | Skype: hugo.manguinhas

 

Be part of Europe's online cultural movement - join the Europeana Network Association: Sign up for the Association | #AllezCulture| @Europeanaeu | Europeana Pro website

Disclaimer: This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are

addressed. If you have received this email in error please notify the system manager. If you are not the named addressee you should not disseminate,

distribute or copy this email. Please notify the sender immediately by email if you have received this email by mistake and delete this email from your

system.