Getting error during downloading the image

[Naitik Agrawal]

I have written a Python code to download images. The image URLs are stored in a JSON file. I have used multithreadin to download images, with max worker = 10, but after downloading some images I am getting error like too many request. so I want to know what is the limit on the API for downloading images?

[Jolan Wuyts]

Hi there! thank you for your question. I have forwarded this query to some of my colleagues so we can give you advice on how to use our thumbnail API without overloading it, I'll get back to you ASAP! You seem to have asked this question in the Google Group twice so I'll delete your first question. We can continue the conversation in this thread. 

Best

Jolan

[Jolan Wuyts]

Hiya,

I've been informed by our technicall staff that we recently saw a spike in our Thumbnail API usage on 26/06 and 27/06 from a user with user agent  python-requests/2.32.3  , making up to 43K requests per minute. Our thumbnail API was failing because of this spike in requests so we temporarily rate-limited the IP address of this user. If this was you, apologies for having to rate-limit your usage, we did so to protect our own infrastructure which was failing under the heavy load. 

We will remove the rate limit we imposed after this spike, for now. The current set rate limit for external requests is set at a maximum of 120 requests per 10 seconds to keep our infrastructure healthy and prevent resource exhaustion. Please feel free to run your code as long as it does not go over this rate limit. 

These rates may be reviewed by our technical staff next week, and I will let you know of these rates change in future. Additionally, we'd love to hear about what you're working on! If you want to share more about your project using Europeana's data, feel free to send an email at a...@europeana.eu or tell us about your creative project using this form.

Best

Jolan