Clean Architecture for a Data Collector with limited API

[Christian Melendez]

Hi Guys,

I'm not sure if this is the correct plase but I would like to have some suggestions from you. I'm creating a service that consumes a web service to grab some data, manipulate it and persist it in our own storage. The problem is that the web service I'm consuming has some limitations, it only accepts a maximum of 500 requests per second and I need to make about 280,000 requests every 15 minutes, the problem is that at some point I will need to make more and more requests (maybe one million every 15 minutes). Has anyone faced a problem where you need to be as fast as possible but there is a bottle neck that doesn't allow you to scale or be faster?

I was thinking that maybe having a queue that will have the list of requests to make and a service can dequeue by having 500 threads every second and consume eache request. I don't know, I'm trying to make this as clean as possible.

Thanks in advance for your help and suggestions! Links, books, videos or blogs are welcome :)

Christian

[James Green]

Not something I would be writing code to solve. If their API stops accepting requests temporarily it's your application's job to re-try. If they have contractual limits, make a component similar to the one you suggest.

As to what to do to overcome the problem, that's a matter for you to highlight to your bosses. If the supplier is not prepared to handle the rate of requests perhaps they can introduce a new service that operates differently (think batches). Or there may be other suppliers.

I can't say this is a clean code issue tbh.

--
The only way to go fast is to go well.
---
You received this message because you are subscribed to the Google Groups "Clean Code Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to clean-code-discu...@googlegroups.com.
To post to this group, send email to clean-code...@googlegroups.com.
Visit this group at http://groups.google.com/group/clean-code-discussion.