API Rate Limits and alternatives

[Robin]

Hello,

first: I really appreciate your impressive tool and data set!

I want to use some of the soil information for evaluations. Ideally, I'd like to use the API (https://rest.soilgrids.org/) In total i have around 10-30k locations across Europe for which I need the soil information by lon/lat.

As my computational resources are limited, I would rather not download the huge tif data set (additionally, I have no idea how to work with them). Is it possible to use the API for that and what are the rate limits I have to adhere to? I can time and balance my requests so I won't stress the servers too much...

Thank you so much in advance,

Robin

[de Sousa, Luis]

Hi Robin,


definitely this is something you can do using the ReST API. I don't think the service was ever subject to stress tests, so I can not give you a hard number. You could start with a request per second, which would complete your location set in the order of hours. You might also run into network issues by issuing so many request, I suggest you take a step-wise approach.


Regards.


--

Luís Moreira de Sousa

ISRIC - World Soil Information

Droevendaalsesteeg 3, Building 101

6708 PB Wageningen

The Netherlands

t: +31 643 84 40 91

e: luis.d...@wur.nl

Ring<https://ring.cx/>: luis-de-sousa

skype: luis.a.de.sousa

________________________________
From: global-soil...@googlegroups.com <global-soil...@googlegroups.com> on behalf of Robin <robi...@gmail.com>
Sent: 11 January 2019 14:23
To: SoilGrids Community Forum
Subject: API Rate Limits and alternatives

--
You received this message because you are subscribed to the Google Groups "SoilGrids Community Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to global-soil-infor...@googlegroups.com<mailto:global-soil-infor...@googlegroups.com>.
Visit this group at https://groups.google.com/group/global-soil-information.
For more options, visit https://groups.google.com/d/optout.

[Jorge de Jesus]

Hi Robin

Thank you for taking into consideration the server stress and asking how to procede. Normally each query takes between 200ms to 1000ms depending on how big is the payload,

 

Big Payload since we query all the layers

https://rest.soilgrids.org/query?lon=51.3&lat=51.57

Small Payload since we only query one layer (organic carbon)

https://rest.soilgrids.org/query?lon=5.39&lat=51.57&attributes=ORCDRC

If you make your queries more specific you will go way faster. If you do all your queries in a sequential way (one query, parse result, send another query) in some loop you will not have any problems with the server or cause problems to other users (or be flagged as denial of service attack). If you send multiple queries in parallel then it is likely that will give us some problems.

I kindly ask that you post the results of your tests

Cheers

Jorge

On Friday, January 11, 2019 at 3:36:15 PM UTC+1, de Sousa, Luis wrote:

Hi Robin,


definitely this is something you can do using the ReST API. I don't think the service was ever subject to stress tests, so I can not give you a hard number. You could start with a request per second, which would complete your location set in the order of hours. You might also run into network issues by issuing so many request, I suggest you take a step-wise approach.


Regards.


--

Luís Moreira de Sousa

ISRIC - World Soil Information

Droevendaalsesteeg 3, Building 101

6708 PB Wageningen

The Netherlands

t: +31 643 84 40 91

e: luis.d...@wur.nl

Ring<https://ring.cx/>: luis-de-sousa

skype: luis.a.de.sousa

________________________________

From: global-soil-information@googlegroups.com <global-soil-information@googlegroups.com> on behalf of Robin <robi...@gmail.com>

Sent: 11 January 2019 14:23
To: SoilGrids Community Forum
Subject: API Rate Limits and alternatives

Hello,

first: I really appreciate your impressive tool and data set!

I want to use some of the soil information for evaluations. Ideally, I'd like to use the API (https://rest.soilgrids.org/) In total i have around 10-30k locations across Europe for which I need the soil information by lon/lat.

As my computational resources are limited, I would rather not download the huge tif data set (additionally, I have no idea how to work with them). Is it possible to use the API for that and what are the rate limits I have to adhere to? I can time and balance my requests so I won't stress the servers too much...

Thank you so much in advance,
Robin

--
You received this message because you are subscribed to the Google Groups "SoilGrids Community Forum" group.

To unsubscribe from this group and stop receiving emails from it, send an email to global-soil-information+unsub...@googlegroups.com<mailto:global-soil-information+unsubscribe@googlegroups.com>.