new limits placed on public api

[mfab]

This information was threaded in response to a question. It could be useful to the build-kiva community as a top-level note.

Kiva recently needed to implement limit restrictions on the public API. The site performance team reported that without the limit restrictions, Kiva servers were seeing resource starvation caused by unbounded requests, and in order to make sure that www.kiva.org is able to stay up, the team had to implement limit restrictions.  At the moment, Kiva needs to be able to protect the public API from serving burdensome, bordering on malicious, requests.

Hope the information is useful.

[YowieFreak]

Could some consideration be given to increasing the rate limits for registered applications please.

With the reduction to the maximum number of items that can be retrieved per call, more calls are now having to be made in order to retrieve the same information, and that is causing us to come up against the rate limits.  (E.g. fundraising loans now exceed 5900 which means, with only 100 loans retrievable per call, I just hit the limit when I made my 60th call.)

[mfab]

Kiva's engineering team is working on procedures to authenticate 3rd party applications for partners and hoping that those procedures and sandboxes will be available to the build-kiva community after the procedures and systems are in place. Until then, rate limitations needed to be placed on the public api. 

Updating the procedures is on the current roadmap for 2020, we thank you for your patience and understanding while the growing pains are being ironed out as Kiva upgrades its internal and external services. 

Hope this information helps.

[Paul]

One solution to prevent apps from needing to regularly download all available loans would be to offer a way to only pull the update since a given server time. It would allow sites to keep their copy of the loans up to date without doing a full refresh of all loan data. Whether through the REST or GraphQL endpoint, I know KivaLens would love to not have to traverse all loans to have fresh data. Also, there used to be a websocket that sites could listen to that would give notice of purchases, new loans posting, etc which helped keep 3rd party apps' copies of the data fresh without going page-by-page through the API to find changes.