Kiva Country Collector

[Blake Swopes]

OK, I'm ready to open this up to a little wider audience... I've put together a tool to search for loans from new (to you) countries.

http://www.bhodisoft.com/kiva-country-collector/

It is written in python, and the code is available on github at https://github.com/bswopes/kiva-country-collector

Input your lender ID and it'll pull your previous loans to see which countries you've already lent to and a count for each. 

Depending on flags passed, it then does a single search for all other countries, if any return, it steps through one by one to identify which country was found and dumps out a link. If no new countries were found, it looks over your loan history and steps through your less frequently loaned to countries and links to loans it finds.

Still some things I want to work out around the most efficient number of api calls for what I'm trying to do... and the big one I need to figure out is how I want to handle when Kiva adds a new country. Currently I have a flag to look against established Kiva countries vs all ISO countries. 

Anyway, give it a try, let me know what you think... Feel free to download it from github and run the command line version locally.

[Martin Butt]

Hey Blake,

Nice app! I can put a ticket in the system for expanding the country value that is returned by: http://api.kivaws.org/v1/methods/GET*%7Cloans%7Csearch.json - so that it returns a list of all the countries we have ever had a loan in.

Let me know when you want me to add it to the app gallery (http://www.kiva.org/apps/all).

Cheers,

    Martin

--
You received this message because you are subscribed to the Google Groups "build-kiva" group.
To unsubscribe from this group and stop receiving emails from it, send an email to build-kiva+...@googlegroups.com.
To post to this group, send email to build...@googlegroups.com.
Visit this group at http://groups.google.com/group/build-kiva?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--

Martin Butt | Software Engineer

Kiva – Loans that Change Lives
Join us on Twitter, Facebook and LinkedIn!

[Kevin O'Brien]

I'd appreciate an API method that would return that Martin, cause then I could use it for the lender maps.

[Blake Swopes]

Oh, that'd be much more usable. The way I pulled it initially was to basically step through each country code's loan page and see if there was anything there, then dump that out to a file for use by the app. That's a lot of hits to the API, though, so it isn't great for maintaining an up to date list. If I could pull it with a single hit, or at least just confirm that the total number of countries hasn't changed, that'd be much better.

Adding to the app gallery would be very cool, thanks! I'm curious to see how my poor little linode box does once its more than just a couple people hitting it.

On Sun, Feb 3, 2013 at 12:57 PM, Martin Butt <mar...@kiva.org> wrote:

[Martin Butt]

This will be in the next build, due for release 2/19.

[Blake Swopes]

I've got my app updated to use the country list from the method description. Thanks for getting this out so fast!

[Chris Means]

Hi Blake,

I just tried your online version with my lender id "cmeans", and it gave me the error:

Error loading lender page. Confirm your ID at http://www.kiva.org/myLenderId

-Chris

[Blake Swopes]

So, while most of the api allows 500 calls per minute on a registered app, the lenders/::id::/loans call is limited to 60/minute. You have 102 pages of loans -- nicely done, btw! -- which means until I add some backoff logic, I won't be able to pull in the full list of countries you've loaned to.

So, some things I need to do... I need to back off as I approach the limit rather than blowing through it. Down side to that is that it'd take two minutes for me to pull in your history. Second, I need to figure out how to store partial data and only do incremental updates for loans I don't already know about... This might need a change to my model, which is currently just a list of countries with a count of loans next to it, e.g "DO",2. I wish there were a better way for me to pull this data, right now its fairly api intensive.

My local copy has at least implemented better error messages so it's obvious that it was a rate limit issue.... Here's the cli version:

$ ./index.py -vpn -i "cmeans"

Lender has made new loans. Updating...

API Calls Remaining: 0 of 60

You are making requests to the server too often and have been temporarily blocked. You can find out more details at http://build.kiva.org/docs/data/rate.

[Blake Swopes]

BTW, I deleted a post because at least to me, I was seeing an email in my reply and didn't want Chris to get spammed.

What I had said was this:

Seems I'm getting rate limited... I'll need to look into that.
{
    "message": "You are making requests to the server too often and have been temporarily blocked. You can find out more details at http://build.kiva.org/docs/data/rate.", 
    "code": "org.kiva.RateLimitExceeded"
}

[Blake Swopes]

This issue should now be resolved. 

• I added support for the rate limit headers. As I hit 10% of the limit remaining, I halt so as not to get blocked. I didn't add exponential backoff, since I figure the web request would time out anyway and it seemed more trouble than its worth for now.
  
• I added incremental caching / updates of lender data. This required a change in the data I was storing from country code and a counter, to country code and a list of loan IDs. This was a big win in terms of how many api calls I make since I no longer throw out everything whenever someone makes a new loan.
  
• I added some intelligence as to where I start in the list of lender data.

For someone with as many loans as you, I can't pull it all in with one run. But a second run through takes care of that. I used your account id as my test case, so the data is already pre-populated and you should be good to go.

[Chris Means]

I ran a test or two and it seemed to work well.

Thanks!

Now, what's the best way to translate (other than manually) the results (the list of country codes) to a filter that can be applied within the Kiva UI to find fundraising loans matching those countries?

-Chris

--
You received this message because you are subscribed to a topic in the Google Groups "build-kiva" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/build-kiva/1QsfKliZ6e4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to build-kiva+...@googlegroups.com.

To post to this group, send email to build...@googlegroups.com.

Visit this group at http://groups.google.com/group/build-kiva.

[Martin Butt]

Hi Blake,

You could present the user with a URL in this format:

http://www.kiva.org/lend?countries[]=PY&countries[]=US

Cheers,

    Martin

--

You received this message because you are subscribed to the Google Groups "build-kiva" group.

To unsubscribe from this group and stop receiving emails from it, send an email to build-kiva+...@googlegroups.com.

To post to this group, send email to build...@googlegroups.com.
Visit this group at http://groups.google.com/group/build-kiva.
For more options, visit https://groups.google.com/groups/opt_out.

--

Martin Butt | Senior Software Engineer

[Blake Swopes]

It should be doing that already. I just ran it and the end output is...

Visit Kiva HERE

HERE is a link to kiva's search page for the countries it recommended. http://www.kiva.org/lend#/?app_id=com.bhodisoft.kcc&countries[]=TL

Or did you want all of the countries you haven't loaned to at once? I currently have that in there as the rss feed link, but that's not exactly what you want... Oh, and its broken. Bah.

Anyway, here's your list:

http://www.kiva.org/lend?countries[]=BD,BZ,BW,BG,TD,GZ,NA,TH,TR

[Blake Swopes]

OK, RSS feed is fixed. 

http://api.kivaws.org/v1/loans/search.rss?status=fundraising&country_code=BD,BZ,BW,BG,TD,GZ,NA,TH,TR

This is what I use, actually. I have a couple recipies on IFTTT watching this and they text me or email me when a new country is found.

https://ifttt.com/people/bswopes

[Chris Means]

I’m checking out IFTTT, thanks!

[Chris Means]

HI Blake,

 

Tried again, and I’m now getting the error:

 

Unable to write to lender file: cmeans.csv

 

I was setting up an IFTTT RSS feed.

 

-Chris

 

From: build...@googlegroups.com [mailto:build...@googlegroups.com] On Behalf Of Blake Swopes
Sent: Thursday, September 19, 2013 10:44 AM
To: build...@googlegroups.com
Subject: Re: Kiva Country Collector

 

OK, RSS feed is fixed. 

[Blake Swopes]

Happened to me recently, too. I think its due to having a loan expire, so the amount of loans in the stored file is higher than the amount of loans kiva has on record. 

I need to add some handling of expired loans.

[Blake Swopes]

That error should be handled now.

[Chris Means]

So far so good.  Thanks!

 

I’ve noticed that your routine produces a longer list than Kiva’s list of countries (that I’ve not supported).

[Blake Swopes]

My list is based on the API call that Martin had added, plus a check that there has been at least one loan listed. Some of those countries are no longer active, so they don't show them in the list on the profile page.