API snapshots refreshed

[Noah Balmer]

As many of you know, we used to share API snapshots, zipped files that include all the loan and lender API objects.  Their publication was always a little buggy, and it ceased entirely at some point in 2010.  Anyone who's downloaded a snapshot since then has had only very stale data to look at.

Last night, we got the API snapshot script to run cleanly for the first time since 2010!  It's not yet configured to run on a regular schedule, but for now, there are fresh snapshots available. 

Enjoy!

JSON version (~350 MB zipped) : http://s3.kiva.org/snapshots/kiva_ds_json.zip

XML version (~380 MB zipped) : http://s3.kiva.org/snapshots/kiva_ds_xml.zip

Noah Balmer

Kiva.org

[Sean Durkin]

Awesome, thanks! Looks like this snapshot has more than twice the data the last one had; the previous snapshot (from Jan 2010) had over 165k loans, and this one has over 410k. Expect an updated world map visualization in... several months :'(

Noah Balmer

Kiva.org

--
You received this message because you are subscribed to the Google Groups "build-kiva" group.
To post to this group, send email to build...@googlegroups.com.
To unsubscribe from this group, send email to build-kiva+...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/build-kiva?hl=en.

[Kevin O'Brien]

Awesome! Very excited by this, well done guys.

-k

Kiva Engineer

--

[Bryan Stroube]

This is excellent news, very much appreciated!

Bryan

[Sean Durkin]

As promised, here's an updated "Kiva world map." I was a little disappointed to find the visualization hadn't changed much since the last snapshot (from 2010), despite a lot of new unique data points being added. They will look different, though; I realized that the previous map didn't look to everyone else as it did to me (my monitor's colors were a little messed up I guess), so this time around it should look how I originally intended...

Jan 2010: https://docs.google.com/open?id=0B-c_cWY4vIisMTI3MTIwNDItMWQwNi00NjllLWE1ZmUtMjNhOTUyNTgzNWRi

Apr 2012: https://docs.google.com/open?id=0B-c_cWY4vIisN1BtY1Z4S0t6Y1E

I was toying with the idea of making a poster out of it; if anyone would be interested in something like that, let me know and I can generate a much higher-res image.

[Kevin O'Brien]

Nicely done. I've talked with some people about how we could put these on lender pages or team pages. 

If you could at some point create an Amazon EC2 image that I could load up and run to generate these, then I could get it working towards running on our side. I see there have been some minor adjustments since the last time I looked at it: https://github.com/sndurkin/kiva-map/commit/0bbb1238379626c5d577c888b84a07bcd4179704

I think we could pretty easily script an install that would be able to generate these.

Anyways, again, nice job!

[Sean Durkin]

Thanks!

Is there still interest in generating maps for lenders & teams? If so, I could look into how to get the script running on Amazon EC2 (maybe you could give me a couple pointers?), but I figured you would want to change the script, because it just uses the API to fetch all the data (which, for teams, can be a lot). I think it would be faster and more efficient if it made specific queries directly to the DB. Or are you talking about more of a proof-of-concept?

Sean

[Kevin O'Brien]

Yes, more just proof-of-concept. 

You're right that we would probably use direct db queries, although if we could do things with the API to make the data available in more easily digestable chunks that'd be good. 

We've also got the API snapshot on our Github page (with hopefully soon to be even more updated data), so makes it easier to pull down changes: https://github.com/kiva/API