World Map Visualization

[Sean Durkin]

Hi,

I downloaded the data snapshot a couple months ago to make a world map
visualization. I realized soon after that I didn't have all the
necessary data to make it (no lender-loan connections, no locations
for the lenders). So I wrote a script to fetch that data from the Kiva
and Google Maps APIs.

This is what it currently looks like:
https://docs.google.com/leaf?id=0B-c_cWY4vIisOTQ1MThjMmUtMGJlMC00NjIzLWE1MDctZWU5NDdlNDQyMjg4&hl=en_US

I have 2 questions:

1. It would be awesome if I could get an updated snapshot. The one
that's available is over 2 years old now (around Jan 2010) and the
more data, the better. I don't even need much, just the loans. I can
continue to slowly make requests for the lenders (for each loan) from
the API. Please?? :)

2. I don't think the visualization looks great right now. The green
lines are too crowded in some places, and I'm not even showing all of
them; I keep a count of all lender-loan connections, and I'm only
showing the ones coming out of the US if the count is > 10, and for
Europe > 5. Looks like I need to do it for Australia too, maybe? What
do you think? Any ideas would be welcomed.


Thanks,
Sean

[Esa Ruoho]

Hi, can you do this for a single kiva-user, to visualize the overall
effect? map looks really really cool. i'd love something like that as
a widget for my music-site.

> --
> You received this message because you are subscribed to the Google Groups "build-kiva" group.
> To post to this group, send email to build...@googlegroups.com.
> To unsubscribe from this group, send email to build-kiva+...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/build-kiva?hl=en.
>

[Kevin O'Brien]

Sean,

Looks awsome.

As for #1, we are definitely working on getting some new snapshots available, though we don't have a date yet for when we'll have that.

As for #2, have you seen the Intercontinental Ballistic Microfinance video? That might inspire some ways to display the data http://vimeo.com/28413747

Cheers,

-k

[Noah Balmer]

Hi Sean,

I think it looks great!

We're working actively on getting a new snapshot live.  I'd say we're most of the way there, but I can't say exactly when it will be available.

For the visualization, you could try using a bright color with a lot of transparency for the lines, with the idea being that a single line looks dim, but a stack of them is bright.

Noah Balmer

Kiva.org

On Mon, Feb 6, 2012 at 7:16 PM, Sean Durkin <sndu...@gmail.com> wrote:

[Sean Durkin]

It never occurred to me to do it for a single lender, that sounds like a great idea. I'm not sure it should be a widget (which sounds to me like something that's loaded every time a user accesses the page), because it involves fetching all the loans for a given lender. If it's just a static image you want for your loan contributions, I'll modify the script to generate that.

[Vijay]

Sean, looks awesome.  What tools did you use to do it?

Same question for Kevin - that video is fantastic.  Was it done with R and the maps combined into a video?
Would be nice to know how it was done

[Sean Durkin]

Thanks, that's great news! I have seen that video (I found it after I started this project), and I didn't think to change the colors based on loan type... that's not a bad idea.

[Esa Ruoho]

Just a static image I could replace every once in a while, I mean, I
don't make loans that often, so it'd be nice to have the script
generate a new one when I do make a new loan - I'd know to run the
script on my own, if at all possible!

[Sean Durkin]

Thanks! That sounds like a good idea, I'll play around with the transparency and brightness of the lines a little.

[Sean Durkin]

Thanks, I used Python to fetch and organize the data and R to draw the map.

[Sean Durkin]

Hi Esa,

Sure, I'll write a script and post it here in a few days.

[Esa Ruoho]

Wonderful news, Sean!
Btw, does the API allow for visualization of a Lending Team? I think
many members of the lending team Finland would be very interested - as
we just met up a while ago to talk about how to bring Kiva to a larger
set of people by making it more visible in our country. . .

On Wed, Feb 8, 2012 at 7:55 PM, Sean Durkin <sndu...@gmail.com> wrote:
> Hi Esa,
>
> Sure, I'll write a script and post it here in a few days.
>

> --
> You received this message because you are subscribed to the Google Groups
> "build-kiva" group.

> To view this discussion on the web visit
> https://groups.google.com/d/msg/build-kiva/-/bH5Ojl_Zn1kJ.

[Noah Balmer]

There's an API method that gives loans by team id, which would allow generating a map like this for a given team.

http://build.kiva.org/api#GET*|teams|:id|loans

[Paul Ericksen]

+1 for different colors by sector, using transparency to make stacks be brighter & allowing visualization for teams/individual lenders (team captains love ways to visualize their stuff to help inspire their members)

Other ideas:
Date ranges (team lending for the month of Feb or all of 2011)

Have you checked out FB's visualization of friendships? https://www.facebook.com/notes/facebook-engineering/visualizing-friendships/469716398919

You may also be interested in seeing www.kiva.org/live even though it's purpose is quite different.

Thanks for sharing, great stuff!

Paul

--
Sent from my computer. No excuses for misspellings or missing words.

[Sean Durkin]

Thanks for the ideas Paul. I have seen FB's visualization, that's actually where I got the idea. In the image above, I stacked the lines so that the higher the count (# of lender-loan pairs found in the data), the brighter it is. It obviously didn't work out as well as I'd hoped, and but I like the transparency/brightness idea and I'll try that. Recently I switched to drawing the shorter lines brighter (like in the FB visualization), and I think it's looking better.

[Andy K]

Hi Sean,

Your viz looks great -- kind of like the ICBMf video squeezed into a
single frame ... ;) In addition to color-by-sector, you could also
assign color by country (either borrower's or lender's), to get an
idea of loan volume from country-to-country. If per-country
granularity is too "colorful", you could break it down by region or
continent instead. Also, re: single-lender and teams, you could start
with your initial "dark green loan lines" image as a base, and overlay
individual lender or team activity on top of it in a brighter color,
to highlight the individual/team contribution to the big picture.
Great work, definitely post an updated pic when you have one!

Vijay - the video was done with PHP and the GD libs -- individual
frames rendered & stitched together with ffmpeg.

Andy

[Esa Ruoho]

I think a lot of "Kiva-Bingo" people (people who try to hit as many
countries via kiva as possible would love what you've done, Sean. I
just heard about this type of people at the Helsinki Kiva meetup, they
like to go for as many countries as at all possible

[Sean Durkin]

Thanks for the great ideas, Andy! I really like the idea to overlay lender/team activity in a brighter color. I've got several cool things to try out, this will keep me busy for a few days.

Esa, that's interesting... I've never met anyone who likes doing that. But then again, I don't know many people IRL who use Kiva :(

[Paul Ericksen]

On Thu, Feb 9, 2012 at 7:47 AM, Sean Durkin <sndu...@gmail.com> wrote:

But then again, I don't know many people IRL who use Kiva :(

Sounds like it's time to do some recruiting. :) Give a Kiva Card.

[Sean Durkin]

Hi,

I've got an updated image which I think looks much better than the original. I liked the idea to color the lines or points by sector, but I never saved anything for the loans except lat/lon points. I ended up coloring the lines based on a function of distance between the lender and loan, and pair count:

Updated: https://docs.google.com/leaf?id=0B-c_cWY4vIisMTdiOTE1MzktNWY0Ny00NDY1LTg5ZmMtMDBiOGY5MTAzYzVk
With Kiva logo: https://docs.google.com/leaf?id=0B-c_cWY4vIisMTI3MTIwNDItMWQwNi00NjllLWE1ZmUtMjNhOTUyNTgzNWRi

Here's the source code, if anyone's interested: http://code.google.com/p/kiva-map/


Esa: I've been working on the script to generate custom maps for a lender or lending team. Right now it's able to generate them, and here are some examples of the buildkiva lending team:

With continents: https://docs.google.com/leaf?id=0B-c_cWY4vIisZmJiMWE2MDctYTU3ZS00ZTlhLTk1NWYtZjNjNjMwNjM2YjE3
With base map: https://docs.google.com/leaf?id=0B-c_cWY4vIisNjUxYzZlZmQtN2M4OS00NjRlLTkwYjAtN2QzOTQyMDA2NzM2

I'm not sure I'm happy with these images, maybe you guys will have some ideas on how to improve them?

One issue with the script is that you won't be able to easily generate an image for a large lending team like Finland. Right now the script doesn't save any data to files until it's completely done, which is a problem because if something goes wrong halfway through processing the loans, all the data is lost. So, I need to revise it to work in increments so that if something goes wrong, you can just restart the script and it will pick up where it left off.

[Kevin O'Brien]

Sean, Neat stuff! Any interest in creating a github account and putting this script up there so we can easily collaborate on it (many of us at Kiva use Github for code sharing).

I'm going to relate below how I got this to work for me, but maybe there is an easier way?

Here is what I did to get this running on an Ubuntu 11.10 (originally tried 10.04) install.

1. Downloaded the script to ~/sandbox/kiva-map "svn checkout http://kiva-map.googlecode.com/svn/trunk/ kiva-map"
2. Downloaded one of our data snapshots in json format and unzipped it in that directory (from http://s3.kiva.org/snapshots/kiva_ds_json.zip)
3. Ran "python process_loans.py 1"
4. Installed R (sudo apt-get install r-base)
5. Install cairo graphics library "sudo apt-get install libcairo2-dev"
6. Installed R packages (R, install.packages("[packagename]"):
• > library(maps)
• > library(mapproj)
• > library(geosphere)
• > library(Cairo)
• > library(png)
• > library(bigmemory)
7. Created a data folder and copied the csv and json files to it
8. Changed the absolute file paths in the various R files to point to my install (e.g., ~/sandbox/kiva-map/data (Note: the first thing I would do is create a config file that you can just input your path once in one place and draw from there...)
9. Found I was still missing a raster function, needed libgeos to install rgeos
• sudo apt-get install libgeos-3.1.0 libgeos-c1 libgeos-dev libgeos-doc
• R, install.packages("rgeos")
• *turns out rgeos needs libgeos-3.2, which in turns needs libstdc++6 > 4.6 
• It was at this point that I decided to spin up an ubuntu 11.10 image on EC2...
• installed libxt-dev for cairo then r, install.packages("cairo") again
10. Finally, all that done, I ran R --file=kiva.R 

Voila! I had generated the image. So next I wanted to see output from Team Reddit's latest loans:

1. wget http://api.kivaws.org/v1/teams/2498/loans.json
2. mv loans.json loans/2.json
3. python process_loans.py 1
4. Wait for output...
5. R --file=kiva.R
6. This produces images/kiva.png

Voila! A generated map of Team Reddit's latest loans! Neat. I also created a modified version in Gimp that more suited my desire for contrast and color.

Maybe we could create a service that would do this for lenders and teams?

Thanks for your contribution, looking forward to working with you more on it! 

Cheers,

-k

--
You received this message because you are subscribed to the Google Groups "build-kiva" group.

To view this discussion on the web visit https://groups.google.com/d/msg/build-kiva/-/UTuAdwFW120J.

[Kevin O'Brien]

I found Sean had made a Github account (woohoo! thanks Sean) and put the code up here: https://github.com/sndurkin/kiva-map

I went ahead and created a fork, and put some instructions in there along with getting the filepath by default and made a pull request to Sean, or you can find my fork here: https://github.com/coderintherye/kiva-map

I'm kind of intrigued by the idea of possibly setting up some service where we could automatically generate these kind of images for Team and Lender pages.

Cheers,

-k

[Sean Durkin]

Hi Kevin,

A service to generate maps for lenders and lending teams sounds awesome! I'm glad you found the github project, I was planning on making that config file change but I see you've already done that. :) When I get home from work today, I'll figure out how to accept that pull request (still learning how to use git).

Later today I'll also commit some scripts that I used to generate the buildkiva map. A Python script fetches all lenders for a team, all loans, and then for each loan it finds the subset of lenders that are within the team and adds them to the dataset. Then it invokes the R script. I was thinking it could be enhanced to work off existing data so that when it's invoked to regenerate a map after N loans, it only fetches up to ~N loans (sorted by newest); once it found a loan id that has already previously been processed, it would write to the csv files with all the new data and then invoke the R script. What do you think?

Sean