data dump

[Matthew Pocock]

Hi,

Is there a way to get a data dump for the kiva info? I ideally need
historical as well as current data. Are there data dumps, or should I
go through the APIs, or something else?

Thanks,

Matthew

[Vijay]

http://build.kiva.org/docs/data/snapshots

I tried it a couple of times, a while ago, but the data was out of date.  just checked this page, looks
like it is still out of date.

It would be awesome to have up-to-date data dump

--
You received this message because you are subscribed to the Google Groups "build-kiva" group.
To post to this group, send email to build...@googlegroups.com.
To unsubscribe from this group, send email to build-kiva+...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/build-kiva?hl=en.

[Paul Ericksen]

Yes, the data dumps are currently broken and there is currently not a
plan to fix them, so you should use the API. What type of historical
data are you looking for?

[Matthew Pocock]

Hi Paul,

We'd like to know about the borrowers, lenders, and the details of loans - their amounts, if they where repaid or not and so on. We'd ideally also like to know how much particular lenders contributed and when they contributed it. The API at http://api.kivaws.org/v1/lending_actions/recent.json covers part of what we need, but is limited to the most recent 50 transactions.

You say there's not a plan to fix the data dumps. Is this something where the donation of some time and expertise could help you? A complete data dump would be the ideal solution for me, and I'd be happy to work with you with any server-side stuff needed to make this work.

Thanks,

Matthew

--

Dr Matthew Pocock

Visitor, School of Computing Science, Newcastle University

mailto: turingate...@gmail.com

gchat: turingate...@gmail.com

msn: matthew...@yahoo.co.uk

irc.freenode.net: drdozer

tel: (0191) 2566550

mob: +447535664143

[Noah Balmer]

Thanks for the offer, Matthew. 

Getting the data dumps working again is an expensive undertaking.  The simplest approach we've discussed so far starts with buying and maintaining more hardware, revisiting some aspects of our architecture, going through some legal and privacy policy review, and then, when all that's ironed out, rewriting the scripts.  There's no way to do it that doesn't take considerable investment on our part, and at this point it's not on the top of anyone's priority list. 

It's also unlikely that we can disclose everything you're looking for... for example, we've never published the exact amount a given lender has lent, and we're unlikely to start. 

I'd like to see some version of the data dumps working again, but I don't see it happening in the near future.

There is a lot of repayment data available via the API.  Is there something particular you're trying to figure out that you can't learn from the API?

-Noah

[Paul Lam]

Speaking as a data scientist, a data dump would be useful to do exploratory analysis. For example, wouldn't you like to know why some loans get funded and others don't? So that you can do something about it?

Instead of taking snapshots of the database, why not push your data at the generation point on the web app into a message queue which then pushes into Amazon S3 directly? This will probably take a few weeks to do, so unfortunately, I doubt you guys have the time.

Matthew

> To unsubscribe from this group, send email to build-kiva+unsubscribe@​googlegroups.com.

> For more options, visit this group at http://groups.google.com/​group/build-kiva?hl=en.
>
>

--
You received this message because you are subscribed to the Google Groups "build-kiva" group.
To post to this group, send email to build...@googlegroups.com.

To unsubscribe from this group, send email to build-kiva+unsubscribe@​googlegroups.com.

For more options, visit this group at http://groups.google.com/​group/build-kiva?hl=en.

--

Dr Matthew Pocock

Visitor, School of Computing Science, Newcastle University

mailto: turingatemyhamster@​gmail.com

gchat: turingatemyhamster@​gmail.com

msn: matthew...@yahoo.co.uk

irc.freenode.net: drdozer

tel: (0191) 2566550

mob: +447535664143

--

You received this message because you are subscribed to the Google Groups "build-kiva" group.
To post to this group, send email to build...@googlegroups.com.

To unsubscribe from this group, send email to build-kiva+unsubscribe@​googlegroups.com.

[Noah Balmer]

We have our own data analysis group in-house who answer questions of the type you describe.  There are a variety of reasons, such as user privacy, that we don't expose all of their tools for public consumption. 

We are actively working on getting the data dumps, as they existed before, working again. Stay tuned for updates.

-Noah

To view this discussion on the web visit https://groups.google.com/d/msg/build-kiva/-/AjufSulHoioJ.

To post to this group, send email to build...@googlegroups.com.

To unsubscribe from this group, send email to build-kiva+...@googlegroups.com.