data dump - build-kiva | Google Groups

The old Google Groups will be going away soon, but your browser is incompatible with the new version.
Message from discussion data dump
Received-SPF: pass (google.com: domain of n...@kiva.org designates 74.125.245.92 as permitted sender) client-ip=74.125.245.92;
MIME-Version: 1.0
In-Reply-To: <3242365.649.1332630777580.JavaMail.geo-discussion-forums@yneo2>
References: <1778c27c-a066-405f-8685-138d05205...@1g2000vbu.googlegroups.com>
	<CAA7u5n7C_7TH-RAFMFt=hp1wps_uanXA4Hdd2Os72B1m2QQ...@mail.gmail.com>
	<CAHQS_ex0tNDA+UrxBWHw+t6Vq4tA=K9BUSN3CO3Qj4fY3cr...@mail.gmail.com>
	<CAOEcmSJ9h-EL2Gn4Ogufk-j_aJXqTjZJcZUY0eBuu-3XpHh...@mail.gmail.com>
	<3242365.649.1332630777580.JavaMail.geo-discussion-forums@yneo2>
Date: Tue, 27 Mar 2012 17:15:24 -0700
Message-ID: <CAOEcmSL_T1MrJ4wmBCWrP4zb_Jsi+T50gABO3atu98LtHBy...@mail.gmail.com>
Subject: Re: data dump
From: Noah Balmer <n...@kiva.org>
To: build-kiva@googlegroups.com
Content-Type: multipart/alternative; boundary=14dae9340e3ff2466904bc4280bb

--14dae9340e3ff2466904bc4280bb
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

We have our own data analysis group in-house who answer questions of the
type you describe.  There are a variety of reasons, such as user privacy,
that we don't expose all of their tools for public consumption.

We are actively working on getting the data dumps, as they existed before,
working again. Stay tuned for updates.

-Noah



On Sat, Mar 24, 2012 at 4:12 PM, Paul Lam <pty...@gmail.com> wrote:

> Speaking as a data scientist, a data dump would be useful to do
> exploratory analysis. For example, wouldn't you like to know why some loa=
ns
> get funded and others don't? So that you can do something about it?
>
> Instead of taking snapshots of the database, why not push your data at th=
e
> generation point on the web app into a message queue which then pushes in=
to
> Amazon S3 directly? This will probably take a few weeks to do, so
> unfortunately, I doubt you guys have the time.
>
>
>
> On Friday, 12 August 2011 07:25:19 UTC+1, Noah Balmer wrote:
>
>> Thanks for the offer, Matthew.
>>
>> Getting the data dumps working again is an expensive undertaking.  The
>> simplest approach we've discussed so far starts with buying and maintain=
ing
>> more hardware, revisiting some aspects of our architecture, going throug=
h
>> some legal and privacy policy review, and then, when all that's ironed o=
ut,
>> rewriting the scripts.  There's no way to do it that doesn't take
>> considerable investment on our part, and at this point it's not on the t=
op
>> of anyone's priority list.
>>
>> It's also unlikely that we can disclose everything you're looking for...
>> for example, we've never published the exact amount a given lender has
>> lent, and we're unlikely to start.
>>
>> I'd like to see some version of the data dumps working again, but I don'=
t
>> see it happening in the near future.
>>
>> There is a lot of repayment data available via the API.  Is there
>> something particular you're trying to figure out that you can't learn fr=
om
>> the API?
>>
>> -Noah
>>
>> On Wed, Aug 10, 2011 at 12:44 PM, Matthew Pocock <
>> turingatemyhams...@gmail.com> wrote:
>>
>>> Hi Paul,
>>>
>>> We'd like to know about the borrowers, lenders, and the details of loan=
s
>>> - their amounts, if they where repaid or not and so on. We'd ideally al=
so
>>> like to know how much particular lenders contributed and when they
>>> contributed it. The API at
>>> http://api.kivaws.org/v1/=E2=80=8Blending_actions/recent.json<http://ap=
i.kivaws.org/v1/lending_actions/recent.json> =E2=80=8Bcovers
>>> part of what we need, but is limited to the most recent 50 transactions=
.
>>>
>>> You say there's not a plan to fix the data dumps. Is this something
>>> where the donation of some time and expertise could help you? A complet=
e
>>> data dump would be the ideal solution for me, and I'd be happy to work =
with
>>> you with any server-side stuff needed to make this work.
>>>
>>> Thanks,
>>>
>>> Matthew
>>>
>>>
>>> On 10 August 2011 20:26, Paul Ericksen <liquidmon...@gmail.com> wrote:
>>>
>>>> Yes, the data dumps are currently broken and there is currently not a
>>>> plan to fix them, so you should use the API. What type of historical
>>>> data are you looking for?
>>>>
>>>> On Tue, Aug 9, 2011 at 12:20 PM, Matthew Pocock
>>>> <turingatemyhams...@gmail.com> wrote:
>>>> > Hi,
>>>> >
>>>> > Is there a way to get a data dump for the kiva info? I ideally need
>>>> > historical as well as current data. Are there data dumps, or should =
I
>>>> > go through the APIs, or something else?
>>>> >
>>>> > Thanks,
>>>> >
>>>> > Matthew
>>>> >
>>>> > --
>>>> > You received this message because you are subscribed to the Google
>>>> Groups "build-kiva" group.
>>>> > To post to this group, send email to build-kiva@googlegroups.com.
>>>> > To unsubscribe from this group, send email to
>>>> build-kiva+unsubscribe@=E2=80=8Bgooglegroups.com<build-kiva%2Bunsubscr=
ibe@googlegroups.com>
>>>> .
>>>> > For more options, visit this group at
>>>> http://groups.google.com/=E2=80=8Bgroup/build-kiva?hl=3Den<http://grou=
ps.google.com/group/build-kiva?hl=3Den>
>>>> .
>>>> >
>>>> >
>>>>
>>>> --
>>>> You received this message because you are subscribed to the Google
>>>> Groups "build-kiva" group.
>>>> To post to this group, send email to build-kiva@googlegroups.com.
>>>> To unsubscribe from this group, send email to
>>>> build-kiva+unsubscribe@=E2=80=8Bgooglegroups.com<build-kiva%2Bunsubscr=
ibe@googlegroups.com>
>>>> .
>>>> For more options, visit this group at
>>>> http://groups.google.com/=E2=80=8Bgroup/build-kiva?hl=3Den<http://grou=
ps.google.com/group/build-kiva?hl=3Den>
>>>> .
>>>>
>>>>
>>>
>>>
>>> --
>>> Dr Matthew Pocock
>>> Visitor, School of Computing Science, Newcastle University
>>> mailto: turingatemyhamster@=E2=80=8Bgmail.com <turingatemyhamster@gmail=
.com>
>>> gchat: turingatemyhamster@=E2=80=8Bgmail.com <turingatemyhamster@gmail.=
com>
>>> msn: matthew_poc...@yahoo.co.uk
>>> irc.freenode.net: drdozer
>>> tel: (0191) 2566550
>>> mob: +447535664143
>>>
>>>  --
>>> You received this message because you are subscribed to the Google
>>> Groups "build-kiva" group.
>>> To post to this group, send email to build-kiva@googlegroups.com.
>>> To unsubscribe from this group, send email to
>>> build-kiva+unsubscribe@=E2=80=8Bgooglegroups.com<build-kiva%2Bunsubscri=
be@googlegroups.com>
>>> .
>>> For more options, visit this group at
>>> http://groups.google.com/=E2=80=8Bgroup/build-kiva?hl=3Den<http://group=
s.google.com/group/build-kiva?hl=3Den>
>>> .
>>>
>>
>>  --
> You received this message because you are subscribed to the Google Groups
> "build-kiva" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/build-kiva/-/AjufSulHoioJ.
>
> To post to this group, send email to build-kiva@googlegroups.com.
> To unsubscribe from this group, send email to
> build-kiva+unsubscribe@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/build-kiva?hl=3Den.
>

--14dae9340e3ff2466904bc4280bb
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

We have our own data analysis group in-house who answer questions of the ty=
pe you describe. =C2=A0There are a variety of reasons, such as user privacy=
, that we don&#39;t expose all of their tools for public consumption.=C2=A0=
<div><br>
</div><div>We are actively working on getting the data dumps, as they exist=
ed before, working again. Stay tuned for updates.</div><div><br></div><div>=
-Noah</div><div><br></div><div><br></div><div><br><div class=3D"gmail_quote=
">
On Sat, Mar 24, 2012 at 4:12 PM, Paul Lam <span dir=3D"ltr">&lt;<a href=3D"=
mailto:pty...@gmail.com">pty...@gmail.com</a>&gt;</span> wrote:<br><blockqu=
ote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc s=
olid;padding-left:1ex">
Speaking as a data scientist, a data dump would be useful to do exploratory=
 analysis. For example, wouldn&#39;t you like to know why some loans get fu=
nded and others don&#39;t? So that you can do something about it?<div><br>
</div><div>Instead of taking snapshots of the database, why not push your d=
ata at the generation point on the web app into a message queue which then =
pushes into Amazon S3 directly? This will probably take a few weeks to do, =
so unfortunately, I doubt you guys have the time.<div class=3D"im">
<div style=3D"border-style:initial;border-color:initial"><br></div><br><br>=
On Friday, 12 August 2011 07:25:19 UTC+1, Noah Balmer  wrote:</div><blockqu=
ote class=3D"gmail_quote" style=3D"margin:0;margin-left:0.8ex;border-left:1=
px #ccc solid;padding-left:1ex">
<div class=3D"im">Thanks for the offer, Matthew.=C2=A0 <br>
<br>
Getting the data dumps working again is an expensive undertaking.=C2=A0 The=
=20
simplest approach we&#39;ve discussed so far starts with buying and=20
maintaining more hardware, revisiting some aspects of our architecture,=20
going through some legal and privacy policy review, and then, when all=20
that&#39;s ironed out, rewriting the scripts.=C2=A0 There&#39;s no way to d=
o it that=20
doesn&#39;t take considerable investment on our part, and at this point it&=
#39;s
 not on the top of anyone&#39;s priority list.=C2=A0 <br>
<br>
It&#39;s also unlikely that we can disclose everything you&#39;re looking f=
or...
 for example, we&#39;ve never published the exact amount a given lender has=
=20
lent, and we&#39;re unlikely to start.=C2=A0 <br>
<br>
I&#39;d like to see some version of the data dumps working again, but I don=
&#39;t see it happening in the near future.<br>
<br>
There is a lot of repayment data available via the API.=C2=A0 Is there=20
something particular you&#39;re trying to figure out that you can&#39;t lea=
rn=20
from the API?<br>
<br>
-Noah<br><br></div><div class=3D"gmail_quote"><div class=3D"im">On Wed, Aug=
 10, 2011 at 12:44 PM, Matthew Pocock <span dir=3D"ltr">&lt;<a href=3D"mail=
to:turingatemyhams...@gmail.com" target=3D"_blank">turingatemyhamster@gmail=
.com</a>&gt;</span> wrote:<br>
</div><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-l=
eft:1px #ccc solid;padding-left:1ex">
Hi Paul,<div><br></div><div>We&#39;d like to know about the borrowers, lend=
ers, and the details of loans - their amounts, if they where repaid or not =
and so on. We&#39;d ideally also like to know how much particular lenders c=
ontributed and when they contributed it. The API at=C2=A0<a href=3D"http://=
api.kivaws.org/v1/lending_actions/recent.json" target=3D"_blank">http://api=
.kivaws.org/v1/=E2=80=8Blending_actions/recent.json</a>=C2=A0=E2=80=8Bcover=
s part of what we need, but is limited to the most recent 50 transactions.<=
/div>
<div><div class=3D"h5">

<div><br></div><div>You say there&#39;s not a plan to fix the data dumps. I=
s this something where the donation of some time and=C2=A0expertise=C2=A0co=
uld help you? A complete data dump would be the ideal solution for me, and =
I&#39;d be happy to work with you with any server-side stuff needed to make=
 this work.</div>


<div><br></div><div>Thanks,</div><div><br></div></div></div><div><div><div =
class=3D"h5">Matthew<div><div></div><div><br><br><div class=3D"gmail_quote"=
>On 10 August 2011 20:26, Paul Ericksen <span dir=3D"ltr">&lt;<a href=3D"ma=
ilto:liquidmon...@gmail.com" target=3D"_blank">liquidmon...@gmail.com</a>&g=
t;</span> wrote:<br>


<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
x #ccc solid;padding-left:1ex">Yes, the data dumps are currently broken and=
 there is currently not a<br>
plan to fix them, so you should use the API. What type of historical<br>
data are you looking for?<br>
<div><div></div><div><br>
On Tue, Aug 9, 2011 at 12:20 PM, Matthew Pocock<br>
&lt;<a href=3D"mailto:turingatemyhams...@gmail.com" target=3D"_blank">turin=
gatemyhams...@gmail.com</a>&gt; wrote:<br>
&gt; Hi,<br>
&gt;<br>
&gt; Is there a way to get a data dump for the kiva info? I ideally need<br=
>
&gt; historical as well as current data. Are there data dumps, or should I<=
br>
&gt; go through the APIs, or something else?<br>
&gt;<br>
&gt; Thanks,<br>
&gt;<br>
&gt; Matthew<br>
&gt;<br>
</div></div><div><div></div><div>&gt; --<br>
&gt; You received this message because you are subscribed to the Google Gro=
ups &quot;build-kiva&quot; group.<br>
&gt; To post to this group, send email to <a href=3D"mailto:build-kiva@goog=
legroups.com" target=3D"_blank">build-kiva@googlegroups.com</a>.<br>
&gt; To unsubscribe from this group, send email to <a href=3D"mailto:build-=
kiva%2Bunsubscribe@googlegroups.com" target=3D"_blank">build-kiva+unsubscri=
be@=E2=80=8Bgooglegroups.com</a>.<br>
&gt; For more options, visit this group at <a href=3D"http://groups.google.=
com/group/build-kiva?hl=3Den" target=3D"_blank">http://groups.google.com/=
=E2=80=8Bgroup/build-kiva?hl=3Den</a>.<br>
&gt;<br>
&gt;<br>
<br>
--<br>
You received this message because you are subscribed to the Google Groups &=
quot;build-kiva&quot; group.<br>
To post to this group, send email to <a href=3D"mailto:build-kiva@googlegro=
ups.com" target=3D"_blank">build-kiva@googlegroups.com</a>.<br>
To unsubscribe from this group, send email to <a href=3D"mailto:build-kiva%=
2Bunsubscribe@googlegroups.com" target=3D"_blank">build-kiva+unsubscribe@=
=E2=80=8Bgooglegroups.com</a>.<br>
For more options, visit this group at <a href=3D"http://groups.google.com/g=
roup/build-kiva?hl=3Den" target=3D"_blank">http://groups.google.com/=E2=80=
=8Bgroup/build-kiva?hl=3Den</a>.<br>
<br>
</div></div></blockquote></div><br><br clear=3D"all"><br></div></div></div>=
</div><div class=3D"im">-- <br><div>Dr Matthew Pocock</div><div>Visitor, Sc=
hool of Computing Science, Newcastle University</div><div>mailto:=C2=A0<a h=
ref=3D"mailto:turingatemyhams...@gmail.com" target=3D"_blank">turingatemyha=
mster@=E2=80=8Bgmail.com</a></div>


<div>gchat:=C2=A0<a href=3D"mailto:turingatemyhams...@gmail.com" target=3D"=
_blank">turingatemyhamster@=E2=80=8Bgmail.com</a></div><div>msn: <a href=3D=
"mailto:matthew_poc...@yahoo.co.uk" target=3D"_blank">matthew_pocock@yahoo.=
co.uk</a></div><div>


<a href=3D"http://irc.freenode.net" target=3D"_blank">irc.freenode.net</a>:=
 drdozer</div><div>tel: (0191) 2566550</div><div>mob:=C2=A0<span style=3D"b=
order-collapse:collapse;font-family:arial,sans-serif;font-size:13px"><a val=
ue=3D"+447535664143">+447535664143</a></span></div>


<br>
</div></div><font color=3D"#888888">

<p></p>

-- <br></font><div><div></div><div><div class=3D"im">
You received this message because you are subscribed to the Google Groups &=
quot;build-kiva&quot; group.<br>
To post to this group, send email to <a href=3D"mailto:build-kiva@googlegro=
ups.com" target=3D"_blank">build-kiva@googlegroups.com</a>.<br>
To unsubscribe from this group, send email to <a href=3D"mailto:build-kiva%=
2Bunsubscribe@googlegroups.com" target=3D"_blank">build-kiva+unsubscribe@=
=E2=80=8Bgooglegroups.com</a>.<br></div><div class=3D"im">

For more options, visit this group at <a href=3D"http://groups.google.com/g=
roup/build-kiva?hl=3Den" target=3D"_blank">http://groups.google.com/=E2=80=
=8Bgroup/build-kiva?hl=3Den</a>.<br>


</div></div></div></blockquote></div><br>
</blockquote></div><div class=3D"im">

<p></p>

-- <br>
You received this message because you are subscribed to the Google Groups &=
quot;build-kiva&quot; group.<br></div>
To view this discussion on the web visit <a href=3D"https://groups.google.c=
om/d/msg/build-kiva/-/AjufSulHoioJ" target=3D"_blank">https://groups.google=
.com/d/msg/build-kiva/-/AjufSulHoioJ</a>.<div class=3D"HOEnZb"><div class=
=3D"h5">
<br>=20
To post to this group, send email to <a href=3D"mailto:build-kiva@googlegro=
ups.com" target=3D"_blank">build-kiva@googlegroups.com</a>.<br>
To unsubscribe from this group, send email to <a href=3D"mailto:build-kiva%=
2Bunsubscribe@googlegroups.com" target=3D"_blank">build-kiva+unsubscribe@go=
oglegroups.com</a>.<br>

For more options, visit this group at <a href=3D"http://groups.google.com/g=
roup/build-kiva?hl=3Den" target=3D"_blank">http://groups.google.com/group/b=
uild-kiva?hl=3Den</a>.<br>


</div></div></blockquote></div><br></div>

--14dae9340e3ff2466904bc4280bb--
Create a group - Google Groups - Google Home - Terms of Service - Privacy Policy
Account Options
