Importing data to Riak - NoSQL Databases

The old Google Groups will be going away soon, but your browser is incompatible with the new version.
Message from discussion Importing data to Riak
Received-SPF: pass (google.com: domain of the.mindstorm.mailinglist+caf_=nosql-databases=googlegroups....@gmail.com designates 209.85.161.46 as permitted sender) client-ip=209.85.161.46;
Received-SPF: neutral (google.com: 72.52.162.75 is neither permitted nor denied by best guess record for domain of riak-users-boun...@lists.basho.com) client-ip=72.52.162.75;
MIME-Version: 1.0
In-Reply-To: <CAHgGLC=JGn-jOqPTdPFptsHczsFXurP3ubeJgjHE1T+97Vo...@mail.gmail.com>
References: <CAEYuqBtq4AWyG7OGC8kAeO6wa7HVcvLxhQgvDHy_R3AjKuE...@mail.gmail.com>
	<EC9EB045-2336-4D00-BF37-0567E761F...@basho.com>
	<CAHgGLC=JGn-jOqPTdPFptsHczsFXurP3ubeJgjHE1T+97Vo...@mail.gmail.com>
Date: Tue, 15 Nov 2011 18:08:12 +0100
Message-ID: <CAEYuqBsfbhXDCXhHW7U_pgw6qaFA3hdV6tGD3TfKLiZU+D+...@mail.gmail.com>
Subject: Re: Importing data to Riak
From: Nitish Sharma <sharmanitishd...@gmail.com>
To: Andres Jaan Tack <andres.jaan.t...@eesti.ee>
Cc: riak-users <riak-us...@lists.basho.com>,
	Russell Brown <russel...@basho.com>
Precedence: list
Content-Type: multipart/mixed; boundary="===============1866951781=="
Errors-To: riak-users-boun...@lists.basho.com
Sender: riak-users-boun...@lists.basho.com

--===============1866951781==
Content-Type: multipart/alternative; boundary=e89a8f23545d4ef4c504b1c908b3

--e89a8f23545d4ef4c504b1c908b3
Content-Type: text/plain; charset=ISO-8859-1

Hi,
I tried importing the data using Python library (with protocol buffers).
After storing several objects, I get thread exception with timeout errors.
Following is the traceback:

  File "/usr/lib/python2.7/threading.py", line 552, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 505, in run
    self.__target(*self.__args, **self.__kwargs)
  File "python_load_data.py", line 23, in worker
    new_obj.store()
  File
"/usr/local/lib/python2.7/dist-packages/riak-1.3.0-py2.7.egg/riak/riak_object.py",
line 296, in store
    Result = t.put(self, w, dw, return_body)
  File
"/usr/local/lib/python2.7/dist-packages/riak-1.3.0-py2.7.egg/riak/transports/pbc.py",
line 188, in put
    msg_code, resp = self.recv_msg()
  File
"/usr/local/lib/python2.7/dist-packages/riak-1.3.0-py2.7.egg/riak/transports/pbc.py",
line 370, in recv_msg
    raise Exception(msg.errmsg)
Exception: timeout

The cluster consists of 3 nodes (Ubuntu 10.04).
Any Suggestions?

Cheers
Nitish

On Mon, Nov 14, 2011 at 2:20 PM, Andres Jaan Tack <andres.jaan.t...@eesti.ee
> wrote:

> I was able to achieve similar results. I wrote a Ruby process that would
> keep at most n (I think n = 10) things at once and reached 2,500ish req/s
> on my macbook pro.
>
> I loaded data to a cluster of six Riak nodes by running several of these
> processes at once and attaching each to a different Riak node, and I hit
> 18,000 req/s. I'm not sure whether loading different nodes affected the
> speed or not, now that I think of it.
>
>
> 2011/11/14 Russell Brown <russel...@basho.com>
>
>>
>> On 14 Nov 2011, at 11:47, Nitish Sharma wrote:
>>
>> > Hi,
>> > This is more sort of a discussion than a question. I am just trying to
>> see the trend in how users import their data to Riak.
>> > For the data I am using, I was able to achieve almost 150
>> records/second with PHP library, and 400 records/second with node.js
>> (fairly new with node; was hitting memory wall when trying to import 1
>> million records).
>> > What are some hacks/tricks/tweaks to import large amount of data to
>> Riak?
>>
>> New keys, new data, straight in for the first time, no fetch before
>> store? I've had reasonable results creating a *number* of threads and using
>> the Java Raw PB client to write.
>>
>> For example, maybe have a 1 or a couple of threads that reads data (from
>> Oracle, a file, what-have-you) and puts it on a queue, and have a bunch of
>> threads that pull data off the queue, create a riak object and store it.
>> From my laptop I've got up to 2500 writes a second like this, and it was
>> just ad hoc, throw away code with 4 threads against a small 3 node cluster
>> (running on desktops.)
>>
>> I imagine others on the list have more direct, real world examples?
>>
>> Cheers
>>
>> Russell
>>
>> >
>> > Cheers
>> > Nitish
>> > _______________________________________________
>> > riak-users mailing list
>> > riak-us...@lists.basho.com
>> > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>
>>
>> _______________________________________________
>> riak-users mailing list
>> riak-us...@lists.basho.com
>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>
>
>

--e89a8f23545d4ef4c504b1c908b3
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Hi,<br>
I tried importing the data using Python library (with protocol buffers).
 After storing several objects, I get thread exception with timeout errors.=
 Following=20
is the traceback:<br>
<br>
=A0 File &quot;/usr/lib/python2.7/threading.py&quot;, line 552, in __bootst=
rap_inner<br>
=A0=A0=A0 self.run()<br>
=A0 File &quot;/usr/lib/python2.7/threading.py&quot;, line 505, in run<br>
=A0=A0=A0 self.__target(*self.__args, **self.__kwargs)<br>
=A0 File &quot;python_load_data.py&quot;, line 23, in worker<br>
=A0=A0=A0 new_obj.store()<br>
=A0 File &quot;/usr/local/lib/python2.7/dist-packages/riak-1.3.0-py2.7.egg/=
riak/riak_object.py&quot;, line 296, in store<br>
=A0=A0=A0 Result =3D t.put(self, w, dw, return_body)<br>
=A0 File &quot;/usr/local/lib/python2.7/dist-packages/riak-1.3.0-py2.7.egg/=
riak/transports/pbc.py&quot;, line 188, in put<br>
=A0=A0=A0 msg_code, resp =3D self.recv_msg()<br>
=A0 File &quot;/usr/local/lib/python2.7/dist-packages/riak-1.3.0-py2.7.egg/=
riak/transports/pbc.py&quot;, line 370, in recv_msg<br>
=A0=A0=A0 raise Exception(msg.errmsg)<br>
Exception: timeout<br><br>
The cluster consists of 3 nodes (Ubuntu 10.04).<br>
Any Suggestions?<br>
<br>
Cheers<br>
Nitish<br><br><div class=3D"gmail_quote">On Mon, Nov 14, 2011 at 2:20 PM, A=
ndres Jaan Tack <span dir=3D"ltr">&lt;<a href=3D"mailto:andres.jaan.tack@ee=
sti.ee">andres.jaan.t...@eesti.ee</a>&gt;</span> wrote:<br><blockquote clas=
s=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;pad=
ding-left:1ex;">
I was able to achieve similar results. I wrote a Ruby process that would ke=
ep at most n (I think n =3D 10) things at once and reached 2,500ish req/s o=
n my macbook pro.<div><br></div><div>I loaded data to a cluster of six Riak=
 nodes by running several of these processes at once and attaching each to =
a different Riak node, and I hit 18,000 req/s. I&#39;m not sure whether loa=
ding different nodes affected the speed or not, now that I think of it.<div=
>
<div></div><div class=3D"h5"><br>
<br><div class=3D"gmail_quote">2011/11/14 Russell Brown <span dir=3D"ltr">&=
lt;<a href=3D"mailto:russel...@basho.com" target=3D"_blank">russelldb@basho=
.com</a>&gt;</span><br><blockquote class=3D"gmail_quote" style=3D"margin:0 =
0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">

<div><div></div><div><br>
On 14 Nov 2011, at 11:47, Nitish Sharma wrote:<br>
<br>
&gt; Hi,<br>
&gt; This is more sort of a discussion than a question. I am just trying to=
 see the trend in how users import their data to Riak.<br>
&gt; For the data I am using, I was able to achieve almost 150 records/seco=
nd with PHP library, and 400 records/second with node.js (fairly new with n=
ode; was hitting memory wall when trying to import 1 million records).<br>


&gt; What are some hacks/tricks/tweaks to import large amount of data to Ri=
ak?<br>
<br>
</div></div>New keys, new data, straight in for the first time, no fetch be=
fore store? I&#39;ve had reasonable results creating a *number* of threads =
and using the Java Raw PB client to write.<br>
<br>
For example, maybe have a 1 or a couple of threads that reads data (from Or=
acle, a file, what-have-you) and puts it on a queue, and have a bunch of th=
reads that pull data off the queue, create a riak object and store it. From=
 my laptop I&#39;ve got up to 2500 writes a second like this, and it was ju=
st ad hoc, throw away code with 4 threads against a small 3 node cluster (r=
unning on desktops.)<br>


<br>
I imagine others on the list have more direct, real world examples?<br>
<br>
Cheers<br>
<br>
Russell<br>
<br>
&gt;<br>
&gt; Cheers<br>
&gt; Nitish<br>
&gt; _______________________________________________<br>
&gt; riak-users mailing list<br>
&gt; <a href=3D"mailto:riak-us...@lists.basho.com" target=3D"_blank">riak-u=
s...@lists.basho.com</a><br>
&gt; <a href=3D"http://lists.basho.com/mailman/listinfo/riak-users_lists.ba=
sho.com" target=3D"_blank">http://lists.basho.com/mailman/listinfo/riak-use=
rs_lists.basho.com</a><br>
<br>
<br>
_______________________________________________<br>
riak-users mailing list<br>
<a href=3D"mailto:riak-us...@lists.basho.com" target=3D"_blank">riak-users@=
lists.basho.com</a><br>
<a href=3D"http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.c=
om" target=3D"_blank">http://lists.basho.com/mailman/listinfo/riak-users_li=
sts.basho.com</a><br>
</blockquote></div><br></div></div></div>
</blockquote></div><br>

--e89a8f23545d4ef4c504b1c908b3--


--===============1866951781==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

_______________________________________________
riak-users mailing list
riak-us...@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

--===============1866951781==--
Create a group - Google Groups - Google Home - Terms of Service - Privacy Policy
Account Options
