batch mutator insert very slow

[Pavan Kumar]

Hi All,

   I am doing a batch insert via Mutator in Pycassa for composite rows and composite column names for counter Column Family. I see very slow write speeds. When i profiled with out network (mutator.send() ) on my MBP I am getting on average 3 seconds for 50K counter increment batch formations. I set the queue size to 0 to avoid network and cassandra write affecting the timing.

Is this common  ? or pycassa needs tweaking . Why should forming the batch packets takes that long ?

Attached is the test code.

regards,

Pavan

[Tyler Hobbs]

On Tue, May 14, 2013 at 12:43 AM, Pavan Kumar <tvp...@gmail.com> wrote:

   I am doing a batch insert via Mutator in Pycassa for composite rows and composite column names for counter Column Family. I see very slow write speeds. When i profiled with out network (mutator.send() ) on my MBP I am getting on average 3 seconds for 50K counter increment batch formations. I set the queue size to 0 to avoid network and cassandra write affecting the timing.

Is this common  ? or pycassa needs tweaking . Why should forming the batch packets takes that long ?

50000 is a really large batch size.  Try much smaller batches (100 to 1000) in parallel with multiple threads.

Also, counter updates are much more expensive than normal writes with Cassandra, so the cluster itself may not be able to handle a high rate of counter updates.

--
Tyler Hobbs
DataStax