HectorException: All host pools marked down. Retry burden pushed out to client.

[rohit reddy]

 Hi,

 I'm pretty new to using Cassandra and Hector. I'm trying to do a POC to get hang of cassandra and hector.

 While i'm trying to insert data into cassandra.. i keep running into this exception:

  me.prettyprint.hector.api.exceptions.HectorException: All host pools marked down. Retry burden pushed out to client.

        at me.prettyprint.cassandra.connection.HConnectionManager.getClientFromLBPolicy(HConnectionManager.java:393)

        at me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:249)

        at me.prettyprint.cassandra.model.ExecutingKeyspace.doExecuteOperation(ExecutingKeyspace.java:97)

        at me.prettyprint.cassandra.model.MutatorImpl.execute(MutatorImpl.java:243)

 Getting this exception when i'm trying to do a batch insert using mutator.addInsertion method. My cassandra cluster is deployed on 2 EC2 instances.

 I have given both the hostnames to CassandraHostConfigurator and also set AutoDiscoverHosts to true.

 Tried searching online for any help, but did not get any useful information.

 Any help is greatly appreciated

 Version i'm using:

 Cassandra - 1.1.4

 Hector - 1.0-5

Thanks

Rohit

 

[Nate McCall]

How big are your batch sizes? It's possible you are overwhelming the
cluster. This will be the case if you see a number of
HTimeoutExceptions coming back before you see this message.

[Patricio Echagüe]

Is hector connecting at all?

[rohit reddy]

Yes, Hector is connecting. I was able to create Column families using the client.

First 4-5 batch inserts work, then the error starts coming.

The error is coming irrespective of the batch size i'm using. I tested with different batch sizes. 

I did not see any error in cassandra logs. The cpu and memory of the EC2 nodes where cassandra is installed also looks good.

Thanks

Rohit

[Robin Verlangen]

The question of Nate is actually a good one: "How big are your batch sizes?"

With kind regards,

Robin Verlangen

Software engineer

W http://www.robinverlangen.nl

E ro...@us2.nl

Disclaimer: The information contained in this message and attachments is intended solely for the attention and use of the named addressee and may be confidential. If you are not the intended recipient, you are reminded that the information remains the property of the sender. You must not use, disclose, distribute, copy, print or rely on this e-mail. If you have received this message in error, please contact the sender immediately and irrevocably delete this message and any copies.

2012/9/4 rohit reddy <rohit.ko...@gmail.com>

[rohit reddy]

Hi,

The batch size is 500. (adding 500 columns at a time). It would be about 250KB - 300KB

Thanks

Rohit

[Nate McCall]

What errors are you seing from the application logs previous to
getting this exception?

Try running 'nodetool tpstats' at the same time and see if there are a
large number of requests in the pending column of the Mutation queue.

What size EC2 instances are these?

[rohit reddy]

Thanks for the replies,

I'm using 2 large EC2 nodes. I stopped getting the errors when i reduced the batch size to 100 (60KB). 

I'll monitor the nodes using tpstats to see if large requests are causing this.

Thanks

Rohit

[rohit reddy]

Decreasing the batch size seems to solve the issue.

But the hector client performance seems to be a issue.

I'm using 2 clients each running 10 threads to insert the data. 

Each thread has its own Mutator which i'm creating using 

HFactory.createMutator(keyspace, IntegerSerializer.get());

keyspace and cluster are one per client.

I'm just getting about 400 writes/sec using this setup. Whereas in cassandra the write latency is about 50microSecs.

I feel the client is becoming a bottleneck. 

Is this the right way to use Hector?

Thanks

Rohit

[Nate McCall]

Not quite sure what you are doing from the description, but here is an
example of doing parallelized inserts into a time series CF:

https://github.com/zznate/strata-west-2012/blob/master/storage-model/src/main/java/org/apigee/tutorial/BucketingTimeSeriesInserter.java

[rohit reddy]

Thanks a lot for your help,

Got it working now. 

Rohit