Hello,
We are using hector-core1.1-1 with Cassandra 1.1.5 (3 nodes).
We are inserting batches in Cassandra using me.prettyprint.hector.api.mutation.Mutator.execute()
At times we get HTimeoutExceptions (see below).
Question:
1. Does this means we lost data? (i.e no insertion was made in Cassandra and we just discarded the data?)
2. How should our client code handle this? (they are RunTime exceptions so technically they shouldn't be catched)
3. In the logs they appear with a WARN. Does this mean that the underlying Hector code recovered from it? or Does it mean that the Hector code tried a number of times and failed? (in which case shouldn't it be reported as an ERROR in the logs?).
4. As I understand the mechanism, Hector operates in failover which for us means: it tries to issue a request to one node a configurable number of times (sleeping in the middle for a configurable amount of time) and if it doesn't manage with that node it tries the next node, and so on until it exhausted all nodes it knows of and then it throws the HTimeoutExceptions to our client code. Is this an accurate description? If yes how can we configure the number of times it tries per node and how much it sleeps in between?
Cheers,
Alex
WARN HConnectionManager - Could not fullfill request on this host CassandraClient<pc-tdq-bst-05:9160-25>
WARN HConnectionManager - Exception:
me.prettyprint.hector.api.exceptions.HTimedOutException: TimedOutException()
at me.prettyprint.cassandra.service.ExceptionsTranslatorImpl.translate(ExceptionsTranslatorImpl.java:35)
at me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:264)
at me.prettyprint.cassandra.model.ExecutingKeyspace.doExecuteOperation(ExecutingKeyspace.java:97)
at me.prettyprint.cassandra.model.MutatorImpl.execute(MutatorImpl.java:243)
at ch.cern.pbeast.insertion.CassandraDBClient.executeBatchInsert(CassandraDBClient.java:3293)
at ch.cern.pbeast.insertion.FilteringTaskNew.run(FilteringTaskNew.java:408)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by: TimedOutException()
at org.apache.cassandra.thrift.Cassandra$batch_mutate_result.read(Cassandra.java:19495)
at org.apache.cassandra.thrift.Cassandra$Client.recv_batch_mutate(Cassandra.java:1035)
at org.apache.cassandra.thrift.Cassandra$Client.batch_mutate(Cassandra.java:1009)
at me.prettyprint.cassandra.model.MutatorImpl$3.execute(MutatorImpl.java:246)
at me.prettyprint.cassandra.model.MutatorImpl$3.execute(MutatorImpl.java:243)
at me.prettyprint.cassandra.service.Operation.executeAndSetResult(Operation.java:103)
at me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:258)
... 7 more