TimeOutException

[Martin Junghanns]

Hi,

I'm using Hector and during some batch inserting the following
exceptions occurs:

WARN [not.interesting.cassandra.Runner.main()]
(HConnectionManager.java:294) - Could not fullfill request on this host
CassandraClient<localhost:9160-50>
WARN [not.interesting.cassandra.Runner.main()]
(HConnectionManager.java:295) - Exception:
me.prettyprint.hector.api.exceptions.HTimedOutException: TimedOutException()
at
me.prettyprint.cassandra.service.ExceptionsTranslatorImpl.translate(ExceptionsTranslatorImpl.java:42)
at
me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:252)
at
me.prettyprint.cassandra.model.ExecutingKeyspace.doExecuteOperation(ExecutingKeyspace.java:97)
at
me.prettyprint.cassandra.model.MutatorImpl.execute(MutatorImpl.java:243)
at
not.interesting.cassandra.batchimport.BatchImport.importFromFile(BatchImport.java:304)
at
not.interesting.cassandra.batchimport.BatchImport.importFromFile(BatchImport.java:182)
at
not.interesting.cassandra.batchimport.BatchImport.importFromDirectory(BatchImport.java:173)
at not.interesting.cassandra.Runner.main(Runner.java:68)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:297)
at java.lang.Thread.run(Thread.java:662)
Caused by: TimedOutException()
at
org.apache.cassandra.thrift.Cassandra$batch_mutate_result.read(Cassandra.java:19493)
at
org.apache.cassandra.thrift.Cassandra$Client.recv_batch_mutate(Cassandra.java:1035)
at
org.apache.cassandra.thrift.Cassandra$Client.batch_mutate(Cassandra.java:1009)
at
me.prettyprint.cassandra.model.MutatorImpl$3.execute(MutatorImpl.java:246)
at
me.prettyprint.cassandra.model.MutatorImpl$3.execute(MutatorImpl.java:243)
at
me.prettyprint.cassandra.service.Operation.executeAndSetResult(Operation.java:103)
at
me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:246)
... 12 more

I currently perform 5000 (also tried with 10000 -> same problem)
Insertions in one Mutator.execute() Call. The exceptions seems to occur
non-deterministic. The application and cassandra are running on the same
machine, so no network problems could cause the problem (imho).

Thanks for your help!

Martin

[Bharatendra Boddu]

Hi Martin,

It could be related to the thrift socket timeout  cassandraThriftSocketTimeout set in CassandraHostConfigurator object. Also check the cassandra.yaml  rpc_timeout_in_ms: 10000. The default is 10 secs.

- Bharat

[Nate McCall]

Cut the size of your mutation down by an order of magnitude to about
100 in a batch and ramp up the load by doing multiple threads instead.
Also, enable GC logging in the cassandra process, you are most likely
getting these co-inciding with a GC pause given that this is a single
machine, while everything is blocking on IO.

[Martin Junghanns]

Bharatendra, Nate,

thanks for your replies. I will try another run with a smaller batch
size and I also currently work on parallel importing from different
machines.

Martin

> >> me.prettyprint.cassandra.model.ExecutingKeyspace.doExecuteOperation(Executi ngKeyspace.java:97)

> >>        at
> >> me.prettyprint.cassandra.model.MutatorImpl.execute(MutatorImpl.java:243)
> >>        at
> >> not.interesting.cassandra.batchimport.BatchImport.importFromFile(BatchImpor t.java:304)
> >>        at
> >> not.interesting.cassandra.batchimport.BatchImport.importFromFile(BatchImpor t.java:182)
> >>        at

> >> not.interesting.cassandra.batchimport.BatchImport.importFromDirectory(Batch Import.java:173)

> >>        at not.interesting.cassandra.Runner.main(Runner.java:68)
> >>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >>        at
> >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:3 9)
> >>        at

> >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImp l.java:25)

> >>        at java.lang.reflect.Method.invoke(Method.java:597)
> >>        at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:297)
> >>        at java.lang.Thread.run(Thread.java:662)
> >> Caused by: TimedOutException()
> >>        at
> >> org.apache.cassandra.thrift.Cassandra$batch_mutate_result.read(Cassandra.ja va:19493)
> >>        at
> >> org.apache.cassandra.thrift.Cassandra$Client.recv_batch_mutate(Cassandra.ja va:1035)
> >>        at
> >> org.apache.cassandra.thrift.Cassandra$Client.batch_mutate(Cassandra.java:10 09)
> >>        at
> >> me.prettyprint.cassandra.model.MutatorImpl$3.execute(MutatorImpl.java:246)
> >>        at
> >> me.prettyprint.cassandra.model.MutatorImpl$3.execute(MutatorImpl.java:243)
> >>        at
> >> me.prettyprint.cassandra.service.Operation.executeAndSetResult(Operation.ja va:103)
> >>        at

> >> me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover( HConnectionManager.java:246)

[Nate McCall]

If it's helpful, there is a basica parallelization example in this data loader:
https://github.com/zznate/cassandra-tutorial/blob/master/src/main/java/com/datastax/tutorial/composite/CompositeDataLoader.java

[KingStark]

Hi Have you resolve your problem ? Because i Have same trouble, i increased cassandraThriftSocketTimeout but nothing works ! :(

[Nate McCall]

To be clear, the general answer here is one of two things:
- that you need more capacity in your cluster
- the operations you are sending are too large