I'm getting unexplained errors. Is Cassandra repeatedly going down?

[Tim Koop]

I've been using Hector for my website back end for many months now.  Usually it works fine, but sometimes the following weird error happens:

I get an error like this:

me.prettyprint.hector.api.exceptions.HCassandraInternalException: Cassandra encountered an internal error processing this request: TApplicationError type: 7 message:Required field 'key' was not present! Struct: get_args(key:null, column_path:null, consistency_level:ONE)
        at me.prettyprint.cassandra.service.ExceptionsTranslatorImpl.translate(ExceptionsTranslatorImpl.java:29)
        at me.prettyprint.cassandra.service.KeyspaceServiceImpl$1.execute(KeyspaceServiceImpl.java:97)
        at me.prettyprint.cassandra.service.KeyspaceServiceImpl$1.execute(KeyspaceServiceImpl.java:90)
        at me.prettyprint.cassandra.service.Operation.executeAndSetResult(Operation.java:101)

Then I refresh the page or do it again and I get this:

me.prettyprint.cassandra.connection.HThriftClient - Could not flush transport (to be expected if the pool is shutting down) in close for client: CassandraClient<localhost:9160-147>
org.apache.thrift.transport.TTransportException: java.net.SocketException: Broken pipe
        at org.apache.thrift.transport.TIOStreamTransport.write(TIOStreamTransport.java:147)
        at org.apache.thrift.transport.TFramedTransport.flush(TFramedTransport.java:156)
        at me.prettyprint.cassandra.connection.HThriftClient.close(HThriftClient.java:85)
        at me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:245)
        at me.prettyprint.cassandra.service.KeyspaceServiceImpl.operateWithFailover(KeyspaceServiceImpl.java:131)
        at me.prettyprint.cassandra.service.KeyspaceServiceImpl.getColumn(KeyspaceServiceImpl.java:857)

and

ERROR TP-Processor3 me.prettyprint.cassandra.connection.HConnectionManager - MARK HOST AS DOWN TRIGGERED for host localhost(127.0.0.1):9160
ERROR TP-Processor3 me.prettyprint.cassandra.connection.HConnectionManager - Pool state on shutdown: <ConcurrentCassandraClientPoolByHost>:{localhost(127.0.0.1):9160}; IsActive?: true; Active: 1; Blocked: 0; Idle: 15; NumBeforeExhausted: 49
INFO TP-Processor3 me.prettyprint.cassandra.connection.ConcurrentHClientPool - Shutdown triggered on <ConcurrentCassandraClientPoolByHost>:{localhost(127.0.0.1):9160}

and

INFO TP-Processor3 me.prettyprint.cassandra.connection.ConcurrentHClientPool - Shutdown complete on <ConcurrentCassandraClientPoolByHost>:{localhost(127.0.0.1):9160}
INFO TP-Processor3 me.prettyprint.cassandra.connection.CassandraHostRetryService - Host detected as down was added to retry queue: localhost(127.0.0.1):9160
WARN TP-Processor3 me.prettyprint.cassandra.connection.HConnectionManager - Could not fullfill request on this host CassandraClient<localhost:9160-147>
WARN TP-Processor3 me.prettyprint.cassandra.connection.HConnectionManager - Exception:
me.prettyprint.hector.api.exceptions.HectorTransportException: org.apache.thrift.transport.TTransportException: java.net.SocketException: Broken pipe

and

INFO TP-Processor3 me.prettyprint.cassandra.connection.HConnectionManager - Client CassandraClient<localhost:9160-147> released to inactive or dead pool. Closing.
INFO TP-Processor3 me.prettyprint.cassandra.connection.HConnectionManager - Client CassandraClient<localhost:9160-147> released to inactive or dead pool. Closing.
ERROR TP-Processor3 com.phelpr.web.pages.register.Register - me.prettyprint.hector.api.exceptions.HectorException: All host pools marked down. Retry burden pushed out to client.

and

INFO pool-4-thread-1 me.prettyprint.cassandra.connection.HConnectionManager - Added host localhost(127.0.0.1):9160 to pool

Then I refresh the page or do it again and it all works fine.


Then it all works good until I delete my cookies or clear my cache.  After I do that I get the errors again.

What is happening?  Is Cassandra going down every few minutes or what?  If so, why in the world is it doing that?

Thanks for any advice.  I'm trying to go live with this website, but a failing datastore is a show-stopper.

--

t...@timkoop.com
www.timkoop.com

[Patricio Echagüe]

What hector version are you using?

Sent from my Android

[Tim Koop]

My class path includes:

hector-core-0.8.0-2.jar
cassandra-all-0.8.1.jar
cassandra-javautils-0.7.0.jar
cassandra-thrift-0.8.1.jar
libthrift-0.6.1.jar

I think my cassandra is version 0.8.7
I know that the api version of the cassandra server is 19.16.0

t...@timkoop.com
www.timkoop.com

[Patricio Echagüe]

I believe I fixed it here:

https://github.com/rantav/hector/issues/299

What I can't recall is for what version I fixed it.

Nate ?

[Tim Koop]

Thanks Patricio.

I just upgraded to hector-core-0.8.0-4.jar (and also two other jars that came with it).

And so far it seems to completely fix the problem.

Thanks a lot.

t...@timkoop.com
www.timkoop.com

[Patricio Echagüe]

Awesome. Glad to hear.