Cassandra java driver connection error

437 views
Skip to first unread message

Havhev Car My

unread,
Nov 15, 2016, 7:05:51 AM11/15/16
to DataStax Java Driver for Apache Cassandra User Mailing List

So i'm having a connection issues from DataStax java cassandra driver to a DataStax cassandra cluster. It initially connects and performs well, then suddenly at some point it looses connection and does not reconnect - at this point all the queries fail.

More info - I'm running DataStax cassandra 2.1 cluster of 3 nodes on CentOS, I'm using DataStax cassandra driver 3.0.0. Everything worked great in the past few months, recently iv'e deployed a some code changes that included some schema changes (namely, adding columns to an existing table) and an increase in the number of queries made. Disconnections started at this point.

So when my app goes up it connects to the cluster and holds a single cluster (and session) object as shown in the code snippet below, at this point everything goes well. After a few hours i start receiving NoHostAvailableException for every query performed. At this point i have other servers performing well with the same cassandra cluster so i know there's nothing wrong with the cluster itself. When i restart my server everything works good again.

After investigating a little more, when the issue start occurring, i see that there's no active connection to neither node. Iv'e set up the driver to log on DEBUG level into a dedicated log file and waited for the issue to reoccur. A few hours later the issue occurred again, at some point the log file shows this message:


Connection[/10.4.116.91:9042-1, inFlight=2, closed=false] connection error
io.netty.handler.codec.DecoderException: com.datastax.driver.core.exceptions.DriverInternalError: Adjusted frame length exceeds 268435456: 326843398 - discarded
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:418)
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:245)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:292)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:278)
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:962)
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131)
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:528)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:485)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:399)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:371)
        at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:112)
        at java.lang.Thread.run(Thread.java:745)
Caused by: com.datastax.driver.core.exceptions.DriverInternalError: Adjusted frame length exceeds 268435456: 326843398 - discarded
        at com.datastax.driver.core.Frame$Decoder$DecoderForStreamIdSize.decode(Frame.java:239)
        at com.datastax.driver.core.Frame$Decoder.decode(Frame.java:205)
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:387)
        ... 11 common frames omitted


And right after that you see this:


Connection[/10.4.116.91:9042-1, inFlight=2, closed=false] connection error
io.netty.handler.codec.DecoderException: com.datastax.driver.core.exceptions.DriverInternalError: Adjusted frame length exceeds 268435456: 326843398 - discarded
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:418)
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:245)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:292)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:278)
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:962)
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131)
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:528)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:485)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:399)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:371)
        at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:112)
        at java.lang.Thread.run(Thread.java:745)
Caused by: com.datastax.driver.core.exceptions.DriverInternalError: Adjusted frame length exceeds 268435456: 326843398 - discarded
        at com.datastax.driver.core.Frame$Decoder$DecoderForStreamIdSize.decode(Frame.java:239)
        at com.datastax.driver.core.Frame$Decoder.decode(Frame.java:205)
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:387)
        ... 11 common frames omitted


From this point on you see only timeouts and retries but the connection doesn't get reestablished.


// CREATION OF CASSANDRA SESSION
PoolingOptions poolingOptions = new PoolingOptions();
poolingOptions
    .setPoolTimeoutMillis(0)
    .setMaxRequestsPerConnection(HostDistance.LOCAL, 32768)
    .setMaxRequestsPerConnection(HostDistance.REMOTE, 2000);
cluster = builder.withPoolingOptions(poolingOptions).build();
cluster.getConfiguration().getCodecRegistry().register(new EnumNameCodec<>(OnBoardingSlide.Type.class));
session = cluster.connect(Global.getServerConfig().CASSANDRA_KEYSPACE_NAME);

Alexandre Dutra

unread,
Nov 15, 2016, 7:27:28 AM11/15/16
to DataStax Java Driver for Apache Cassandra User Mailing List
Hi,

It seems that you are hitting JAVA-1292. Could you please upgrade your driver to 3.0.5 or 3.1.2 to see if that solves the problem?

Also, if your Cassandra cluster is using frame lengths greater than 256 MB, you might want to adjust the maximum frame length on the driver side as well, see JAVA-1293.

Thanks,

Alexandre

--
You received this message because you are subscribed to the Google Groups "DataStax Java Driver for Apache Cassandra User Mailing List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to java-driver-us...@lists.datastax.com.
--
Alexandre Dutra
Driver & Tools Engineer @ DataStax

Havhev Car My

unread,
Nov 15, 2016, 8:56:15 AM11/15/16
to DataStax Java Driver for Apache Cassandra User Mailing List
Hi Alexandre, thanks for the quick reply.
I've migrated the driver.
Frame size is set to default so tweaking needed there.

I'll wait to see if the problem repeats.
Let you know either way.

Olivier Michallat

unread,
Nov 15, 2016, 5:53:41 PM11/15/16
to java-dri...@lists.datastax.com
On a side note, you might want to investigate why your frames get so large, 256 MB is a lot. This could be caused by a large fetch size, and/or large blob columns in your rows (which you generally want to split into chunks, or store outside of Cassandra, see this recent discussion on cassandra-user).

Compression might also be interesting if you're not already using it.

--

Olivier Michallat

Driver & tools engineer, DataStax


--
You received this message because you are subscribed to the Google Groups "DataStax Java Driver for Apache Cassandra User Mailing List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to java-driver-user+unsubscribe@lists.datastax.com.

Havhev Car My

unread,
Nov 18, 2016, 4:47:05 AM11/18/16
to DataStax Java Driver for Apache Cassandra User Mailing List
Solved!
Reply all
Reply to author
Forward
0 new messages