select query throwing NoHostAvailabkeException in cassandra driver 3.5

[hemabs]

Hi All,

I have below code which is throwing error in execute call:

                Select selectQuery = QueryBuilder.select("column1").from("mytable").limit(1);

ResultSet results = session.execute(selectQuery);

Im using Cassandra server 2.1.13 with driver 3.5. Seeing below exception

com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: host/ip:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_ONE (1 required but only 0 alive)))

at com.datastax.driver.core.exceptions.NoHostAvailableException.copy(NoHostAvailableException.java:84)

at com.datastax.driver.core.exceptions.NoHostAvailableException.copy(NoHostAvailableException.java:37)

at com.datastax.driver.core.DriverThrowables.propagateCause(DriverThrowables.java:37)

at com.datastax.driver.core.DefaultResultSetFuture.getUninterruptibly(DefaultResultSetFuture.java:245)

at com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:68)

......

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at java.lang.reflect.Method.invoke(Method.java:498)

Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: host/ip:port (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_ONE (1 required but only 0 alive)))

at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:232)

at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:50)

at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:318)

at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:486)

at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:464)

at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:688)

at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1087)

at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1010)

at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)

at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)

at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)

at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)

at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)

at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)

at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:312)

at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:286)

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)

at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)

at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)

at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)

at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)

at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1304)

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)

at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:921)

at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:135)

at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:646)

at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:581)

at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498)

at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:460)

at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131)

at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)

at java.lang.Thread.run(Thread.java:748)

Can anyone let me know what is going wrong here?

Thanks

[Andrew Tolbert]

Hi Hemabs,

This indicates that there is only 1 node that the driver is able to contact, and that node responded to the query with an 'unavailable' error meaning the node coordinator observes that there are no replicas alive to handle that request.

Is this a multi-node cluster?  What is the keyspace replication factor you have configured for the keyspace you are querying?  If RF is 1, that means if one node goes down all data belonging to that replica is unavailable until the node comes back up.

Thanks,

Andy

[hemabs]

Hi Andy,

I have below settings done for the keyspace:

       mykeyspace |           True |          org.apache.cassandra.locator.SimpleStrategy | {"replication_factor":"1"}

And yes it is a multi-node cluster.But for my test purpose I have this keyspace created with RF=1.

Thanks

hbs

[hemabs]

May I know if you are seeing node availability issue given this configuration.

Thanks

hbs

[Andrew Tolbert]

Since your keyspace replication factor is only 1 when any of your nodes goes down you will see unavailable errors.

Thanks,

Andy

[hemabs]

But in my case all nodes are up and running.

[root@Host bin]# ./nodetool status

Datacenter: dc1

===============

Status=Up/Down

|/ State=Normal/Leaving/Joining/Moving

--  Address         Load       Tokens  Owns    Host ID    Rack

UN  aadr1           6.97 MB    256     ?       host_id01  rac1

Datacenter: dc2

===============

Status=Up/Down

|/ State=Normal/Leaving/Joining/Moving

--  Address         Load       Tokens  Owns    Host ID     Rack

UN  aadr2           8.02 MB    256     ?       host_id02   rac1

What could be the cause?

Thanks

hbs

[hemabs]

Something strange happened with the nodes. After re-imaging the nodes, things are working as expected.

Thanks Andy on the help.