Confluent 6 || Frequently KsqlDB host is getting marked as dead

[Anup Tiwari]

Hi,

Since upgrading of confluent ksqlDB from 5.5 to 6, i can see frequent lines in logs which are as follows :-

Feb 17 10:45:12 ip-10-*-*-14* ksql-server-start: [2021-02-17 10:45:12,916] INFO Host: KsqlHostInfo{host='10.*.*.13*', port=8088} marked as dead. (io.confluent.ksql.rest.server.HeartbeatAgent:250)

This message is getting logged on all nodes for all other nodes of the cluster. I checked connectivity from my end on 8088 port using telnet and can see connectivity is fine.

I tried restarting nodes but still it is coming.

Is it a bug OR i am missing something ?

Regards,
Anup Tiwari

[Anup Tiwari]

I can also see below ERROR while starting ksqlDB :-

Feb 18 13:13:17 ip-10-51-*-* ksql-server-start: [2021-02-18 13:13:17,313] ERROR Failed to handle request 503 /heartbeat (io.confluent.ksql.api.server.
FailureHandler:38)
Feb 18 13:13:17 ip-10-51-*-* ksql-server-start: io.confluent.ksql.api.server.KsqlApiException: KSQL is not yet ready to serve requests.
Feb 18 13:13:17 ip-10-51-*-* ksql-server-start: at io.confluent.ksql.api.server.ServerStateHandler.handle(ServerStateHandler.java:43)
Feb 18 13:13:17 ip-10-51-*-* ksql-server-start: at io.confluent.ksql.api.server.ServerStateHandler.handle(ServerStateHandler.java:28)
Feb 18 13:13:17 ip-10-51-*-* ksql-server-start: at io.vertx.ext.web.impl.RouteState.handleContext(RouteState.java:1034)
Feb 18 13:13:17 ip-10-51-*-* ksql-server-start: at io.vertx.ext.web.impl.RoutingContextImplBase.iterateNext(RoutingContextImplBase.java:131)
Feb 18 13:13:17 ip-10-51-*-* ksql-server-start: at io.vertx.ext.web.impl.RoutingContextImpl.next(RoutingContextImpl.java:133)
Feb 18 13:13:17 ip-10-51-*-* ksql-server-start: at io.vertx.ext.web.impl.RouterImpl.handle(RouterImpl.java:54)
Feb 18 13:13:17 ip-10-51-*-* ksql-server-start: at io.vertx.ext.web.impl.RouterImpl.handle(RouterImpl.java:36)
Feb 18 13:13:17 ip-10-51-*-* ksql-server-start: at io.vertx.core.http.impl.WebSocketRequestHandler.handle(WebSocketRequestHandler.java:50)
Feb 18 13:13:17 ip-10-51-*-* ksql-server-start: at io.vertx.core.http.impl.WebSocketRequestHandler.handle(WebSocketRequestHandler.java:32)
Feb 18 13:13:17 ip-10-51-*-* ksql-server-start: at io.vertx.core.http.impl.Http1xServerConnection.handleMessage(Http1xServerConnection.java:136)
Feb 18 13:13:17 ip-10-51-*-* ksql-server-start: at io.vertx.core.impl.ContextImpl.executeTask(ContextImpl.java:366)
Feb 18 13:13:17 ip-10-51-*-* ksql-server-start: at io.vertx.core.impl.EventLoopContext.execute(EventLoopContext.java:43)
Feb 18 13:13:17 ip-10-51-*-* ksql-server-start: at io.vertx.core.impl.ContextImpl.executeFromIO(ContextImpl.java:229)
Feb 18 13:13:17 ip-10-51-*-* ksql-server-start: at io.vertx.core.net.impl.VertxHandler.channelRead(VertxHandler.java:173)
Feb 18 13:13:17 ip-10-51-*-* ksql-server-start: at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java
:379)
Feb 18 13:13:17 ip-10-51-*-* ksql-server-start: at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java
:365)
Feb 18 13:13:17 ip-10-51-*-* ksql-server-start: at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:3
57)
Feb 18 13:13:17 ip-10-51-*-* ksql-server-start: at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
Feb 18 13:13:17 ip-10-51-*-* ksql-server-start: at io.netty.handler.codec.http.websocketx.extensions.WebSocketServerExtensionHandler.channelRead(WebSo
cketServerExtensionHandler.java:101)
Feb 18 13:13:17 ip-10-51-*-* ksql-server-start: at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java
:379)
Feb 18 13:13:17 ip-10-51-*-* ksql-server-start: at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java
:365)
Feb 18 13:13:17 ip-10-51-*-* ksql-server-start: at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
Feb 18 13:13:17 ip-10-51-*-* ksql-server-start: at io.vertx.core.http.impl.Http1xUpgradeToH2CHandler.channelRead(Http1xUpgradeToH2CHandler.java:109)
Feb 18 13:13:17 ip-10-51-*-* ksql-server-start: at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
Feb 18 13:13:17 ip-10-51-*-* ksql-server-start: at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
Feb 18 13:13:17 ip-10-51-*-* ksql-server-start: at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
Feb 18 13:13:17 ip-10-51-*-* ksql-server-start: at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)
Feb 18 13:13:17 ip-10-51-*-* ksql-server-start: at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
Feb 18 13:13:17 ip-10-51-*-* ksql-server-start: at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
Feb 18 13:13:17 ip-10-51-*-* ksql-server-start: at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
Feb 18 13:13:17 ip-10-51-*-* ksql-server-start: at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324)
Feb 18 13:13:17 ip-10-51-*-* ksql-server-start: at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:311)
Feb 18 13:13:17 ip-10-51-*-* ksql-server-start: at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:425)
Feb 18 13:13:17 ip-10-51-*-* ksql-server-start: at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
Feb 18 13:13:17 ip-10-51-*-* ksql-server-start: at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
Feb 18 13:13:17 ip-10-51-*-* ksql-server-start: at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
Feb 18 13:13:17 ip-10-51-*-* ksql-server-start: at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
Feb 18 13:13:17 ip-10-51-*-* ksql-server-start: at io.vertx.core.http.impl.Http1xOrH2CHandler.end(Http1xOrH2CHandler.java:61)
Feb 18 13:13:17 ip-10-51-*-* ksql-server-start: at io.vertx.core.http.impl.Http1xOrH2CHandler.channelRead(Http1xOrH2CHandler.java:38)
Feb 18 13:13:17 ip-10-51-*-* ksql-server-start: at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
Feb 18 13:13:17 ip-10-51-*-* ksql-server-start: at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
Feb 18 13:13:17 ip-10-51-*-* ksql-server-start: at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
Feb 18 13:13:17 ip-10-51-*-* ksql-server-start: at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)
Feb 18 13:13:17 ip-10-51-*-* ksql-server-start: at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
Feb 18 13:13:17 ip-10-51-*-* ksql-server-start: at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
Feb 18 13:13:17 ip-10-51-*-* ksql-server-start: at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
Feb 18 13:13:17 ip-10-51-*-* ksql-server-start: at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
Feb 18 13:13:17 ip-10-51-*-* ksql-server-start: at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
Feb 18 13:13:17 ip-10-51-*-* ksql-server-start: at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
Feb 18 13:13:17 ip-10-51-*-* ksql-server-start: at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
Feb 18 13:13:17 ip-10-51-*-* ksql-server-start: at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
Feb 18 13:13:17 ip-10-51-*-* ksql-server-start: at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
Feb 18 13:13:17 ip-10-51-*-* ksql-server-start: at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
Feb 18 13:13:17 ip-10-51-*-* ksql-server-start: at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
Feb 18 13:13:17 ip-10-51-*-* ksql-server-start: at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
Feb 18 13:13:17 ip-10-51-*-* ksql-server-start: at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
Feb 18 13:13:17 ip-10-51-*-* ksql-server-start: at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
Feb 18 13:13:17 ip-10-51-*-* ksql-server-start: at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
Feb 18 13:13:17 ip-10-51-*-* ksql-server-start: at java.lang.Thread.run(Thread.java:748)

Regards,
Anup Tiwari

[Anup Tiwari]

Hi Team,

Can somebody look into this and help me narrow down the issue ? Because even after restarting the servers(suspecting network related issue) , we are getting this error.

ERROR Failed to handle request 503 /heartbeat (io.confluent.ksql.api.server.FailureHandler:38)

Regards,
Anup Tiwari