Spark streaming is continously writing the below log statements and killing the regular processing

[Charan Adabala]

Hi,

Spark is indefinitely writing the below statements while running our application.
We are using 3 kafka servers with 5 topics and 5 spark and 5 Cassandra servers, objective is to receive the events from kafka and compute the events and update the events to Cassandra.

Spark and Cassandra servers are having 16 cores,64GB EC2 machines and we are using 5 receivers in spark.
Can anyone please help me here.

07:32:22 INFO storage.BlockManagerInfo: Added input-3-1439983941250 in memory on 172.31.51.10:37813 (size: 11.0 MB, free: 25.3 GB)
15/08/19 07:32:22 INFO storage.BlockManagerInfo: Added input-3-1439983941250 in memory on 172.31.51.7:49791 (size: 11.0 MB, free: 29.5 GB)
15/08/19 07:32:23 INFO storage.BlockManagerInfo: Added input-3-1439983942500 in memory on 172.31.51.10:37813 (size: 11.3 MB, free: 25.3 GB)
15/08/19 07:32:23 INFO storage.BlockManagerInfo: Added input-3-1439983942500 in memory on 172.31.51.9:60521 (size: 11.3 MB, free: 29.7 GB)
15/08/19 07:32:25 INFO storage.BlockManagerInfo: Added input-3-1439983943750 in memory on 172.31.51.10:37813 (size: 10.7 MB, free: 25.3 GB)

[Russell Spitzer]

When you say "fails" what do you mean? The below log messages are normal.

To unsubscribe from this group and stop receiving emails from it, send an email to spark-connector-...@lists.datastax.com.

--

http://datastax.com/all/images/cs_logo_color_sm.png

[Charan Adabala]

Hi,

I found the below errors during this time.Nodes are continuously trying to reconnect and failing.

15/08/19 05:10:20 ERROR server.TransportChannelHandler: Connection to /172.31.51.11:44493 has been quiet for 120000 ms while there are outstanding requests. Assuming connection is dead; please adjust spark.network.timeout if this is wrong.
15/08/19 05:10:20 ERROR client.TransportResponseHandler: Still have 81 requests outstanding when connection from /172.31.51.11:44493 is closed
15/08/19 05:10:20 INFO shuffle.RetryingBlockFetcher: Retrying fetch (1/3) for 20 outstanding blocks after 5000 ms
15/08/19 05:10:20 INFO shuffle.RetryingBlockFetcher: Retrying fetch (1/3) for 30 outstanding blocks after 5000 ms
15/08/19 05:10:20 INFO shuffle.RetryingBlockFetcher: Retrying fetch (1/3) for 30 outstanding blocks after 5000 ms
15/08/19 05:10:20 ERROR shuffle.OneForOneBlockFetcher: Failed while starting block fetches
java.io.IOException: Connection from /172.31.51.11:44493 closed
at org.apache.spark.network.client.TransportResponseHandler.channelUnregistered(TransportResponseHandler.java:104)
at org.apache.spark.network.server.TransportChannelHandler.channelUnregistered(TransportChannelHandler.java:91)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelUnregistered(AbstractChannelHandlerContext.java:183)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelUnregistered(AbstractChannelHandlerContext.java:169)
at io.netty.channel.ChannelInboundHandlerAdapter.channelUnregistered(ChannelInboundHandlerAdapter.java:53)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelUnregistered(AbstractChannelHandlerContext.java:183)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelUnregistered(AbstractChannelHandlerContext.java:169)
at io.netty.channel.ChannelInboundHandlerAdapter.channelUnregistered(ChannelInboundHandlerAdapter.java:53)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelUnregistered(AbstractChannelHandlerContext.java:183)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelUnregistered(AbstractChannelHandlerContext.java:169)
at io.netty.channel.ChannelInboundHandlerAdapter.channelUnregistered(ChannelInboundHandlerAdapter.java:53)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelUnregistered(AbstractChannelHandlerContext.java:183)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelUnregistered(AbstractChannelHandlerContext.java:169)
at io.netty.channel.DefaultChannelPipeline.fireChannelUnregistered(DefaultChannelPipeline.java:738)
at io.netty.channel.AbstractChannel$AbstractUnsafe$6.run(AbstractChannel.java:606)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:380)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:357)
at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
at java.lang.Thread.run(Thread.java:745)