Exception caught in channel processing. java.io.IOException: Connection reset by peer

[vishwanadh G B S C]

We can see messaging service log flooded with these errors. We saw this issue previously but was more intermittent. Now  entire log is filled with these errors.

2018-01-22 23:32:21,797 - ERROR [messaging.service-worker-thread-25:c.c.h.HttpDispatcher@139] - Exception caught in channel processing.

java.io.IOException: Connection reset by peer

        at sun.nio.ch.FileDispatcherImpl.read0(Native Method) ~[na:1.8.0_77]

        at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) ~[na:1.8.0_77]

        at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) ~[na:1.8.0_77]

        at sun.nio.ch.IOUtil.read(IOUtil.java:192) ~[na:1.8.0_77]

        at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380) ~[na:1.8.0_77]

        at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:64) ~[io.netty.netty-3.6.6.Final.jar:na]

        at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:109) ~[io.netty.netty-3.6.6.Final.jar:na]

        at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312) ~[io.netty.netty-3.6.6.Final.jar:na]

        at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:90) ~[io.netty.netty-3.6.6.Final.jar:na]

        at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) ~[io.netty.netty-3.6.6.Final.jar:na]

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_77]

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_77]

        at java.lang.Thread.run(Thread.java:745) [na:1.8.0_77]

[Ali Anwar]

Hi Vishwanadh.

This likely indicates that some writes to hbase (from a cdap system service called metrics processor) are going slowly. However, it shouldn't be problematic, as it will retry the operation.
It shouldn't be making any user-facing impact (besides appearing in the logs and some delay in metrics availability). Are you noticing any other impact to the system?

Regards,
Ali Anwar

--
You received this message because you are subscribed to the Google Groups "CDAP User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cdap-user+unsubscribe@googlegroups.com.
To post to this group, send email to cdap...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cdap-user/e7d9d910-98f4-4424-83b4-d86ac6b93404%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[vishwanadh G B S C]

Hi Ali,

Thanks for the response. We can see messaging service go down frequently and that is the error.

To unsubscribe from this group and stop receiving emails from it, send an email to cdap-user+...@googlegroups.com.

[Ali Anwar]

Hi Vishwanadh.

How much memory is configured for the messaging service?
In cdap-site.xml, you can look for a parameter named 'messaging.container.memory.mb'. If it is missing from there, it likely defaults to 1G.
Perhaps YARN NodeManager is killing the container because it is going over the configured memory limits. You should be able to find logs indicating this in cdap master logs or the YARN NodeManager logs.

Regards,
Ali Anwar

To unsubscribe from this group and stop receiving emails from it, send an email to cdap-user+unsubscribe@googlegroups.com.

To post to this group, send email to cdap...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/cdap-user/e8c1221b-0aa9-414a-b9e2-a5a477760238%40googlegroups.com.