WARN Got error produce response with correlation id - ERROR NETWORK_EXCEPTION

[Saravanan Tirugnanum]

We are using Kafka Streams v 0.10.2.0 and encountered NETWORK_EXCEPTION while our streams app publishing messages to a target topic which is in same Data center using custom publisher Processor.  After this error , our streams app tasks have failed and did not recover back. 

Ops team have confirmed there is no issue with Network , Kafka , ZK nodes and all were in good state.

Wondering what could be the cause of this issue and why our stream apps not recovered back. How could we make our streams more fault tolerant in case of environmental failures. Any pointers would help

Regards

Saravanan

[Saravanan Tirugnanum]

We also got other errors on subsequent runs -  Any pointers please

2017-08-29 07:26:27,885] ERROR Uncaught exception in kafka-coordinator-heartbeat-thread | ei-transformation-pos-client-1.cdc: (org.apache.kafka.clients.consumer.internals.AbstractCoordinator$HeartbeatThread)
java.lang.OutOfMemoryError: Java heap space
at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57)
at java.nio.ByteBuffer.allocate(ByteBuffer.java:335)
at org.apache.kafka.common.network.NetworkReceive.readFromReadableChannel(NetworkReceive.java:93)
at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:71)
at org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:169)
at org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:150)
at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:355)
at org.apache.kafka.common.network.Selector.poll(Selector.java:303)
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:349)
at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:226)
at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.pollNoWakeup(ConsumerNetworkClient.java:263)
at org.apache.kafka.clients.consumer.internals.AbstractCoordinator$HeartbeatThread.run(AbstractCoordinator.java:887)
[2017-08-29 07:26:39,338] WARN Unexpected error from kafka-197807230-4-205927040.prod1.kafka-cluster.ms-df-messaging.cdcprod7.prod.walmart.com/10.227.152.156; closing connection (org.apache.kafka.common.network.Selector)
java.lang.NullPointerException
at org.apache.kafka.common.network.NetworkReceive.complete(NetworkReceive.java:67)
at org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:151)
at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:355)
at org.apache.kafka.common.network.Selector.poll(Selector.java:303)
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:349)
at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:226)
at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:188)
at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.commitOffsetsSync(ConsumerCoordinator.java:578)
at org.apache.kafka.clients.consumer.KafkaConsumer.commitSync(KafkaConsumer.java:1125)
at org.apache.kafka.streams.processor.internals.StreamTask.commitOffsets(StreamTask.java:296)
at org.apache.kafka.streams.processor.internals.StreamTask$1.run(StreamTask.java:79)
at org.apache.kafka.streams.processor.internals.StreamsMetricsImpl.measureLatencyNs(StreamsMetricsImpl.java:188)
at org.apache.kafka.streams.processor.internals.StreamTask.commit(StreamTask.java:280)
at org.apache.kafka.streams.processor.internals.StreamThread.commitOne(StreamThread.java:777)
at org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:650)
at org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:368)

Regards

Saravanan