Kafka HIght Level Consumer OutOfMemoryError

[Mariano Mirabelli]

Hello

I am working with Kafka Java api 0.8.2.2 and  I developed a High Level Kafka Consumer. The Consumer is listening all time the new incoming events for specific kafka topic. However, after two or three days since the App is started ( but is not used, because is a development enviroment and It has not loading), it crashes by java.lang.OutOfMemoryError: Java heap space.

The logs show the following:

Error for /api/fraud-screening/health java.lang.OutOfMemoryError: Java heap space

 Session 0x153393be296021c for server null, unexpected error, closing socket connection and attempting reconnect java.lang.OutOfMemoryError: Java heap space

 Opening socket connection to server 10.10.1.82/10.10.1.82:2181. Will not attempt to authenticate using SASL (unknown error)

Failed to invoke @ExceptionHandler method: public org.springframework.http.ResponseEntity<com.almundo.commons.exception.error.ApiError> com.almundo.commons.exception.controller.ExceptionControllerAdvice.handleExceptions(javax.servlet.http.HttpServletRequest,java.lang.Exception) org.eclipse.jetty.io.EofException: Closed

at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:317) ~[application.jar:na]

at com.fas...

An unexpected error has occurred org.eclipse.jetty.io.EofException: null

at org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:192) ~[application.jar:na]

at org...

Opening socket connection to server 10.10.1.82/10.10.1.82:2181. Will not attempt to authenticate using SASL (unknown error)

Opening socket connection to server 10.10.1.82/10.10.1.82:2181. Will not attempt to authenticate using SASL (unknown error)

Client session timed out, have not heard from server in 5396ms for sessionid 0x153393be296021d, closing socket connection and attempting reconnect

An unexpected error has occurred org.eclipse.jetty.io.EofException: null

at org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:192) ~[application.jar:na]

at org...

Session establishment complete on server 10.10.1.82/10.10.1.82:2181, sessionid = 0x153393be296021d, negotiated timeout = 6000

Socket connection established to 10.10.1.82/10.10.1.82:2181, initiating session

Client session timed out, have not heard from server in 14061ms for sessionid 0x153393be296021c, closing socket connection and attempting reconnect

Client session timed out, have not heard from server in 18649ms for sessionid 0x153393be296021a, closing socket connection and attempting reconnect

Client session timed out, have not heard from server in 18649ms for sessionid 0x153393be296021b, closing socket connection and attempting reconnect

Opening socket connection to server 10.10.1.82/10.10.1.82:2181. Will not attempt to authenticate using SASL (unknown error)

Opening socket connection to server 10.10.1.82/10.10.1.82:2181. Will not attempt to authenticate using SASL (unknown error)

Session establishment complete on server 10.10.1.82/10.10.1.82:2181, sessionid = 0x153393be296021c, negotiated timeout = 10000

Socket connection established to 10.10.1.82/10.10.1.82:2181, initiating session

Client session timed out, have not heard from server in 16483ms for sessionid 0x153393be296021b, closing socket connection and attempting reconnect

Client session timed out, have not heard from server in 16742ms for sessionid 0x153393be296021a, closing socket connection and attempting reconnect

Opening socket connection to server 10.10.1.82/10.10.1.82:2181. Will not attempt to authenticate using SASL (unknown error)

Opening socket connection to server 10.10.1.82/10.10.1.82:2181. Will not attempt to authenticate using SASL (unknown error)

Failed to invoke @ExceptionHandler method: public org.springframework.http.ResponseEntity<com.almundo.commons.exception.error.ApiError> com.almundo.commons.exception.controller.ExceptionControllerAdvice.handleExceptions(javax.servlet.http.HttpServletRequest,java.lang.Exception) org.eclipse.jetty.io.EofException: Closed

at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:317) ~[application.jar:na]

at com.fas...

An unexpected error has occurred org.eclipse.jetty.io.EofException: null

at org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:192) ~[application.jar:na]

at org...

EventThread shut down

EventThread shut down

Session establishment complete on server 10.10.1.82/10.10.1.82:2181, sessionid = 0x153393be296021b, negotiated timeout = 6000

Socket connection established to 10.10.1.82/10.10.1.82:2181, initiating session

Opening socket connection to server 10.10.1.82/10.10.1.82:2181. Will not attempt to authenticate using SASL (unknown error)

Session establishment complete on server 10.10.1.82/10.10.1.82:2181, sessionid = 0x153393be296021a, negotiated timeout = 6000

Initiating client connection, connectString=kafka01dv.almundo.it:2181 sessionTimeout=10000 watcher=org.I0Itec.zkclient.ZkClient@44fb1361

Unable to reconnect to ZooKeeper service, session 0x153393be2960217 has expired, closing socket connection

Initiating client connection, connectString=kafka01dv.almundo.it:2181 sessionTimeout=6000 watcher=org.I0Itec.zkclient.ZkClient@7261235d

Socket connection established to 10.10.1.82/10.10.1.82:2181, initiating session

Socket connection established to 10.10.1.82/10.10.1.82:2181, initiating session

Unable to reconnect to ZooKeeper service, session 0x153393be2960218 has expired, closing socket connection

Socket connection established to 10.10.1.82/10.10.1.82:2181, initiating session

Opening socket connection to server 10.10.1.82/10.10.1.82:2181. Will not attempt to authenticate using SASL (unknown error)

Opening socket connection to server 10.10.1.82/10.10.1.82:2181. Will not attempt to authenticate using SASL (unknown error)

Opening socket connection to server 10.10.1.82/10.10.1.82:2181. Will not attempt to authenticate using SASL (unknown error)

Client session timed out, have not heard from server in 29237ms for sessionid 0x0, closing socket connection and attempting reconnect

The last that I can see before the App crashes is the Zookeeper Connector open/close sessions.

Someone have any idea about this?

Thanks in advance!.

Greetings.

[Manikumar Reddy]

Hi, 

 This may be due to object leak in your application.  Profiling helps us to find out the  root cause. 

 You can also take periodic Jmap command output for memory analysis

    

  $jmap  -histo:live <pid> //to print histogram of  live java object heap

ManiKumar

--
You received this message because you are subscribed to the Google Groups "kafka-clients" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kafka-client...@googlegroups.com.
To post to this group, send email to kafka-...@googlegroups.com.
Visit this group at https://groups.google.com/group/kafka-clients.
To view this discussion on the web visit https://groups.google.com/d/msgid/kafka-clients/afb6084e-759e-4255-ab2b-febc3f51b6f3%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.