Heap memory issues ConsumerNetworkClient ClientUnsentRequest ConcurrentHashMap

53 views

Skip to first unread message

Win Kidzz

unread,

Mar 16, 2023, 3:13:59 PM3/16/23

to kafka-clients

We have Spring Kafka Listener consumer running into heap issue causing oom. Upon investigating the heap there are lot of nulls in the ConcurrentHashMap that is maintained in the ConsumerNetworkClient. Single listener was holding 1.2 million null objects eventually crashing. The Client requests has messages like heartbeat and other requests that are completed successfully and removed from hashmap. However after the removal the nulls are not cleaned up eventually causing OOM.

https://github.com/a0x8o/kafka/blob/master/clients/src/main/java/org/apache/kafka/clients/consumer/internals/ConsumerNetworkClient.java#L61

Is there a configuration to limit or force cleanup of these null messages in the heap?