KAFKA consumer - Message batch listener reprocessing issue.

[Amol Badgujar]

Hello All, 

We are facing an issue while processing KAFKA messages using JAVA + Spring Batch listener APIs. We have observed that most of the time the same kafka message is getting reprocessed multiple times. As per our initial analysis we arrived at conclusion that if one message is getting processed and if it takes more time to finish processing then the same message gets delivered to another Thread in the same consumer group. 

It's observed that it takes 110 seconds to finish the batch of transactions which are getting reprocessed again.This is causing deducting customer balance multiple times for single transaction.

Below is Consumer property configurations in our project and all the values are more than 120 seconds.

#group.id=tran-request-group
enable.auto.commit=true
#2min
auto.commit.interval.ms=120000
#100 sec
session.timeout.ms=100000
request.timeout.ms=200000
fetch.max.wait.ms=20000
heartbeat.interval.ms=30000
auto.offset.reset=latest
key.deserializer=org.apache.kafka.common.serialization.StringDeserializer
value.deserializer=org.apache.kafka.common.serialization.StringDeserializer
max.poll.records=1
receive.buffer.bytes=16777216

-Rgds,

Amol

[Amol]

Hello All,

Can someone please help me here ? to configure the Consumer configuration.

Rgds,

-Amol

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platf...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/CAP7uFdzWGsT%2BmL8dSTLpSmScm-OfKYY7U37atmqeE%3D5iFA8jeQ%40mail.gmail.com.