Axon 4.0.3 Messages from Kafka are missed

Ranadheer Pendru

unread,

Jul 10, 2020, 8:59:21 AM7/10/20

to Axon Framework Users

HI Team,

We have issue from the Consumer side when we give an initial segment count more than one. Event handlers missing some of the messages from Kafka. Currently, Kafka has 50 partitions.

Application Yml:

axon:
eventhandling:
processors:
"[kafkaProcessor]":
source: kafkaMessageSource
mode: TRACKING
threadCount: 1
initialSegmentCount: 1
batchSize: 500

Allard Buijze

unread,

Jul 10, 2020, 12:10:00 PM7/10/20

to axonfr...@googlegroups.com

The total number of threads (across all instances of the component) needs to be at least equal to the number of segments.

--
You received this message because you are subscribed to the Google Groups "Axon Framework Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to axonframewor...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/axonframework/1a240001-4533-4228-82c7-183b7982fcfcn%40googlegroups.com.

--

Allard Buijze

CTO

Ranadheer Pendru

unread,

Jul 10, 2020, 12:15:49 PM7/10/20

to Axon Framework Users

yes, that's true. I have given 2 threads per container and 4 segments. We will have at least 2 containers at any time.

Ranadheer Pendru

unread,

Jul 13, 2020, 8:43:06 AM7/13/20

to Axon Framework Users

i have tried to bring up one instance with 50 segments and 50 threads, but it missed the events. anyone having the same issue?

Ranadheer Pendru

unread,

Jul 17, 2020, 11:04:44 AM7/17/20

to Axon Framework Users

It seems like missing some message .

I have debugged it by publishing 5 messages and in a single instance of the consumer with 2 segments then while polling eventStream.hasNextAvailable() is returning false and buffer.poll(timeout, unit) returning null after getting 2 or 3 messages. So it missing at least 1 each time.

Ranadheer Pendru

unread,

Jul 17, 2020, 11:05:25 AM7/17/20

to Axon Framework Users

i am missing any config or is it bug ?

Steven van Beelen

unread,

Sep 16, 2020, 5:32:56 AM9/16/20

to Axon Framework Users

Hi Ranadheer,

Fifty segments and threads is a very, very large amount.

Not trying to be offensive here, but the chances are very slim you actually need that much work going on at the same time.

On the other note, it is good to know that the Kafka extension is not out of beta yet.

Thus, some bugs can be expected. If you find something promising, I would kindly ask you if you can file an issue for it on the Kafka Extension GitHub page.

If you want a more thorough solution fo dispatching events without bugs, I would highly recommend giving Axon Server a try.

That's my two cents to the situation.

Cheers,

Steven

PS. It is important to note that this mailing list will be discontinued as specified in this thread.
Instead of this mailing list we have moved to use a forum style of communication, at https://discuss.axoniq.io/.

Hope to see you there!

To view this discussion on the web visit https://groups.google.com/d/msgid/axonframework/ddb4360d-bfd4-4fde-9ac3-e18c811220d2n%40googlegroups.com.

Reply all

Reply to author

Forward