Broadcast message duplication

[Anton Mostovenko]

My team faced a problem with a duplication of broadcast messages. It started to occur after we increased number of server nodes using HAProxy.

Currently we have 8 nodes which are receiving BMs, initiated by several processes. Each node shows 1-6 BMs at once after each run. Number of messages looks like a random number. But it persists for some time (until reboot or something else). I mean that if node #1 shows 3 messages, it will show same amounts repeatedly. While node #2 may work normally.

Has anyone encountered similar behavior?

[Anton Mostovenko]

New information discovered.
Looks like Hazelcast creates a new connection when ZK plugin or ZK dependant is refreshed. Each refresh adds one messages to each run of broadcast publishing process. Server restart resets it back.

вторник, 2 июня 2020 г., 11:46:56 UTC+3 пользователь Anton Mostovenko написал:

[Anton Mostovenko]

Steps to reproduce:

1. You need at least 2 nodes inside a local network with a default Hazelcast configuration (or configured to the same Hazelcast group)

2. Send broadcast message with "Immediate" type from first node. You should receive it on second node

3. Refresh any plugin with ZK dependencie (not ZK itself) on both nodes. If both sessions were reseted - you did it right!

4. Send broadcast message with "Immediate" type from first node. You should get 2 messages on each node

вторник, 2 июня 2020 г., 11:46:56 UTC+3 пользователь Anton Mostovenko написал:

My team faced a problem with a duplication of broadcast messages. It started to occur after we increased number of server nodes using HAProxy.