Hi Fuad,
On Mar 6, 4:47 am, Fuad Malikov <
f...@hazelcast.com> wrote:
> Hi Tim,
>
> Any listener notification is done asynchronous. And actually we require you
> not to do any heavy weight operation inside the listener. Upon receiving
> the event you should spin your own thread and do all the processing there.
> In my previous code to keep it simple I didn't obey this rule. I called the
> getCluster().getMembers() inside the memberAdded method. Normally you
> shouldn't do that. The main reason is not to block a Hazelcast thread.
>
> The set of Members represents the current alive members. It will always
> return the current true state.
Well, as you know "current" is something hard to define in a
distributed system ;)
I can workaround not doing any "heavyweight" operations in an event
handler, but the issue still remains that the membership still might
get updated while the handler is being executed, e.g.
Node 1 might have these sequence of events:
public synchronized void memberRemoved(MembershipEvent
membershipEvent) {
// membershipEvent A is received
<<===== The members set gets updated concurrently here
Set<Member> members = instance.getCluster().getMembers();
}
But Node 2 might see these sequence of events:
public synchronized void memberRemoved(MembershipEvent
membershipEvent) {
// membershipEvent A is received
Set<Member> members = instance.getCluster().getMembers();
<<===== The members set gets updated concurrently here
}
Resulting in Node 1 and Node 2 seeing _different_ sets of members for
the _same_ membership event that was received.
If different nodes see different sets of members for the same sequence
of membership events then any calculations that require each node to
see the same set of members (e.g. calculating a failover node by
hashing the list of members) don't work.
If Hazelcast cannot guarantee that different nodes see the same
membership set per sequence of member events then is there some other
way I can do this with Hazelcast? If not this would be as showstopper
for Vert.x and I would have to start looking elsewhere for clustering
support in Vert.x.
If the members list cannot be trusted one thing I thought would be to
maintain our own member state ourselves, i.e. each node maintains its
own set of Members and adds/removes them as it receives
MemberShipEvents. This would mean the member state is a proper
deterministic state machine, which is what we want, however the
problem then becomes how does a new node sync it's initial state with
that of other nodes when it joins.
Any help greatly appreciated here. I don't want to have to look at
other clustering technologies at this point.