Is Shared Lock suitable for a leader election?

[黄宇龙]

sd.getLock("leaderLock", res -> {
  if (res.succeeded()) {
    
    Lock lock = res.result();
    // I'm leader now , i can do something!    //Never release my lock unless I die.   } else {     //I'm not leader     vertx.setTimer(1000, res -> {         // try to get leaderLock      })  }
});


Will the lock been released when a verticle exits? 

[Tim Fox]

It should do. Can you check?

If it doesn't can you add an issue?

Thanks

--
You received this message because you are subscribed to the Google Groups "vert.x" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vertx+un...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Jordan Halterman]

You can use locks for leader election. I've done it directly with the Hazelcast API. The lock will be released when the leader exits, but I did run in to some issues with Hazelcast exceptions on slaves when nodes were killed. You should do a test and kill -9 a node that holds a lock. I suspect it was mostly user error so it's possible (probable?) that Vert.x handles this issue better than I did.

Also, it's important to note that however unlikely the scenario, Hazelcast cannot guarantee that two or more processes that can't see each other (due to network failures) won't hold a lock at the same time (this amounts to split brain). I actually saw this behavior when I was testing Hazelcast locks for leader election in EC2. Mine was a case where avoiding split brain was simpler than accepting it, but there's nothing wrong with eventually consistent or probabilistic leader election; just be cognizant of potential side effects. If your leaders' responsibilities are critical and you happen to have a ZooKeeper server laying around, it may be prudent to go that route (though ZK leader election is not perfect, it's very close).

Speaking of which, @tim: me being me, I would love to see that ZooKeeper cluster manager made available to the community. Do you suppose if I clean up the atomicity issues we can merge it in to the vert-x3 organization? (I suppose I should be asking this in that thread)

--

[Tim Fox]

On 13/05/15 09:28, Jordan Halterman wrote:

You can use locks for leader election. I've done it directly with the Hazelcast API. The lock will be released when the leader exits, but I did run in to some issues with Hazelcast exceptions on slaves when nodes were killed. You should do a test and kill -9 a node that holds a lock. I suspect it was mostly user error so it's possible (probable?) that Vert.x handles this issue better than I did.

Also, it's important to note that however unlikely the scenario, Hazelcast cannot guarantee that two or more processes that can't see each other (due to network failures) won't hold a lock at the same time (this amounts to split brain).

This is true. However, Vert.x has quorum functionality built in, and if a verticle is deployed as HA, then it will be automatically undeployed if the node can't see a quorum of nodes, and undeploying the verticle should release any locks (we need to check!). So it should avoid split brain :)

I actually saw this behavior when I was testing Hazelcast locks for leader election in EC2. Mine was a case where avoiding split brain was simpler than accepting it, but there's nothing wrong with eventually consistent or probabilistic leader election; just be cognizant of potential side effects. If your leaders' responsibilities are critical and you happen to have a ZooKeeper server laying around, it may be prudent to go that route (though ZK leader election is not perfect, it's very close).

Speaking of which, @tim: me being me, I would love to see that ZooKeeper cluster manager made available to the community. Do you suppose if I clean up the atomicity issues we can merge it in to the vert-x3 organization?

Yes! I just don't have the cycles to do this, but if you'd like to that would be great.

[Jordan Halterman]

On May 13, 2015, at 1:38 AM, Tim Fox <timv...@gmail.com> wrote:

On 13/05/15 09:28, Jordan Halterman wrote:

You can use locks for leader election. I've done it directly with the Hazelcast API. The lock will be released when the leader exits, but I did run in to some issues with Hazelcast exceptions on slaves when nodes were killed. You should do a test and kill -9 a node that holds a lock. I suspect it was mostly user error so it's possible (probable?) that Vert.x handles this issue better than I did.

Also, it's important to note that however unlikely the scenario, Hazelcast cannot guarantee that two or more processes that can't see each other (due to network failures) won't hold a lock at the same time (this amounts to split brain).

This is true. However, Vert.x has quorum functionality built in, and if a verticle is deployed as HA, then it will be automatically undeployed if the node can't see a quorum of nodes, and undeploying the verticle should release any locks (we need to check!). So it should avoid split brain :)

That's good to know about the quorum. Does the quorum size still have to be set by the user though? I suppose it does since the cluster membership is dynamic.

If that's the case, perhaps some docs on safe locking would be useful to users.

This actually reminded me of https://github.com/elastic/elasticsearch/issues/2488

In a three node cluster - nodes A, B, and C - if a split occurs between nodes A and C, but not A and B and not C and B, both A and C will still be able to see majority of the cluster. In that event, it seems that if an HA verticle were deployed on node A, and such an "intersecting partition" occurred between nodes A and C only, because both nodes can see a majority of the cluster, node C would redeploy the HA verticle from node A, and node A would not undeploy the HA verticle. Then again, this all depends on how Hazelcast detects failures.

But I digress. I just can't help myself :-)

Obviously this level of coordination is a problem for consensus which is a bit of overkill for most basic Vert.x use cases. But that's why I'm interested in making the ZooKeeper cluster manager available.

Some day I will bring my Raft implementation to Vert.x. ZooKeeper will be a great option, but it's reliant upon an external cluster. The best aspect of Vert.x's use of Hazelcast is obviously its dynamism. But the issue with Raft being embedded in a system like Vert.x is that clusters for consensus are necessarily strict. I did some work to make Raft's strict membership a bit more palatable by implementing a gossip-based membership protocol for dynamic cluster members in Copycat.

Speaking of which, with regard to the partition scenario I described above, it could be possible for Hazelcast to prevent such a scenario. But the Hazelcast documentation isn't entirely clear to me; it seems that the oldest member is responsible for maintaining the membership list. So, if the oldest member is node A then that's still an issue.

Something I did in Copycat to prevent this type of scenario is probing. The first time a node fails to contact another member, the member's marked suspicious in that node's membership list. When the membership list containing the suspicious member is gossiped to another node, that node too will attempt to probe the suspicious member. While this process continues, the gossiped membership list carries with it a vector clock and the suspicious member in that list a set of nodes (a CRDT) that have probed the suspicious member and failed. Once the set has been merged into some satisfactory percentage of the cluster, the suspicious member is marked dead.

Of course, unlike Hazelcast which needs to replicate data lost on failed nodes, Copycat has the luxury of taking its time detecting failures outside the consensus algorithm since they're inconsequential to user state.

K now I went way off topic... Better shut up :-)

[黄宇龙]

Great to know that,thanks!     I  actually needs a clustered timer do some work periodly , right now I think I can use this trick.


在 2015年5月13日星期三 UTC+8下午3:42:16，Tim Fox写道：