Creating a periodic timer only once in the entire cluster. Not as simple as it sounds

[bytor99999]

OK. So we have a cluster of Vert.x instances and we want a 5 minute periodic timer to manage some game tables. where if there is inactivity, we inactivate that table. Problem is if we have code in the onStart of the verticle, it will still call it for each and ever verticle deployed as well as in each Vert.x instance. How can we makes sure the setPeriodic is called only once for the entire cluster? We can't use sharedData to store the timerID because we know sharedData isn't shared across the cluster. One solution we thought of was storing the id in Hazelcast, but to make an entire Set/Map to store one value and one value only is a bit overkill.

So wondering if anyone had any suggestions?

Thanks

Mark

[bytor99999]

Ended up going with a lastUpdated timestamp in the table. Then have the timer started once by having a simple Set in Hazelcast with the timerID in it. If there is nothing in the Set, then setPeriodic (start the timer) if it is in there, don't start another timer.

Works.

Thanks

Mark

[bytor99999]

Damn, the idea of a Set of one in Hazelcast to make a Once and only once set of periodic timer throughout the whole cluster failed. Problem is I have to stop all my server types, the Spring Web MVC apps as well as the Vert.x instances to make sure that Hazelcast stops. Because it just one of those servers are still running, Hazelcast still has the data, so f the Vert.x instances stop, but not the Spring Web MVC apps with Vert.x embedded, then the timer is no longer actually running, but I still have that one entry in the Set that is storing the timerID. And therefore, when restarting a Vert.x instance that calls setPeriodic will skip that code because set.size() > 0.

Maybe a feature for 3.x vertx.setClusterPeriodic(long time, Handler)

Mark

[Nick Scavelli]

We will have a distributed counter in 3.x https://github.com/eclipse/vert.x/blob/master/vertx-core/src/main/java/io/vertx/core/shareddata/SharedData.java.

[bytor99999]

Yeah, but a counter isn't a timer. Timers are also using the cool feature that is in Netty with a  timer wheel. So I guess use the counter and only let it get to one. It would be nice to not have to do that. But I guess also that that would just need to be the same code but internal to the timer API.

Thanks for the response Nick.

Mark

[Jordan Halterman]

What happens when node A checks the counter, sees it's 0, node B checks the counter, sees it's zero, and both start the counter. You have to use optimistic locking to ensure two timers aren't set, i.e. call incrementAndGet() and if the return value is > 1 then don't set the timer (as opposed to checking the counter and then setting the timer).

Sent from my iPhone

--
You received this message because you are subscribed to the Google Groups "vert.x" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vertx+un...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Nick Scavelli]

I was suggesting in 3.x you could use the distributed counter to know when to start your periodic timer :)  http://git.io/L4Kp-A

[Frederic Masion]

I guy's,

I've faced the same problem and here is how I did it :

Star a timer on some or all nodes of your cluster (when starting your verticle ie)

This would lead in duplicate timers BUT : 

on each timer's callback you try to get an Hazelcast LOCK 

only one node will get the lock so he will be the "current cluster master" for scheduling 

only the node that has the lock can perform the callback action.

The other timers will do nothing if they don't have acquired the lock

so only the "master" will really trigger the action

The good thing with hazelcast locks is that : if the master node leaves the cluster the lock is automaticaly released

So on the next timer a new node will be able to get the lock and become the new master and trigger the action.

Using Locks is a good way only if specifics nodes launch the timer

If all nodes have the same role in the hazelcast cluster you can just use hazelcast cluster "oldest member" (check the api) to achieve this without lock (the oldest becomes the master and if it leaves a new one is the new oldest...)

In both case adding a new node the the cluster will work

Be careful (if it is important in your use case) to "sync" the timers if you don't want to have less than the delay each time a master leaves

Fred

[Tim Fox]

In Vert.x 3.0 we have an AsynchronousLock class which makes this kind of problem easy to solve.

Internally it uses the Hazelcast ISemaphore - beware using this (and Hazelcast Lock) directly as they are blocking - you don't want to be blocking the event loop.