Ordered distributed queue

[Chris Cleveland]

I'd like to implement a distributed task scheduler using Hazelcast. It could be done using a distributed queue if I could pull items out of the queue based on their nextFireTime. In other words, if I could specify the order of items in the queue.

Unfortunately, it appears that Hazelcast queues are strictly first-in first-out.

If I could get a different ordering, then I could have a bunch of machines polling the queue, looking for a task that had a nextFireTime >= now. The first machine to grab the task would execute it. It would make a very nice distributed job scheduler.

Is this possible?

Is there a different approach which is better?

[Peter Veentjer]

On Mon, Nov 4, 2013 at 11:29 PM, Chris Cleveland <cclev...@gmail.com> wrote:

I'd like to implement a distributed task scheduler using Hazelcast. It could be done using a distributed queue if I could pull items out of the queue based on their nextFireTime. In other words, if I could specify the order of items in the queue.

You would like to have a PriorityBlockingQueue

http://docs.oracle.com/javase/7/docs/api/java/util/concurrent/PriorityBlockingQueue.html

 

Unfortunately, it appears that Hazelcast queues are strictly first-in first-out.

Correct.

 

If I could get a different ordering, then I could have a bunch of machines polling the queue, looking for a task that had a nextFireTime >= now. The first machine to grab the task would execute it. It would make a very nice distributed job scheduler.

Is this possible?

With the ordinary queue it isn't possible, but that doesn't mean it can't be done.

It can always be done by writing your own PriorityBlockingQueue using our SPI; the API that is exposed and used internally to write e.g. the Hazelcast distributed queue/map/locks etc. But this is quite a lot of work and perhaps not a solution you immediately want to go for. So the question is if there is a simpler solution.

I'm sure it can be simulated using a map. The only question is what the key of this map should be; the value can be the job. 

The other question is how to effectively get items from this map. One possible solution would be to use a distributed query to load all jobs whose nextFireTime is less than X seconds and every X seconds execute this query and schedule the found jobs.

Can you live with this solution?

 

Is there a different approach which is better?

--
You received this message because you are subscribed to the Google Groups "Hazelcast" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hazelcast+...@googlegroups.com.
To post to this group, send email to haze...@googlegroups.com.
Visit this group at http://groups.google.com/group/hazelcast.
For more options, visit https://groups.google.com/groups/opt_out.

[eden.fe...@gmail.com]

Can you share what you decided on?  I was thinking of using HazelCast also but I also need a PriorityBlockingQueue.  

Cheers,

Mitch

[Chris Cleveland]

At the moment, we're having each machine do brute force polling of all of their local keys every five seconds. Seems to work, but we're not yet using it in production.

--
You received this message because you are subscribed to a topic in the Google Groups "Hazelcast" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/hazelcast/POdgnE-pgFg/unsubscribe.
To unsubscribe from this group and all its topics, send an email to hazelcast+...@googlegroups.com.

To post to this group, send email to haze...@googlegroups.com.
Visit this group at http://groups.google.com/group/hazelcast.

To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/2d33aace-87c3-4076-a21b-f9cd3e3f3073%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.