Re: Building a distributed delayed queue on top of queues

[Ryan Thomas]

Hi Todd,

My suggestion for a delayed delivery would be to do something like: Use two queues and an intermediate set. Messages are posted to the first queue (Queue A), where they are popped off immediately by some service that you will write, here they are entered into the intermediate set (so they are not lost if a node dies); this service also has a timer task that will periodically wake and transition any messages that have expired.

When a message expires, it should then be posted to the second queue (Queue B) - where it continues its journey to the rest of your application.

So; your consumers will register with Queue B, and you will post messages to Queue A.

This will allow you to have a delay in messages, but I am a little confused with the expiration and the term durable queues - are you using this in the same context as used with ActiveMQ / RabbitMQ?

Hope that helps,

ryan

On 17 November 2012 03:01, Todd Nine <tn...@apigee.com> wrote:

Hi guys,

  I'm building some prototypes for our application using hazelcast.  I have queues and topics working perfectly, but now I need a durable queue, that will only deliver messages after the expiration time in the message has lapsed.  Is it possible to accomplish this within hazelcast? 

Thanks,

Todd  

--
You received this message because you are subscribed to the Google Groups "Hazelcast" group.
To view this discussion on the web visit https://groups.google.com/d/msg/hazelcast/-/FEGb04-7mCQJ.
To post to this group, send email to haze...@googlegroups.com.
To unsubscribe from this group, send email to hazelcast+...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/hazelcast?hl=en.

[Todd Nine]

Hi Ryan,

  Correct, same as ActiveMQ/ Rabbit MQ.  We use cassandra internally, so I was going to create a backing map impl that writes to Cassandra.  My biggest issue is I haven't found a way to ensure that messages on a queue are delivered eventually if there are no listeners at the time the event is added to the queue.  Any settings/advice on how to accomplish this?

I've done as you suggested, except I've removed the first queue.  The future "queue" operation simply writes to a multimap with the time it should fire.  Then each node has a daemon task that reads it's local keys, and removes entries that need to be fired and writes them to a queue.  It works well.  I'm using topics to notify all nodes when a new queue is created, to ensure that they're scanning it to load info.  Is it possible to have ordered keys in a multimap?  This would make the daemon task more efficient, since it would read the lowest value first and could short circuit once it hits a future key.  At the moment I have to get all local keys, sort them, then check the fire time.

Thanks,

Todd

[Matt Hurne]

Todd,

Obviously it has been quite some time since your last message discussing your approach to implementing delayed delivery queuing with Hazelcast. We are considering doing something similar, and I was wondering if you might fill me in on how your solution panned out. Did it pass the test of time and prove effective?

Thanks,

Matt

[Peter Veentjer]

You could also have a look at the SPI and how the IQueue is implemented. It isn't that complicated once you get the hang of it and it probably will be more efficient than building it on top of an IQueue.

The SPI is the API we use internally to create our distributed datastructures like the queue, map lock etc.

To unsubscribe from this group and stop receiving emails from it, send an email to hazelcast+...@googlegroups.com.

To post to this group, send email to haze...@googlegroups.com.

Visit this group at http://groups.google.com/group/hazelcast.
To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/7812ce5f-791a-4592-9e5e-0366ec4b68f8%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.