dynamically creating queues for pub/sub

[novi border]

Hi,

It is not exactly clear to me from docs and forums is it possible to create new queues in a cluster dynamically?
What I mean is will getQueue("aName") generate the the queue if it is not already present in the cluster (it is not defined in the xml file before cluster startup). If it will how can it be configured, possibly from template queue defined in xml?
Also is it possible to have expiration (TTL) for such queues, so unused queues don't use up system resources?

I would like to create the queues dynamically and use them for point to point (unlike topic) pub/sub. Is there a better/recommended approach to accomplish this with hazelcast?

Best regards

[novi border]

Hi

I am evaluating Hazelcast for usage in dynamic, async, point to point, pub sub scenario, and I am wandering what the best approach is.
Publishers and subscribes communicate over dynamically created queues, of which they know names in advance, but publisher may connect before subscriber and vice versa. Publishers simply get the queue and put some data, and subscribes get the queue and get some data if available, or listen to modifications if data is not yet available. I have a few questions about this:

1. I am not clear whether queues can be dynamically created in a running cluster. Will getQueue on a non existing name create a new queue. If it will how can capacity and other parameters be configured? There is some mention in the groups of maybe being able to use predefined queue from the xml as a template, is that possible and how to select from which template queues get created?
2. As the usage scenario is dynamic, consumers may never connect to get the data. Can the old data be made to expire (TTL) if the queue consumer has not accessed it in a long time, so as to not consume resources indefinitely.
3. If the client reads available data from a queue, and then attaches a listener for any more additions does it need to wrap such an operation in transaction? Which should be done first adding a listener or draining the contents? Is it possible to guarantee no messages will be lost or duplicated?

This may not at all be the best (or even workable) approach to do it. Any suggestions? Maybe a with some maps, multimaps or lists in combination with transactions. Multimap by id for queue data, what about ordering of messages, ttl and listening to changes?


Best regards

[Fuad Malikov]

On Fri, Jan 27, 2012 at 1:35 AM, novi border <novib...@gmail.com> wrote:

Hi,

It is not exactly clear to me from docs and forums is it possible to create new queues in a cluster dynamically?

Yes, you can create every instance dynamically without defining it. 

What I mean is will getQueue("aName") generate the the queue if it is not already present in the cluster (it is not defined in the xml file before cluster startup). If it will how can it be configured, possibly from template queue defined in xml?

It will use the configuration for "default" or Hazelcast's internal default configuration.

Also is it possible to have expiration (TTL) for such queues, so unused queues don't use up system resources?

Queue's are backup by Map's. So you can set the TTL on the backed map. It will evict the unused entries.

I would like to create the queues dynamically and use them for point to point (unlike topic) pub/sub. Is there a better/recommended approach to accomplish this with hazelcast?

Yes, you can use Queue's for that but the topics are way more cheaper. I don't know your use case, but you can have a topic where only one point adds a listener and the other point publishes. 

 

Best regards

--
You received this message because you are subscribed to the Google Groups "Hazelcast" group.
To view this discussion on the web visit https://groups.google.com/d/msg/hazelcast/-/VUilA4GNiUsJ.
To post to this group, send email to haze...@googlegroups.com.
To unsubscribe from this group, send email to hazelcast+...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/hazelcast?hl=en.

[Fuad Malikov]

On Fri, Jan 27, 2012 at 10:24 AM, novi border <novib...@gmail.com> wrote:

Hi

I am evaluating Hazelcast for usage in dynamic, async, point to point, pub sub scenario, and I am wandering what the best approach is.
Publishers and subscribes communicate over dynamically created queues, of which they know names in advance, but publisher may connect before subscriber and vice versa. Publishers simply get the queue and put some data, and subscribes get the queue and get some data if available, or listen to modifications if data is not yet available. I have a few questions about this:

Instead of listening on Queue a subscriber can wait on queue.Take(). This is the blocking operation and will wait until an item is available.

1. I am not clear whether queues can be dynamically created in a running cluster. Will getQueue on a non existing name create a new queue. If it will how can capacity and other parameters be configured? There is some mention in the groups of maybe being able to use predefined queue from the xml as a template, is that possible and how to select from which template queues get created?

The documentation should be clear if you didn't see it yet: http://hazelcast.com/docs/2.0/manual/single_html/#Queue
 

2. As the usage scenario is dynamic, consumers may never connect to get the data. Can the old data be made to expire (TTL) if the queue consumer has not accessed it in a long time, so as to not consume resources indefinitely.

Yes, see the docs for setting TTL on Map(which backs the Queue)
 

3. If the client reads available data from a queue, and then attaches a listener for any more additions does it need to wrap such an operation in transaction? Which should be done first adding a listener or draining the contents? Is it possible to guarantee no messages will be lost or duplicated?

You should first add listener and then drain the entries. Now imagine that in between you adding a listener and draining publisher offered some new item. You will be receiving it both with listener and as the result of drain. You should somehow make sure that you haven't processed the item. As I said before, you don't need to use listeners. And note that receiving an entry from listener doesn't remove it from Queue.  

This may not at all be the best (or even workable) approach to do it. Any suggestions? Maybe a with some maps, multimaps or lists in combination with transactions. Multimap by id for queue data, what about ordering of messages, ttl and listening to changes?

 

Best regards

 

--

You received this message because you are subscribed to the Google Groups "Hazelcast" group.

To view this discussion on the web visit https://groups.google.com/d/msg/hazelcast/-/4Lk5biSb-rMJ.

[novi border]

Hi

thanks for the answer I have just a few more questions...

 

Instead of listening on Queue a subscriber can wait on queue.Take(). This is the blocking operation and will wait until an item is available.

I would like the wait operation to be nonblocking. Presumably I could use queue.take() from inside listener (or actually dranAll)..

The documentation should be clear if you didn't see it yet: http://hazelcast.com/docs/2.0/manual/single_html/#Queue

I am sorry, the front page links to 1.9 documentation, and this seems to be a new feature. This answers my questions.

I have one more question regarding queue creation. In certain conditions I would like getQueue to NOT create a new queue if it is not already existing. Is that possible? If it is not, perhaps I could use some queue creation listener and conditionally delete that queue or otherwise indicate it is not to be used yet?

Regards

[Fuad Malikov]

Fuad Malikov
Co-founder & Managing Partner
Hazelcast | Open source in-memory data grid
Mobile: +90.538.378.9777 | @fuadm

On Mon, Jan 30, 2012 at 4:26 PM, novi border <novib...@gmail.com> wrote:

Hi

thanks for the answer I have just a few more questions...

 

Instead of listening on Queue a subscriber can wait on queue.Take(). This is the blocking operation and will wait until an item is available.

I would like the wait operation to be nonblocking. Presumably I could use queue.take() from inside listener (or actually dranAll)..

Hazelcast Queue supports both blocking/non blocking and timely operations. Please have a look at them. 

The documentation should be clear if you didn't see it yet: http://hazelcast.com/docs/2.0/manual/single_html/#Queue

I am sorry, the front page links to 1.9 documentation, and this seems to be a new feature. This answers my questions.

I have one more question regarding queue creation. In certain conditions I would like getQueue to NOT create a new queue if it is not already existing. Is that possible? If it is not, perhaps I could use some queue creation listener and conditionally delete that queue or otherwise indicate it is not to be used yet?

Hazelcast will always create the new one if it doesn't exist. However you can listen Hazelcast.addInstanceListener() or get the existing instances Hazelcast.getInstances().
 

Regards

--
You received this message because you are subscribed to the Google Groups "Hazelcast" group.

To view this discussion on the web visit https://groups.google.com/d/msg/hazelcast/-/kMelJWy3ncUJ.