Conceptual overview of how Queues work

[Odin Hole Standal]

I've been looking for high level documentation on how the queue mechanism in Hazelcast works. I.e how messages are distributed in a cluster, how the guarantee that a message is only consumed by a single consumer, how throughput is affected by adding nodes to a network, what happens when a node fails during consumption / production etc.

Does anyone know of such documentation or blogs or anything related? I found some information in the "Getting started with Hazelcast" and the "Book of Hazelcast", but it was mostly code examples and little about the implementation.

[Peter Veentjer]

The queue is not a partitioned data-structure. So if you have 1M items in a Queue, there will be a member in the cluster containing 1M items. And of course, if you have x backups, there will be another x members that contain 1M items.

Guarantee: when an item is taken of that queue, the internal data-structure is updated and the response is returned as soon as all backups have synced. If your taking node crashes before it has processed the message, the item is lost. 

Throughput: since the queue is not a partitioned data-structure, adding/removing nodes to the cluster will not influence the performance of that queue. Since all communication will be done on the member containing that partitions (and the members containing the backup). 

Is there more you want to know?

On Wed, Apr 23, 2014 at 9:58 AM, Odin Hole Standal <odin.hol...@gmail.com> wrote:

I've been looking for high level documentation on how the queue mechanism in Hazelcast works. I.e how messages are distributed in a cluster, how the guarantee that a message is only consumed by a single consumer, how throughput is affected by adding nodes to a network, what happens when a node fails during consumption / production etc.

Does anyone know of such documentation or blogs or anything related? I found some information in the "Getting started with Hazelcast" and the "Book of Hazelcast", but it was mostly code examples and little about the implementation.

--
You received this message because you are subscribed to the Google Groups "Hazelcast" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hazelcast+...@googlegroups.com.
To post to this group, send email to haze...@googlegroups.com.
Visit this group at http://groups.google.com/group/hazelcast.
To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/ad3e0c88-096c-457a-8e32-41b524d48920%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Odin Hole Standal]

Thanks for the quick reply, Peter. 

This information would be really great to have in the documentation, it certainly helps me understand the queuing mechanism.

I am interested in your comment on taking messages from the queue; Is it possible to take a certain number of elements off a queue (through the drainTo(Collection, maxElements method) in a transactional manner? I.e if the processing fails on the consumer, the elements will not be removed from the queue? 

/Odin

[Ali Gurbuz]

Transactional queue does not support drain. But it is easy to implement, can you file an enhancement issue

To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/2bebdada-b5a8-4e42-b86f-1334f9e3743d%40googlegroups.com.

[Odin Hole Standal]

I will create an issue for it. 

Can you suggest a workaround in the meantime? 

I.e can I start a Hazelcast transaction, take x items, process them and then commit?

/Odin

[Odin Hole Standal]

Issue created. https://github.com/hazelcast/hazelcast/issues/2334

[Ali Gurbuz]

The workaround is to poll x times from queue and process, performance will not be good since every poll will make a network hop

To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/98e925d7-9ac1-48d2-9291-ce7ebd016687%40googlegroups.com.

[Odin Hole Standal]

Just to check that I understand you correctly, do you mean the following?

Use a transactionalQueue.

1. Start a hazelcast transaction

2. poll x times from the transactional queue

3. process messages

4. commit

Additionally, how does starting a transaction while reading from a queue affect the other consumers of the queue? Are they blocked?

/Odin 

[Ali Gurbuz]

Yes you are correct, and queue does not block

To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/06e664dc-74b4-42bd-ae3d-fae8fc4ffd06%40googlegroups.com.

[Odin Hole Standal]

Thank you, again this would be really helpful to have in the documentation. Perhaps along with a diagram of some sort that illustrates the distributed queues in a high level fashion.

/Odin