Hazelcast ITopic - reliable message delivery

[Afkham Azeez]

Hi folks,

What type of reliable message guarantees does Hazelcast topics provide? Is this the similar to persistent messaging system guarantees? Does it support durable subscriptions?

For example, if a member who has subscribed to a topic, momentarily loses connectivity, and reconnects, will messages that were published to the topic during the time the member was disconnected delivered to the member when it reconnects?

Thanks

Azeez

[Christoph Engelbert]

Hi Azeez,

For know there is neither reliable transportation nor caching / retransmission in case of node failure (as far as I know).

Eventually it could make sense to offer that features. What is your actual use case for that? Maybe there's some other way to realize it.

Chris

[Afkham Azeez]

Hi Chris,

In our case, we have a mechanism called deployment synchronization. When new deployment artifacts become available in a deployment repo, the master node notifies all others in the cluster to get an update of the deployment repo. This message has to be reliably delivered. Otherwise, we could end up in a situation where some deployment artifacts are available on some nodes, and it is not available in others. The way the master node (we call it the management node), notifies others is through publishing a message to a Hazelcast topic. I wanted to know whether the Hazelcast topics provides these delivery guarantees. Is there a way to ensure this?

Thanks

Azeez

--
You received this message because you are subscribed to the Google Groups "Hazelcast" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hazelcast+...@googlegroups.com.
To post to this group, send email to haze...@googlegroups.com.
Visit this group at http://groups.google.com/group/hazelcast.
For more options, visit https://groups.google.com/groups/opt_out.

[Peter Veentjer]

Afaik this isn't possible using a ITopic.

An idea:

Instead of relying on a topic, you could perhaps try to use an IMap. 

You can use the timestamp as key and the original message as a value. Set a ttl on the map to get rid of old data, this will also help to create your durable subscriptions since you can go back in time. I don't know if it can happen that multiple messages are send on the same timestamp, but otherwise you could combine the timestamp with the senderid.

And for processing you register a map listener; and you need to update your local timestamp once you receive an update from the map so you know where you are.

It requires a bit more work, but I think this could be a step in the right direction.

[Nick Pratt]

A reasonable workaround.  For what it's worth, I've used durable topic subscription in several platforms now (outside of Hazelcast) so it's a useful feature. Some other messaging platforms have a bridge setup whereby the topic is bridged to a persistent queue per registered listener (each registered listener having its own persistent queue).  Shared durable queues are also useful for load balancing scenarios (multiple queue listeners on the same queue, but if all listeners disconnect, the queue should remain) - I think this functionality is available now (persistent queue with multiple listeners)?

[Christoph Engelbert]

But when they decide to remove the queues for not returning members?

[Nick Pratt]

They dont - its a durable subscription.  Has to be monitored by ops. 

[Christoph Engelbert]

Ok I see. So you would need a persistent queue for every listener that is at best handled by something that looks like a topic?

What listeners do you use, only nodes or clients too?

I guess I can hack something together that looks easy to use (like a topic) but has listener and persistence support.

Chris

[Nick Pratt]

For this use case (the persistent topic listener) its for ensuring that a series of different clients running different programs/algos receive each and every message.  In a Hazelcast scenario Id likely use Clients only - however the issue then becomes what to do in between the nodes - if all clients are subscribed to Node#2 and Node#2 fails, does Node#1 ensure that any messages broadcast in the interim (while the clients are failing over to Node#1) are delivered upon successful reconnection to the cluster?

[Christoph Engelbert]

Every entry has at least one backup in the cluster, so if node 2 fails and the client reconnects to one of the other nodes no data are lost.

[vdines...@gmail.com]

Hello,

I just wanted to know, if there has been any update on the implementation of Topics in terms of guaranteed delivery of messages. Let me quote Azeez's example :

" For example, if a member who has subscribed to a topic, momentarily loses connectivity, and reconnects, will messages that were published to the topic during the time the member was disconnected delivered to the member when it reconnects? "

Thanks,
Dinesh

[Noctarius]

Hey Dinesh,

Hazelcast 3.5 will include a reliable topic implementation backed by a ringbuffer structure. Unfortunately there is not yet any documentation about it.

Chris

To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/a8478945-88ab-4322-a510-0f4a6abf8848%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[wenxing zheng]

Regarding the Reliable topic, if the consumer crashed and then restarted again, how to recover from the crash? In this case, there may be some events which hadn't been processed. But it seems the Reliable topic didn't provide the ACK mechanism?

[Sertuğ Kaya]

You should use ReliableTopic and ReliableMessageListener's. As you can see from the interface, it stores a sequence id and can resume from the latest sequence in case of a disconnection.

[wenxing zheng]

Thanks to Sertuğ.

With the sequence id, then the messages in the ReliableTopic are delivered in exactly once, at most once, or at least once?

[Sertuğ Kaya]

Both is possible based on your listener implementation. Please see: https://github.com/hazelcast/hazelcast/blob/b1c425e8598cde123ab11493b4dc7896dbfec15e/hazelcast/src/main/java/com/hazelcast/topic/ReliableMessageListener.java#L43

[wenxing zheng]

From the description of https://github.com/hazelcast/hazelcast/blob/b1c425e8598cde123ab11493b4dc7896dbfec15e/hazelcast/src/main/java/com/hazelcast/topic/ReliableMessageListener.java#L73:

/**
	* Informs the ReliableMessageListener that it should store the sequence. This method is called before the message is
	* processed. Can be used to make a durable subscription.
	*
	* @param sequence the sequence
	*/
	void storeSequence(long sequence);

storeSequence will be called before the message is processed, therefore, we can't make sure the message corresponding to the "sequence“ has already been processed in expectation. In other words, the "exactly once" could be a problem here.

And from the Message signature, there is not sequence id attribute in the definition.

[Sertuğ Kaya]

There's also an `isTerminal(Throwable failure)` setting which terminates the listener in case of a faliure during message processing, which counters that case.