Reliable topic configuration

[pooja.ac...@gmail.com]

Hi,

I have few concerns on topic configuration,

1. is it advisable to have multiple hazelcast instances per application??
2. should we set configuration before creating hazelcast instance?
3. Can we set some of the config later after creating hazelcast instance??

[Ahmet Mircik]

Hi, answers inlined.

On Thu, Dec 27, 2018 at 2:26 PM pooja.ac...@gmail.com wrote:

Hi,

I have few concerns on topic configuration,

1. is it advisable to have multiple hazelcast instances per application??

It depends on the case. One instance should be ok in general.

1. should we set configuration before creating hazelcast instance?

Better to set it before instance creation.

1. Can we set some of the config later after creating hazelcast instance??

You cannot change existing config but can add new configurations dynamically.

--
You received this message because you are subscribed to the Google Groups "Hazelcast" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hazelcast+...@googlegroups.com.
To post to this group, send email to haze...@googlegroups.com.
Visit this group at https://groups.google.com/group/hazelcast.
To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/733c27ca-3870-4340-bf15-cfcb13c476cb%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Pooja Sahu]

Thank you Ahmet. 

I have few more concerns.

In my application, already one Hazelcast instance and Hazelcast client is running and currently the instance and client are very much, module specific in terms of configurations. I need to implement something new and I don't want to disturb existing instance and client. So I am planning to instantiate new hazelcast instance or client. I am not able to decide that whether I should go for Hazelcast instance or Hazelcast client ?? I would do refactoring later and probably whole application would use single instance or client. 

newHazelcastInstance or newHazelcastClient ??

Thanks,

Pooja

To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/CAEb73hEUNoVg9vgkCq7eWHfh2%2BAnvnXpzgbJEzkX25TVnnXTDQ%40mail.gmail.com.

[Pooja Sahu]

++ info,

We are using Hazelcast v3.8.3.

Thanks,

Pooja

[Sertuğ Kaya]

Pooja;

It depends on if you want embedded or client/server architecture. If you want more storage/process capability you can add more members to your cluster but if you just want to read/write data than you can simply spawn a client and connect to your existing cluster. Please keep in mind that you can add/remove members to your cluster without any disruption so both options are applicable for you.

Also please remember that members backup each other and cluster can recover in case some of them fails. Having a single member removes this advantage so I suggest you have a cluster of 3 at least.

Please also see: https://docs.hazelcast.org/docs/latest/manual/html-single/#hazelcast-topology

[Pooja Sahu]

Thank you Sertug for elaborate explanation.I have one scenario where there would be only one member as it would be an stand alone node, however for this standalone node also Hazelcast topic logic is same as cluster nodes. In this standalone scenario I would just have one cluster member.

Regards,

Pooja

To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/a7c57bbf-7423-4095-b3fc-4baccbd468e5%40googlegroups.com.

[Pooja Sahu]

Hi,

I am writing a generic Message Listener for ReliableTopic. And OnMessage(), I want to identify that for which topic I have received message and based on some property of the received message. However I am not sure yet that if it is possible to identify that for which topic we have received message.

Thanks,

Pooja

[Sertuğ Kaya]

Pooja;

Every listener works with a single topic, and you can get the name of the source topic for a message by `message.getSource();` from inside the listener.

[Pooja Sahu]

Thank you Sertug for quick response,

I had figured out 'getSource()' method. Actually I was trying to write a generic class. And in onMessage() function based on 'getSource()' I can decide further course of action.

Sample code,

public class MyMsgListener<E> implements ReliableMessageListener<E> {

    

    long seq;

    

    @Override

    public void onMessage(Message<E> m) {

              switch(m.getSource())

              {      

              }      

    }

public static void main()

{

        MyMsgListener<SomeData> msgListener = new MyMsgListener<SomeData>();

        if(null != msgListener) {

            ITopic<SomeData> topic = client.getReliableTopic(TopicsConstants.TopicName);

            topic.addMessageListener(msgListener);

        } 

} 

I think above code should work though not sure as I have not tested it yet. Please let me know if above code is allowed or work. However now we are planning to have a separate message listener for each topic else it would lead to a big switch case which is also not good or a second level of observer/listener hierarchy which definitely we don't want to have as to avoid multiple listeners we were planning to come up with generic listener concept. Moreover with specific message listeners we can set the properties which are specific to listener.

To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/493cdbaa-3106-4f0d-b6ae-fa93164402b3%40googlegroups.com.

[Pooja Sahu]

Hi,

Brief on Reliable Topic config is below,

The reliable topic makes use of the Ringbuffer to store the actual messages. To configure the ringbuffer for a reliable topic, define a ringbuffer in the config with exactly the same name. It is very unlikely that you want to run with the default settings. When a ReliableTopic starts, it will always start from the tail+1 item from the RingBuffer. It will not chew its way through all available events but it will wait for the next item being published. 

Here, I am not able to understand the meaning of the line "It will not chew its way through all available events but it will wait for the next item being published" .

Please help me to understand the statement.

Thanks.

Pooja

[Sertuğ Kaya]

Pooja;

Each listener instance works for a single topic, so having a separate message listener for each topic is the correct approach indeed.

The statement in the javadoc means that your listener will be triggered on each publish event so it will work as intended in a pub/sub mechanism.

[Pooja Sahu]

Hi Sertug,

I have doubts for setting 'read-batch-size' of Reliable topic, Below is the description for the config param,

read-batch-size : Minimum number of messages that Reliable Topic will try to read in batches. Its default value is 10. The ReliableTopic tries to read a batch of messages from the ringbuffer. It will get at least one, but if there are more available, then it will try to get more to increase throughput. The minimal read batch size can be influenced using the read batch size. Apart from influencing the number of messages to download, the readBatchSize also determines how many messages will be processed by the thread running the MessageListener before it returns back to the pool to look for other MessageListeners that need to be processed. The problem with returning to the pool and looking for new work is that interacting with an Executor is quite expensive due to contention on the work-queue. The more work that can be done without retuning to the pool, the smaller the overhead. If the readBatchSize is 10 and there are 50 messages available, 10 items are retrieved and processed consecutively before the thread goes back to the pool and helps out with the processing of other messages. If the readBatchSize is 10 and there are 2 items available, 2 items are retrieved and processed consecutively. If the readBatchSize is an issue because a thread will be busy too long with processing a single MessageListener and it can't help out other MessageListeners, increase the size of the threadpool so the other MessageListeners don't need to wait for a thread, but can be processed in parallel.

> Apart from influencing the number of messages to download, the readBatchSize also determines how many messages will be processed by the thread running the MessageListener before it returns back to the pool to look for other MessageListeners that need to be processed.

Here Another message listener means the the listener for another topic or it can be another message listener for the same topic for which messages are being processed currently??

And this thread pool is in terms of single cluster node. Each cluster node would have their own thread pool??

To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/765700a9-3df0-4830-acb0-28b121a29dfc%40googlegroups.com.

[Pooja Sahu]

Hi,

I have few more concerns,

1. Does the item gets deleted from ring buffer once it is consumed by all the subcribers?? As we know that based on topic overload policy if we set it to DISCARD OLDEST then the oldest item would be deleted only if the ring buffer is full and there is another message. But I want to know if message gets deleted even when it is read by all the consumer and the ring buffer is not full yet.

2. Whether it is necessary to override ‘storeSequence( )’ and ‘retrieveInitialSequence( )’ methods. I am not able to understand the exact usecase of preserving sequence parameter.

3. What happens when MessageListener gets terminated, does it restart by itself or we need to monitor the termination and need to restart the message listener.

Based on values of ‘isLossTolerant’ and ‘isTerminal’, MessageListener can be terminated. 

Please put some light on thread pool mechanism for message listener also.

Thanks,

Pooja

Tr

To unsubscribe from this group and stop receiving emails from it, send an email to hazelcast+unsubscribe@googlegroups.com.

[Sertuğ Kaya]

>Here Another message listener means the the listener for another topic or it can be another message listener for the same topic for which messages are being processed currently??

It's the executor of this ReliableTopic. You can use the same executor thread pool for multiple topic or use one for each and isolate them from each other. In this sentence it assumes you might be using the same executor service for multiple listeners, on a given node. Please see https://github.com/hazelcast/hazelcast/blob/master/hazelcast/src/main/java/com/hazelcast/config/ReliableTopicConfig.java#L161 

>And this thread pool is in terms of single cluster node. Each cluster node would have their own thread pool??

It's a per node setting.

>Does the item gets deleted from ring buffer once it is consumed by all the subcribers?

No it doesn't. As you have guessed only oldest items are discarded in case the underlying ringbuffer is full, per your configuration of course.

>2. Whether it is necessary to override ‘storeSequence( )’ and ‘retrieveInitialSequence( )’ methods. I am not able to understand the exact usecase of preserving sequence parameter.

It gives you the ability to start listening from a specific sequence, so in case of a brief network disconnection for instance, you can start listening from the latest event you stopped at, and not miss any events came in when you were disconnected. So it's necessary. Please see the javadocs at https://github.com/hazelcast/hazelcast/blob/master/hazelcast/src/main/java/com/hazelcast/topic/ReliableMessageListener.java

>3. What happens when MessageListener gets terminated, does it restart by itself or we need to monitor the termination and need to restart the message listener.

Afaik it doesn't restart itself and expects user to take necessary action and restart upon the termination.

[Pooja Sahu]

Hi Sertug,

To figure out that whether the items from Ringbuffer gets removed once they are consumed by all subscribers, I have done below exercise and it looked like somehow items from Ringbuffer gets removed for some condition.

1. Test case 1: Reliable Topic 
1. Set RingBuffer capacity to 4
2. Set TopicOverloadPolicy to ‘ERROR’ (The publish call immediately fails.)
3. Publish messages > 4

‘publish()’ was supposed to fail for 5^th message but I have not seen any failure for Test case 1 and have received all the messages with correct data values. So based on the testing it looks like that messages from RingBuffer gets removed for some condition though not sure yet about the condition of item removal from ring buffer. 

Please put some light on above scenario.

Thanks,

Pooja

To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/a35949a5-9d66-4156-b465-3e4c38ab679c%40googlegroups.com.

[Pooja Sahu]

++ Message Listener config,

I am planning to have below configuration for Message Listener, please provide your input if anything is out of sync.

1. void storeSequence(long sequence): Would store sequence in a data member say ‘seq’ , with initial value seq = -1;
2. long retrieveInitialSequence(): if (seq == -1) return -1; else return seq + 1; (This is as per the method’s description provided in documentation)
3. isLossTolerant():  return true (Message listener won’t terminate for failures)
4. boolean isTerminal(Throwable failure): return false (Message listener won’t terminate for failures)

Thanks,

Pooja

[Sertuğ Kaya]

That sounds fine. You can also check out this example impl in hazelcast tests https://github.com/hazelcast/hazelcast/blob/master/hazelcast/src/test/java/com/hazelcast/topic/impl/reliable/DurableSubscriptionTest.java#L80