Multiple threads for MessageListener

[Ron Siemens]

I couldn't find anywhere in the documentation describing what the thread activation pattern for onMessage() of a MessageListener is.

It seems that all my onMessage() calls are handled by the same thread ("hz.client.1.Listener"), and if it is still busy, subsequent messages just queue up.

Can I configure hazelcast to some more robust thread activation mechanism than this?

I thought maybe adding multiple listeners to the same topic may give me multiple threads too. Alas no, the same behavior with just random(?) individual listeners getting a message, still only one at a time.

Is this as expected? No configurable thread pool or the like? That's up to the user?

Ron

[Fuad Malikov]

The thing is topic guaranties the message order. That's why it have to be one thread delivering them. Your on message implementation shouldn't keep thread busy. Better hand off the message to another thread and let the processing done there.

Regards,

Fuad Malikov
Sent from Mobile

> --
> You received this message because you are subscribed to the Google Groups "Hazelcast" group.
> To post to this group, send email to haze...@googlegroups.com.
> To unsubscribe from this group, send email to hazelcast+...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/hazelcast?hl=en.
>

[Ron Siemens]

Thanks for the clarification. I've also already assumed as much and went ahead and used ExecutorService to handle messages. It works fine enough, but I do find that pattern odd.

If the framework is going to introduce threads that run my application code, it would be good to allow robust configuration of such. Otherwise it would seem preferable just to have your single sequencing thread under the covers, and keep all threading up to the user. Any thread could block on reception and a message could wake any of them to be handled.

Not being one way or the other, feels like an awkward hybrid with a framework thread leaking into my application.

Ron

[Tim Peierls]

Allowing users to configure framework threading would mean, among other things, that the user would be responsible for enforcing message ordering guarantees. That's a heavy burden to impose, especially when it's so easy to off-load the work using something like Guava's EventBus (in this case, AsyncEventBus). 

It's not unreasonable for a framework to require that application code get in and get out quickly. The Fork Join framework in Java SE7, for example, says:

The efficiency of ForkJoinTasks stems from a set of restrictions (that are only partially statically enforceable) reflecting their intended use as computational tasks calculating pure functions or operating on purely isolated objects. The primary coordination mechanisms are fork(), that arranges asynchronous execution, and join(), that doesn't proceed until the task's result has been computed. Computations should avoid synchronized methods or blocks, and should minimize other blocking synchronization apart from joining other tasks or using synchronizers such as Phasers that are advertised to cooperate with fork/join scheduling. Tasks should also not perform blocking IO, and should ideally access variables that are completely independent of those accessed by other running tasks. 

So I think Hazelcast has it right.

--tim

[Ron Siemens]

On Nov 17, 2011, at 6:05 AM, Tim Peierls wrote:

Allowing users to configure framework threading would mean, among other things, that the user would be responsible for enforcing message ordering guarantees.

Not sure if you don't understand, or you're playing with straw men.  The alternatives I suggested wouldn't mean this at all.  There's no reason you can't keep a sequencing thread under the covers.

That's a heavy burden to impose, especially when it's so easy to off-load the work using something like Guava's EventBus (in this case, AsyncEventBus). 

It's not unreasonable for a framework to require that application code get in and get out quickly. The Fork Join framework in Java SE7, for example, says:

The efficiency of ForkJoinTasks stems from a set of restrictions (that are only partially statically enforceable) reflecting their intended use as computational tasks calculating pure functions or operating on purely isolated objects. The primary coordination mechanisms are fork(), that arranges asynchronous execution, and join(), that doesn't proceed until the task's result has been computed. Computations should avoid synchronized methods or blocks, and should minimize other blocking synchronization apart from joining other tasks or using synchronizers such as Phasers that are advertised to cooperate with fork/join scheduling. Tasks should also not perform blocking IO, and should ideally access variables that are completely independent of those accessed by other running tasks. 

Because something can be used advantageously, means it's always advantageous to use it?  

So I think Hazelcast has it right.

The alternatives may have required a little more effort but would have been more robust.  It seems quite likely this was a trade-off made for expediency or convenience, because it doesn't offer any other advantages, only drawbacks.

This feature set can easily be interrupted or brought down by the user because it has been exposed this way.  At the very least, the expected pattern of usage should be clearly documented because of its drawbacks and risks.

Ron

[Fuad Malikov]

I documented this a bit more in Topic documentation.

Thanks,
Fuad
http://twitter.com/fuadm

On Thu, Nov 17, 2011 at 8:54 PM, Tim Peierls <t...@peierls.net> wrote:

On Thu, Nov 17, 2011 at 11:39 AM, Ron Siemens <rsie...@greatergood.com> wrote:

On Nov 17, 2011, at 6:05 AM, Tim Peierls wrote:

Allowing users to configure framework threading would mean, among other things, that the user would be responsible for enforcing message ordering guarantees.

Not sure if you don't understand, or you're playing with straw men.  The alternatives I suggested wouldn't mean this at all.  There's no reason you can't keep a sequencing thread under the covers.

Having a separate thread to dispatch message events introduces extra task allocation and potential thread context switching that not all users would tolerate, whether this separate thread is managed by the user or internally to Hazelcast. You might not care about that overhead, but I wouldn't want to impose it on those who do.

It's not unreasonable for a framework to require that application code get in and get out quickly. The Fork Join framework in Java SE7, for example, [does this]

Because something can be used advantageously, means it's always advantageous to use it?  

No, I'm saying that Hazelcast is not unique in placing requirements on the nature of user-provided callbacks.

So I think Hazelcast has it right.

The alternatives may have required a little more effort but would have been more robust.  It seems quite likely this was a trade-off made for expediency or convenience, because it doesn't offer any other advantages, only drawbacks.

You can layer thread-pool-based dispatching on top of this design -- sounds like that's what you ended up doing -- but you can't go the other way, from thread-pool-based dispatching to the raw performance that this design allows.

I don't think it's fair to call this design expedient or convenient; the Hazelcast developers have worked too hard at keeping the engine highly performant to deserve that characterization. OTOH, if you can find a way to rewrite the internals to make them tolerant of blocking message handlers without sacrificing any performance, please share that code with us.

 

This feature set can easily be interrupted or brought down by the user because it has been exposed this way.  At the very least, the expected pattern of usage should be clearly documented because of its drawbacks and risks.

I agree that the requirements for onMessage() should be documented clearly, and an example provided of how to dispatch via an executor.

Your objection is surprising to me. The usual complaint we get on the concurrency mailing lists is that something is too protective by not offering access to the underlying raw internals. It's rare to hear someone complain about being given low-level access.

--tim

[Tim Peierls]

On Thu, Nov 17, 2011 at 11:39 AM, Ron Siemens <rsie...@greatergood.com> wrote:

On Nov 17, 2011, at 6:05 AM, Tim Peierls wrote:

Allowing users to configure framework threading would mean, among other things, that the user would be responsible for enforcing message ordering guarantees.

Not sure if you don't understand, or you're playing with straw men.  The alternatives I suggested wouldn't mean this at all.  There's no reason you can't keep a sequencing thread under the covers.

Having a separate thread to dispatch message events introduces extra task allocation and potential thread context switching that not all users would tolerate, whether this separate thread is managed by the user or internally to Hazelcast. You might not care about that overhead, but I wouldn't want to impose it on those who do.

It's not unreasonable for a framework to require that application code get in and get out quickly. The Fork Join framework in Java SE7, for example, [does this]

Because something can be used advantageously, means it's always advantageous to use it?  

No, I'm saying that Hazelcast is not unique in placing requirements on the nature of user-provided callbacks.

So I think Hazelcast has it right.

The alternatives may have required a little more effort but would have been more robust.  It seems quite likely this was a trade-off made for expediency or convenience, because it doesn't offer any other advantages, only drawbacks.

You can layer thread-pool-based dispatching on top of this design -- sounds like that's what you ended up doing -- but you can't go the other way, from thread-pool-based dispatching to the raw performance that this design allows.

I don't think it's fair to call this design expedient or convenient; the Hazelcast developers have worked too hard at keeping the engine highly performant to deserve that characterization. OTOH, if you can find a way to rewrite the internals to make them tolerant of blocking message handlers without sacrificing any performance, please share that code with us.

 

This feature set can easily be interrupted or brought down by the user because it has been exposed this way.  At the very least, the expected pattern of usage should be clearly documented because of its drawbacks and risks.

[Timothy Peierls]

Great, thanks! I think the sample code should have ITopic instead of Topic.

Here's how one might adapt that sample to include the idea of dispatching heavyweight tasks via an Executor.

public class Sample implements MessageListener<MyEvent> {

    public static void main(String[] args) { 

        Sample sample = new Sample();

        ITopic topic = Hazelcast.getTopic ("default");  

        topic.addMessageListener(sample);         

        topic.publish (new MyEvent());

    }  

     

    public void onMessage(MyEvent myEvent) {

        System.out.println("Message received = " + myEvent.toString());

        if (myEvent.isHeavyweight()) {

            messageExecutor.execute(new Runnable() {

                public void run() {

                    doHeavyweightStuff(myEvent);

                }

            });

        }

    }

    // ...

    private static final Executor messageExecutor = Executors.newSingleThreadExecutor(); 

}

--tim

[Fuad Malikov]

Thanks Tim,

I added your code. Much appreciated!

Fuad
http://twitter.com/fuadm

--

You received this message because you are subscribed to the Google Groups "Hazelcast" group.

To view this discussion on the web visit https://groups.google.com/d/msg/hazelcast/-/XZYg9Xyxf-oJ.

[joaqui...@pointwest.com.ph]

When should I shutdown the executor?

CONFIDENTIALITY NOTICE: This email may contain confidential and privileged material for the sole use of the intended recipient(s). Any review, use, distribution or disclosure by others is strictly prohibited. If you have received this communication in error, please notify the sender immediately by e-mail and delete the message and any file attachments from your computer. There is no warranty that this email is error, virus or defect free. If this is a private communication it does not represent the views of Pointwest Technologies Corporation or their related entities.

[Nick Pratt]

After you unregister the listener from HZ.

--

You received this message because you are subscribed to the Google Groups "Hazelcast" group.

To unsubscribe from this group and stop receiving emails from it, send an email to hazelcast+unsubscribe@googlegroups.com.

To post to this group, send email to haze...@googlegroups.com.

Visit this group at https://groups.google.com/group/hazelcast.
To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/4467a235-e60e-48d2-99df-2d39e0f7f8f9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Joaquin Cheng Toral]

public void onMessage(MyEvent myEvent) {

        System.out.println("Message received = " + myEvent.toString());

        if (myEvent.isHeavyweight()) {

            messageExecutor.execute(new Runnable() {

                public void run() {

                    doHeavyweightStuff(myEvent);

                }

            });

        }

        myEvent.shutdown();

    }

Is this valid? Thank you for your time. I am kinda new with MessageListener and spring multithreading and I came across your discussion. So excited to get back to the office on monday to try this out.

--
You received this message because you are subscribed to a topic in the Google Groups "Hazelcast" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/hazelcast/OW0i9un0cDA/unsubscribe.
To unsubscribe from this group and all its topics, send an email to hazelcast+unsubscribe@googlegroups.com.

To post to this group, send email to haze...@googlegroups.com.
Visit this group at https://groups.google.com/group/hazelcast.

To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/CALpvOjuWGOjEFRdC1oDnbionJ3BW-mXQYBbpa6Y6yugROB2%3Dbw%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Joaquin Cheng Toral

Software Engineer/Developer

Pointwest Technologies Corporation

3/F Building Alpha UP-AyalaLand, Commonwealth Ave. QC

Email: joaquin...@pointwest.com.ph

Website: www.pointwest.com.ph

   

This e-mail message may contain confidential or legally privileged information and is intended only for the use of the intended recipient(s). Any unauthorized disclosure, dissemination, distribution, copying or the taking of any action in reliance on the information herein is prohibited. E-mails are not secure and cannot be guaranteed to be error free as they can be intercepted, amended, or contain viruses. Anyone who communicates with us by e-mail is deemed to have accepted these risks. Pointwest Technologies Corporation is not responsible for errors or omissions in this message and denies any responsibility for any damage arising from the use of e-mail. Any opinion and other statement contained in this message and any attachment are solely those of the author and do not necessarily represent those of the company.

[Noctarius]

No you need to shutdown the executor when you shutdown your application. With this code you’d shut it down right after the first event.

To unsubscribe from this group and stop receiving emails from it, send an email to hazelcast+...@googlegroups.com.

To post to this group, send email to haze...@googlegroups.com.
Visit this group at https://groups.google.com/group/hazelcast.

To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/CAGFUHG44FzLxqwhXrv10XP2VAM25FJ%2BbrqF%3Dw50jDfPu-kz71g%40mail.gmail.com.

[Joaquin Cheng Toral]

I see. My application is a listener app that gets data through messages coming from several sources. The app will run almost forever unless a maintenance is scheduled. So if the app is running constantly, it won't be necessary to shutdown the executor anymore? Is this correct? The thread will gracefully end each time a task is processed and will take up new tasks without the risk of having memory leak. Working in different projects kinda makes it hard to read about this. 

Thank you for your time man! 

On Sat, Jun 3, 2017 at 1:21 PM, 'Noctarius' via Hazelcast <haze...@googlegroups.com> wrote:

No you need to shutdown the executor when you shutdown your application. With this code you’d shut it down right after the first event.

On 3. Jun 2017, at 07:16, Joaquin Cheng Toral <joaqui...@pointwest.com.ph> wrote:

public void onMessage(MyEvent myEvent) {

        System.out.println("Message received = " + myEvent.toString());

        if (myEvent.isHeavyweight()) {

            messageExecutor.execute(new Runnable() {

                public void run() {

                    doHeavyweightStuff(myEvent);

                }

            });

        }

        myEvent.shutdown();

    }

Is this valid? Thank you for your time. I am kinda new with MessageListener and spring multithreading and I came across your discussion. So excited to get back to the office on monday to try this out.

On Fri, Jun 2, 2017 at 8:35 PM, Nick Pratt <nbp...@gmail.com> wrote:

After you unregister the listener from HZ.

-- 

Email: joaquin.toral@pointwest.com.ph

Website: www.pointwest.com.ph

   

This e-mail message may contain confidential or legally privileged information and is intended only for the use of the intended recipient(s). Any unauthorized disclosure, dissemination, distribution, copying or the taking of any action in reliance on the information herein is prohibited. E-mails are not secure and cannot be guaranteed to be error free as they can be intercepted, amended, or contain viruses. Anyone who communicates with us by e-mail is deemed to have accepted these risks. Pointwest Technologies Corporation is not responsible for errors or omissions in this message and denies any responsibility for any damage arising from the use of e-mail. Any opinion and other statement contained in this message and any attachment are solely those of the author and do not necessarily represent those of the company.

CONFIDENTIALITY NOTICE: This email may contain confidential and privileged material for the sole use of the intended recipient(s). Any review, use, distribution or disclosure by others is strictly prohibited. If you have received this communication in error, please notify the sender immediately by e-mail and delete the message and any file attachments from your computer. There is no warranty that this email is error, virus or defect free. If this is a private communication it does not represent the views of Pointwest Technologies Corporation or their related entities.

-- 
You received this message because you are subscribed to the Google Groups "Hazelcast" group.

To unsubscribe from this group and stop receiving emails from it, send an email to hazelcast+unsubscribe@googlegroups.com.
To post to this group, send email to haze...@googlegroups.com.
Visit this group at https://groups.google.com/group/hazelcast.

To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/CAGFUHG44FzLxqwhXrv10XP2VAM25FJ%2BbrqF%3Dw50jDfPu-kz71g%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "Hazelcast" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/hazelcast/OW0i9un0cDA/unsubscribe.
To unsubscribe from this group and all its topics, send an email to hazelcast+unsubscribe@googlegroups.com.
To post to this group, send email to haze...@googlegroups.com.
Visit this group at https://groups.google.com/group/hazelcast.

To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/6AD402BF-DF1C-4F3F-B55F-EA97DAD7B127%40googlemail.com.

For more options, visit https://groups.google.com/d/optout.

--

Joaquin Cheng Toral

Software Engineer/Developer

Pointwest Technologies Corporation

3/F Building Alpha UP-AyalaLand, Commonwealth Ave. QC

Email: joaquin...@pointwest.com.ph

Website: www.pointwest.com.ph

   

This e-mail message may contain confidential or legally privileged information and is intended only for the use of the intended recipient(s). Any unauthorized disclosure, dissemination, distribution, copying or the taking of any action in reliance on the information herein is prohibited. E-mails are not secure and cannot be guaranteed to be error free as they can be intercepted, amended, or contain viruses. Anyone who communicates with us by e-mail is deemed to have accepted these risks. Pointwest Technologies Corporation is not responsible for errors or omissions in this message and denies any responsibility for any damage arising from the use of e-mail. Any opinion and other statement contained in this message and any attachment are solely those of the author and do not necessarily represent those of the company.

[Nick Pratt]

No - that will shut down your executor immediately after registering the listener.  Typically you would have a shutdown hook / close / stop method on your class.  In there you would unregister your listener from Hazelcast (look at the return value from addMessageListener()) and then after unregistering you would shut down your Executor.

N

Email: joaquin.toral@pointwest.com.ph

Website: www.pointwest.com.ph

   

This e-mail message may contain confidential or legally privileged information and is intended only for the use of the intended recipient(s). Any unauthorized disclosure, dissemination, distribution, copying or the taking of any action in reliance on the information herein is prohibited. E-mails are not secure and cannot be guaranteed to be error free as they can be intercepted, amended, or contain viruses. Anyone who communicates with us by e-mail is deemed to have accepted these risks. Pointwest Technologies Corporation is not responsible for errors or omissions in this message and denies any responsibility for any damage arising from the use of e-mail. Any opinion and other statement contained in this message and any attachment are solely those of the author and do not necessarily represent those of the company.

CONFIDENTIALITY NOTICE: This email may contain confidential and privileged material for the sole use of the intended recipient(s). Any review, use, distribution or disclosure by others is strictly prohibited. If you have received this communication in error, please notify the sender immediately by e-mail and delete the message and any file attachments from your computer. There is no warranty that this email is error, virus or defect free. If this is a private communication it does not represent the views of Pointwest Technologies Corporation or their related entities.

--

You received this message because you are subscribed to the Google Groups "Hazelcast" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hazelcast+unsubscribe@googlegroups.com.
To post to this group, send email to haze...@googlegroups.com.
Visit this group at https://groups.google.com/group/hazelcast.

To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/CAGFUHG44FzLxqwhXrv10XP2VAM25FJ%2BbrqF%3Dw50jDfPu-kz71g%40mail.gmail.com.