ClusterClient and graceful shutdown of ClusterReceptionist

[Frederic]

Hi all,

The ClusterClient is designed to connect to a single ClusterReceptionist. When a cluster node running a receptionist gracefully leaves the cluster (i.e., because the node is shutting down), the cluster clients connected to that receptionist will keep sending message to it until the failure detector actually detects a failure. Detecting the failure can take quite long, depending on the configured heartbeat-interval and acceptable-heartbeat-pause (by default, up to 15 seconds).

In the case of a graceful shutdown of the cluster receptionist, couldn't we make the cluster client switch to another receptionist much faster?

There are a couple ways we could do that, but I feel the most elegant would require to modify ClusterClient and ClusterReceptionist. Here's what I tried... I'm wondering if there's an interest in discussing/improving it for inclusion upstream, I could create an issue/PR if there's interest.

I've created a new message: ClusterReceptionist.Internal.ReceptionistShutdown.

When a receptionist detects that the node it is running on is removed from the cluster, in addition to stopping itself, it sends the ReceptionistShutdown message to all clients which interacted (using ClusterReceptionist.clientInteractions).

When a cluster client get the ReceptionistShutdown message from the receptionist it is currently connect to, it behaves as if a failure had been detected, effectively looking for another receptionist to connect to.

Should I do a PR out of this? Are there better ways?

Thanks, Fred

[Patrik Nordwall]

Sounds like an excellent suggestion and contribution is welcome.
Thanks, Patrik

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

[frederic arno]

Thanks Patrick.

What do you think would be the best way to do it, what I described in my original message or rather having the cluster client setup a death watch on the receptionist?

Fred

To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+unsubscribe@googlegroups.com.

To post to this group, send email to akka...@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

--

>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---

You received this message because you are subscribed to a topic in the Google Groups "Akka User List" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/akka-user/rjYuKPibNyQ/unsubscribe.
To unsubscribe from this group and all its topics, send an email to akka-user+unsubscribe@googlegroups.com.

[Patrik Nordwall]

As you suggested, when the receptionist is stopped it should notify “connected” clients about that.

/Patrik

lör 16 dec. 2017 kl. 11:38 skrev frederic arno <freder...@gmail.com>:

Thanks Patrick.

What do you think would be the best way to do it, what I described in my original message or rather having the cluster client setup a death watch on the receptionist?

Fred

Le 16 déc. 2017 16:38, "Patrik Nordwall" <patrik....@gmail.com> a écrit :

Sounds like an excellent suggestion and contribution is welcome.
Thanks, Patrik

lör 16 dec. 2017 kl. 09:15 skrev Frederic <freder...@gmail.com>:

Hi all,

The ClusterClient is designed to connect to a single ClusterReceptionist. When a cluster node running a receptionist gracefully leaves the cluster (i.e., because the node is shutting down), the cluster clients connected to that receptionist will keep sending message to it until the failure detector actually detects a failure. Detecting the failure can take quite long, depending on the configured heartbeat-interval and acceptable-heartbeat-pause (by default, up to 15 seconds).

In the case of a graceful shutdown of the cluster receptionist, couldn't we make the cluster client switch to another receptionist much faster?

There are a couple ways we could do that, but I feel the most elegant would require to modify ClusterClient and ClusterReceptionist. Here's what I tried... I'm wondering if there's an interest in discussing/improving it for inclusion upstream, I could create an issue/PR if there's interest.

I've created a new message: ClusterReceptionist.Internal.ReceptionistShutdown.

When a receptionist detects that the node it is running on is removed from the cluster, in addition to stopping itself, it sends the ReceptionistShutdown message to all clients which interacted (using ClusterReceptionist.clientInteractions).

When a cluster client get the ReceptionistShutdown message from the receptionist it is currently connect to, it behaves as if a failure had been detected, effectively looking for another receptionist to connect to.

Should I do a PR out of this? Are there better ways?

Thanks, Fred

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---

You received this message because you are subscribed to the Google Groups "Akka User List" group.

To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.

To post to this group, send email to akka...@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---

You received this message because you are subscribed to a topic in the Google Groups "Akka User List" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/akka-user/rjYuKPibNyQ/unsubscribe.

To unsubscribe from this group and all its topics, send an email to akka-user+...@googlegroups.com.

To post to this group, send email to akka...@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---

You received this message because you are subscribed to the Google Groups "Akka User List" group.

To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.

[frederic arno]

Please have a look at https://github.com/akka/akka/pull/24167

Thanks, Fred