Migration of data from a leaving node

[Fedor Tokarev]

Hi guys,

What is the right way to migrate data from a leaving node? The docs say that ClusterEvent.MemberExited may be received when a node is already down. Does it mean that a leaving node must initiate data migration itself? How a node can detect that it is about to leave? How to make sure that the leader won't shutdown a leaving node in the middle of the data migration process?

[√iktor Ҡlang]

If you get Exited it's already too late.

Compare this to a nomber of people out in the forest, communicating over walkie-talkies.

If someone goes silent, after a while the others must assume he/she is dead and move on,

at that point it's too late for that lost individual to start migrating data (who would he migrate data to?)

cheers,

√

On Wed, Oct 23, 2013 at 4:29 PM, Fedor Tokarev <ftok...@gmail.com> wrote:

Hi guys,

What is the right way to migrate data from a leaving node? The docs say that ClusterEvent.MemberExited may be received when a node is already down. Does it mean that a leaving node must initiate data migration itself? How a node can detect that it is about to leave? How to make sure that the leader won't shutdown a leaving node in the middle of the data migration process?

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://akka.io/faq/
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/groups/opt_out.

--

Cheers,

√

Viktor Klang
Director of Engineering

Typesafe

Twitter: @viktorklang

[Fedor Tokarev]

Hi Viktor, 

Yeah, when a node is died unexpectedly, data migration is not possible. But what if a node is about to be gracefully stopped? What are the ways to perform a data migration in this case?

A possible scenario: we have a cluster of N nodes. We shut down one node at a time, update its software, then bring it back to the cluster. Data should be migrated from a node before the maintenance, and then should be returned to the node after the maintenance. 

среда, 23 октября 2013 г., 18:41:44 UTC+4 пользователь √ написал:

[Akka Team]

Hi Fedor,

Akka has currently no specific support for handling the handoff—which is an application-specific concern anyway. In your case the node should do whatever is necessary to hand off its data and only then initiate the LEAVING state to finally exit.

Regards,

Roland

Akka Team

Typesafe - The software stack for applications that scale

Blog: letitcrash.com
Twitter: @akkateam

[Anders Båtstrand]

I am struggling with the same issue.

I am using Cluster Sharding and Cluster Singleton. If I have to programmatically move the actors away from the node that is leaving the cluster, how can I do it?

If I kill the actor, it would take time to get up, and it might even get up on the same node.

I can see from my logs that when a node calls cluster.leave, the other node is sending HandOverToMe-messages (1). However, I don't know how to wait for the process to finish. cluster.leave does not wait (afaik), and shuts down the cluster-system before the migration is complete.

Is this process documented somewhere? Can I send "HandOverToMe" myself?

I don't mind the GUI hanging a few seconds if a node goes down unexpected, but I am trying to avoid it during upgrades, maintenance etc.

Best regards,

Anders

1) From my log: Retry[X], sending HandOverToMe to [None]

To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+unsubscribe@googlegroups.com.

To post to this group, send email to akka...@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/groups/opt_out.

[Patrik Nordwall]

For cluster sharding I have implemented that, but need to get in some other things before I can submit that pull request.

https://github.com/akka/akka/issues/16050

Perhaps next week (no promises).

/Patrik

Akka Team

Typesafe - The software stack for applications that scale

Blog: letitcrash.com
Twitter: @akkateam

--
>>>>>>>>>> Read the docs: http://akka.io/docs/

>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html

>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.

For more options, visit https://groups.google.com/d/optout.

--

Patrik Nordwall
Typesafe -  Reactive apps on the JVM
Twitter: @patriknw

[Anders Båtstrand]

That sounds perfect! graceful handoff of Cluster Sharding is most
important for my project.

As I understand the ClusterSingleton is always using the leader as the
host of the actor. Is there a way to elect/appoint a new leader if the
leader is the one going down?

Best regards,

Anders

> You received this message because you are subscribed to a topic in the

> Google Groups "Akka User List" group.

> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/akka-user/pCZDusmt2_Q/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to

[Patrik Nordwall]

On Thu, May 7, 2015 at 2:29 PM, Anders Båtstrand <ande...@gmail.com> wrote:

That sounds perfect! graceful handoff of Cluster Sharding is most
important for my project.

As I understand the ClusterSingleton is always using the leader as the
host of the actor.

No, it is using the oldest node.

 

Is there a way to elect/appoint a new leader if the
leader is the one going down?

No, it will be the next oldest node.

[Anders Båtstrand]

Ah, sorry about that. I thought the leader was always the oldest node.