rename cluster node A->B, B->A

[Giri Chintala]

Hi, 

    I'm testing 'rabbitmqctl rename_cluster_node' functionality. Successfully renamed NodeA to NodeB, But getting Error: {rename_in_progress,"Restart node under old name to roll back"} when trying to rename NodeB to NodeA.

I am not quite sure if RabbitMQ supports renaming node back and forth.

Steps to re-produce error. 

Using RabbitMQ 3.6.9

1. Start Rabbit a Server whose Hostname is NodeA
2. Stop rabbit@NodeA.
3. change Server's hostname to Node
4. rename cluster node_name to rabbit@NodeB using '# rabbitmqctl rename_cluster_node rabbit@NodeA rabbit@NodeB'
5. Stop rabbit@NodeB
6. Try renaming rabbit@NodeB back to rabbit@NodeA using '# rabbitmqctl rename_cluster_node rabbit@NodeB rabbit@NodeA', you will see following error 

Error: {rename_in_progress,"Restart node under old name to roll back"}

Regards,

Giri

[Michael Klishin]

You can't do more than one rename at a time. You can rename multiple nodes at once.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Giri Chintala]

I am not trying to rename at once. I am Trying to rename A->B, start as B and then rename it back to   A.

Get Outlook for iOS

_____________________________
From: Michael Klishin <mkli...@pivotal.io>
Sent: Sunday, April 23, 2017 12:05 PM
Subject: Re: [rabbitmq-users] rename cluster node A->B, B->A
To: <rabbitm...@googlegroups.com>

You received this message because you are subscribed to a topic in the Google Groups "rabbitmq-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/rabbitmq-users/ESxu7ee-9VQ/unsubscribe.
To unsubscribe from this group and all its topics, send an email to rabbitmq-user...@googlegroups.com.

[Michael Klishin]

Giri,

I did not imply that you are trying to do that. You MUST start a node after

it has been renamed: part of the renaming operation only happens on node boot.

I don't know why you would want to rename a node to B then back to A

but that command was created with one very specific case in mind: you must

migrate all nodes in a cluster because node naming scheme has changed,

which was at some point unavoidable in the Pivotal CF RabbitMQ tile.

So it works like this: you run it when all nodes that are being renamed are down,

then they are started. Then they can be shut down again and the whole thing can be repeated but you cannot rename without starting the node. Which is what the error

is trying to hint it: there is another rename operation already in progress.

[Giri Chintala]

Our RabbitMQ brokers are deployed on AWC EC2 in Private VPC. EC2 host name mapped from its private IP addresses. 

An EC2 instance may get randomly terminated by either AWS or by our internal chaos testing tool.

In such cases, I would like to launch new EC2 instance (most likely with different IP address as IP is choosen randomly from subnet CIDR), attach EBS volume from previously terminated EC2 instance to the newly launched instance, so that Rabbit on new EC2 starts from where previous one was stopped.

Example:- Rabbit is running on instance A (HostnameA and mnesia is stored on additional EBS volume), chaos test decided to terminate instanceA. Auto Scale Group launches instanceB( hostnameB) immediately after instanceA is terminated. our internal tooling retrieves EBS that was attached to instanceA and attaches to instanceB and rename RabbitMQ node name from instanceA to instanceB and start rabbit@hostnameB. 

So far no issues.

But problem starts when instanceB gets terminated and ASG launches new instanceC and decided to use IP-A and hostnameB (which was previously associated with instanceA). I am getting error on renaming rabbit@hostnamB to rabbit@HostnameA.

 

Get Outlook for iOS

_____________________________
From: Michael Klishin <mkli...@pivotal.io>

Sent: Sunday, April 23, 2017 3:02 PM

[Michael Klishin]

`rename_cluster_nodes` creates temporary files among other things that are deleted on successful boot.

If they exist, another rename will be refused. If the node starts under its old name, they are deleted without applying IIRC.

On Sun, Apr 23, 2017 at 11:42 PM, Giri Chintala <chintala....@gmail.com> wrote:

Our RabbitMQ brokers are deployed on AWC EC2 in Private VPC. EC2 host name mapped from its private IP addresses. 

An EC2 instance may get randomly terminated by either AWS or by our internal chaos testing tool.

In such cases, I would like to launch new EC2 instance (most likely with different IP address as IP is choosen randomly from subnet CIDR), attach EBS volume from previously terminated EC2 instance to the newly launched instance, so that Rabbit on new EC2 starts from where previous one was stopped.

Example:- Rabbit is running on instance A (HostnameA and mnesia is stored on additional EBS volume), chaos test decided to terminate instanceA. Auto Scale Group launches instanceB( hostnameB) immediately after instanceA is terminated. our internal tooling retrieves EBS that was attached to instanceA and attaches to instanceB and rename RabbitMQ node name from instanceA to instanceB and start rabbit@hostnameB. 

So far no issues.

But problem starts when instanceB gets terminated and ASG launches new instanceC and decided to use IP-A and hostnameB (which was previously associated with instanceA). I am getting error on renaming rabbit@hostnamB to rabbit@HostnameA.

 

Get Outlook for iOS

_____________________________
From: Michael Klishin <mkli...@pivotal.io>
Sent: Sunday, April 23, 2017 3:02 PM

Subject: Re: [rabbitmq-users] rename cluster node A->B, B->A

To: <rabbitmq-users@googlegroups.com>

Giri,

I did not imply that you are trying to do that. You MUST start a node after

it has been renamed: part of the renaming operation only happens on node boot.

I don't know why you would want to rename a node to B then back to A

but that command was created with one very specific case in mind: you must

migrate all nodes in a cluster because node naming scheme has changed,

which was at some point unavoidable in the Pivotal CF RabbitMQ tile.

So it works like this: you run it when all nodes that are being renamed are down,

then they are started. Then they can be shut down again and the whole thing can be repeated but you cannot rename without starting the node. Which is what the error

is trying to hint it: there is another rename operation already in progress.

On 23 Apr 2017, at 19:13, Giri Chintala <chintala....@gmail.com> wrote:

I am not trying to rename at once. I am Trying to rename A->B, start as B and then rename it back to   A.

Get Outlook for iOS

_____________________________
From: Michael Klishin <mkli...@pivotal.io>
Sent: Sunday, April 23, 2017 12:05 PM
Subject: Re: [rabbitmq-users] rename cluster node A->B, B->A

To: <rabbitmq-users@googlegroups.com>

You can't do more than one rename at a time. You can rename multiple nodes at once.

On 23 Apr 2017, at 13:10, Giri Chintala <chintala....@gmail.com> wrote:

Hi, 

    I'm testing 'rabbitmqctl rename_cluster_node' functionality. Successfully renamed NodeA to NodeB, But getting Error: {rename_in_progress,"Restart node under old name to roll back"} when trying to rename NodeB to NodeA.

I am not quite sure if RabbitMQ supports renaming node back and forth.

Steps to re-produce error. 

Using RabbitMQ 3.6.9

1. Start Rabbit a Server whose Hostname is NodeA
2. Stop rabbit@NodeA.
3. change Server's hostname to Node
4. rename cluster node_name to rabbit@NodeB using '# rabbitmqctl rename_cluster_node rabbit@NodeA rabbit@NodeB'
5. Stop rabbit@NodeB
6. Try renaming rabbit@NodeB back to rabbit@NodeA using '# rabbitmqctl rename_cluster_node rabbit@NodeB rabbit@NodeA', you will see following error 

Error: {rename_in_progress,"Restart node under old name to roll back"}

Regards,

Giri

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "rabbitmq-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/rabbitmq-users/ESxu7ee-9VQ/unsubscribe.

To unsubscribe from this group and all its topics, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "rabbitmq-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/rabbitmq-users/ESxu7ee-9VQ/unsubscribe.

To unsubscribe from this group and all its topics, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ