Rabbitmq mnesia database import issue

[om tiwari]

Hi,

I am facing issue with mnesia data movement from one standalone single node A RabbitMQ to another node B.

Using Rabbitmq3.8.2Erlang 22.1.8

I created a single node RabbitMQ A and there i created multiple user, virtual hosts, exchanges and queues. After that I need to move these queues and exchanges to a fresh single node RabbitMQ B. I followed the official doc (https://www.rabbitmq.com/backup.html), exported the metadata from node A and then copied to the "/var/lib/rabbitmq/mnesia"  path and tried using "rabbitmqctl rename_cluster_node " command but it did not help me .

So, I manually copied the data files with extension (.DCL / .DCD) from the node A mnesia dump to the new node B, also I copied msg_stores.

after doing this I was able to get all my exchanges and virtual host/users working on RabbitMQ B UI. But all the queues that I imported via manual copying mnesia DB dump of node A are in stopped state.

Please help me in getting all my queues in working state on this new node, also let me know that If by doing this all manually is there possibility of data loss( i.e some queues/ exchanges/user gets missing)  

PS: I cannot do the import/export of metadata from UI or command line because of some limitations. 

Using Rabbitmq3.8.2Erlang 22.1.8

[Luke Bakken]

Hello,

Please see this guide: https://www.rabbitmq.com/rabbitmqctl.8.html#rename_cluster_node

Stop node A, rename it using rabbitmqctl rename_cluster_node, and copy the mnesia data to node B. Ensure that the database directory names match node B's name. You should not have to export any definitions.

If the above process does not work, you must capture all of the commands you run and their output into a file. Attach that file as well as your RabbitMQ log files to your response. Do not paste output into your response as it will fill the screen.

Thanks,

Luke

[om tiwari]

Hi Luke,

Thanks for your quick helping response.
I followed the steps you mentioned, and I was able to migrate the
persistant mnesia db to the new node.
But I am still facing issue with messages that were on Node A in
pending state (ready) but were not consumed for multiple queues.
Since all the queues were successfully copied for node B, my
expectation was all the pending message will also persist in the
queues and will get acknowledged by the consumers attached to this new
node.
But this is not happening for me and all the messages got lost after
the data movement on new node and all queues were in idle state.

Please help me with this issue and provide me a way to move the ready
state messages of multiple queues from older node A to node B.

On 12/26/19, Luke Bakken <lba...@pivotal.io> wrote:
> Hello,
>
> Please see this guide:
> https://www.rabbitmq.com/rabbitmqctl.8.html#rename_cluster_node
>
> Stop node A, rename it using rabbitmqctl rename_cluster_node, and copy the
> mnesia data to node B. Ensure that the database directory names match node
> B's name. You should not have to export any definitions.
>
> If the above process does not work, you must capture all of the commands
> you run and their output into a file. Attach that file as well as your

> RabbitMQ log files to your response. *Do not* paste output into your

> response as it will fill the screen.
>
> Thanks,
> Luke
>
> On Thursday, December 26, 2019 at 8:49:58 AM UTC-8, om tiwari wrote:
>>
>> Hi,
>>
>> I am facing issue with mnesia data movement from one standalone single
>> node A RabbitMQ to another node B.

>> *Using Rabbitmq*3.8.2Erlang 22.1.8

>>
>> I created a single node RabbitMQ A and there i created multiple user,
>> virtual hosts, exchanges and queues. After that I need to move these
>> queues
>> and exchanges to a fresh single node RabbitMQ B. I followed the official
>> doc (https://www.rabbitmq.com/backup.html), exported the metadata from

>> node A and then copied to the *"/var/lib/rabbitmq/mnesia"* path and
>> tried using *"rabbitmqctl rename_cluster_node " *command but it did not
>> help me .
>>
>> So, I manually copied the data files with extension *(.DCL / .DCD)* from
>> the node A mnesia dump to the new node B, also I copied *msg_stores.*

>> after doing this I was able to get all my exchanges and virtual host/users
>>
>> working on RabbitMQ B UI. But all the queues that I imported via manual
>> copying mnesia DB dump of node A are in stopped state.
>>
>> Please help me in getting all my queues in working state on this new node,
>>
>> also let me know that If by doing this all manually is there possibility
>> of
>> data loss( i.e some queues/ exchanges/user gets missing)
>>

>> * PS: I cannot do the import/export of metadata from UI or command line
>> because of some limitations. *
>> *Using Rabbitmq*3.8.2Erlang 22.1.8
>>
>
> --
> You received this message because you are subscribed to the Google Groups
> "rabbitmq-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to rabbitmq-user...@googlegroups.com.
> To view this discussion on the web, visit
> https://groups.google.com/d/msgid/rabbitmq-users/16d625b5-4a1c-42a7-be2b-7b9c6922bb5b%40googlegroups.com.
>

[Luke Bakken]

Hello,

Are you certain all messages were published as "persistent"? If not, they won't be preserved across restarts.

Thanks,

Luke

[Mashrin Srivastava]

Luke,
I have a doubt here. Any message sent to a durable queue, isn't it persistent by default?

Thanks,
Manoj

[Luke Bakken]

No. You must publish the message with the "persistent" flag set to true for it to survive restarts of the broker.

The "durable" flag for a queue only affects the queue, not the messages in it.

[om tiwari]

Hi Luke,

Thanks for your suggestion, we are working on persistent message flag
true in our usecase.
Meanwhile we are trying to use the rename cluster command in our
Docker container based RabbitMQ, the issue we are facing is after
executing the "rabbitmqctl stop/shutdown" the container stops and then
we are not able to execute the next command i.e "rabbitmqctl
rename_cluster_node". After this if we restart the container again
then we are not able to execute the  "rabbitmqctl rename_cluster_node"
command as it requires "rabbitmqctl stop/shutdown" command to be
executed first.

Is there document or helpful link through which we can take help in
implementing this rabbitmqctl command in docker container of RabbitMQ.

> --
> You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.

> To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/9c330b22-fcea-4407-acf3-aa6ab66a4605%40googlegroups.com.

[Luke Bakken]

Hello,

You're going to have to change how docker starts RabbitMQ in your container. Rather than running the rabbitmq-server command, it could execute the rename_cluster_node command upon startup, and then do the necessary mnesia directory renames.

I'm assuming you're using some sort of persistent data store. You can mount that in a different container or linux instance to run the rename_cluster_node command on the mnesia data.

Thanks,

Luke