RabbitMQ upgrade procedures for stand alone nodes

[thoma...@gmail.com]

Hello RabbitMQ users,

I have a question on recommended upgrade procedures for stand alone systems.

We have many running instances of RabbitMQ, some of them on very old releases like 3.8.X.

We currently deploy 3.10.25, but are working on a 3.12.X release that will be available for our internal customers in a little while.

When upgrading we have just asked for downtime and upgraded RabbitMQ + Erlang. 

https://www.rabbitmq.com/upgrade.html#single-node-upgrade

However, the matrix https://www.rabbitmq.com/upgrade.html#rabbitmq-version-upgradability shows that the normal upgrade step is just one major release.

Is it ok to go from a 3.8.X directly to 3.12.X or do we need to do this stepwise as suggested in the matrix above?

Best Regards,

Thomas

[Michal Kuratczyk]

As the matrix says - you should go through each minor release: ideally, upgrade to the latest patch release of the current minor

and then jump to the latest release of the next minor until you reach 3.12.x. Don't forget to enable all feature flags after each upgrade.

There could have been cases where skipping a version would work, but this is not something we test.

If these are dev instances or empty nodes with no data you care about, you can just delete the data folder

and upgrade directly to 3.12.

Best,

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/43b88565-739b-41be-a763-c0806d282ef3n%40googlegroups.com.

--

Michał

RabbitMQ team

[thoma...@gmail.com]

Thank you Michal,

Looks like we would need to go by the matrix and not skip minor releases in the future.

The matrix is then very important for single node installations as well.

Maybe worth considering adding a link to the matrix in the "single-node" section of the documentation?

https://www.rabbitmq.com/upgrade.html#single-node-upgrade

I have some follow on questions:

1. Are feature flags relevant for single nodes?

2. If we simplify upgrades by skipping minor releases by dropping the data folder, will we able to restore a configuration backup from 3.8.X to 3.12.X?

I am thinking about the "eiffel-rabbitmqadmin export eiffel_backup_mb.json" type of backup. 

This will restore all exchanges, federations, users etc. but will drop any pending messages in the db.

3. Are there any procedures to do a full backup of the data folder?  Can we somehow do a full restore of messages + config from an old 3.8.X release and restore that on a new 3.12.X?

4. If we lift our current systems by stepwise upgrades like 3.8.X -> 3.9.X -> 3.10.X -3.11.X -> 3.12.X is there a way to keep the node in maintenance mode until all upgrades are completed?

Kind Regards,

Thomas

[Michal Kuratczyk]

> Maybe worth considering adding a link to the matrix in the "single-node" section of the documentation?

Indeed, I've created a PR with some updates. They will show up on the website later today:

https://github.com/rabbitmq/rabbitmq-website/pull/1738

> 1. Are feature flags relevant for single nodes?

Yes.

> 2. If we simplify upgrades by skipping minor releases by dropping the data folder, will we able to restore a configuration backup from 3.8.X to 3.12.X?

> I am thinking about the "eiffel-rabbitmqadmin export eiffel_backup_mb.json" type of backup. 

> This will restore all exchanges, federations, users etc. but will drop any pending messages in the db.

Yes.

> 3. Are there any procedures to do a full backup of the data folder?  Can we somehow do a full restore of messages + config from an old 3.8.X release and restore that on a new 3.12.X?

There's https://www.rabbitmq.com/backup.html#manual-messages-backup, but I can see it's also outdated - it doesn't mention "quorum" and "streams"

folder that contain, surprise ;), quorum queues and streams messages, but you probably don't use them if you are on 3.8.

> 4. If we lift our current systems by stepwise upgrades like 3.8.X -> 3.9.X -> 3.10.X -3.11.X -> 3.12.X is there a way to keep the node in maintenance mode until all upgrades are completed?

I'm afraid not. The maintenance mode is reset/cleared on node startup.

Best,

To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/2fc2f60d-13b4-4b67-899b-398177102d45n%40googlegroups.com.

--

Michał

RabbitMQ team

[thoma...@gmail.com]

Thank you Michal,

We have a very simple implementation of RabbitMQ.

Just plain CQv1 and the federation plugin. All running in the default vhost "/".

We have no streams-, quorum- or mirrored queues. Very basic really.

For step 3, isn't the message store split so that you also have the queue index? 

And will not the broker store small messages in the index alone?

I assume that the backup has to be done when RabbitMQ is shutdown?

Best Regards,

Thomas

[Michal Kuratczyk]

The docs are clear on that "To back up messages on a node it must be first stopped."

Yes, message store (messages above 4kb) and queue indices (which include embedded messages below 4kb)

need to be backed up.

PS Please practice this process before performing in production. :)

To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/019dd00a-0cf0-4f2f-af22-6441fb026e42n%40googlegroups.com.

--

Michał

RabbitMQ team

[thoma...@gmail.com]

Thank you Michal,

I will for sure test first.. :-)

Best Regards,

Thomas