On windows machine I am getting either previous upgrade is in progress or has failed during an upgrade

[Sri Reddy]

Hi,

I am trying to test the erlang/rabbitmq upgrade as we are going to do it in production. I am testing this on my local windows 10 machine, locally I have ERLANG OTP 17 (6.3) and RabbitMQ Server 3.4.3 installed. It was working for last 5 years. Today, I have upgraded both to the latest versions.

ERLANG OTP 22 (10.5)

RabbitMQ Server 3.8.3

- I uninstalled the older version of ERLANG before installing the new one (new install prompted me to remove the old one).

- I installed new RabbitMQ Server, the install prompted me saying it found the older version and will upgrade. I accepted it.

The upgrade went well, I updated the environment variables to point to the right paths for the newer versions. 

Also ran the following

- Run SET HOMEDRIVE=C:

- Run the following command to enable the plugin rabbitmq-plugins.bat enable rabbitmq_management

- rabbitmq-service.bat stop  

- rabbitmq-service.bat install  

- rabbitmq-service.bat start  

But, the RabbitMQ windows service starts and then stops automatically. I checked the logs and see the following errors:

2020-05-07 18:04:11.132 [info] <0.9.0> Feature flags: list of feature flags found:

2020-05-07 18:04:11.132 [info] <0.9.0> Feature flags:   [ ] drop_unroutable_metric

2020-05-07 18:04:11.132 [info] <0.9.0> Feature flags:   [ ] empty_basic_get_metric

2020-05-07 18:04:11.132 [info] <0.9.0> Feature flags:   [ ] implicit_default_bindings

2020-05-07 18:04:11.132 [info] <0.9.0> Feature flags:   [ ] quorum_queue

2020-05-07 18:04:11.132 [info] <0.9.0> Feature flags:   [ ] virtual_host_metadata

2020-05-07 18:04:11.132 [info] <0.9.0> Feature flags: feature flag states written to disk: yes

2020-05-07 18:04:11.221 [error] <0.9.0> Found lock file at c:/Program Files (x86)/RabbitMQ Server/db/rabbit@MyLocal-mnesia/schema_upgrade_lock.

            Either previous upgrade is in progress or has failed.

            Database backup path: c:/Program Files (x86)/RabbitMQ Server/db/rabbit@MyLocal-mnesia-upgrade-backup

2020-05-07 18:04:11.221 [error] <0.9.0> 

Error description:

    init:do_boot/3 line 817

    init:start_em/1 line 1109

    rabbit:start_it/1 line 484

    rabbit:'-boot/0-fun-0-'/0 line 333

    rabbit_upgrade:maybe_upgrade_mnesia/0 line 142

    rabbit_upgrade:ensure_backup_taken/0 line 98

throw:{error,previous_upgrade_failed}

I replace the content of "C:\Program Files (x86)\RabbitMQ Server\db\rabbit@MYMAC-LT-mnesia" with the content of previous (working) version, still it won't work. I am trying to see if my messages that are sitting in the queues can be moved to the upgraded rabbitMQ or not.

What is wrong, please help. This has to go in production this weekend.

[Luke Bakken]

Hello,

Did you read the documentation with regard to upgrades?

https://www.rabbitmq.com/upgrade.html#rabbitmq-version-upgradability

You can only upgrade from 3.4.3 to 3.6.16, then 3.8.3.

I strongly suggest standing up a second cluster with the newest RabbitMQ and Erlang, and using the blue-green strategy to upgrade:

https://www.rabbitmq.com/blue-green-upgrade.html

If you absolutely must upgrade in-place, let us know.

Thanks,

Luke

[Sri Reddy]

We will be upgrading at off hours when there is no real-time activities in the queues. We are worried about the existing messages that did process yet. We want to back that up and after upgrade restore it.

Thanks for pointing out to the upgrade routes, which I completely ignored.

If we upgrade from 3.4.3 to 3.6.16 first, we upgrade erlang to supported version and then to 3.8.3, right? And, then the the exiting messages would migrate automatically? Will this solution works or I still better off blue-green deployment?

I never did cluster of any sort. Can I add cluster to an existing single instance/node and how? Is there a simpler step by step guide for window users?

I was looking for the simpler solution and one more piece of information that we are not expecting lot of messages (couple of thousands at most). But, we have to do this for lot of clients.

Thanks in advance.

[Luke Bakken]

Hello,

Blue-green will probably be the simplest method, because you won't have to upgrade to intermediate versions of Erlang and RabbitMQ.

If you upgrade to 3.6.x first, please note that it requires an older version of Erlang - https://www.rabbitmq.com/which-erlang.html#eol-series

So, you will be upgrading Erlang and RabbitMQ twice. When you upgrade Erlang, please uninstall the previous version. Do not install the new one because it will be side-by-side with the older version. You want to only have one version of Erlang installed at a time.

Thanks,

Luke

[Sri Reddy]

Thank you Luke. Today I tried upgrading it from 3.4.x to 3.6.x and then to 3.8.x along with the supported Erlang version. It worked as expected. I know, it is little longer approach than Blue-green, but i didn't find a great example post to do it yet. I was not sure how do you do that, so i opted for the first approach.

If you can point me to an article that explains step by step on how to use blue-green in windows, i can try that and push it to production.

[Luke Bakken]

Hi Sri,

The documentation for blue-green should apply to Windows as well. Do you have specific questions about it?

[Sri Reddy]

Yes, the documentation doesn't even explain how to setup another instance of ERLANG and RabbitMQ Server. How to make two instances (Green and Blue) work together? I have very limited knowledge about RabbitMQ. 

Here's what i have done so far:

- I have ERLANG OTP 17 running on Windows 10.

- ERLANG_HOME is pointing to C:\Program Files\erl6.3 in Environment Variables

- RabbitMQ Server running at port 5672

- 

- Have 100s of queues with messages in many queues

- I installed ERLANG OTP 20.3

- Added a new ERLANG_HOME_NEW = C:\Program Files\erl10.7 in Environment Variables

- Downloaded the zip file for RabbitMQ Server 3.8.3

- Copied the folder rabbitmq_server-3.8.3 to C:\Program Files (x86)\RabbitMQ Server

Also copied bunch of environment variables, here is the screen shot of all the new variables with _NEW to distinguish it with the existing ones:

Then updated all the .bat files in C:\Program Files (x86)\RabbitMQ Server\rabbitmq_server-3.8.3\sbin to reflect above variable names. 

When i run the rabbitmq-service.bat install, it says that the `RabbitMQ service is already present - only updating service parameters` and `ERROR: node with name "rabbit" already running on "MYMAC"`

So, can you help me on how to setup the second instance of rabbitmq  (GREEN cluster). What all environemtn variables i have to override and what .bat files to update. I am stuck in first step itself, then i have to make sure i can setup the cluster and then federated queues.

[Luke Bakken]

Hi Sri,

Blue-Green requires at least two separate servers. You might be able to get RabbitMQ running twice on the same machine but we don't support it.

https://www.rabbitmq.com/blue-green-upgrade.html

Thanks,

Luke

[Sri Reddy]

Ah ok. I think in that case, we are better off with upgrading to 3.6.16 and then to 3.8.3.