Upgrade from RabbitMQ 3.8.0 and Erlang 22.1to RabbitMQ 3.11.5 and Erlang 25.1.2 fails

1,582 views
Skip to first unread message

Steven Ogilvie

unread,
Jan 18, 2023, 5:23:21 PM1/18/23
to rabbitm...@googlegroups.com

Hi Luke,

 

The install of RabbitMQ 3.8.0 and Erlang 22.1 seemed to run fine (we have created a MSI installer to wrap around RabbitMQ Server, Erlang, and OpenSSL light).

I then did an upgrade to RabbitMQ 3.11.5 and Erlang 25.1.2 and OpenSSL 3.0.7, it hangs and the message is:
“RabbitMQ node is not running” I verified that all 4 .erlang.cookies have the same value…

I have attached a zip with the logs and the erl_crash.dmp and etc.

 

Sigh, unfortunately this is time sensitive.

 

Thank you,

 

Steve

 

Steven Ogilvie

Software Developer, SCMT - Builds & Installers

RabbitMQ.zip

Michal Kuratczyk

unread,
Jan 18, 2023, 5:32:06 PM1/18/23
to rabbitm...@googlegroups.com
Hi,

The important message is here:
"Feature flags: `maintenance_mode_status`: required feature flag not enabled! It must be enabled before upgrading RabbitMQ."

And that's documented here:


--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/DM4PR18MB435012C84074782F0D4F04B5BEC79%40DM4PR18MB4350.namprd18.prod.outlook.com.


--
Michał
RabbitMQ team

Steven Ogilvie

unread,
Jan 18, 2023, 5:52:38 PM1/18/23
to rabbitm...@googlegroups.com

Hi Michal,

 

Happy New Year and thanks for the quick response

 

I presume then at the very beginning of the upgrade of 3.11.5 I have to run
C:\Program Files\<path to sbin>\sbin\rabbitmqctl.bat enable_feature_flag all

 

Which will be run in RabbitMQ 3.8.0 RabbitMQ, then the upgrade to 3.11.5 should be smooth?

 

Cheers,

 

Steve

 

Michal Kuratczyk

unread,
Jan 18, 2023, 5:57:14 PM1/18/23
to rabbitm...@googlegroups.com
Hi,

Yes, you have to enable them before upgrading to 3.11. Whether it'll be smooth - I can't guarantee, we only test upgrading to the next minor,
so 3.8->3.9->3.10->3.11.

Best,



--
Michał
RabbitMQ team

Steven Ogilvie

unread,
Jan 20, 2023, 2:32:13 PM1/20/23
to rabbitm...@googlegroups.com

Hi Michal,

 

So I have it enabling the feature flags for 3.8.0 before the upgrade 3.11.5

Of course it is a .MSI so it will do a major upgrade.

 

It is failing in the same area verifying the node.

 

I have attached a zip file with the logs and other info…

 

Cheers,

 

Steve

 

RabbitMQ.zip

Daniel Faust

unread,
Jan 22, 2023, 7:57:20 AM1/22/23
to rabbitmq-users
Hi Steve,

There are other feature flags added to RabbitMQ after version  3.8, so enabling all in version 3.8 won't allow you to directly upgrade to 3.11 since some will be missing and therefore assumed to be disabled.

Since version 3.11 is the first one to enforce flag compatibility, you could directly upgrade to 3.10 without changing anything to your 3.8 database, then enable all the flags there, and finally upgrade to 3.11,

Kind regards,
Daniel

Steven Ogilvie

unread,
Jan 23, 2023, 10:14:43 AM1/23/23
to rabbitm...@googlegroups.com

Isn’t there a complete list of the feature flags I need to enable, I tried All but I got an error that the node doesn’t have that flag

 

Cheers,

 

Steve

 

Steven Ogilvie

unread,
Jan 23, 2023, 3:36:58 PM1/23/23
to rabbitm...@googlegroups.com

Luke/Michal,

 

Is there anyway of knowing which feature flags to enable?
I am upgrading from 3.8.0 to 3.11.5 (and we can’t have a 2 step process, i.e.:  3.8.0 to 3.9.x or 3.10.x then to 3.11.5 as someone else has suggested)

 

These are the flags I have enabled:

var featureFlags = new List<string>

{

    EnableQuorumQueueArgument,

    EnableDropUnroutableMetricFeatureFlag,

    EnableEmptyBasicGetMetricFeatureFlag,

    EnableImplicitDefaultBindingsFeatureFlag,

    EnableVirtualHostMetadataFeatureFlag

};

 

Thank you very much!

 

Steve

RabbitMQ.zip

Michal Kuratczyk

unread,
Jan 23, 2023, 3:43:43 PM1/23/23
to rabbitm...@googlegroups.com
`rabbitmqctl list_feature_flags` to get a list, then enable all of them (see also, the first yellow box here: https://www.rabbitmq.com/feature-flags.html)

`rabbitmqctl enable_feature_flag all` was introduced in 3.8.10.

Best,



--
Michał
RabbitMQ team

Steven Ogilvie

unread,
Jan 23, 2023, 3:48:03 PM1/23/23
to rabbitm...@googlegroups.com

That is what I did in 3.8.0, and then in the upgrade to 3.11.5 BEFORE the upgrade actually starts to remove any files etc.

Do I need to enable those flags in 3.11.5 BEFORE I verify that the RabbitMQ node is running?

 

The below email about the flags I have enabled still does not help to start the RabbitMQ node…

 

Cheers,

 

Steve

 

Michal Kuratczyk

unread,
Jan 23, 2023, 4:05:04 PM1/23/23
to rabbitm...@googlegroups.com
Have you followed Daniel's suggestion? Some flags were added in 3.8.x, so if you enable all flags of 3.8.0, then you will miss some 3.8 flags (since they were added later in the 3.8 lifecycle).
So upgrade to the latest 3.8, enable all and try again.

No offence, but I'm going to stop responding here. Support for 3.8 ended half a year ago.

Best,



--
Michał
RabbitMQ team

Steven Ogilvie

unread,
Jan 23, 2023, 4:28:10 PM1/23/23
to rabbitm...@googlegroups.com

We can’t do a 2 step process for our customers, i.e. go from 3.8.x to 3.10.x then to 3.11.5…

thanks

 

Steve

 

Daniel Faust

unread,
Jan 23, 2023, 6:55:54 PM1/23/23
to rabbitmq-users
> Do I need to enable those flags in 3.11.5 BEFORE I verify that the RabbitMQ node is running?

Those flags need to be enabled through the actual use of a pre-3.11 node. You cannot enable those flags without the node running, this is something that is done on a running instance.

I did this last week, I had a old database which was on 3.8.9 (contrary to Micael's comment "`rabbitmqctl enable_feature_flag all` was introduced in 3.8.10." I think `rabbitmqctl enable_feature_flag all` was able to enable all the flags in that version, but not all the neccesary flags for 3.11 were already introduced in 3.8.9).

I was using this with Docker. So I upgraded the container to the latest 3.11 and it did quit on start, due to the disabled flags. So I started that old 3.8.9 container and `docker exec -it rabbitmq /bin/bash` into it, then I executed `rabbitmqctl enable_feature_flag all`. Then I stopped the container and started 3.11 but it would still stop because some new flags which weren't in 3.8.9 but were introduced later were still disabled. So I created a 3.10.17 container (because the docs said that 3.11 would enfocre the flags requirement, so 3.10 was still ok) and did a `rabbitmqctl enable_feature_flag all` in it, so that all the flags got enabled, then I was able to successfully run the 3.11.17 container.

From the docs:
> rabbitmqctl is the main command line tool for managing a RabbitMQ server node [...].
> It performs all actions by connecting to the target RabbitMQ node on a dedicated CLI tool communication port and authenticating using a shared secret (known as the cookie file).

If you are not using Docker, you need to uninstall 3.11 and can then directly install 3.10, then use `rabbitmqctl enable_feature_flag all` on the running node, then uninstall 3.10 and reinstall 3.11.

While doing this I did take some notes very sluggshly:

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2023-01-21 -- upgrade to 3.11
https://blog.rabbitmq.com/posts/2022/07/required-feature-flags-in-rabbitmq-3.11/

docker exec -it smanager-amqp /bin/bash
bash-5.0# rabbitmqctl list_feature_flags
Listing feature flags ...

{Flags known to 3.8.9:}
name    state
drop_unroutable_metric    disabled
empty_basic_get_metric    disabled
implicit_default_bindings enabled
maintenance_mode_status   enabled
quorum_queue              enabled
virtual_host_metadata     enabled
{End Flags known to 3.8.9}

------------------------------------------

docker exec -it  smanager-amqp-3.8.9-inactive /bin/bash
docker exec -it  smanager-amqp /bin/bash


implicit_default_bindings       enabled
maintenance_mode_status enabled
quorum_queue    enabled
virtual_host_metadata   enabled

quorum_queue
implicit_default_bindings
virtual_host_metadata
maintenance_mode_status
user_limits


rabbitmqctl enable_feature_flag user_limits

rabbitmqctl enable_feature_flag all


{Flags known to 3.10.17:}
name    state
classic_mirrored_queue_version  enabled
classic_queue_type_delivery_support     enabled
drop_unroutable_metric  enabled
empty_basic_get_metric  enabled
implicit_default_bindings       enabled
maintenance_mode_status enabled
quorum_queue    enabled
stream_queue    enabled
user_limits     enabled
virtual_host_metadata   enabled
{ End Flags known to 3.10.17  }
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

But keep in mind that in my case it was OK to have downtime and I am not using a cluster, so there was nothing to be careful about.

Hope this helps.

Daniel

Daniel Faust

unread,
Jan 23, 2023, 7:02:16 PM1/23/23
to rabbitmq-users

>  We can’t do a 2 step process for our customers, i.e. go from 3.8.x to 3.10.x then to 3.11.5…

Then you can't upgrade them to 3.11.


stog...@genetec.com schrieb am Montag, 23. Januar 2023 um 22:28:10 UTC+1:
Reply all
Reply to author
Forward
0 new messages