Query about Rolling Rollback from RabbitMq 3.9.6 to 3.8.17 on K8s

[Utkarsh Khare]

Hi, 

My current setup has RabbitMQ 3.8.17 , Erlang 24.0.2 on K8s. 

I want to eventually upgrade to RabbitMQ 3.9.6, Erlang 24.0.5 

My upgrades and rollbacks are done in a rolling fashion, with 1 pod going down and coming up with a new version, at a time. 

From 3.8.17, I can upgrade successfully to 3.8.21, Erlang 24.0.5. Rollback also works. 

From 3.8.21, I can upgrade successfully to 3.9.6, Erlang 24.0.5. Rollback also works. 

From 3.8.17, I can upgrade successfully to 3.9.6, Erlang 24.05. Rollback creates mismatched cookies across my pods.  

It is very much possible that this is a bug in my cookie injection logic, which I need to debug. 

My question to this group is if this is a supported rollback. 

I have seen a couple of documentation links about - 

- Upgrade from 3.8.x to 3.9.x requiring a full cluster stop? Is my reading correct? 

  https://www.rabbitmq.com/upgrade.html#rabbitmq-version-upgradability 

- Requirement of Feature Flags that were added in 3.8.x. 

  I don't see a problem with upgrades to 3.9.6, am not sure if this is indeed the case. 

  https://www.rabbitmq.com/feature-flags.html 

  

  

Thanks and Regards, 

Utkarsh 

[Utkarsh Khare]

I should add, I don't create any queues in these upgrade tests. There is no data being pushed. 

The error I see after rollback from 3.9.6 to 3.8.17 says : 

2021-10-25 15:25:07.346 [error] <0.786.0> ** Connection attempt from node '<rabbit 'at' 'node-name'>' rejected. Invalid challenge reply. **

The erlang cookies are different on the nodes post rollback. 

I've verified that manually. 

[Michal Kuratczyk]

Well, as you already know - the cookie has to be the same. It probably changes because of the way you deploy it (ConfigMap/Secret gets recreated or something like that). Without knowing how you generate it, we can't help much.

As for downgrades, from the page you linked to: "RabbitMQ does not support downgrades; it's strongly advised to back node's data directory up before upgrading."

As with any software, there is a difference between "supported" and "technically possible" - as you already know, downgrades are often possible but we don't guarantee or test them.

If you test downgrade before a specific upgrade and it works, you may be in luck. Cookie changes are unrelated though - if the cookie changes, even a node restarted with the same version won't be able to re-join the cluster.

Best,

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/e231249a-44e1-47e3-8498-a9785384ab45n%40googlegroups.com.

--

Michał

RabbitMQ team