RabbitMQ v3.11: Invalid CA certificate file ..data/ca.crt

[Alex Szakaly]

Dear All, I wanted to upgrade RabbitMQ from v3.10.7 to v3.11.0 but it throws error during start up: "Cannot accept TLS distribution connection: Invalid CA certificate file ..data/ca.crt: no such file or directory"

RabbitMQ clusters are managed by RabbitMQ Cluster Kubernetes Operator, RabbitMQ configuration files can be found here (incl. inter-node-tls configuration file): https://github.com/rabbitmq/cluster-operator/tree/main/docs/examples/mtls-inter-node

The ca.crt is a Kubernetes secret which is mounted into the container. It looks like RabbitMQ uses symlink path which is a relative path. I think the previous behaviour was resolving symlink and and using it as absolute path.

$ ls -lah .

total 0

drwxrwsrwt. 3 root root 140 Sep 27 05:48 .

drwxr-xr-x. 1 root root 58 Sep 27 06:04 ..

drwxr-sr-x. 2 root root 100 Sep 27 05:48 ..2022_09_27_05_48_10.829304596

lrwxrwxrwx. 1 root root 31 Sep 27 05:48 ..data -> ..2022_09_27_05_48_10.829304596

lrwxrwxrwx. 1 root root 13 Sep 27 05:48 ca.crt -> ..data/ca.crt

lrwxrwxrwx. 1 root root 14 Sep 27 05:48 tls.crt -> ..data/tls.crt

lrwxrwxrwx. 1 root root 14 Sep 27 05:48 tls.key -> ..data/tls.key

Container log:

2022-09-27 05:48:40.459722+00:00 [warning] <0.132.0> cluster_formation.randomized_startup_delay_range.min and cluster_formation.randomized_startup_delay_range.max are deprecated

2022-09-27 05:48:45.760257+00:00 [warning] <0.158.0> TLS distribution: unexpected message: {'EXIT',<0.157.0>,shutdown}

2022-09-27 05:48:45.760257+00:00 [warning] <0.158.0>

2022-09-27 05:48:45.960614+00:00 [error] <0.183.0> Cannot accept TLS distribution connection: Invalid CA certificate file ..data/ca.crt: no such file or directory

2022-09-27 05:48:45.960614+00:00 [error] <0.183.0>

2022-09-27 05:48:45.960813+00:00 [error] <0.181.0> Cannot accept TLS distribution connection: Invalid CA certificate file ..data/ca.crt: no such file or directory

Best regards,

Alex Szakaly

[Luke Bakken]

Hi Alex,

I've passed this on to our team that works on the Kubernetes Operator.

Thanks,

Luke

[Luke Bakken]

Hi Alex,

You're running into this bug: https://github.com/erlang/otp/issues/6328

For now the solution is to use a RabbitMQ docker image that does not use Erlang 25.1

[Alex Szakaly]

Dear Luke, thank you for the quick root cause analysis. I started to watch the issue you referenced and the other related issues as well.

Best regards,

Alex