RabbitMQ cluster within Red Hat OpenShift Service Mesh (Istio/Envoy)

[Tsimon Dorakh]

Good day,

I'm trying to put RabbitMQ cluster inside Red Hat Service Mesh, but without success.

What we have:

- Openshift cluster 4.11

- OpenShift Service Mesh 2.3.1

- RabbitMQ 3.8.12 cluster

When enabling sidecar istio-proxy using "sidecar.istio.io/inject: 'true'" annotation it fails with the next logs:

2023-03-20 13:14:47.589 [info] <0.293.0> Will wait for 870 milliseconds before proceeding with registration...
2023-03-20 13:14:48.463 [info] <0.44.0> Application mnesia exited with reason: stopped

2023-03-20 13:14:48.463 [error] <0.293.0>
2023-03-20 13:14:48.463 [info] <0.44.0> Application mnesia exited with reason: stopped
2023-03-20 13:14:48.463 [error] <0.293.0> BOOT FAILED
BOOT FAILED
2023-03-20 13:14:48.463 [error] <0.293.0> ===========
===========
2023-03-20 13:14:48.463 [error] <0.293.0> Exception during startup:
2023-03-20 13:14:48.463 [error] <0.293.0>
2023-03-20 13:14:48.463 [error] <0.293.0>     rabbit_boot_steps:run_step/2 line 46
2023-03-20 13:14:48.464 [error] <0.293.0>     rabbit_boot_steps:'-run_step/2-lc$^0/1-0-'/2 line 41
2023-03-20 13:14:48.464 [error] <0.293.0>     rabbit_mnesia:init/0 line 76
2023-03-20 13:14:48.464 [error] <0.293.0>     rabbit_mnesia:init_with_lock/3 line 111
2023-03-20 13:14:48.464 [error] <0.293.0>     rabbit_mnesia:run_peer_discovery_with_retries/2 line 139
2023-03-20 13:14:48.464 [error] <0.293.0>     rabbit_peer_discovery:discover_cluster_nodes/0 line 121
2023-03-20 13:14:48.464 [error] <0.293.0>     rabbit_peer_discovery_k8s:list_nodes/0 line 41
2023-03-20 13:14:48.464 [error] <0.293.0>     rabbit_peer_discovery_k8s:make_request/0 line 108
2023-03-20 13:14:48.464 [error] <0.293.0> error:{badmatch,{error,eacces}}
2023-03-20 13:14:48.465 [error] <0.293.0>
Exception during startup:

    rabbit_boot_steps:run_step/2 line 46
    rabbit_boot_steps:'-run_step/2-lc$^0/1-0-'/2 line 41
    rabbit_mnesia:init/0 line 76
    rabbit_mnesia:init_with_lock/3 line 111
    rabbit_mnesia:run_peer_discovery_with_retries/2 line 139
    rabbit_peer_discovery:discover_cluster_nodes/0 line 121
    rabbit_peer_discovery_k8s:list_nodes/0 line 41
    rabbit_peer_discovery_k8s:make_request/0 line 108
error:{badmatch,{error,eacces}}

2023-03-20 13:14:49.466 [info] <0.292.0> [{initial_call,{application_master,init,['Argument__1','Argument__2','Argument__3','Argument__4']}},{pid,<0.292.0>},{registered_name,[]},{error_info,{exit,{{badmatch,{error,eacces}},{rabbit,start,[normal,[]]}},[{application_master,init,4,[{file,"application_master.erl"},{line,138}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]}},{ancestors,[<0.291.0>]},{message_queue_len,1},{messages,[{'EXIT',<0.293.0>,normal}]},{links,[<0.291.0>,<0.44.0>]},{dictionary,[]},{trap_exit,true},{status,running},{heap_size,376},{stack_size,28},{reductions,366}], []
2023-03-20 13:14:49.466 [error] <0.292.0> CRASH REPORT Process <0.292.0> with 0 neighbours exited with reason: {{badmatch,{error,eacces}},{rabbit,start,[normal,[]]}} in application_master:init/4 line 138
2023-03-20 13:14:49.466 [info] <0.44.0> Application rabbit exited with reason: {{badmatch,{error,eacces}},{rabbit,start,[normal,[]]}}
2023-03-20 13:14:49.466 [info] <0.44.0> Application rabbit exited with reason: {{badmatch,{error,eacces}},{rabbit,start,[normal,[]]}}
{"Kernel pid terminated",application_controller,"{application_start_failure,rabbit,{{badmatch,{error,eacces}},{rabbit,start,[normal,[]]}}}"}
Kernel pid terminated (application_controller) ({application_start_failure,rabbit,{{badmatch,{error,eacces}},{rabbit,start,[normal,[]]}}})

Crash dump is being written to: /var/log/rabbitmq/erl_crash.dump...done

Could you please help to troubleshoot what might be the issue. 

Attached the full logs as well 

[Luke Bakken]

Hello,

RabbitMQ 3.8.12 is no longer supported.

error,eacces means that a permission error happened. Here is the line where that error happens:

https://github.com/rabbitmq/rabbitmq-server/blob/v3.8.12/deps/rabbitmq_peer_discovery_k8s/src/rabbit_peer_discovery_k8s.erl#L108

Apparently the user ID as which RabbitMQ is running does not have access to the token file. k8s_token_path is the setting that points to the file.

Thanks,

Luke

[Tsimon Dorakh]

Hello Luke,

Thank you very much for the hint!

Indeed rabbitmq user didn't have permissions to k8s_token_path and once I fixed securityContext RabbitMq started as expected.

It's worth noting that k8s token mount permissions differ with/without istio-proxy sidecar.. and it was a bit of surprise for me :)

Thanks,

Tsimon