Rabbitmq containers throwing error "Bad characters in cookie"

[Edmore Tshuma]

I am trying to setup a Rabbitmq cluster and when the containers start they are failing with error [error] CRASH REPORT Process <0.200.0> with 0 neighbours crashed with reason: "Bad characters in cookie" in auth:init_no_setcookie/0 line 313. This suggests that the erlang cookie value passed in is not valid :

kubectl -n demos get pods 

NAME READY STATUS RESTARTS AGE 

rabbitmq-0 0/1 CrashLoopBackOff 9 25m 

rabbitmq-1 0/1 CrashLoopBackOff 9 24m

 rabbitmq-2 0/1 CrashLoopBackOff 9 23m

And when I query the logs for one of the pods :

kubectl -n demos logs -p rabbitmq-0 --previous

I get as part of the output :

warning: /var/lib/rabbitmq/.erlang.cookie contents do not match RABBITMQ_ERLANG_COOKIE

WARNING: '/var/lib/rabbitmq/.erlang.cookie' was populated from '$RABBITMQ_ERLANG_COOKIE', which will no longer happen in 3.9 and later! (https://github.com/docker-library/rabbitmq/pull/424)
Configuring logger redirection
06:39:57.022 [error]

06:39:57.019 [error] Bad characters in cookie
06:39:57.019 [error] Supervisor net_sup had child auth started with auth:start_link() at undefined exit with reason "Bad characters in cookie" in auth:init_no_setcookie/0 line 313 in context start_error
06:39:57.019 [error] CRASH REPORT Process <0.200.0> with 0 neighbours crashed with reason: "Bad characters in cookie" in auth:init_no_setcookie/0 line 313
06:39:57.030 [error] BOOT FAILED
BOOT FAILED
06:39:57.030 [error] ===========
===========
06:39:57.030 [error] Exception during startup:
Exception during startup:

06:39:57.030 [error]
06:39:57.030 [error]     supervisor:children_map/4 line 1250
    supervisor:children_map/4 line 1250
    supervisor:'-start_children/2-fun-0-'/3 line 384

....

....

....

This is how I am generating the cookie in kubectl :

kubectl create secret generic rabbitmqs \ --from-literal= RABBITMQ_ERLANG_COOKIE =$(dd if=/dev/urandom bs=30 count=1 | base64)

And in the statefulset  I have:

...

...

- name: RABBITMQ_ERLANG_COOKIE

          valueFrom:

            secretKeyRef:

              name: rabbitmqs

              key: RABBITMQ_ERLANG_COOKIE

.....

.....

I have tried looking at the solutions here but I am not really sure how to fix this. On the link there is a suggestion to set the erlang cookie and mirror policy in the config map but Im not sure either how to do this (or if thats the fix).

Another suggestion I found on the internet was to "Check the node home dir and for each node, modify a file called .erlang.cookie and have the file contain the same value e.g. echo mycookie > $NODE_HOME_DIR/.erlang.cookie for each node"

I have tried to exec into 1 of the pods with kubectl exec -it -n labs rabbitmq-0 -- sh 

but I get error:

Defaulted container "rabbitmq" out of: rabbitmq, config (init)
error: unable to upgrade connection: container not found ("rabbitmq")

I am literally at a loss how to solve this exactly. Any pointers appreciated.

[Michal Kuratczyk]

Hi,

1. There is a Kubernetes Operator - you can just use it and not worry about such details: https://www.rabbitmq.com/kubernetes/operator/quickstart-operator.html

2. I guess the cookie is too long or you get some additional output from dd or something. Just use a bunch of random characters and you should be good.

Best,

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/07587234-5574-4a51-8cd5-62906db7cda1n%40googlegroups.com.

--

Michał

RabbitMQ team