`rabbitmqctl` doesn't use the erlang cookie value that is specified in $HOME/.erlang.cookie

[chebyshev]

Refer to https://github.com/rabbitmq/rabbitmq-cli/issues/252

I've set docker container's environment variable HOME to /var/lib/rabbitmqso that the `rabbitmqctl`and broker will share the same cookie value.

the erlang cookie file's owner has been set to rabbitmq:rabbitmq and its permissions is 400.

After rabbitmq-server started, the erlang cookie value in file /var/lib/rabbitmq/.erlang.cookie was used.

broker's erlang cookie hash was ilFMuK/7ghwxZmBkF0VDLg==  according to its log file.

node           : rabbit@rabbitmq-0.rabbitmq.default.svc.cluster.local.
 home dir       : /var/lib/rabbitmq
 config file(s) : /etc/rabbitmq/rabbitmq.conf
 cookie hash    : ilFMuK/7ghwxZmBkF0VDLg==
 log(s)         : /var/log/rabbitmq/rabbit@rabbitmq-0.rabbitmq.default.svc.cluster.local..log
                : /var/log/rabbitmq/rabbit@rabbitmq-0.rabbitmq.default.svc.cluster.local._upgrade.log
 database dir   : /var/lib/rabbitmq/mnesia/rabbit@rabbitmq-0.rabbitmq.default.svc.cluster.local.

when I executed command `rabbitmqctl -l status`, it said that the cli tool used a different erlang cookie and rejected by remote node ...

DIAGNOSTICS
===========
attempted to contact: ['rab...@rabbitmq-0.rabbitmq.default.svc.cluster.local.']

rabbit@rabbitmq-0.rabbitmq.default.svc.cluster.local.:
  * connected to epmd (port 4369) on rabbitmq-0.rabbitmq.default.svc.cluster.local.
  * epmd reports node 'rabbit' uses port 25672 for inter-node and CLI tool traffic 
  * TCP connection succeeded but Erlang distribution failed 

  * Authentication failed (rejected by the remote node), please check the Erlang cookie

Current node details:
 * node name: 'rabbit...@rabbitmq-0.rabbitmq.default.svc.cluster.local.'
 * effective user's home directory: /var/lib/rabbitmq
 * Erlang cookie hash: +MX//PtiDHrEvoWfQDsuuQ==

From the output , the cli tool's cookie hash is different from that of broker.   And, the cli tool's cookie value remains the same each time `rabbitmqctl` is executed. 

Why didn't  rabbitmqctl use the same cookie value in file $HOME/.erlang.cookie that was shared by broker ?

I've to specify erlang cookie value explicitly with command line option `--erlang-cookie` to make rabbitmqctl work normally. 

Is there any possibility that broker and CLI tool 's cookie hash is calculated in a different way , say that one gets the cookie string with a line separator (`\n`) and another not ?

?????????????????????????????????????????????????????????????????????????????????????????????????????????????

[chebyshev]

According to the author's advice, I tried to run rabbitmq-server and rabbitmqctl under the same user `rabbitmq` and on the same container， the problem didn't go away .

@Michael Klishin

[Michael Klishin]

There is a doc section that lists several scenarios that cover 99% of cookie-based authentication [1].

Running rabbitmqctl under strace will reveal what path is actually used. Besides that, node name mismatch is another

common reason. Node name should be mentioned in the log file when the node boots [2].

Lastly, there may be something in the container that affects cookie path. I will not speculate what that might be since

we have no details about what image is used and we don't guess on this list if we can help it.

Except for more details returned to the connecting side (in this case, rabbitmqctl) which we contributed

and default cookie path on Windows in 20.2, nothing has changed around how the runtime loads or uses cookie files

in many years.

1. http://www.rabbitmq.com/cli.html#cli-authentication-failures

2. http://www.rabbitmq.com/logging.html

On Fri, Jun 1, 2018 at 3:08 AM, chebyshev <zha...@gmail.com> wrote:

According to the author's advice, I tried to run rabbitmq-server and rabbitmqctl under the same user `rabbitmq` and on the same container， the problem didn't go away .

@Michael Klishin

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ

[chebyshev]

I have thoroughly read related official documents and it just indicates that the CLI tool's erlang cookie is not the same as broker's.  That is just the case I encountered and other possible reason don't meet my situation. 

I verified by guess. The CLI tool actually used the same cookie value as broker, but it appended a line separator to the cookie string and then calculated its MD5 hash. 

Based on that assumption, I did a test and calculated the MD5 hash of erlang cookie string in two different way.

As for the erlang cookie string `7818a0c141d65621ba2006ea52127617`, I get its MD5 checksum by command 

echo -n '7818a0c141d65621ba2006ea52127617' | md5sum

# 8831cb23bc57ffc759517e3f5c1d9a5e  -

echo '7818a0c141d65621ba2006ea52127617' | md5sum 

# 2e544297faec097fffc487e034691f65 -

I convert the two hexadecimal hash string to the form of Base64. 

they are exactly the two cookie hash string reported by rabbitmq server log and rabbitmqctl.

server log screenshot:

rabbitmqctl's output:

a trace report generated by `strace`  when I execute `strace rabbitmqctl -l status` is attached to this post 

[Michael Klishin]

I will need more than a few minutes to go through a 107 kB strace output file

but I failed to find any places in which RabbitMQ may append any new lines characters to the

cookie file. CLI tools support alternative ways to specif the cookie [1] as of 3.7.0 and earlier

versions do even less.

Does the cookie file have a newline? None of my environments do. Have you tried restarting

the node since the cookie file was last modified?

The easiest way to track this down would be to start two Erlang shells and try to connect them

and make them authenticate with each other [2].

My guess here is that the cookie file was modified after the node was started (e.g. the trailing newline character was appended).

1. https://github.com/rabbitmq/rabbitmq-cli/search?q=cookie&unscoped_q=cookie

2. http://learnyousomeerlang.com/distribunomicon

--

You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Michael Klishin]

To verify that that's not the case I'd do the following:

 * Remove the trailing newline character from $HOME/.erlang.cookie

 * Restart the node

 * Try again

With RabbitMQ 3.7.x you can also avoid relying on the runtime to load the cookie and do something like

rabbitmqctl status --erlang-cookie `cat $HOME/.erlang.cookie`

to compare.

 

To post to this group, send email to rabbitm...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ

[chebyshev]

Thank you for you patience. Finally I've solved this problem in my case. 

[final solution]

I don't know why and how this can happen. But this problem can be avoided by leaving the environment variable `RABBITMQ_ERLANG_COOKIE` not set. 

[analysis]

I set this environment variable in my statefulset.

When the container starts up,  my entrypoint script will read  `RABBITMQ_ERLANG_COOKIE` and echo this cookie string to file `/var/lib/rabbitmq/.erlang.cookie` (without any line separator, i.e. `echo -n $RABBITMQ_ERLANG_COOKIE /var/lib/rabbitmq/.erlang.cookie`) . 

In new version (3.7.x) of RabbitMQ， the environment variable `RABBITMQ_ERLANG_COOKIE` seems to act as a cookie file. 

It turns out that rabbitmqctl will fail to authenticate due to cookie hash mismatch if `RABBITMQ_ERLANG_COOKIE` and erlang cookie file are both set.

[Michael Klishin]

RABBITMQ_ERLANG_COOKIE is one of the ways to specify the cookie in 3.7.x.

So what was the value of RABBITMQ_ERLANG_COOKIE? It can be that environment variable

values have a new line character appended to them at some point, including by one of the Elixir functions

we did not expect to do that.

I tried 3 options:

1. Rely on the file: `rabbitmqctl status`

2. Use --erlang-cookie: `rabbitmqctl status --erlang-cookie `cat $HOME/.erlang.cookie``

3. Use the environment variable: `RABBITMQ_ERLANG_COOKIE=`cat $HOME/.erlang.cookie` rabbitmqctl status`

and they all succeed for me. Now this wasn't on Kubernetes and I am using zsh in this specific case but I somewhat doubt

any of those things really matter. It is more likely than one of the operations your automation code performs unexpectedly appended

a newline character.

In any case, I'd recommend using the file, in general and specifically on Kubernetes.

HTH.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.

To post to this group, send email to rabbitmq-users@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Michael Klishin]

I will update the docs to spell out that `RABBITMQ_ERLANG_COOKIE` will be used by CLI tools

now that 3.7.x has long been shipped.

It never occurred to us that someone would use that variable as a way to generate the file from it :)

To post to this group, send email to rabbitm...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ

[rfa...@360incentives.com]

Hello,

I encountered the exact same problem on Kubernetes where the erlang cookie was stored in a Secret, which was used to populate the RABBITMQ_ERLANG_COOKIE environment variable inside the RabbitMQ Pod. The value in the .erlang-cookie file (created by the Docker container from the environment variable) had a newline character as part of its value.

The Secret was initially created manually from the command line. Almost certainly the problem was a newline introduced by using echo to base64 encode the secret values. When we created the Secret from files (i.e. kubectl create secret), the problem went away.

See this issue for details: https://github.com/kubernetes/kubernetes/issues/23404

Hope this helps the OP.

Regards,

Ray

On Thursday, 31 May 2018 21:55:33 UTC-4, chebyshev wrote:

Refer to https://github.com/rabbitmq/rabbitmq-cli/issues/252

I've set docker container's environment variable HOME to /var/lib/rabbitmqso that the `rabbitmqctl`and broker will share the same cookie value.

the erlang cookie file's owner has been set to rabbitmq:rabbitmq and its permissions is 400.

After rabbitmq-server started, the erlang cookie value in file /var/lib/rabbitmq/.erlang.cookie was used.

broker's erlang cookie hash was ilFMuK/7ghwxZmBkF0VDLg==  according to its log file.

node           : rabbit@rabbitmq-0.rabbitmq.default.svc.cluster.local.
 home dir       : /var/lib/rabbitmq
 config file(s) : /etc/rabbitmq/rabbitmq.conf
 cookie hash    : ilFMuK/7ghwxZmBkF0VDLg==
 log(s)         : /var/log/rabbitmq/rabbit@rabbitmq-0.rabbitmq.default.svc.cluster.local..log
                : /var/log/rabbitmq/rabbit@rabbitmq-0.rabbitmq.default.svc.cluster.local._upgrade.log
 database dir   : /var/lib/rabbitmq/mnesia/rabbit@rabbitmq-0.rabbitmq.default.svc.cluster.local.

when I executed command `rabbitmqctl -l status`, it said that the cli tool used a different erlang cookie and rejected by remote node ...

DIAGNOSTICS
===========
attempted to contact: ['rab...@rabbitmq-0.rabbitmq.default.svc.cluster.local.']

rabbit@rabbitmq-0.rabbitmq.default.svc.cluster.local.:
  * connected to epmd (port 4369) on rabbitmq-0.rabbitmq.default.svc.cluster.local.
  * epmd reports node 'rabbit' uses port 25672 for inter-node and CLI tool traffic 
  * TCP connection succeeded but Erlang distribution failed 

  * Authentication failed (rejected by the remote node), please check the Erlang cookie

Current node details:

 * node name: 'rabbitmqcli18@rabbitmq-0.rabbitmq.default.svc.cluster.local.'

 * effective user's home directory: /var/lib/rabbitmq
 * Erlang cookie hash: +MX//PtiDHrEvoWfQDsuuQ==