Problem using CLI tools with a remote RabbitMQ cluster in K8S

[jose.sa...@osoco.es]

Hi,

I have a RabbitMQ cluster of 1 node deployed in Kubernetes (AWS EKS) using the RabbitMQ Kubernetes Operator.

I have a script that uses rabbitmqctl and rabbitmqadmin to create users, exchanges, queues, bindings, and so on.

That script works fine when used in a local RabbitMQ installation (Dockerized or otherwise).

The RabbitMQ container defines this environment variables:

RABBITMQ_NODENAME=rab...@rabbitmqcluster-staging-server-0.rabbitmqcluster-staging-nodes.default

RABBITMQ_USE_LONGNAME=true

Its hostname is rabbitmqcluster-staging-server-0.

Its hostname (fqdn version) is rabbitmqcluster-staging-server-0.rabbitmqcluster-staging-nodes.default.svc.cluster.local

I'm using port-forwarding from the remote Pod to localhost:

> kubectl port-forward  pod/rabbitmqcluster-staging-server-0 15672:15672 4369:4369 25672:25672

I'm trying to run the script inside a local container with RabbitMQ and the CLI tools, launched with --net=host and --hostname=rabbitmqcluster-staging-server-0.

I've copied the Erlang cookie from the remote RabbitMQ container to the local container.

If I use as RABBITMQ_NODENAME rab...@rabbitmqcluster-staging-server-0.rabbitmqcluster-staging-nodes.default, while horoning RABBITMQ_USE_LONG_NAME=true, I get this error:

Error: unable to connect to node 'rab...@rabbitmqcluster-staging-server-0.rabbitmqcluster-staging-nodes.default': nodedown



DIAGNOSTICS
===========


attempted to contact: ['rab...@rabbitmqcluster-staging-server-0.rabbitmqcluster-staging-nodes.default']


rab...@rabbitmqcluster-staging-server-0.rabbitmqcluster-staging-nodes.default:
* connected to epmd (port 4369) on rabbitmqcluster-staging-server-0.rabbitmqcluster-staging-nodes.default
* epmd reports node 'rabbit' running on port 25672
* TCP connection succeeded but Erlang distribution failed


* Authentication failed (rejected by the remote node), please check the Erlang cookie


current node details:
- node name: 'rabbitm...@localhost.no-domain'
- home dir: /home/rabbitmq
- cookie hash: CNCANb4CVAWndRLrelkI9g==


And the logs print:


2020-12-09 18:08:19.221 [error] <0.27772.0> ** Connection attempt from node 'rabbitm...@localhost.no-domain' rejected. Invalid challenge reply. *

I'm unable to change the fqdn of the hostname in the local container. It prints localhost.

If I use rab...@rabbitmqcluster-staging-server-0.rabbitmqcluster-staging-nodes.default as RABBITMQ_NODENAME and remove RABBITMQ_USE_LONG_NAME, I get this other error:

Error: unable to connect to node 'rab...@rabbitmqcluster-staging-server-0.rabbitmqcluster-staging-nodes.default': nodedown

DIAGNOSTICS

===========

attempted to contact: ['rab...@rabbitmqcluster-staging-server-0.rabbitmqcluster-staging-nodes.default']

rab...@rabbitmqcluster-staging-server-0.rabbitmqcluster-staging-nodes.default:

  * connected to epmd (port 4369) on rabbitmqcluster-staging-server-0.rabbitmqcluster-staging-nodes.default

  * epmd reports node 'rabbit' running on port 25672

  * TCP connection succeeded but Erlang distribution failed

  * suggestion: hostname mismatch?

  * suggestion: is the cookie set correctly?

  * suggestion: is the Erlang distribution using TLS?

current node details:

- node name: 'rabbitmq-cli-28@localhost'

- home dir: /home/rabbitmq

- cookie hash: 7BcrUJ6T5n25PlR3VV6Oag==

The same error is reported when I use the long name in RABBITMQ_NODENAME, and set RABBITMQ_USE_LONG_NAME=true.

In the logs I see the same error as before:

** Connection attempt from node 'rabbitm...@localhost.no-domain' rejected. Invalid challenge reply. **

The command I use to check the connection is rabbitmqctl list_exchanges.

What I'm doing wrong? Is my setup feasible?

Kind regards,

Jose

[Luke Bakken]

Hi Jose,

Use a hex dump program like xxd or a checksum to ensure the Erlang cookie file is identical between your server and client nodes.

Notice in the output you provided the "cookie hash" is different, which is unexpected. Be sure you have copied the server's cookie file to /home/rabbitmq on the client node.

Thanks,

Luke

[Michal Kuratczyk]

What Luke said but also:

1. You can run the CLIs like this from your local machine:

kubectl exec foo-server-0 -- rabbitmqadmin declare user name=foo password=bar tags=baz

2. Perhaps what you really want is to import definitions?

https://github.com/rabbitmq/cluster-operator/tree/main/docs/examples/import-definitions

Best,

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/51069d64-0a97-4729-8517-44ff38ec5397n%40googlegroups.com.

--

Michał

[jose.sa...@osoco.es]

Thanks a lot Luke.

Instead of copying the contents, I copied the file.

Now rabbitmqadmin succeeds, but rabbitmqctl doesn't.

Here is my current test, after copying the .erlang.cookie file to the local container:

> rabbitmqadmin -N rab...@rabbitmqcluster-staging-server-0.rabbitmqcluster-staging-nodes.default -u '[user]' -p '[password]' -f bash list exchanges

 amq.direct amq.fanout amq.headers amq.match amq.rabbitmq.trace amq.topic 

However, I'm not able to make rabbitmqctl work. With RABBITMQ_NODENAME=rabbit@rabbitmqcluster-staging-server-0 (or RABBITMQ_NODENAME=rabbit), regardless of the value of RABBITMQ_USE_LONG_NAME, I get

Error: unable to connect to node rabbit@localhost: nodedown

DIAGNOSTICS

===========

attempted to contact: [rabbit@localhost]

rabbit@localhost:

  * connected to epmd (port 4369) on rabbitmqcluster-staging-server-0

  * epmd reports node 'rabbit' running on port 25672

  * TCP connection succeeded but Erlang distribution failed

  * Hostname mismatch: node "rab...@rabbitmqcluster-staging-server-0.rabbitmqcluster-staging-nodes.default" believes its host is different. Please ensure that hostnames resolve the same way locally and on "rab...@rabbitmqcluster-staging-server-0.rabbitmqcluster-staging-nodes.default"

current node details:

- node name: 'rabbitmq-cli-91@localhost'

- home dir: /root

- cookie hash: 7BcrUJ6T5n25PlR3VV6Oag==

If I set RABBITMQ_NODENAME as rab...@rabbitmqcluster-staging-server-0.rabbitmqcluster-staging-nodes.default, regardless of RABBITMQ_USE_LONG_NAME, I get

Error: unable to connect to node 'rab...@rabbitmqcluster-staging-server-0.rabbitmqcluster-staging-nodes.default': nodedown

DIAGNOSTICS

===========

attempted to contact: ['rab...@rabbitmqcluster-staging-server-0.rabbitmqcluster-staging-nodes.default']

rab...@rabbitmqcluster-staging-server-0.rabbitmqcluster-staging-nodes.default:

  * connected to epmd (port 4369) on rabbitmqcluster-staging-server-0.rabbitmqcluster-staging-nodes.default

  * epmd reports node 'rabbit' running on port 25672

  * TCP connection succeeded but Erlang distribution failed

  * suggestion: hostname mismatch?

  * suggestion: is the cookie set correctly?

  * suggestion: is the Erlang distribution using TLS?

current node details:

- node name: 'rabbitmq-cli-36@localhost'

- home dir: /root

- cookie hash: 7BcrUJ6T5n25PlR3VV6Oag==

What am I doing wrong?

[jose.sa...@osoco.es]

Hi Michał,

 I used your approach and it worked fine. I exported the definitions from the RabbitMQ instance in the cluster, and merged them with the definitions of a local container with my changes already applied. It worked fine.

Thank you very much for your help.