K8s Raspberry PI - RabbitMQ cluster

[Mark Olliver]

Hi,

I have setup a raspberry pi K8s cluster using MicroK8s. On top of that I have installed RabbitMq using three node clustering. As far as I can tell all mostly looks ok.

With the exception of two issues:

1, I can connect ok with port-forwarding to the management UI on 15672 but not using TLS on port 15671? I can see the server logs say it is listening on that port so it looks like it started but no errors.

2, If i exec into a pod any `rabbitmq-diagnostic` command (other than just blank) gives me no output. As if the command died.

e.g

rabbitmq@rabbitmq-0:/$ rabbitmq-diagnostics status &

[1] 1178

:system_limit

[1]+  Exit 70                 rabbitmq-diagnostics status

rabbitmq@rabbitmq-0:/$ cat /etc/rabbitmq/rabbitmq.conf

cluster_formation.peer_discovery_backend = k8s

cluster_formation.k8s.host = kubernetes.default.svc.cluster.local

cluster_formation.k8s.address_type = hostname

cluster_formation.k8s.service_name = rabbitmq-headless

cluster_formation.k8s.hostname_suffix = .rabbitmq-headless.rabbitmq.svc.cluster.local

queue_master_locator=min-masters

management.tcp.port = 15672

listeners.ssl.default = 5671

ssl_options.verify = verify_peer

ssl_options.fail_if_no_peer_cert = true

ssl_options.cacertfile = /certificates/ca.crt

ssl_options.certfile = /certificates/tls.pem

management.listener.ssl = true

management.listener.ssl_opts.cacertfile = /certificates/ca.crt

management.listener.ssl_opts.certfile   = /certificates/tls.pem

prometheus.tcp.port = 15692

prometheus.ssl.port       = 15691

prometheus.ssl.cacertfile = /certificates/ca.crt

prometheus.ssl.certfile   = /certificates/tls.pem

mqtt.listeners.ssl.default = 8883

mqtt.listeners.tcp.default = 1883

loopback_users.guest = false

total_memory_available_override_value = 1073741824

listeners.tcp.default = 5672

rabbitmq@rabbitmq-0:/$ cat /etc/rabbitmq/enabled_plugins

[rabbitmq_peer_discovery_k8s, rabbitmq_management, rabbitmq_prometheus, rabbitmq_mqtt].

Thanks for any ideas

Regards

Mark

[Luke Bakken]

Hello -

Your TLS settings do not specify private key files via the keyfile setting, which is necessary - https://www.rabbitmq.com/ssl.html#enabling-tls

Please use this document to further troubleshoot TLS - https://www.rabbitmq.com/troubleshooting-ssl.html

This search reveals some likely causes - https://www.google.com/search?q=erlang+system_limit

My guess is that your open file descriptor limit is too low. Check the output of "uname -a".

Thanks,

Luke

[Mark Olliver]

Hi,

I will check the file descriptors, it just seemed odd the UI works ok.

As for the TLS settings i followed one of the other guides that said you could put the key into the cert. I do also note the service does say it is listening on the TLS port in the logs which I would not expect if there was a cert/key issue?

Regards

Mark

[Luke Bakken]

Hello,

What is this "other guide" that says you can put the key into the cert? The official documentation is what you should follow.

Unfortunately the way TLS currently works you really don't know there is a configuration problem until a connection attempt is made.

Thanks,

Luke

On Thursday, December 3, 2020 at 11:40:40 PM UTC-8 ma...@digitalpatterns.io wrote:

Hi,