Cannot use "cluster_formation.k8s.address_type=ip"

[Marcelo da Cruz Pinto]

I'm trying to use the rabbit_peer_discovery_k8s plugin for auto clustering with RabbitMQ 3.7.8, and Kubernetes 1.2. The hostname-based discovery option cannot be used with this version of Kubernetes, but the IP addresses should work just fine. The problem is that when I set "cluster_formation.k8s.address_type=ip", the plugin can talk to Kubernetes and retrieve the ip addresses associated to the k8s service, but then it complains about the IP address not being legal (not that in this particular case, this IP address is the one from the same node that is starting, but the same happens in any other node):

2018-11-09 20:13:04.639 [debug] <0.222.0> GET https://kubernetes.default.svc.skydns.local:443/api/v1/namespaces/tc02runtime/endpoints/rmq
2018-11-09 20:13:04.663 [debug] <0.222.0> Response: {ok,{{"HTTP/1.1",200,"OK"},[{"date","Fri, 09 Nov 2018 20:13:04 GMT"},{"content-length","665"},{"content-type","application/json"}],"{\"kind\":\"Endpoints\",\"apiVersion\":\"v1\",\"metadata\":{\"name\":\"rmq\",\"namespace\":\"tc02runtime\",\"selfLink\":\"/api/v1/namespaces/tc02runtime/endpoints/rmq\",\"uid\":\"2994ec0d-e45b-11e8-85ef-06436deee198\",\"resourceVersion\":\"50923715\",\"creationTimestamp\":\"2018-11-09T20:08:02Z\",\"labels\":{\"k8s-app\":\"rmq\",\"kubernetes.io/cluster-service\":\"true\"}},\"subsets\":[{\"addresses\":[{\"ip\":\"10.244.33.3\",\"targetRef\":{\"kind\":\"Pod\",\"namespace\":\"tc02runtime\",\"name\":\"celery-runtime-mqueue--0-0-1--rabbitmq-n352u\",\"uid\":\"29b3fe52-e45b-11e8-85ef-06436deee198\",\"resourceVersion\":\"50923714\"}}],\"ports\":[{\"name\":\"api\",\"port\":15672,\"protocol\":\"TCP\"},{\"name\":\"service\",\"port\":5672,\"protocol\":\"TCP\"}]}]}\n"}}
2018-11-09 20:13:04.663 [info] <0.222.0> All discovered existing cluster peers: rabbit@10.244.33.3
2018-11-09 20:13:04.663 [info] <0.222.0> Peer nodes we can cluster with: rabbit@10.244.33.3
2018-11-09 20:13:04.664 [warning] <0.222.0> Could not auto-cluster with node rabbit@10.244.33.3: {badrpc,nodedown}
2018-11-09 20:13:04.664 [error] <0.243.0> ** System NOT running to use fully qualified hostnames **
** Hostname 10.244.33.3 is illegal **
2018-11-09 20:13:04.664 [warning] <0.222.0> Could not successfully contact any node of: rabbit@10.244.33.3 (as in Erlang distribution). Starting as a blank standalone node...

Again, in this case 10.244.33.3 is the same node that is generating this log, so I guess the plugin is not realizing that it should not try to cluster with itself (the node fails to start properly). I've tried the setting for using FQDN's, but that doesn't work either. What's the correct configuration in order to be able to use IP-based discovery? BTW, when using a newer version of Kubernetes (with support for StatefulSets and headless services) I can use hostname-based discovery and it works just fine, but in this particular case I need to use ip-based discovery.

[Marcelo da Cruz Pinto]

I found the solution to this issue, the following environment variables have to be set:

• "RABBITMQ_USE_LONGNAME" has to be set to "true"
• "RABBITMQ_NODENAME" (or "NODENAME") has to be set to "rabbitmq@$IP_ADDRESS", where IP_ADDRESS is the node's IP address.

These variables have to be set not only for starting the rabbitmq server, but also when using rabbitmqctl. 

[Michael Klishin]

We discourage environment variable use.

The configuration key you are looking for is cluster_formation.k8s.host [1][2].

We are not aware of any reasons why "the hostname-based discovery option cannot be used with this version of Kubernetes" so please

provide evidence to your claims.

As [1] explains using hostnames implies using stateful sets which is the only supported deployment scenario anyway.

1. http://www.rabbitmq.com/cluster-formation.html#peer-discovery-k8s

2. https://github.com/rabbitmq/rabbitmq-peer-discovery-k8s/blob/master/priv/schema/rabbitmq_peer_discovery_k8s.schema#L3

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ