Dynamic EPMD port when using rabbitmq-peer-discovery-consul

[Andy Davies]

Hi,

I am trying to setup a rabbitmq cluster (for a demo) inside Nomad, and am using the Consul peer discovery mechanism to do so.

I have a working rabbitmq container, which when used manually (e.g. on one host, start a consul container, 3 rmq containers) works fine, but when I try it as a job inside Nomad, the cluster won't form.

As far as I can work out, it is becase the EPMD port (4369) gets mapped from the container to some arbitrary port on the host, and this doesn't seem to get broadcast in the Consul registrations.

Setting the CONSUL_SVC_PORT environment variable fixes the service in Consul to have the right (generated) port mapping, but I can't find any kind of setting to do something similar for the EPMD port, and setting ERL_EPMD_PORT seemed to have no discernable effect on it either.

I might of course be completely wrong on this assumption, but any help would be great.

Thanks.

My entire rabbit.nomad file is as follows:

job "rabbit" {

  datacenters = ["dc1"]
  type = "service"

  group "cluster" {
    count = 3

    task "rabbit" {
      driver = "docker"

      config {
        image = "piomin/rabbitmq:1.0"
        hostname = "rabbit${NOMAD_ALLOC_INDEX}"
        port_map {
          amqp = 5672
          ui = 15672
          clustering = 4369
        }
      }

      env {
        RABBITMQ_ERLANG_COOKIE = "rabbitmq"
        CONSUL_HOST = "${attr.unique.network.ip-address}"
        CONSUL_SVC_PORT = "${NOMAD_HOST_PORT_amqp}"
      }

      resources {
        network {
          port "amqp" {}
          port "ui" {}
          port "clustering" {}
        }
      }
    }
  }
}

[Luke Bakken]

Hi Andy,

ERL_EPMD_PORT should work as of RabbitMQ 3.7.9 -

https://github.com/rabbitmq/rabbitmq-server/issues/1709

Thanks,

Luke

[Andy Davies]

Hi Luke,

After I built the alpine docker container locally (as the docker repo only has 3.7.8 available), and tested it out, it still seems to fail.  I have managed to make a repro that doesn't involve Nomad, so that might be easier to test with what I am trying to do:

docker run -d --name consul -p 8500:8500 consul

wait 10

docker run -d --rm --name rabbit1 -h rabbit1 \
    -p 30000:5672 -p 30010:15672 -p 30020:4369 \
    -e CONSUL_HOST='192.168.141.168' \
    -e RABBITMQ_ERLANG_COOKIE='rabbitmq' \
    -e ERL_EPMD_PORT=30020 \
    rabbit:local-consul

docker run -d --rm --name rabbit2 -h rabbit2 \
    -p 30001:5672 -p 30011:15672 -p 30021:4369 \
    -e CONSUL_HOST='192.168.141.168' \
    -e RABBITMQ_ERLANG_COOKIE='rabbitmq' \
    -e ERL_EPMD_PORT=30021 \
    rabbit:local-consul

docker run -d --rm --name rabbit3 -h rabbit3 \
    -p 30002:5672 -p 30012:15672 -p 30022:4369 \
    -e CONSUL_HOST='192.168.141.168' \
    -e RABBITMQ_ERLANG_COOKIE='rabbitmq' \
    -e ERL_EPMD_PORT=30022 \
    rabbit:local-consul

I've also tried changing the port mapping for EPMD to map to the same value as ERL_EPMD_PORT e.g. -p 30022:30022 -e ERL_EPMD_PORT=30022, which also didn't help.

I guess for my demo, I can use a Nomad cluster with 3 nodes so that I can hardcode the port mappings to their default values, but I kinda wish this worked also.

Thanks again,

[Luke Bakken]

Hi Andy,

Can you use netstat or some other tool to see what port epmd is actually listening on?

Thanks -

Luke

[Andy Davies]

Hi Luke,

ERL_EPMD_PORT definitely works (rabbit version 3.7.9) as this is the output of netstat (for the container with it set to 30020):

    tcp 0 0 localhost:41124 localhost:30020 ESTABLISHED
    tcp 0 0 localhost:30020 localhost:41124 ESTABLISHED

So, if you set ERL_EPMD_PORT on one node to 111111, and ERL_EPMD_PORT on the other node to 22222 (etc.), how does each node know which port to talk to on the other nodes? or does each node assume all other nodes are listenening on the same port that it is?

Andy

[Andy Davies]

A bit more investigation shows this:

If I run all 3 containers using the same value for ERL_EPMD_PORT, then clustering works, no matter what port is used.

If I run all 3 containers using different values for ERL_EPMD_PORT (e.g. 11111, 22222, 33333) then clustering does not work.

[Luke Bakken]

Hey Andy,

You beat me to the answer. All nodes must be using the same ERL_EPMD_PORT value in order for clustering to work. When an Erlang node needs to communicate with a node on another server, it first connects to epmd on that server and then asks what distribution port is in use for a particular node name, then it opens a connection to that distribution port.

99.99% of the time there's no need to change from the default value of 4369.

The gory details can be found here: http://erlang.org/doc/man/epmd.html

Thanks,

Luke

[Andy Davies]

Hi Luke,

Thanks for all your help/time.  I have managed to make it work eventually (after hitting a few other problems).  After getting the container running on 3 machines (vagrant vms in this case), it still was not clustering, but eventually I figured out it was down to the host name not being dns resolvable (the host of the machine is resolvable, but the name I was giving to the host within the container was not).  This was the resulting command:

docker run --rm -d --name rabbit \

-h $(cat /etc/hostname) \

-p 5672:5672 -p 15672:15672 -p 25672:25672 -p 4369:4369 \

-e CONSUL_HOST="$(ip route get 1 | awk '{print $NF;exit}')" \

-e RABBITMQ_ERLANG_COOKIE='rabbitmq' \

piomin/rabbitmq:1.0

For anyone who finds this looking for Nomad help, you can set the hostname property in the config to this and it will have the same effect:

hostname = "${attr.unique.hostname}"

Thanks again!