RabbitMQ Bug on rabbitmq-server rabbit_prelaunch using epmd bound to IPV6 interface

[Gabriele Santomaggio]

Can't start RabbitMQ with IPV6 interface, rabbitmq can't bind to EPMD.

There is a possible bug during the rabbit_prelaunch.

This is what's happening here [0]

Enviroment:

# ip addr
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 52:54:00:77:77:70 brd ff:ff:ff:ff:ff:ff
    inet6 fd3b:f495:9651:f04:5054:ff:fe77:7770/64 scope global

# cat /usr/lib/systemd/system/epmd.socket
...
[Socket]
ListenStream=[::1]:4369
Accept=false
..

# systemctl start epmd

# ss -tulpen | grep 4369
tcp    LISTEN     0      128      [fd3b:f495:9651:f04:5054:ff:fe77:7770]:4369               [::]:*                   users:(("epmd",pid=30365,fd=4),("systemd",pid=1,fd=28)) ino:1082557 sk:2b v6only:1 <->
tcp    LISTEN     0      128       [::1]:4369               [::]:*                   users:(("epmd",pid=30365,fd=3),("systemd",pid=1,fd=27)) ino:1082556 sk:2c v6only:1 <->

Can't start an erlang node using tcp4

# erl -sname hello
Protocol 'inet_tcp': register/listen error: econnrefused

With  inet6_tcp, it works:

# erl -proto_dist inet6_tcp -sname hello -kernel inet_dist_use_interface "{64827,62613,38481,3844,20564,255,65143,30576}"
Erlang/OTP 20 [erts-9.3.3.3] [source] [64-bit] [smp:2:2] [ds:2:2:10] [async-threads:10] [hipe] [kernel-poll:false]

Eshell V9.3.3.3  (abort with ^G)
(hello@crowbar)1>

Here the (possible) RabbitMQ bug:
RMQ can't start, this because here [1] it is not possible to pass enviroment variables, it ignores rabbitmq-env.conf file.
 

#  /usr/lib64/rabbitmq/lib/rabbitmq_server-3.6.16/sbin/rabbitmq-server
Protocol 'inet_tcp': register/listen error: econnrefused

by modifying the code manually:
RABBITMQ_DIST_PORT=$RABBITMQ_DIST_PORT \
    ${ERL_DIR}erl  -pa "$RABBITMQ_EBIN_ROOT" \
    -boot "${CLEAN_BOOT_FILE}" \
    -noinput \
    -hidden \
    -proto_dist inet6_tcp \  %%% <<--- ADDED THIS MANUALLY, IT IS NOT POSSIBLE USING rabbitmq-env.conf
    -s rabbit_prelaunch \
    ${RABBITMQ_NAME_TYPE} ${RABBITMQ_PRELAUNCH_NODENAME} \
    -extra "${RABBITMQ_NODENAME}"

Here I have a different error:

 # /usr/lib64/rabbitmq/lib/rabbitmq_server-3.6.16/sbin/rabbitmq-server
ERROR: epmd error for host localhost: address (cannot connect to host/port)

This is a possible Erlang bug,  net_adm:names(localhost) does not work even the -proto_dist is set to inet6_tcp:

# erl -proto_dist inet6_tcp -sname hello -kernel inet_dist_use_interface "{64827,62613,38481,3844,20564,255,65143,30576}"
Erlang/OTP 20 [erts-9.3.3.3] [source] [64-bit] [smp:2:2] [ds:2:2:10] [async-threads:10] [hipe] [kernel-poll:false]
Eshell V9.3.3.3  (abort with ^G)
(hello@crowbar)1> net_adm:names(localhost).
{error,address}

That in RabbitMQ is here [2]

A possible solution for the RabbitMQ bug it to use RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS variable here [1]

For Erlang side I am using 20.3, have to see if the same error is still in 21.x.

This could be also the bug [3].

0 - https://github.com/rabbitmq/rabbitmq-server/issues/116

1- https://github.com/rabbitmq/rabbitmq-server/blob/v3.6.x/scripts/rabbitmq-server#L82
2- https://github.com/rabbitmq/rabbitmq-server/blob/v3.6.x/src/rabbit_nodes.erl#L51
3- https://github.com/rabbitmq/rabbitmq-server/issues/1222
-
Gabriele Santomaggio
Cloud Developer @SUSE

[Gabriele Santomaggio]

p.s.

I tried with RabbitMQ 3.6.x, I could not try with RabbitMQ 3.7.x, but looking the code [1], i think there is the same problem:

1 - https://github.com/rabbitmq/rabbitmq-server/blob/v3.7.x/scripts/rabbitmq-server#L191

[Gabriele Santomaggio]

Ok,

Just to conlude:

RIght now it is not possible to use epmd with ipv6.

There is an open PR with it [1]


1 - https://github.com/erlang/otp/pull/602

[Luke Bakken]

Thanks for all of this valuable information Gabriele