Erlang error when running rename_cluster_node

[Anton Aleksandrov]

To whom it may concern,

We are running rabbitmq cluster(3 nodes RabbitMQ 3.8.9 Erlang 23.2.2) in ECS(Docker containers). Our goal is to rename on of the nodes without host restart.

We are using rabbitmqctl rename_cluster_node as indicated in documentation:

1. rabbitmqctl stop_app

2. rabbitmqctl rename_cluster_node rabbit@REDACTED rab...@REDACTED.eu-west-1.compute.internal -l

3. rabbitmqctl start_app

However, on the second step(rename_cluster_node command), we are encountering an error. Please see logs:

rabbitmqctl rename_cluster_node rabbit@REDACTED rab...@REDACTED.eu-west-1.compute.internal -l

Renaming cluster nodes:
 rabbit@REDACTED -> rab...@REDACTED.eu-west-1.compute.internal

  * Impersonating node: rabbit@REDACTED...Error:
{:rename_failed, {:badmatch, {:error, {{:shutdown, {:failed_to_start_child, :net_kernel, {:EXIT, :nodistribution}}}, {:child, :undefined, :net_sup_dynamic, {:erl_distribution, :start_link, [[:"rabbit@REDACTED", :shortnames], false, :net_sup_dynamic]}, :permanent, 1000, :supervisor, [:erl_distribution]}}}}}

While similar issues are related to networking, I have checked it as well:
epmd -names

epmd: up and running on port 4369 with data:
name rabbit at port 25672

If I try start rabbitmq using start_app, node starts with an old name as expected. It's  available almost immediately.

Worth noting, there's a set of variables exported such as:

RABBITMQ_FEATURE_FLAGS_FILE=/var/lib/rabbitmq/mnesia/rabbit@REDACTED-feature_flags

RABBITMQ_MNESIA_DIR=/var/lib/rabbitmq/mnesia/rabbit@REDACTED