Node name changed after 3.8.3 -> 3.8.4 upgrade

230 views
Skip to first unread message

Drew Mazurek

unread,
May 26, 2020, 9:44:03 AM5/26/20
to rabbitmq-users
I just upgraded one node in my RabbitMQ cluster from 3.8.3 to 3.8.4 from the Ubuntu 16.04 PPA, and after restarting it came up with a different node name. The node name was originally set using the NODENAME var in /etc/rabbitmq/rabbitmq-env.conf, but that seems to be ignored now and it's creating a name based on the machine's hostname, which doesn't work for my AWS cluster. Where should the node name now be configured? Is there anything else I should know about the 3.8.3 to 3.8.4 update that's not in the release notes?

Thanks,
Drew

Michael Klishin

unread,
May 26, 2020, 11:35:02 AM5/26/20
to rabbitm...@googlegroups.com

Can you share a little bit more info about your environment? What OS is used, how was RabbitMQ provisioned, what the contents of rabbitmq-env.conf are? There were no changes to what environment variables are used in 3.8.4. There were changes in what nodes do early on boot

so it could potentially have an effect but we haven’t noticed anything after months of

first making the changes in master.

 

So we need as much information as possible to reproduce.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/efead52f-730b-4c44-b0e1-b9197d538c11%40googlegroups.com.

Michael Klishin

unread,
May 26, 2020, 11:42:21 AM5/26/20
to rabbitm...@googlegroups.com

I have tried the following rabbitmq-env.conf file:

 

NODENAME=hare@warp10

 

And started a local node like so:

 

RABBITMQ_CONF_ENV_FILE=./rabbitmq-env.conf rabbitmq-server

 

and the node did pick up the node name on boot.

 

So my hypothesis is that the env config file location is computed differently in your environment.

You can verify this using a process tree inspector or, in 3.8.4, a new rabbitmq-diagnostics command:

 

rabbitmq-diagnostics os_env -n {node name}

 

which will print all environment variables relevant to RabbitMQ, as observed by the node.

Drew Mazurek

unread,
May 26, 2020, 11:54:16 AM5/26/20
to rabbitmq-users
Sure, it's RabbitMQ 3.8.4 on Erlang 22.3.4.1, and I assume this issue may be more related to the Ubuntu packaging than RabbitMQ itself. I'm using the packagecloud.io repository --


/etc/rabbitmq/rabbitmq.conf only contains:

cluster_formation.aws.region = us-east-1
cluster_formation.aws.use_autoscaling_group = true
cluster_formation.aws.use_private_ip = true
cluster_formation.peer_discovery_backend = rabbit_peer_discovery_aws
heartbeat = 30
listeners.tcp.default = 5672
log.file.level = debug

and /etc/rabbitmq/rabbitmq-env.conf contains:

USE_LONGNAME=true

One thing I noticed is that the command line arguments changed between 3.8.3-1 and 3.8.4-1. 3.8.3-1 started up with:

/usr/lib/erlang/erts-10.7.2.1/bin/beam.smp -W w -A 64 -MBas ageffcbf -MHas ageffcbf -MBlmbcs 512 -MHlmbcs 512 -MMmcs 30 -P 1048576 -t 5000000 -stbt db -zdbbl 128000 -K true -- -root /usr/lib/erlang -progname erl -- -home /var/lib/rabbitmq -- -pa /usr/lib/rabbitmq/lib/rabbitmq_server-3.8.3/ebin  -noshell -noinput -s rabbit boot -name rab...@172.16.42.14 -boot start_sasl -conf /etc/rabbitmq/rabbitmq.conf -conf_dir /var/lib/rabbitmq/config -conf_script_dir /usr/lib/rabbitmq/bin -conf_schema_dir /var/lib/rabbitmq/schema -conf_advanced /etc/rabbitmq/advanced.config -kernel inet_default_connect_options [{nodelay,true}] -sasl errlog_type error -sasl sasl_error_logger false -rabbit lager_log_root "/var/log/rabbitmq" -rabbit lager_default_file "/var/log/rabbitmq/rab...@172.16.42.14.log" -rabbit lager_upgrade_file "/var/log/rabbitmq/rab...@172.16.42.14_upgrade.log" -rabbit feature_flags_file "/var/lib/rabbitmq/mnesia/rab...@172.16.42.14-feature_flags" -rabbit enabled_plugins_file "/etc/rabbitmq/enabled_plugins" -rabbit plugins_dir "/usr/lib/rabbitmq/plugins:/usr/lib/rabbitmq/lib/rabbitmq_server-3.8.3/plugins" -rabbit plugins_expand_dir "/var/lib/rabbitmq/mnesia/rab...@172.16.42.14-plugins-expand" -os_mon start_cpu_sup false -os_mon start_disksup false -os_mon start_memsup false -mnesia dir "/var/lib/rabbitmq/mnesia/rab...@172.16.42.14" -ra data_dir "/var/lib/rabbitmq/mnesia/rab...@172.16.42.14/quorum" -kernel inet_dist_listen_min 25672 -kernel inet_dist_listen_max 25672 --

[You can ignore the IP difference -- 172.16.42.14 is a different instance that's still running the older version.]

While 3.8.4-1 has a much more slim:

/usr/lib/erlang/erts-10.7.2.1/bin/beam.smp -W w -K true -A 64 -MBas ageffcbf -MHas ageffcbf -MBlmbcs 512 -MHlmbcs 512 -MMmcs 30 -P 1048576 -t 5000000 -stbt db -zdbbl 128000 -- -root /usr/lib/erlang -progname erl -- -home /var/lib/rabbitmq -- -pa  -noshell -noinput -s rabbit boot -boot start_sasl -lager crash_log false -lager handlers [] --

I saw your followup post regarding rabbitmq-diagnostics, and it shows NODENAME properly:

> rabbitmq-diagnostics os_env -n rab...@172.16.32.6
Listing RabbitMQ-specific environment variables defined on node rab...@172.16.32.6...
ADVANCED_CONFIG_FILE=/etc/rabbitmq/advanced.config
CONFIG_FILE=/etc/rabbitmq/rabbitmq
ENABLED_PLUGINS_FILE=/etc/rabbitmq/enabled_plugins
MNESIA_BASE=/var/lib/rabbitmq/mnesia
PLUGINS_DIR=/usr/lib/rabbitmq/plugins:/usr/lib/rabbitmq/lib/rabbitmq_server-3.8.4/plugins
USE_LONGNAME=true

I noticed the config file var doesn't have .conf at the end, so I tried renaming the previous config file from /etc/rabbitmq/rabbitmq.conf, but that didn't make a difference.

The full error when I try to run rabbitmqctl status is:

Error: unable to perform an operation on node 'rabbit@rabbitmq-[...].com'. Please see diagnostics information and suggestions below.

Most common reasons for this are:

 * Target node is unreachable (e.g. due to hostname resolution, TCP connection or firewall issues)
 * CLI tool fails to authenticate with the server (e.g. due to CLI tool's Erlang cookie not matching that of the server)
 * Target node is not running

In addition to the diagnostics info below:

 * See the CLI, clustering and networking guides on https://rabbitmq.com/documentation.html to learn more
 * Consult server logs on node rabbit@rabbitmq-[...].com
 * If target node is configured to use long node names, don't forget to use --longnames with CLI tools

DIAGNOSTICS
===========

attempted to contact: ['rabbit@rabbitmq-[...].com']

rabbit@rabbitmq-[...].com:
  * connected to epmd (port 4369) on rabbitmq-[...].com
  * epmd reports node 'rabbit' uses port 25672 for inter-node and CLI tool traffic 
  * TCP connection succeeded but Erlang distribution failed 

  * Node name (or hostname) mismatch: node "rab...@172.16.32.6" believes its node name is not "rab...@172.16.32.6" but something else.
    All nodes and CLI tools must refer to node "rab...@172.16.32.6" using the same name the node itself uses (see its logs to find out what it is)


Current node details:
 * node name: 'rabbitmqcli-26933-rabbit@rabbitmq-[...].com'
 * effective user's home directory: /var/lib/rabbitmq
 * Erlang cookie hash: [...]


So the issue is that it thinks its name is something different than it was -- the name before the upgrade was rab...@172.16.32.6.

Does anything there stand out for you?

Thanks,
Drew

On Tuesday, May 26, 2020 at 11:35:02 AM UTC-4, Michael Klishin wrote:
 

Can you share a little bit more info about your environment? What OS is used, how was RabbitMQ provisioned, what the contents of rabbitmq-env.conf are? There were no changes to what environment variables are used in 3.8.4. There were changes in what nodes do early on boot

so it could potentially have an effect but we haven’t noticed anything after months of

first making the changes in master.

 

So we need as much information as possible to reproduce.

 

On 26.05.2020, 16:44, rabbit...@googlegroups.com on behalf of Drew Mazurek wrote:

 

I just upgraded one node in my RabbitMQ cluster from 3.8.3 to 3.8.4 from the Ubuntu 16.04 PPA, and after restarting it came up with a different node name. The node name was originally set using the NODENAME var in /etc/rabbitmq/rabbitmq-env.conf, but that seems to be ignored now and it's creating a name based on the machine's hostname, which doesn't work for my AWS cluster. Where should the node name now be configured? Is there anything else I should know about the 3.8.3 to 3.8.4 update that's not in the release notes?

 

Thanks,

Drew

 

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitm...@googlegroups.com.

Michael Klishin

unread,
May 26, 2020, 12:07:48 PM5/26/20
to rabbitm...@googlegroups.com

The difference in VM startup arguments comes down to the fact that RabbitMQ scripts start

a “pre-launch” application which performs environment and configuration discovery, translation and so on. Then it starts RabbitMQ itself.

 

Since os_env -n {node name} works it means that {node name} is indeed what the node

believes its name is.

 

If it reports the value you’d expect then my thinking is that perhaps CLI tools compute the default name differently.

 

rabbitmqctl eval 'node().' -n {node name}

 

will report node’s name (which must match {node name}, of course). I’m not sure what can affect

CLI tool node computation in 3.8.4. A quick look at the code suggests it uses RABBITMQ_NODENAME

like before. This is where in 3.8.4 it could use the same modules as the server does.

 

So the node name did not change. What CLI tools use as default name did. You can export

RABBITMQ_NODENAME for the user used to run CLI commands for the time being.

 

I toyed around with a 3.8.4 node on Ubuntu just yesterday and haven’t found anything unusual. That said, it was not on AWS and my environment was very minimalistic.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/0f04151c-45ff-44ab-a328-848d071cfa01%40googlegroups.com.

Drew Mazurek

unread,
May 26, 2020, 1:34:59 PM5/26/20
to rabbitmq-users
Thanks, Michael, that indeed seems to be the case -- the node name didn't change, it's that the CLI tools aren't using the correct name anymore. I discovered that adding:

RABBITMQ_NODENAME=rab...@172.16.32.6

to /etc/rabbitmq/rabbitmq-env.conf fixed this issue for me. As mentioned before, rabbitmq-env.conf only contained:

USE_LONGNAME=true

It appears I need to set RABBITMQ_NODENAME there now too.

Thanks again for your help!

- Drew

/usr/lib/erlang/erts-10.7.2.1/bin/beam.smp -W w -A 64 -MBas ageffcbf -MHas ageffcbf -MBlmbcs 512 -MHlmbcs 512 -MMmcs 30 -P 1048576 -t 5000000 -stbt db -zdbbl 128000 -K true -- -root /usr/lib/erlang -progname erl -- -home /var/lib/rabbitmq -- -pa /usr/lib/rabbitmq/lib/rabbitmq_server-3.8.3/ebin  -noshell -noinput -s rabbit boot -name rab...@172.16.42.14 -boot start_sasl -conf /etc/rabbitmq/rabbitmq.conf -conf_dir /var/lib/rabbitmq/config -conf_script_dir /usr/lib/rabbitmq/bin -conf_schema_dir /var/lib/rabbitmq/schema -conf_advanced /etc/rabbitmq/advanced.config -kernel inet_default_connect_options [{nodelay,true}] -sasl errlog_type error -sasl sasl_error_logger false -rabbit lager_log_root "/var/log/rabbitmq" -rabbit lager_default_file "/var/log/rabbitmq/rabbit@172.16.42.14.log" -rabbit lager_upgrade_file "/var/log/rabbitmq/rabbit@172.16.42.14_upgrade.log" -rabbit feature_flags_file "/var/lib/rabbitmq/mnesia/rab...@172.16.42.14-feature_flags" -rabbit enabled_plugins_file "/etc/rabbitmq/enabled_plugins" -rabbit plugins_dir "/usr/lib/rabbitmq/plugins:/usr/lib/rabbitmq/lib/rabbitmq_server-3.8.3/plugins" -rabbit plugins_expand_dir "/var/lib/rabbitmq/mnesia/rab...@172.16.42.14-plugins-expand" -os_mon start_cpu_sup false -os_mon start_disksup false -os_mon start_memsup false -mnesia dir "/var/lib/rabbitmq/mnesia/rabbi...@172.16.42.14" -ra data_dir "/var/lib/rabbitmq/mnesia/rabbi...@172.16.42.14/quorum" -kernel inet_dist_listen_min 25672 -kernel inet_dist_listen_max 25672 --

Michael Klishin

unread,
May 26, 2020, 3:18:25 PM5/26/20
to rabbitm...@googlegroups.com

Thank you for confirming. We will see what can be done and whether it is practical for CLI tools to reuse the environment discovery code from RabbitMQ common library. At the very least we could

take NODENAME into account, that’d be a trivial one liner. You can watch [1] if interested.

 

  1. https://github.com/rabbitmq/rabbitmq-cli/issues/421

/usr/lib/erlang/erts-10.7.2.1/bin/beam.smp -W w -A 64 -MBas ageffcbf -MHas ageffcbf -MBlmbcs 512 -MHlmbcs 512 -MMmcs 30 -P 1048576 -t 5000000 -stbt db -zdbbl 128000 -K true -- -root /usr/lib/erlang -progname erl -- -home /var/lib/rabbitmq -- -pa /usr/lib/rabbitmq/lib/rabbitmq_server-3.8.3/ebin  -noshell -noinput -s rabbit boot -name rab...@172.16.42.14 -boot start_sasl -conf /etc/rabbitmq/rabbitmq.conf -conf_dir /var/lib/rabbitmq/config -conf_script_dir /usr/lib/rabbitmq/bin -conf_schema_dir /var/lib/rabbitmq/schema -conf_advanced /etc/rabbitmq/advanced.config -kernel inet_default_connect_options [{nodelay,true}] -sasl errlog_type error -sasl sasl_error_logger false -rabbit lager_log_root "/var/log/rabbitmq" -rabbit lager_default_file "/var/log/rabbitmq/rab...@172.16.42.14.log" -rabbit lager_upgrade_file "/var/log/rabbitmq/rab...@172.16.42.14_upgrade.log" -rabbit feature_flags_file "/var/lib/rabbitmq/mnesia/rab...@172.16.42.14-feature_flags" -rabbit enabled_plugins_file "/etc/rabbitmq/enabled_plugins" -rabbit plugins_dir "/usr/lib/rabbitmq/plugins:/usr/lib/rabbitmq/lib/rabbitmq_server-3.8.3/plugins" -rabbit plugins_expand_dir "/var/lib/rabbitmq/mnesia/rab...@172.16.42.14-plugins-expand" -os_mon start_cpu_sup false -os_mon start_disksup false -os_mon start_memsup false -mnesia dir "/var/lib/rabbitmq/mnesia/rab...@172.16.42.14" -ra data_dir "/var/lib/rabbitmq/mnesia/rab...@172.16.42.14/quorum" -kernel inet_dist_listen_min 25672 -kernel inet_dist_listen_max 25672 --

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/4d54ef87-885c-4d78-bb47-c8722f9cafdd%40googlegroups.com.

Reply all
Reply to author
Forward
0 new messages