3.8.4+ Breaks rabbitmqctl for my configuration

[Lance Kinley]

When upgrading to 3.8.4 or 3.8.5, rabbitmqctl no longer works.  When I downgrade back to 3.8.3, it works again.

DIAGNOSTICS

===========

attempted to contact: ['rabbit@mq1.-redacted-']

rabbit@mq1.-redacted-:

  * connected to epmd (port 4369) on mq1.-redacted-

  * epmd reports node 'rabbit' uses port 25672 for inter-node and CLI tool traffic

  * TCP connection succeeded but Erlang distribution failed

  * Remote host closed TCP connection before completing authentication. Is the Erlang distribution using TLS?

Current node details:

 * node name: 'rabbitmqcli-12927-rabbit@mq1.-redacted-'

 * effective user's home directory: /var/lib/rabbitmq

 * Erlang cookie hash: -redacted-

Any insight?  I've looked through the release notes and don't see anything relevant.

I am using TLS for inter-node communication.  Here are my configs:

rabbitmq.conf:

mirroring_sync_batch_size = 1024

queue_master_locator = min-masters

listeners.ssl.default = 5671

ssl_options.cacertfile = /mq/certs/mq-ca.pem

ssl_options.certfile = /mq/certs/mq-server.crt

ssl_options.keyfile = /mq/certs/mq-server.key

ssl_options.verify = verify_peer

ssl_options.fail_if_no_peer_cert = true

ssl_options.versions.1 = tlsv1.2

auth_mechanisms.1 = EXTERNAL

ssl_cert_login_from = common_name

auth_backends.1 = internal

management.listener.port = 15672

management.ssl.port = 15671

management.ssl.cacertfile = /mq/certs/mq-ca.pem

management.ssl.certfile = /mq/certs/mq-server.crt

management.ssl.keyfile = /mq/certs/mq-server.key

management.ssl.honor_cipher_order   = true

management.ssl.honor_ecc_order      = true

management.ssl.client_renegotiation = false

management.ssl.secure_renegotiate   = true

management.ssl.versions.1 = tlsv1.2

management.ssl.versions.2 = tlsv1.1

management.ssl.ciphers.1 = ECDHE-ECDSA-AES256-GCM-SHA384

management.ssl.ciphers.2 = ECDHE-RSA-AES256-GCM-SHA384

management.ssl.ciphers.3 = ECDHE-ECDSA-AES256-SHA384

management.ssl.ciphers.4 = ECDHE-RSA-AES256-SHA384

management.ssl.ciphers.5 = ECDH-ECDSA-AES256-GCM-SHA384

management.ssl.ciphers.6 = ECDH-RSA-AES256-GCM-SHA384

management.ssl.ciphers.7 = ECDH-ECDSA-AES256-SHA384

management.ssl.ciphers.8 = ECDH-RSA-AES256-SHA384

management.ssl.ciphers.9 = DHE-RSA-AES256-GCM-SHA384

rabbitmq-env.conf:

ERL_SSL_PATH="/usr/lib64/erlang/lib/ssl-9.5/ebin"

RABBITMQ_USE_LONGNAME=true

RABBITMQ_DISTRIBUTION_BUFFER_SIZE=256000

SERVER_ADDITIONAL_ERL_ARGS="-pa $ERL_SSL_PATH \

  -proto_dist inet_tls \

  -ssl_dist_optfile /etc/rabbitmq/ssl_dist.config"

CTL_ERL_ARGS="-pa $ERL_SSL_PATH \

  -proto_dist inet_tls \

  -ssl_dist_optfile /etc/rabbitmq/ssl_dist.config"

ssl_dist.config:

[

  {server, [

    {cacertfile, "/mq/certs/mq-ca.pem"},

    {certfile, "/mq/certs/mq-server.crt"},

    {keyfile,  "/mq/certs/mq-server.key"},

    {secure_renegotiate, true},

    {verify, verify_peer},

    {fail_if_no_peer_cert, true}

  ]},

  {client, [

    {cacertfile, "/mq/certs/mq-ca.pem"},

    {certfile, "/mq/certs/mq-admin.crt"},

    {keyfile, "/mq/certs/mq-admin.key"},

    {secure_renegotiate, true},

    {verify, verify_peer},

    {fail_if_no_peer_cert, true}

  ]}

].

[Wesley Peng]

Can you disable TLS for inner nodes then give a try?

I have no problem when upgrading to 3.8.5.

root@ubuntu:~# rabbitmqctl status
Status of node rabbit@ubuntu ...
Runtime

OS PID: 6792
OS: Linux
Uptime (seconds): 583767
RabbitMQ version: 3.8.5
Node name: rabbit@ubuntu
Erlang configuration: Erlang/OTP 23 [erts-11.0.2] [source] [64-bit] [smp:1:1] [ds:1:1:10] [async-threads:64]
Erlang processes: 442 used, 1048576 limit
Scheduler run queue: 1
Cluster heartbeat timeout (net_ticktime): 60

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/84e230d4-dfc0-45d4-882b-d2aabaff4411o%40googlegroups.com.

[Jean-Sébastien Pédron]

On 23/06/2020 21:00, Lance Kinley wrote:
> When upgrading to 3.8.4 or 3.8.5, rabbitmqctl no longer works.  When I
> downgrade back to 3.8.3, it works again.
>

> (...)

>
> CTL_ERL_ARGS="-pa $ERL_SSL_PATH \
>   -proto_dist inet_tls \
>   -ssl_dist_optfile /etc/rabbitmq/ssl_dist.config"

Hi!

Could you please try to rename "CTL_ERL_ARGS" to "RABBITMQ_CTL_ERL_ARGS"?

--
Jean-Sébastien Pédron
RabbitMQ Core team
VMware, Inc.

[Luke Bakken]

Hi Lance and JSP -

Lance - Jean-Sébastien's suggestion should be a workaround for you.

I have filed this issue to fix this bug - https://github.com/rabbitmq/rabbitmq-server/issues/2394

Thank you for taking the time to report it to the mailing list.

Luke

----

Senior Member of Technical Staff

VMware / RabbitMQ

[Lance Kinley]

This indeed fixes it!  Thanks.

On Thursday, June 25, 2020 at 3:48:44 AM UTC-7, Jean-Sébastien Pédron wrote:

[Jean-Sébastien Pédron]

On 28/06/2020 21:42, Lance Kinley wrote:
> This indeed fixes it!  Thanks.

Thank you for testing and reporting!

The next release of RabbitMQ 3.8.x should have a bugfix to restore the
handling of the $CTL_* variables.