Inter-node communication via TLS

[Alexander B]

Hi,

I went through the guide on how to setup the inter-node TLS communication.

I've got rabbit.pem file as per the guide, rabbitmq is owner of it.

My exported variables:

ERL_SSL_PATH=/usr/lib64/erlang/lib/ssl-7.3.3.1/ebin


RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS=-pa /usr/lib64/erlang/lib/ssl-7.3.3.1/ebin -proto_dist inet_tls -ssl_dist_opt server_certfile <my_path_to_rabbit.pem> -ssl_dist_opt server_secure_renegotiate true client_secure_renegotiate true

RABBITMQ_CTL_ERL_ARGS=-pa /usr/lib64/erlang/lib/ssl-7.3.3.1/ebin -proto_dist inet_tls -ssl_dist_opt server_certfile <my_path_to_rabbit.pem> -ssl_dist_opt server_secure_renegotiate true client_secure_renegotiate true

I replaced the real path to rabbit.pem with a stub in the above.

Environment: 2 node test cluster, Erlang 18.3 + Rabbit 3.6.6 + Centos7

I restart the cluster by stopping both nodes and then starting them one at a time. It looks like Rabbit nodes continue to communicate over plain TCP (I can see it from wireshark). Cluster is healthy according to Management UI.

Not sure what to do from here, how to troubleshoot it. Any help is appreciated.

[Michael Klishin]

`ps aux | grep beam` will display the effective Erlang VM command line flags.

You haven't mentioned how RabbitMQ is installed. If you are not running it in the foreground,
e.g. using a generic UNIX binary download, exploring environment variables is insufficient: they won't be picked
up as they will only have effect in your terminal's session.

See RabbitMQ Environment Variables on http://www.rabbitmq.com/configure.html
as well as http://www.rabbitmq.com/man/rabbitmq-env.conf.5.man.html.

On 15 March 2017 at 06:04:56, Alexander B (alex...@gmail.com) wrote:
> Hi,
> I went through the guide on
> how to setup the inter-node TLS communication.
>
> I've got rabbit.pem file as per the guide, rabbitmq is owner of it.
>
> My exported variables:
>
>
> ERL_SSL_PATH=/usr/lib64/erlang/lib/ssl-7.3.3.1/ebin
>
>
> RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS=-pa /usr/lib64/erlang/lib/ssl-7.3.3.1/ebin

> -proto_dist inet_tls -ssl_dist_opt server_certfile -ssl_dist_opt

> server_secure_renegotiate true client_secure_renegotiate true
>
> RABBITMQ_CTL_ERL_ARGS=-pa /usr/lib64/erlang/lib/ssl-7.3.3.1/ebin -proto_dist

> inet_tls -ssl_dist_opt server_certfile -ssl_dist_opt

> server_secure_renegotiate true client_secure_renegotiate true
>
>
> I replaced the real path to rabbit.pem with a stub in the above.
>
> Environment: 2 node test cluster, Erlang 18.3 + Rabbit 3.6.6 + Centos7
>
> I restart the cluster by stopping both nodes and then starting them one at
> a time. It looks like Rabbit nodes continue to communicate over plain TCP
> (I can see it from wireshark). Cluster is healthy according to Management
> UI.
>
> Not sure what to do from here, how to troubleshoot it. Any help is
> appreciated.
>

> --
> You received this message because you are subscribed to the Google Groups "rabbitmq-users"
> group.
> To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
> To post to this group, send an email to rabbitm...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

--
MK

Staff Software Engineer, Pivotal/RabbitMQ

[Alexander B]

Hi Michael, thank you for your help.

You right, environment variables were local to my session :(

Created the /etc/rabbitmq/rabbitmq-env.conf  file with 

SERVER_ADDITIONAL_ERL_ARGS=-pa /usr/lib64/erlang/lib/ssl-7.3.3.1/ebin -proto_dist inet_tls -ssl_dist_opt server_certfile /cert_path/rabbit.pem -ssl_dist_opt server_secure_renegotiate true client_secure_renegotiate true

on both nodes and restart rabbitmq-server service. When I run ps aux | grep beam I dont see the new arguments being appended. I do notice that if I put invalid strings in the .conf file, rabbit fails to start, so I assume file is being read.

[Michael Klishin]

I'd start with quoting the value of SERVER_ADDITIONAL_ERL_ARGS. Spaces in values

in shell script can completely confuse CLI parsers.

I can't think of a reason why your env config wouldn't work otherwise. 

[Alexander B]

Hi Michael,

thank you,

quoting helped to a degree - now doing ps aux | grep beam shows me my TLS arguments.

I restarted nodes and they are back in the cluster.

However Wireshark shows just TCP communications between nodes, no TLS ones. Have no idea how to troubleshoot this...

[Alexander B]

Also tried to use rabbitmqctl without setting "CTL_ERL_ARGS". rabbitmqctl fails to check node status. It makes me think that TLS may be enabled actually. But still don't understand why wireshark is showing me the TCP and not TLS communications.

[Michael Klishin]

Because Wireshark cannot know if a particular connection actually uses TLS unless it's a
well known TLS port (e.g. 443 for HTTPS). Try inspecting the traffic ;)

On 17 March 2017 at 01:02:35, Alexander B (alex...@gmail.com) wrote:
> Also tried to use rabbitmqctl without setting "CTL_ERL_ARGS". rabbitmqctl
> fails to check node status. It makes me think that TLS may be enabled
> actually. But still don't understand why wireshark is showing me the TCP
> and not TLS communications.
>
> On Friday, March 17, 2017 at 10:28:51 AM UTC+13, Alexander B wrote:
> >
> > Hi Michael,
> >
> > thank you,

> > quoting helped to a degree - now doing *ps aux | grep beam* shows me my

> > TLS arguments.
> > I restarted nodes and they are back in the cluster.
> > However Wireshark shows just TCP communications between nodes, no TLS
> > ones. Have no idea how to troubleshoot this...
> >
> > On Friday, March 17, 2017 at 4:31:56 AM UTC+13, Michael Klishin wrote:
> >>
> >> I'd start with quoting the value of SERVER_ADDITIONAL_ERL_ARGS. Spaces
> >> in values
> >> in shell script can completely confuse CLI parsers.
> >>
> >> I can't think of a reason why your env config wouldn't work otherwise.
> >>
> >> On Thursday, March 16, 2017 at 12:37:25 AM UTC+3, Alexander B wrote:
> >>>
> >>> Hi Michael, thank you for your help.
> >>> You right, environment variables were local to my session :(
> >>>

> >>> Created the */etc/rabbitmq/rabbitmq-env.conf* file with

> >>>
> >>> SERVER_ADDITIONAL_ERL_ARGS=-pa /usr/lib64/erlang/lib/ssl-7.3.3.1/ebin
> -proto_dist
> >>> inet_tls -ssl_dist_opt server_certfile /cert_path/rabbit.pem -ssl_dist_opt
> >>> server_secure_renegotiate true client_secure_renegotiate true
> >>>

> >>> on both nodes and restart rabbitmq-server service. When I run *ps aux |
> >>> grep beam *I dont see the new arguments being appended. I do notice

[Alexander B]

Hi Michael,

thank you for your help. All looks good now.