How to setup epmd and RabbitMQ for SSL communication.

[Narasimha Karumanchi]

Hi Team,

   I am trying to setup RabbitMQ cluster nodes with SSL communication between nodes in the cluster.

   I have referred, https://www.rabbitmq.com/clustering-ssl.html

   I ran the following on each of the nodes in the cluster:

export RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS='-pa "/usr/lib64/erlang/lib/ssl-5.3.7/ebin" -proto_dist inet_tls -ssl_dist_opt server_cacertfile /etc/rabbitmq/ssl/cacert.pem server_certfile /etc/rabbitmq/ssl/cert.pem server_keyfile /etc/rabbitmq/ssl/key.pem -ssl_dist_opt server_secure_renegotiate true client_secure_renegotiate true'

export RABBITMQ_CTL_ERL_ARGS='-pa "/usr/lib64/erlang/lib/ssl-5.3.7/ebin" -proto_dist inet_tls -ssl_dist_opt server_cacertfile /etc/rabbitmq/ssl/cacert.pem server_certfile /etc/rabbitmq/ssl/cert.pem server_keyfile /etc/rabbitmq/ssl/key.pem -ssl_dist_opt server_secure_renegotiate true client_secure_renegotiate true'

   Assuming same certificates on all the nodes in cluster, restarting rabbitmq servers is not picking the SSL communication.

   When we set up a rabbitmq cluster, the rabbitmq servers need to talk to each other, which is done over separate "erlang distribution" ports (4369 and other port for communication between nodes).  I think this involves the Erlang Port Mapper Daemon (epmd service).

   Do I need to set any configuration in epmd daemon as well?

   Please help me with an example on this.

Thanks and Regards,

Narasimha

[Jean-Sébastien Pédron]

On 06.04.2015 20:37, Narasimha Karumanchi wrote:
> Hi Team,

Hi!

> *export RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS='-pa

> "/usr/lib64/erlang/lib/ssl-5.3.7/ebin" -proto_dist inet_tls
> -ssl_dist_opt server_cacertfile /etc/rabbitmq/ssl/cacert.pem
> server_certfile /etc/rabbitmq/ssl/cert.pem server_keyfile
> /etc/rabbitmq/ssl/key.pem -ssl_dist_opt server_secure_renegotiate true

> client_secure_renegotiate true'*

After setting the two environment variables, how do you start RabbitMQ?

> Do I need to set any configuration in epmd daemon as well?

I don't think epmd supports SSL. But there is no sensitive data going
between Erlang nodes and epmd.

--
Jean-Sébastien Pédron
Pivotal / RabbitMQ

[Narasimha Karumanchi]

I am restarting the rabbitmq with:

rabbitmq-server restart

I have the following queries in this regard:

1. With above command I see that epmd is getting started with two ports: 4369 and 25672. I understood that 4369  is for node registrations and 25672 is for inter node communications. If we want to make 25672 with SSL, is there a way?
   
2. Why are we using this extra port? Cannot we use rabbitmq main port, 5671?
3. Is it possible to do internode communication with SSL and epmd?

Thanks and Regards,
-Narasimha Karumanchi
+91-986654231

--
You received this message because you are subscribed to a topic in the Google Groups "rabbitmq-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/rabbitmq-users/rJaJWctOYKQ/unsubscribe.
To unsubscribe from this group and all its topics, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send an email to rabbitm...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Jean-Sébastien Pédron]

On 07.04.2015 11:14, Narasimha Karumanchi wrote:
> I am restarting the rabbitmq with:
>

> *rabbitmq-server restart*

If this is the same shell than the one you set the variables, then it
should be using SSL inter-node communication. How do you tell it's not
doing this?

> I have the following queries in this regard:
>

> 1. With above command I see that epmd is getting started with two

> ports: 4369 and 25672. I understood that 4369 is for node
> registrations and 25672 is for inter node communications. If we want
> to make 25672 with SSL, is there a way?

epmd uses 4369 only. Inter-node communication on 25672 is made directly
between nodes (ie. they don't go through epmd) when using non-SSL
distribution.

There is one known issue in Erlang SSL distribution: it doesn't honor
the distribution ports configuration. That means Erlang picks a random
ephemere port.

> 2. Why are we using this extra port? Cannot we use rabbitmq main
> port, 5671?

They are not used for the same purpose:
o 5672/5672: AMQP messages
o 25672/random port if SSL: inter-node communication (Erlang-level
messages)

> 3. Is it possible to do internode communication with SSL and epmd?

Inter-node communication, yes. epmd, no, but it has nothing to do with
inter-node communication beside keeping the registry.

[Narasimha Karumanchi]

Thank you Jean for the reply.

Since, epmd is just for node registry and from local machine, I think it seems OK if we skip SSL for epmd.

1. Also, if we set the flags for SSL on all nodes in cluster, does the inter-node communication happen with SSL and random port?
2. Can you please point me to any sample code SSL communication between nodes in cluster? I could not see any such in tutorial section.

Thanks and Regards,
-Narasimha Karumanchi
+91-986654231

[Jean-Sébastien Pédron]

On 07.04.2015 11:39, Narasimha Karumanchi wrote:
> 1. Also, if we set the flags for SSL on all nodes in cluster, does the

> inter-node communication happen with SSL and random port?

SSL inter-node communication happens on random ports on both sides of
the connection (the listening port and the client port). This is a
limitation in Erlang which we already reported upstream.

> 2. Can you please point me to any sample code SSL communication between

> nodes in cluster? I could not see any such in tutorial section.

What do you mean?

There is no need to develop anything to enable SSL inter-node
communication. This feature is even outside of RabbitMQ: it is provided
by Erlang itself.

Our documentation just explains how to set this up in the context of
RabbitMQ.

[Narasimha Karumanchi]

You mean, inter-node communication always happen with SSL?

https://www.rabbitmq.com/clustering-ssl.html

http://www.erlang.org/doc/apps/ssl/ssl_distribution.html

I was referring these two.

I did the setup (setting the ENV and restarted) on all nodes in cluster. But to verify that SSL communication is happening is there a way to confirm?

Thanks and Regards,
-Narasimha Karumanchi
+91-986654231

[Jean-Sébastien Pédron]

On 07.04.2015 11:57, Narasimha Karumanchi wrote:
> You mean, inter-node communication always happen with SSL?

No, out of the box, inter-node communication is done unencrypted,
because most of the time, this is uneeded (as Erlang nodes are often in
the same private trusted LAN) and SSL requires certificates and private
keys Erlang can't create automatically.

> https://www.rabbitmq.com/clustering-ssl.html
> http://www.erlang.org/doc/apps/ssl/ssl_distribution.html
>
> I was referring these two.

This is correct.

> I did the setup (setting the ENV and restarted) on all nodes in cluster.
> But to verify that SSL communication is happening is there a way to confirm?

I'm not sure the Erlang SSL distribution module provide a tool for that.
But if your node doesn't listen on 25672, I guess SSL is being used
(because of the Erlang limitation not honoring the distribution port
configuration).

[Narasimha Karumanchi]

I am still trying to convince myself and understand if inter-node communication happen without SSL, why does the page below says we can do that.

https://www.rabbitmq.com/clustering-ssl.html

Thanks and Regards,
-Narasimha Karumanchi
+91-986654231

[Narasimha Karumanchi]

I tried to setup the flags, restart of rabbitmg. Also, I set the flag (in config) to allow only SSL communications. But, the communication is happening without SSL. I disabled the tcp listeners but it is still using.

{tcp_listeners, []}

export RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS='-pa "/usr/lib64/erlang/lib/ssl-5.3.7/ebin" -proto_dist inet_tls -ssl_dist_opt server_cacertfile /etc/rabbitmq/ssl/cacert.pem server_certfile /etc/rabbitmq/ssl/cert.pem server_keyfile /etc/rabbitmq/ssl/key.pem -ssl_dist_opt server_secure_renegotiate true client_secure_renegotiate true'

export RABBITMQ_CTL_ERL_ARGS='-pa "/usr/lib64/erlang/lib/ssl-5.3.7/ebin" -proto_dist inet_tls -ssl_dist_opt server_cacertfile /etc/rabbitmq/ssl/cacert.pem server_certfile /etc/rabbitmq/ssl/cert.pem server_keyfile /etc/rabbitmq/ssl/key.pem -ssl_dist_opt server_secure_renegotiate true client_secure_renegotiate true'

Sample Config file:

[

  {kernel, [

  ]},

{tcp_listeners, []},

  {ssl, [{versions, ['tlsv1.2', 'tlsv1.1', tlsv1]}]},

  {rabbit, [

    {cluster_nodes, {[], disc}},

    {cluster_partition_handling,pause_minority},

    {ssl_listeners, [5671]},

    {ssl_options, [{cacertfile,"/etc/rabbitmq/ssl/cacert.pem"},

                    {certfile,"/etc/rabbitmq/ssl/cert.pem"},

                    {keyfile,"/etc/rabbitmq/ssl/key.pem"},

                    {verify,verify_none},

                    {fail_if_no_peer_cert,false}]},

    {tcp_listen_options, [binary, {packet,raw},

                                  {reuseaddr,true},

                                  {backlog,128},

                                  {nodelay,true},

                                  {exit_on_close,false},

                                  {keepalive,false}]},

    {default_user, <<"guest">>},

    {default_pass, <<"PASSWORDHERE">>},

    {heartbeat, 580},

    {loopback_users, []}

  ]}

].

After setting the above flags, and with below config file, it is will using 5672 port.

[root@test]#  ps aux | grep epmd;ps aux | grep erl

epmd       912  0.0  0.0  17884   792 ?        Ss   04:31   0:00 /usr/bin/epmd -systemd

root     10299  0.0  0.0 112644   972 pts/0    S+   06:30   0:00 grep --color=auto epmd

rabbitmq  1238  0.6  0.8 2316356 70268 ?       Ssl  04:31   0:47 /usr/lib64/erlang/erts-6.2.1/bin/beam.smp -W w -K true -A30 -P 1048576 -- -root /usr/lib64/erlang -progname erl -- -home /var/lib/rabbitmq -- -pa /usr/lib/rabbitmq/lib/rabbitmq_server-3.3.4/sbin/../ebin -noshell -noinput -s rabbit boot -sname rabbit@test -boot start_sasl -config /etc/rabbitmq/rabbitmq -kernel inet_default_connect_options [{nodelay,true}] -rabbit tcp_listeners [{"IP",5672}] -sasl errlog_type error -sasl sasl_error_logger false -rabbit error_logger {file,"/var/log/rabbitmq/rab...@test.log"} -rabbit sasl_error_logger {file,"/var/log/rabbitmq/rab...@test.log"} -rabbit enabled_plugins_file "/etc/rabbitmq/enabled_plugins" -rabbit plugins_dir "/usr/lib/rabbitmq/lib/rabbitmq_server-3.3.4/sbin/../plugins" -rabbit plugins_expand_dir "/var/lib/rabbitmq/mnesia/rabbit@test-plugins-expand" -os_mon start_cpu_sup false -os_mon start_disksup false -os_mon start_memsup false -mnesia dir "/var/lib/rabbitmq/mnesia/rabbit@test" -kernel inet_dist_listen_min 25672 -kernel inet_dist_listen_max 25672

root     10301  0.0  0.0 112644   972 pts/0    S+   06:30   0:00 grep --color=auto erl

Thanks and Regards,
-Narasimha Karumanchi
+91-986654231

[Jean-Sébastien Pédron]

On 07.04.2015 12:32, Narasimha Karumanchi wrote:
> I tried to setup the flags, restart of rabbitmg. Also, I set the flag
> (in config) to allow only SSL communications.

In fact, I'm not sure we are talking about the same thing.

SSL for inter-node communication is distinct from SSL for AMQP
connections. The former is used to cipher the connections between Erlang
nodes (only used if you setup a cluster and also when you run
rabbitmqctl or rabbitmq-plugins).

The latter is used to cipher the AMQP messages published and consumed by
clients (http://www.rabbitmq.com/ssl.html).

Both are independent from each other.

> export RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS='-pa
> "/usr/lib64/erlang/lib/ssl-5.3.7/ebin" -proto_dist inet_tls
> -ssl_dist_opt server_cacertfile /etc/rabbitmq/ssl/cacert.pem
> server_certfile /etc/rabbitmq/ssl/cert.pem server_keyfile
> /etc/rabbitmq/ssl/key.pem -ssl_dist_opt server_secure_renegotiate true
> client_secure_renegotiate true'
>
> export RABBITMQ_CTL_ERL_ARGS='-pa "/usr/lib64/erlang/lib/ssl-5.3.7/ebin"
> -proto_dist inet_tls -ssl_dist_opt server_cacertfile
> /etc/rabbitmq/ssl/cacert.pem server_certfile /etc/rabbitmq/ssl/cert.pem
> server_keyfile /etc/rabbitmq/ssl/key.pem -ssl_dist_opt
> server_secure_renegotiate true client_secure_renegotiate true'

To be clear, those variables are only useful to setup inter-node
communication.

> *Sample Config file:*
>
> [
> {kernel, [
>
> ]},
> {tcp_listeners, []},

Here, this line is misplaced: it should be inside {rabbit, ...}, like
ssl_listeners.

For instance:

{rabbit, [
{cluster_nodes, {[], disc}},
{cluster_partition_handling,pause_minority},

{tcp_listeners, []},
{ssl_listeners, [5671]},
...

Other than that, your configuration seems correct and sets up SSL for
connections between AMQP consumers/producers and RabbitMQ.

[Narasimha Karumanchi]

Correct Jean.

Is there a way to confirm that the internodes communication happen with SSL?

Any netstat or similr commands?

Coming to config file setting, my thought was, since it is using non-SSL communication I set it to disable that totally.

Thanks and Regards,
-Narasimha Karumanchi
+91-986654231

[Norberto Meijome]

You could try tcpdump and compare what you see over clear text and expected over-SSL  traffic...

You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.

[Jean-Sébastien Pédron]

On 07.04.2015 13:00, Narasimha Karumanchi wrote:
> Is there a way to confirm that the internodes communication happen with SSL?

Like I said, one possibility is to check which port beam listens to. To
go further, you can follow Norberto Meijome's advice.

> Coming to config file setting, my thought was, since it is using non-SSL
> communication I set it to disable that totally.

You can do that, just move the tcp_listeners line to the right place.