TLS authentication: handshake_timeout error

[Lorenzo Chianura]

Hi there,

I am trying to enable TLS authentication for RabbitMQ clients and despite all the useful information

I've found on this mailing list I am still not capable to solve my problem(s).

First of all, I should clarify that I don't have any previous experience with RabbitMQ nor SSL/TLS, 

hence I'll appreciate any suggestion/correction about what I've done since now.

Before proceeding into the description of the steps I made, some information:

[server]

Ubuntu 18.04.2 LTS
Linux 4.15.0-50-generic x86_64

[RabbitMQ]

3.7.15

[RabbitMQ - enabled plugins]

[E*] rabbitmq_auth_backend_http        3.7.15
[E*] rabbitmq_auth_mechanism_ssl       3.7.15
[E*] rabbitmq_management               3.7.15
[e*] rabbitmq_management_agent         3.7.15
[E*] rabbitmq_mqtt                     3.7.15
[E*] rabbitmq_shovel                   3.7.15
[E*] rabbitmq_shovel_management        3.7.15
[E*] rabbitmq_top                      3.7.15
[e*] rabbitmq_web_dispatch             3.7.15
[E*] rabbitmq_web_mqtt                 3.7.15

[Erlang]

1:20.2.2+dfsg-1ubuntu2

[Erlang - VM TLS support]

tlsv1.3
tlsv1.2
tlsv1.1
tlsv1
sslv3

Following the RabbitMQ ssl guide I created CA and certificate/key pairs using tls-gen (basic profile), ending up with the following files:

ca_certificate.pem
ca_key.pem
client_certificate.pem
client_key.p12
client_key.pem
server_certificate.pem
server_key.p12
server_key.pem

I moved all of them to /etc/rabbitmq/certs and checked that both directories and files have the correct owner (which I found using ps aux | grep rabbitmq):

ls -l /etc/rabbitmq/
drwxr-sr-x   2 rabbitmq rabbitmq 4096 Jun 12 00:58 certs/
-rw-r--r--   1 root     rabbitmq  166 Jun 12 01:23 enabled_plugins
-rw-r--r--   1 rabbitmq rabbitmq  570 Jun 12 02:19 rabbitmq.conf

and 

ls -l /etc/rabbitmq/certs
-rw-r--r-- 1 rabbitmq rabbitmq 1196 Jun 12 00:58 ca_certificate.pem
-rw------- 1 rabbitmq rabbitmq 1704 Jun 12 00:58 ca_key.pem
-rw-r--r-- 1 rabbitmq rabbitmq 1147 Jun 12 00:58 client_certificate.pem
-rw------- 1 rabbitmq rabbitmq 2405 Jun 12 00:58 client_key.p12
-rw------- 1 rabbitmq rabbitmq 1675 Jun 12 00:58 client_key.pem
-rw-r--r-- 1 rabbitmq rabbitmq 1237 Jun 12 00:58 server_certificate.pem
-rw------- 1 rabbitmq rabbitmq 2477 Jun 12 00:58 server_key.p12
-rw------- 1 rabbitmq rabbitmq 1679 Jun 12 00:58 server_key.pem

Once I recognised that my client (which is paho-mqtt) had some issues during TLS authentication,

I started the troubleshooting process described here https://www.rabbitmq.com/troubleshooting-ssl.html.

NOTE: 

server -> Ubuntu VM (VirtualBox) with bridge network (IP: 192.168.0.26)

client -> Ubuntu Host, same version of the server (also same openssl version: 1.1.1)

Protocol listeners are:

Interface: [::], port: 25672, protocol: clustering, purpose: inter-node and CLI tool communication
Interface: [::], port: 5672, protocol: amqp, purpose: AMQP 0-9-1 and AMQP 1.0
Interface: [::], port: 5671, protocol: amqp/ssl, purpose: AMQP 0-9-1 and AMQP 1.0 over TLS
Interface: [::], port: 1883, protocol: mqtt, purpose: MQTT
Interface: [::], port: 15675, protocol: http/web-mqtt, purpose: MQTT over WebSockets
Interface: [::], port: 15672, protocol: http, purpose: HTTP API

double checking with netstat confirm the previous results:

netstat -tulnp | grep beam
tcp        0      0 0.0.0.0:25672           0.0.0.0:*               LISTEN      7232/beam.smp
tcp        0      0 0.0.0.0:15672           0.0.0.0:*               LISTEN      7232/beam.smp
tcp        0      0 0.0.0.0:15675           0.0.0.0:*               LISTEN      7232/beam.smp
tcp6       0      0 :::5671                 :::*                    LISTEN      7232/beam.smp
tcp6       0      0 :::5672                 :::*                    LISTEN      7232/beam.smp
tcp6       0      0 :::1883                 :::*                    LISTEN      7232/beam.smp

Openssl tools to test TLS connections confirm that certificates and keys are OK, running:

openssl s_server -accept 8443 -cert server_certificate.pem -key server_key.pem -CAfile ca_certificate.pem

on the server and:

openssl s_client -connect 192.168.0.26:8443 -cert client_certificate.pem -key client_key.pem -CAfile ca_certificate.pem -verify 8 -verify_hostname caldev

the trust chain is established and the client shows:

Verify return code: 0 (ok)

But when I run the same test against the RabbitMQ broker using:

openssl s_client -connect 192.168.0.26:5671 -cert client_certificate.pem -key client_key.pem -CAfile ca_certificate.pem

on server side I can see (tail -f /var/log/rabbitmq/rab...@caldev.log):

2019-06-12 04:14:23.287 [info] <0.2844.0> accepting AMQP connection <0.2844.0> (192.168.0.25:51481 -> 192.168.0.26:5671)
2019-06-12 04:14:23.288 [error] <0.2844.0> closing AMQP connection <0.2844.0> (192.168.0.25:51481 -> 192.168.0.26:5671): {handshake_timeout,handshake}

which cause the client to log:

read:errno=0

or, when I run openssl s_client in -debug mode:

read from 0x7fffbfde0990 [0x7fffbfde92a3] (5 bytes => 5 (0x5))
0000 - 15 03 03 00 1a                                    .....
read from 0x7fffbfde0990 [0x7fffbfde92a8] (26 bytes => 26 (0x1A))
0000 - 04 ff 3a 82 48 b8 1c 57-79 91 ab 03 25 71 e3 e5   ..:.H..Wy...%q..
0010 - 94 e0 6b 4c e4 5a 84 bd-33 78                     ..kL.Z..3x
closed
write to 0x7fffbfde0990 [0x7fffbfded3f3] (31 bytes => 0 (0x0))
read from 0x7fffbfde0990 [0x7fffbfdd4e50] (8192 bytes => 0 (0x0))

Since the RabbitMQ and openssl documentation are both well written, I think that due to my lack of experience of the whole thing I am missing something important, could someone of you point me in the right direction?

Also, please find attached my:

• rabbitmq.conf
• rabbitmq.logs

Thank you for your time,

Lorenzo

[Arnaud Cogoluègnes]

Which Paho MQTT client are you using (Python, Java, etc)?

I recently stumbled on an issue between Erlang 22.0 and Java 11 when
using TLS (I'm still investigating this issue). So you may be hitting
the same problem if you're using the Paho MQTT Java client on Java
11+. You can downgrade to Erlang 21.3 for a try.

Logs from the client could be useful as well, to see the different
steps of the handshake before the failure.

> --
> You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
> To post to this group, send email to rabbitm...@googlegroups.com.
> To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/43c88bce-0876-45c4-bfb3-c07c9f1d6d48%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Lorenzo Chianura]

Hi Arnaud,

thanks for you answer.

I am using Python paho-mqtt (1.3.1), checking the disconnect reason with:

def on_disconnect(client, userdata, rc):
    if rc == 0:
        print("MQTT CONNACK returned 0: Connection successful")
    elif rc == 1:
        print("MQTT CONNACK returned 1: Connection refused - incorrect"
             " protocol version")
    elif rc == 2:
        print("MQTT CONNACK returned 2: Connection refused - invalid"
             " client identifier")
    elif rc == 3:
        print("MQTT CONNACK returned 3: Connection refused - server"
             " unavailable")
    elif rc == 4:
        print("MQTT CONNACK returned 4: Connection refused - bad"
             " username or password")
    elif rc == 5:
        print("MQTT CONNACK returned 5: Connection refused - not"
             " authorised")
    else:
        print("MQTT CONNACK returned 6-255: Currently unused")

it returns:

MQTT CONNACK returned 1: Connection refused - incorrect protocol version

while RabbitMQ logs:

2019-06-13 11:15:17.364 [info] <0.2270.0> accepting AMQP connection <0.2270.0> (192.168.0.25:53993 -> 192.168.0.26:5671)
2019-06-13 11:15:17.364 [error] <0.2270.0> closing AMQP connection <0.2270.0> (192.168.0.25:53993 -> 192.168.0.26:5671):
{bad_header,<<16,30,0,4,77,81,84,84>>}

I tried different protocols but nothing changed.

I don't think it is something related to the client configuration because I obtain the exactly same RabbitMQ logs using mosquitto_sub:

mosquitto_sub -h 192.168.0.26 -p 5671 --cafile certs/ca_certificate.pem -t "#" -d -v

and checking the certificates on both client and server with openssl verify confirm that they are ok.

> on server side I can see (tail -f /var/log/rabbitmq/rabbit@caldev.log):

>
> 2019-06-12 04:14:23.287 [info] <0.2844.0> accepting AMQP connection <0.2844.0> (192.168.0.25:51481 -> 192.168.0.26:5671)
> 2019-06-12 04:14:23.288 [error] <0.2844.0> closing AMQP connection <0.2844.0> (192.168.0.25:51481 -> 192.168.0.26:5671): {handshake_timeout,handshake}
>
> which cause the client to log:
>
> read:errno=0
>
> or, when I run openssl s_client in -debug mode:
>
> read from 0x7fffbfde0990 [0x7fffbfde92a3] (5 bytes => 5 (0x5))
> 0000 - 15 03 03 00 1a                                    .....
> read from 0x7fffbfde0990 [0x7fffbfde92a8] (26 bytes => 26 (0x1A))
> 0000 - 04 ff 3a 82 48 b8 1c 57-79 91 ab 03 25 71 e3 e5   ..:.H..Wy...%q..
> 0010 - 94 e0 6b 4c e4 5a 84 bd-33 78                     ..kL.Z..3x
> closed
> write to 0x7fffbfde0990 [0x7fffbfded3f3] (31 bytes => 0 (0x0))
> read from 0x7fffbfde0990 [0x7fffbfdd4e50] (8192 bytes => 0 (0x0))
>
>
> Since the RabbitMQ and openssl documentation are both well written, I think that due to my lack of experience of the whole thing I am missing something important, could someone of you point me in the right direction?
>
> Also, please find attached my:
>
> rabbitmq.conf
> rabbitmq.logs
>
>
> Thank you for your time,
> Lorenzo
>
> --
> You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to rabbitm...@googlegroups.com.

[Lorenzo Chianura]

Also, my issue is really similar to this one https://groups.google.com/d/msg/rabbitmq-users/v6zR74ZJxtY/eZuCBXeKCgAJ, except for the fact that my configuration

is simpler since I am not using any intermediate CA, but still I am not able to proceed in any direction.

To the extent to exclude any environment related issue I have configured another Ubuntu server from scratch with the same RabbitMQ and Erlang versions,

following the RabbitMQ guides to enable TLS authentication I've ended up with the same error:

2019-06-13 12:13:01.731 [info] <0.6007.0> accepting AMQP connection <0.6007.0> (192.168.0.25:54300 -> 192.168.0.26:5671)
2019-06-13 12:13:01.731 [error] <0.6007.0> closing AMQP connection <0.6007.0> (192.168.0.25:54300 -> 192.168.0.26:5671):
{handshake_timeout,handshake}

just wondering, could it be that https://www.rabbitmq.com/ssl.html#enabling-tls should be updated since following it on different machines and setups led every time

to the same error?

[Lorenzo Chianura]

Hello again,

I repeated the tests described in my first post, this time capturing network traffic with Wireshark.

This time I used:

ssl_options.verify               = verify_none
ssl_options.fail_if_no_peer_cert = false

because first Wireshark sessions showed that using openssl s_client against RabbitMQ 5671 caused the server to ask for client certificate, while this was not the case using openssl s_client/s_server, 

so I wanted the logs to be as similar as possible and I disabled the peer verification.

NOTE, although is pretty obvious from the screenshots:

CLIENT - 192.168.0.19
SERVER - 192.168.0.80

The first screenshot (debug_handshake_ok.png) describes the s_client/s_server openssl handshake, as you can see everything was OK and the connection was established.

The second screenshot (debug_handshake_problem.png) describes the s_client/rabbitmq handshake, in this case there are a few things that we can notice that match the issues described in the first post:

- the confirmation packet from server (frame 50) lacks the New Session Ticket information

- everything hangs for 10 seconds between frames 54 and 84

- then we receive an Encrypted Alert (21)

- the connection is closed (RST packet)

As you can see from the third screenshot (debug_encrypted_alert_21.png) the Encrypted Alert message has content:

Content Type: Alert (21)

that could be a normal close notify but also a decryption problem: we don't know since the message is encrypted.

Any further suggestion on how to proceed from here will be really appreciated.

Thank you,

Lorenzo

[Arnaud Cogoluègnes]

Could you give a try to the latest Erlang 21.3? Erlang 22.0 introduced
a couple of regressions in TLS, I'd like to make sure we're not
hitting one of them. Thanks!

>>>> > on server side I can see (tail -f /var/log/rabbitmq/rab...@caldev.log):

> To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.

> To post to this group, send email to rabbitm...@googlegroups.com.

> To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/12553d87-7f1a-4aac-9779-9bedd37a08ee%40googlegroups.com.

[Lorenzo Chianura]

Hi Arnaud,

(I don't know why my posts are being deleted today)

unfortunately nothing changes with Erlang 21.3.2:

cat /usr/lib/erlang/releases/21/OTP_VERSION
21.3.2

the Wireshark sessions gives the same result as the previous post.

Can you suggest a way to debug this from RabbitMQ & Erlang side?

[Michael Klishin]

[1] is what we recommend. Unfortunately troubleshooting Erlang TLS implementation regressions is only possible

with Wireshark and a lot of time (and access to private keys).

The latest 21.x version is not 21.3.2 but 21.3.8.4, however.

--

You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.

To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/dde316cf-b852-4b46-b8fa-9204d6d452d4%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ

[Shuming Xin]

Hello,

Have you resolved this issue?

I encountered with the same issue. (Ubuntu 18.04.2, Rabbitmq is 3.7.15 and Erlang 22.0.4) . With openssl client, there is the same error message - saying {handshake_timeout,handshake}

But I can connect to Rabbitmq on TLS with Java client.

[Lorenzo Chianura]

Hi,

unfortunately not.

I also tried with Erlang 21.3.8.4 as suggested by @Michael, but still no luck.

I've generated last certificates and keys using letsencrypt (https://letsencrypt.org/) but the results are the same (with both Erlang 22 and 21.3.8.4).

Can you share more details about your working Java client?

thanks,

Lorenzo

[Shuming Xin]

I just followed the Rabbitmq online documentation - https://www.rabbitmq.com/ssl.html#java-client-connecting-with-peer-verification

Below is the sample code. You can get the complete code from the online documentation.

As java client can connect but openssl doesn't work, there may be caused by version mismatch (Openssl, Erlang, etc). But I can't get a clue to fix it :(

shuming@shuming-VirtualBox:/media/sf_ubuntu-shared/demo/build/libs$ java -jar gradle-demo-0.0.1.jar
Received: Hello, World

public static void main(String[] args) throws Exception {
    char[] keyPassphrase = "MySecretPassword".toCharArray();
    KeyStore ks = KeyStore.getInstance("PKCS12");
    ks.load(new FileInputStream("/home/shuming/client/client_certificate.p12"), keyPassphrase);

    KeyManagerFactory kmf = KeyManagerFactory.getInstance("SunX509");
    kmf.init(ks, keyPassphrase);

    char[] trustPassphrase = "rabbitmq".toCharArray();
    KeyStore tks = KeyStore.getInstance("JKS");
    tks.load(new FileInputStream("/home/shuming/client/rabbitstore"), trustPassphrase);

    TrustManagerFactory tmf = TrustManagerFactory.getInstance("SunX509");
    tmf.init(tks);

    SSLContext c = SSLContext.getInstance("TLSv1.2");
    c.init(kmf.getKeyManagers(), tmf.getTrustManagers(), null);

    ConnectionFactory factory = new ConnectionFactory();
    factory.setHost("shuming-VirtualBox");
    factory.setPort(5671);
    factory.useSslProtocol(c);
    factory.enableHostnameVerification();

[Michael Klishin]

Please start a new thread for your questions.

We cannot suggest much without seeing what exactly is the output of `openssl s_client`.

--

You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.

To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/04eaa081-77f5-47aa-9d32-0df69984a7ed%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--