TLS hanging connections issue

[jan.go...@enervalis.com]

I'm trying to set up RabbitMQ for TLS 1.2. Unfortunately the connections seem to hang. And there is no logging. I'm most probably omitting some basic option since this is my first setup of this kind. :-)

> Erlang supports TLS 1.2 (see below)

> The straightforward openssl client/server connection test works. (see below)

> The RabbitMQ server config file is using the very same key- and certificate files. (see below)

> Connections just hang when connecting to the RabbitMQ server with the openssl client. It says "CONNECTED(00000003)" and that's it.

$ openssl s_client -pass "pass:**************" -connect localhost:5671 -cert ../certs/client.cert.pem -key ../private/client.key.pem -CAfile ./ca.cert.pem
CONNECTED(00000003)

> The same happens when using the Java client. A timeout is exception is thrown. It looks like it is related to the handshake since extending the handshake timeout delays the exception accordingly. Unfortunately the RabbitMQ logs remain empty.

> Connections *seem* to be established

$ netstat -a | grep amqp
tcp        3      0 localhost:amqps         0.0.0.0:*               LISTEN    
tcp      315      0 localhost:amqps         localhost:40822         CLOSE_WAIT
tcp      306      0 localhost:amqps         localhost:36920         CLOSE_WAIT
tcp        0      0 localhost:58500         localhost:amqps         ESTABLISHED
tcp      306      0 localhost:amqps         localhost:40508         CLOSE_WAIT
tcp      305      0 localhost:amqps         localhost:58500         ESTABLISHED
tcp      305      0 localhost:amqps         localhost:58500         ESTABLISHED
tcp6       0      0 [::]:amqp               [::]:*                  LISTEN

Anybody has an idea of the beginner's mistake I am making ? :-)

Thanks !

The environment

xUbuntu 16.10 (amd64)
Erlang 19.2 (esl-erlang package)
RabbitMQ 3.6.6 (rabbitmq-server package)

Erlang ssl support

$ erl
Erlang/OTP 19 [erts-8.2] [source-fbd2db2] [64-bit] [smp:8:8] [async-threads:10] [hipe] [kernel-poll:false]


Eshell V8.2  (abort with ^G)
1> ssl:versions().
[{ssl_app,"8.1"},
 {supported,['tlsv1.2','tlsv1.1',tlsv1]},
 {available,['tlsv1.2','tlsv1.1',tlsv1,sslv3]}]
2>

RabbitMQ server config /etc/rabbitmq/rabbitmq.config

$ cat /etc/rabbitmq/rabbitmq.config
[
 {ssl, [{versions, ['tlsv1.2']}]},
 {rabbit, [
   {ssl_listeners, [{"127.0.0.1", 5671}]},
   {ssl_options, [
     {cacertfile, "/opt/dev/certificate/ca/intermediate/amqp/ca.cert.pem" },
     {certfile, "/opt/dev/certificate/ca/intermediate/amqp/server.cert.pem" },
     {keyfile, "/opt/dev/certificate/ca/intermediate/amqp/server.key.pem" },
     {password, "**********************"},
     {verify, verify_none},
     {fail_if_no_peer_cert, false },
     {versions, ['tlsv1.2']}
    ]}
  ]}
].

openssl client/server connect log

$ openssl s_client -connect localhost:8443 -cert ../certs/client.cert.pem -key ../private/client.key.pem -CAfile ./ca.cert.pem
Enter pass phrase for ../private/client.key.pem:
CONNECTED(00000003)
depth=2 CN = root, OU = Control, O = Enervalis, L = Houthalen, ST = Limburg, C = BE
verify return:1
depth=1 C = BE, ST = Limburg, O = Enervalis, OU = Control, CN = intermediate
verify return:1
depth=0 C = BE, ST = Limburg, O = Enervalis, OU = Control, CN = amqp
verify return:1
---
Certificate chain
 0 s:/C=BE/ST=Limburg/O=Enervalis/OU=Control/CN=amqp
   i:/C=BE/ST=Limburg/O=Enervalis/OU=Control/CN=intermediate
 1 s:/C=BE/ST=Limburg/O=Enervalis/OU=Control/CN=intermediate
   i:/CN=root/OU=Control/O=Enervalis/L=Houthalen/ST=Limburg/C=BE
 2 s:/CN=root/OU=Control/O=Enervalis/L=Houthalen/ST=Limburg/C=BE
   i:/CN=root/OU=Control/O=Enervalis/L=Houthalen/ST=Limburg/C=BE
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIGXzCCBEegAwIBAgICEAEwDQYJKoZIhvcNAQELBQAwXDELMAkGA1UEBhMCQkUx
EDAOBgNVBAgMB0xpbWJ1cmcxEjAQBgNVBAoMCUVuZXJ2YWxpczEQMA4GA1UECwwH
Q29udHJvbDEVMBMGA1UEAwwMaW50ZXJtZWRpYXRlMB4XDTE3MDExNjA5NTk1MloX
DTI3MDEyNDA5NTk1MlowVDELMAkGA1UEBhMCQkUxEDAOBgNVBAgMB0xpbWJ1cmcx
EjAQBgNVBAoMCUVuZXJ2YWxpczEQMA4GA1UECwwHQ29udHJvbDENMAsGA1UEAwwE
YW1xcDCCAiIwDQYJKoZIhvcNAQEBBQADggIPADCCAgoCggIBALyyDin60IEh1g9F
zVY83tcfRcbrmKpM/4kllyEQpKAmUCxnLY0oi4jfjKHM8xgs/ajnhBBYg9pMWttP
YX9NmaYg3hMJYV4xVNxNLbPTbsWrIxwOqDoEzSC7TAYR1hWJXi3Zj9uexhDapN+b
V/Iquur+eBqQvGYz13LVTemBFex6WZ915kQIU8Zdy8A786VZDbj4LQjArS/23x2o
/vwpKYMyfiB4iA/9GtxaY8ZHsx9g52xF7rjwjM29hkOM2uLWhy2Krx/sSEZJBHip
1Rebg/2dBd5kmeC0wJR4wIQBrslD8vN/lnE8yL2wYdNf4edJ/mye8NcoiMGwKoXJ
kiOO8f24ELc3A9xr/rELbdrgK70kYMMeGHWRmKTBwUpR1QoUTOpGl1NQg6KyMLss
0KOn0hG72QGE+xqfZ7m3DSeVKxtrD05g5fKOBZINu3Q7sgn647CMU/bKdKqaE3L+
oPvdysrqOtRgLYCL0ZTUlHg75HOZfr1TmtQuHCXPZ9Jfa0J7hGPtD0LBAbgtYUFH
ekTndWvnPEoOnytRnOrEsniIps0S6lXGpBMHbY1ZLShMM3lpcCtxgGNuQ6Q/AcwC
9qT0GgIhGTx62xsYmzXWie/1HU7TgMG0k0tp61HT9goJYUG0qNjFxLCqqEVW7056
bAS64oCA73s8v54qqKBYt1BeNxq/AgMBAAGjggExMIIBLTAJBgNVHRMEAjAAMBEG
CWCGSAGG+EIBAQQEAwIGQDAzBglghkgBhvhCAQ0EJhYkT3BlblNTTCBHZW5lcmF0
ZWQgU2VydmVyIENlcnRpZmljYXRlMB0GA1UdDgQWBBSd7DMbsPXd5+17qo3znHmo
AWIxBjCBkwYDVR0jBIGLMIGIgBSZd5gLw4eRXKteEDUgGdRqomAirqFspGowaDEN
MAsGA1UEAwwEcm9vdDEQMA4GA1UECwwHQ29udHJvbDESMBAGA1UECgwJRW5lcnZh
bGlzMRIwEAYDVQQHDAlIb3V0aGFsZW4xEDAOBgNVBAgMB0xpbWJ1cmcxCzAJBgNV
BAYTAkJFggIQADAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0lBAwwCgYIKwYBBQUHAwEw
DQYJKoZIhvcNAQELBQADggIBAH7sABoC3c6wukTxm/1e70xlov7qFJlw6buGr6K6
drpa6uZh+CZHqyvlwc/2swxgyOzEbZtPpEEi/XKteTwi2KneXz8aaIIb9jImyzFN
EGlUSQ/8FBuj1SbgoHRQ1LyhMgiPKh2l4Ov507iGI70fquBhuZ4D08dyB3v3sk+l
0bs+zt0cP2Fa9SUwO3p4wkapNCv9tWAElTYBR2CIiO/VHtEgr7hCtMO4RsopOz1E
iHAeB57x0jtAnPdKq+PMOZvm16tqqxAc5+2gWFiRcNaorOZAPGBiQcneEtZqzOxK
VQqmDBsgUk0JLdA+ITRxoI6HYd6EFEl8pm1dK6/EjlEElmiKNDiXa3wXyDjrlf37
056/sN312Btmj/wDS03KTqfOaFdEGDoEkER3/sMWfzX88nYHa88OOCAWyB0qz3F5
fociyQGDFOK3uVooxcNHRR2FtMspV2AKBwDrtMEpA40IJ8PYHHS41xvHcKrNkPYE
w7/lRkB9spqkBIVhRaM05Z0Z71UjLbRfTHvMoAKmI+lr8plg38aw6X08jLYrJyqm
7eGSyXgy5pVbj1UYI0MuTjld8kwL4Yu66QcrtQJiEkd3RB4JJiV9tvq8AabQhAMq
JkebnTxkU56aVW+R/vsiPtiCvAX/bBR4YHKbdNeTgsBZUe9fXzxYmmnHb4ZLHRR5
nCvK
-----END CERTIFICATE-----
subject=/C=BE/ST=Limburg/O=Enervalis/OU=Control/CN=amqp
issuer=/C=BE/ST=Limburg/O=Enervalis/OU=Control/CN=intermediate
---
No client certificate CA names sent
Peer signing digest: SHA512
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 5472 bytes and written 431 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 4096 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES256-GCM-SHA384
    Session-ID: 86C62B04E749B09EB9668DAE260161B4DB7A33D53566DB2D2046985E2FEF383C
    Session-ID-ctx:
    Master-Key: D53D5F7FA09B0C5986A877354D9E0D8A38B104E123D662B24E84CC9453AEFA1C6D354E1099C2829592BF6BB499BB65D1
    Key-Arg   : None
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 300 (seconds)
    TLS session ticket:
    0000 - c5 fb 34 bc 99 85 8d 06-99 51 ad d0 c9 6e aa 72   ..4......Q...n.r
    0010 - 5e da d1 b3 c2 db c8 f5-91 1d 7f 13 81 f7 1e fe   ^...............
    0020 - 07 67 87 1d f7 d2 c2 d0-96 47 70 c5 cf a2 25 dd   .g.......Gp...%.
    0030 - 01 43 1b 09 9f 34 52 1f-23 da 41 41 be d9 23 21   .C...4R.#.AA..#!
    0040 - 55 5f c2 d3 02 b3 ac 40-c1 0e 45 a1 bb 9e 95 53   U_.....@..E....S
    0050 - 2a 23 2c a1 81 78 2e cd-88 54 35 6a f9 99 8a 26   *#,..x...T5j...&
    0060 - ab 7b 10 ba c4 cd 90 55-7a d7 33 46 31 3e 9e c8   .{.....Uz.3F1>..
    0070 - 62 99 81 1f 35 a8 c4 f0-db fc 33 a2 63 71 ca ca   b...5.....3.cq..
    0080 - d3 29 db 8e e0 c4 88 f1-82 8a 49 ec a0 76 20 90   .)........I..v .
    0090 - d8 8b d4 34 18 01 cb 4d-57 14 45 55 4a eb 00 98   ...4...MW.EUJ...


    Start Time: 1484561388
    Timeout   : 300 (sec)
    Verify return code: 0 (ok)
---

[Michael Klishin]

And what is in server logs?

[Michael Klishin]

Also, what does your connecting code look like?

On Monday, January 16, 2017 at 1:57:36 PM UTC+3, jan.goyvaerts wrote:

[jan.go...@enervalis.com]

The sasl log is empty, the default log

=INFO REPORT==== 16-Jan-2017::13:25:15 ===
Starting RabbitMQ 3.6.6 on Erlang 19.2
Copyright (C) 2007-2016 Pivotal Software, Inc.
Licensed under the MPL.  See http://www.rabbitmq.com/


=INFO REPORT==== 16-Jan-2017::13:25:15 ===
node           : rabbit@black
home dir       : /var/lib/rabbitmq
config file(s) : /etc/rabbitmq/rabbitmq.config
cookie hash    : rpDRsQVxXm0hUzR6qjFO7A==
log            : /var/log/rabbitmq/rabbit@black.log
sasl log       : /var/log/rabbitmq/rabbit@black-sasl.log
database dir   : /var/lib/rabbitmq/mnesia/rabbit@black


=INFO REPORT==== 16-Jan-2017::13:25:15 ===
Memory limit set to 6347MB of 15868MB total.


=INFO REPORT==== 16-Jan-2017::13:25:15 ===
Disk free limit set to 50MB


=INFO REPORT==== 16-Jan-2017::13:25:15 ===
Limiting to approx 924 file handles (829 sockets)


=INFO REPORT==== 16-Jan-2017::13:25:15 ===
FHC read buffering:  OFF
FHC write buffering: ON


=INFO REPORT==== 16-Jan-2017::13:25:16 ===
Priority queues enabled, real BQ is rabbit_variable_queue


=INFO REPORT==== 16-Jan-2017::13:25:16 ===
Starting rabbit_node_monitor


=INFO REPORT==== 16-Jan-2017::13:25:16 ===
msg_store_transient: using rabbit_msg_store_ets_index to provide index


=INFO REPORT==== 16-Jan-2017::13:25:16 ===
msg_store_persistent: using rabbit_msg_store_ets_index to provide index


=INFO REPORT==== 16-Jan-2017::13:25:16 ===
started TCP Listener on [::]:5672


=INFO REPORT==== 16-Jan-2017::13:25:16 ===
started SSL Listener on 127.0.0.1:5671


=INFO REPORT==== 16-Jan-2017::13:25:16 ===
Server startup complete; 0 plugins started.

[jan.go...@enervalis.com]

openssl command line

$ openssl s_client -pass "pass:CfYD5GKdE9tcItAK6QdHgCPCDyWxoC" -connect localhost:5671 -cert ../certs/client.cert.pem -key ../private/client.key.pem -CAfile ./ca.cert.pem

Java client code

  @Test
  public void testMakeConnectionWithCertificate() throws Exception {
    final String trustPassword = "***********************";
    final String clientPassword = "*************************";

    // Load the client keystore
    final KeyStore ks = KeyStore.getInstance("PKCS12");
    ks.load(new FileInputStream("/opt/dev/certificate/ca/intermediate/amqp/client.p12"), clientPassword.toCharArray());
    final KeyManagerFactory kmf = KeyManagerFactory.getInstance("SunX509");
    kmf.init(ks, clientPassword.toCharArray());

    // Load the trust store
    final KeyStore tks = KeyStore.getInstance("JKS");
    tks.load(new FileInputStream("/opt/dev/certificate/ca/intermediate/amqp/trust.jks"), trustPassword.toCharArray());
    final TrustManagerFactory tmf = TrustManagerFactory.getInstance("SunX509");
    tmf.init(tks);

    // initialize SSL with the various keys and shit
    SSLContext c = SSLContext.getInstance("tlsv1.2");
    c.init(kmf.getKeyManagers(), tmf.getTrustManagers(), null);

    final ConnectionFactory factory = new ConnectionFactory();
    factory.setHost("localhost");
    factory.setPort(5671);
    factory.useSslProtocol(c);
    factory.setHandshakeTimeout(180000);

    Connection connection = null;
    try {
      connection = factory.newConnection();
    } finally {
      if (connection != null) connection.close();
    }
  }

[jan.go...@enervalis.com]

Maybe this sheds some light on the problem ?

 

openssl command line

$ openssl s_client -pass "pass:CfYD5GKdE9tcItAK6QdHgCPCDyWxoC" -connect localhost:5671 -cert ../certs/client.cert.pem -key ../private/client.key.pem -CAfile ./ca.cert.pem

 

CONNECTED(00000003)
turning on non blocking io
SSL_connect:before/connect initialization
SSL_connect:SSLv2/v3 write client hello A
SSL_connect:error in SSLv2/v3 read server hello A
write R BLOCK

 

[Michael Klishin]

There are no inbound connections as far as RabbitMQ is concerned.

With nearly every other connection issue and TLS upgrades specifically,

there should be at least some kind of log entries, even if only in the SASL log

(which logs unhandled exceptions).

So I'd do a traffic capture and see if there may be a firewall involved.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ

[Jan Goyvaerts]

Thought about that too - apparmor is removed and there are no iptables rules active. I just build the latest openssl from source to make sure.

It's all on the same host anyway. So there can't be much in the way ! :-)

Netstat indicates established connections. So *something* picks op the phone, or is it ?

To post to this group, send email to rabbitm...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ

--
You received this message because you are subscribed to a topic in the Google Groups "rabbitmq-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/rabbitmq-users/HeBCYVa3eFM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to rabbitmq-users+unsubscribe@googlegroups.com.

To post to this group, send email to rabbitmq-users@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Jan Goyvaerts

Senior Software Engineer

jan.go...@enervalis.com

+32 (0) 11 39 75 79

[Michael Klishin]

This recent Erlang/OTP change looks relevant: https://github.com/erlang/otp/commit/1364c7308e17d43d1a2244e3f2bf11cfec3789ef.

--

Jan Goyvaerts

Senior Software Engineer

jan.go...@enervalis.com

+32 (0) 11 39 75 79

--

You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.

To post to this group, send email to rabbitmq-users@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

[Daniel Gong]

I am having the same issue here. Have you found out what the issue was? 

Thanks

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.

To post to this group, send email to rabbitm...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ

--
You received this message because you are subscribed to a topic in the Google Groups "rabbitmq-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/rabbitmq-users/HeBCYVa3eFM/unsubscribe.

To unsubscribe from this group and all its topics, send an email to rabbitmq-user...@googlegroups.com.

To post to this group, send email to rabbitm...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

[Jan Goyvaerts]

Hey Daniel,

We gave up on it because we couldn't make it work. We're using HAProxy in front of RabbitMQ now. 

Maybe that's a solution for you too ?

Regards,

Jan

To unsubscribe from this group and all its topics, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Daniel Gong]

Thanks Jan. I finally found out what the problem was for us. We had a typo in our deployment script for the passphrase of the SSL used by the server. The nodejs rabbitmq client (amqp & amqplib) we use just didn't show any error or log for this type of errors. Hope this could help other people with similar problems in the future.