Unable to connect to RabbitMQ server from phpAmqpLib with TLS

[Romain WeMove]

Hello,

I am having trouble connecting to RabbitMQ with TLS using a phpAmqpLib client even though it works fine from a bunny client, using the same key and certificate.

My set-up is the following:

Server

RabbitMQ v3.7.15 deployed in docker

SSL listener on port 5671 restricted to protocol TLSv1.2 without terminating proxy

Peer verification enabled and client certificate required

All the ciphers listed by rabbitmq-diagnostics are allowed

Letsencrypt certificate

Client

PhpAmqpLib v2.9.2 with PHP 7.1

Wildcard certificate from a trusted company

N.B. I had to comment one line of code in the library (StreamIO.php:96) to get "meaningful" information about the error, otherwise it just reports "Unknown error"...

Connection code:

 37 new AMQPSSLConnection(
 38     AMQP_HOST, AMQP_PORT,
 39     AMQP_USER, AMQP_PASSWORD, AMQP_VHOST,
 40     array(
 41       'local_cert' => PATH_TO_SSL_CERT,
 42       'local_pk' => PATH_TO_SSL_KEY,
 43       'cafile' => '/etc/ssl/certs/ca-certificates.crt',
 44  //     'verify_peer' => FALSE,
 45  //     'verify_peer_name' => FALSE,
 46  //     'ciphers' => 'DEFAULT:!TLSv1.0:!SSLv3',
 48     ),
 49     array(),
 50     'ssl'       // 'tls', 'tlsv1.2'
 51   );

The client and the server are on different machines, but have the same CA file, which is the default Ubuntu store appended with intermediate certificates of Letsencrypt and the Wildcard certificate provider.

Error

PHP Warning:  stream_socket_client(): SSL operation failed with code 1. OpenSSL Error messages:
error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure in /var/www/civi.custom/extensions/mailjet/amqp/vendor/php-amqplib/php-amqplib/PhpAmqpLib/Wire/IO/StreamIO.php on line 105
PHP Warning:  stream_socket_client(): Failed to enable crypto in /var/www/civi.custom/extensions/mailjet/amqp/vendor/php-amqplib/php-amqplib/PhpAmqpLib/Wire/IO/StreamIO.php on line 105
PHP Warning:  stream_socket_client(): unable to connect to ssl://-----.---------.--:5671 (Unknown error)

On the server side, nothing much in logs (log level set to debug for console):

2019-08-13 09:48:50.201 [debug] <0.4091.0> Supervisor {<0.4091.0>,rabbit_connection_sup} started rabbit_connection_helper_sup:start_link() at pid <0.4092.0>
2019-08-13 09:48:50.201 [debug] <0.4091.0> Supervisor {<0.4091.0>,rabbit_connection_sup} started rabbit_reader:start_link(<0.4092.0>, {acceptor,{0,0,0,0,0,0,0,0},5671}) at pid <0.4093.0>

As you can see, the client seems to try to negociate a SSLv3 handshake, even though the server allows only TLSv1.2

I've tried to specify the protocol to 'tls', but it gives a protocol error because it tries to negociate a TLSv1.0 handshake, or specify 'tlsv1.2' (which is listed as a valid protocol by "php -i") but it gives the same "sslv3 alert handshake failure". So I'm not sure this is actually a SSLv3 or TLSv1.2 that's being negociated...

I've also tried limiting ciphers to the ones accepted by TLSv1.2 (see line 46), same error message.

I've tried disabling peer verification on client side, no difference.

I've also tried bundling the client key and certificate in the same file, as I've seen in (old) forums this can solve SSL problems with PHP, no difference.

I've also tried the following tests to find the root cause:

 - phpAmqpLib -> openssl s_server -tlsv1_2 (with same certificate as rabbitmq, ran on docker host) => Connect OK (and dies with no hearbeat from server)

 - openssl s_client (with same certificate as phpAmqpLib) -> RabbitMQ  => Connect OK on TLSv1.2 (and server closes the connection because no AMQP input)

 - bunny (with same certificate, from another machine) -> RabbitMQ => Connect OK

These tests seem to show that both the php client and server are configured correctly as both are able to establish connections, yet they don't want to talk to each other :(

Any idea what I might be missing?

Thanks for your suggestions
Romain

[Luke Bakken]

Hi Romain -

Your tests using openssl s_server and s_client show that this is probably not a RabbitMQ issue nor an issue with your certs.  I did a search on "php failed to enable crypto" and that seems to be a common PHP error that is caused by a wide range of issues.

I recommend modifying this code to connect to RabbitMQ - https://github.com/php-http/socket-client/blob/master/tests/server/tcp-ssl-server-client.php

You should be able to get more information from a test client like the above.

We do have some sample openssl s_client usage here that enables host name verification - https://www.rabbitmq.com/troubleshooting-ssl.html#openssl-tools

Maybe there is something up with how PHP validates a wildcard certificate (?)

Thanks,

Luke

[Romain WeMove]

Thanks for the answer, Luke.

I am not able to get additional information about the problem, even with a test client. This is what I wrote:

<?php
$context = stream_context_create([
    'ssl' => [
        'local_cert' => 'key-and-cert.pem',
        'cafile' => '/etc/ssl/certs/ca-certificates.crt',
        'capture_peer_cert' => true,
    ],
]);

$errno = NULL;
$errstr = NULL;
$sock = stream_socket_client(
    'tcp://-----.-----------.--:5671',
    $errno,
    $errstr,
    3,
    STREAM_CLIENT_CONNECT,
    $context
);

stream_set_blocking($sock, true);
stream_socket_enable_crypto($sock, true, STREAM_CRYPTO_METHOD_TLSv1_2_CLIENT);

I get the same error message. Thanks to the capture_peer_cert option I am able to display the received certificate, which tells me the failure happens after receiving it but still doesn't tell me why.

I'll continue searching...

Romain

[Luke Bakken]

Hi Romain,

Thanks for taking the time to test with that code. If you don't limit the SSL/TLS version in RabbitMQ or in this test code, does it work?

[Romain WeMove]

Le mardi 13 août 2019 23:08:44 UTC+2, Luke Bakken a écrit :

Hi Romain,

Thanks for taking the time to test with that code. If you don't limit the SSL/TLS version in RabbitMQ or in this test code, does it work?

No, it still doesn't work if I don't limit the version, I get the same error message.

But I finally found something I think: the full chain of certificates of the client has 4 certificates. In the client code I was sending the full chain because when I was setting up another client some time ago this is what solved my problems (because the intermediate certificates were not trusted by the server). Now I tried my luck by sending a truncated chain with only the client certificate and the first intermediate, and this worked! Both in the test code and with the actual AMQP PHP client.

Am I right that this sounds like a bug in the PHP bindings of OpenSSL, or am I misunderstanding something about how SSL works? Sending the full chain does work from openssl s_client and from the bunny client. I have also seen online that it is recommended to send the full chain because the root is more likely to be trusted. And in my case it works only because I have added the intermediate CA certificates to the default Ubuntu store, otherwise it couldn't work I believe. I'd be really surprised to have found a bug in such a wildly used library, but can't see any other explanation...

Anyway, thanks for providing support and suggesting different leads. I was starting to feel really desperate!

Romain

[Luke Bakken]

Hi Romain,

That is an unexpected solution, for sure. On your RabbitMQ server, have you concatenated all of these certificates into a single file that is used as the cacertfile configuration argument, or just the first intermediate?

[Romain WeMove]

Hi Luke,

On both the server and the client, I use the same single cacertfile with all the certificates. I in fact use the CA store of the host system ('/etc/ssl/certs/ca-certificates.crt'), which contains the default list of certificates of Ubuntu and some custom certificates (that I have added with the command update-ca-certificates): Letsencrypt and Sectigo intermediate certificates (Sectigo is the provider of the wildcard certificate used by the client).

Could my situation somehow be because I have added intermediate certificates to the store and RabbitMQ always trusts only the root of a chain (I've read this in the documentation)?

Given the results of the tests I've made with different implementations, it looks like the problem comes from a combination of some SSL implementation details of both PHP and RabbitMQ.

Romain

[Luke Bakken]

Hi Romain,

The Erlang TLS implementation is a bit "unusual" in that it requires that clients send the entire certificate chain when client certificate authentication is enabled. At least, the last time I tested with intermediaries that is what I have seen.

In your case I would recommend configuring RabbitMQ to use a cacertfile containing only the necessary certs and not use the entire host's CA store.

If you can generate a set of certificates that I could use to test maybe we could get to the bottom of this. At the moment the RabbitMQ team tests with certs generated by this project: https://github.com/michaelklishin/tls-gen

Thanks,

Luke

[Romain WeMove]

Hi Luke,

Today I followed your suggestion to do some tests with self-signed certificates, and I must say I am even more confused because I did not reproduce what I saw on my production server. I get similar troubles to connect from a chained CA, but I am not able to solve them in the same manner.

What I did is:

•  Use tls-gen to generate certificates in 2 profiles: basic and two_shared_intermediates. The server is using the cert and key from basic but trusting the CA for two_shared. The client is using cert and key from two_shared and not doing peer verification (to simplify the tests).
• I've compared results from PHP client (a simple script like the one mentioned earlier) and openssl s_client, and from servers RabbitMQ and openssl s_server

Here is a recap of the results (I hope each line is self-explanatory of what set-up is being tested, let me know if not):

1. basic CA PHP         => basic CA rabbit                       : OK
2. chained CA PHP       => basic CA rabbit trusting whole chain  : KO!
3. short chained CA PHP => basic CA rabbit trusting whole chain  : KO!
4. chained CA PHP       => basic CA rabbit trusting only root    : KO!
5. chained CA PHP       => basic s_server trusting whole chain   : OK
6. chained CA PHP       => basic s_server trusting only root     : OK
7. chained CA s_client  => basic CA rabbit trusting only root    : KO!
8. chained CA s_client  => basic CA rabbit trusting whole chain  : KO!

Test 1 was just to give me confidence I'm doing everything right.

What puzzles me is that from what I found last week, I should have got a successful connection for tests 3 and 8.

Do you want me to attach all the keys and certificates I used?

Romain

[Luke Bakken]

Hi Romain,

When you use tls-gen to generate certs in different profiles (basic and two_shared_intermediates), they have completely different trust chains ending with two different self-signed root certs.

I have a question about how you set up test #2, 3, 5 and 8 -

2. chained CA PHP       => basic CA rabbit trusting whole chain  : KO!

When you say "basic CA rabbit trusting whole chain" did you concatenate the root certificates from both profiles? Like this:

$ cat basic/result/ca_certificate.pem \

    two_shared_intermediates/result/chained_ca_certificate.pem > ca_certificates.pem

To reproduce your test #8, I did the above and used the attached rabbitmq.config file with RabbitMQ 3.7.17 and Erlang 22.0.7. Notice that I enabled all available ciphers (https://www.rabbitmq.com/ssl.html#cipher-suites)

I then ran this command, which works successfully:

openssl s_client -connect localhost:5671 \

    -cert ./two_shared_intermediates/result/client_certificate.pem \

    -key ./two_shared_intermediates/result/client_key.pem \

    -CAfile ./two_shared_intermediates/result/chained_ca_certificate.pem -verify 16 -verify_hostname shostakovich

Would you mind taking a second to review my test and point out what is different in your #8 test scenario?

Thanks,

Luke

[Romain WeMove]

Hi Luke, thanks for taking a look at the tests.

My answers below

Le samedi 24 août 2019 03:05:26 UTC+2, Luke Bakken a écrit :

Hi Romain,

When you use tls-gen to generate certs in different profiles (basic and two_shared_intermediates), they have completely different trust chains ending with two different self-signed root certs.

Yes, I thought this would be the closest setup to my real setup (client and server having certs from different CAs)

 

I have a question about how you set up test #2, 3, 5 and 8 -

2. chained CA PHP       => basic CA rabbit trusting whole chain  : KO!

When you say "basic CA rabbit trusting whole chain" did you concatenate the root certificates from both profiles? Like this:

$ cat basic/result/ca_certificate.pem \

    two_shared_intermediates/result/chained_ca_certificate.pem > ca_certificates.pem

No, by trusting whole chain I mean trusting intermediate certificates, as opposed to only the root.So that's exactly the file two_shared_intermediates/result/chained_ca_certificate.pem and nothing else. I assume the CA cert from basic profile is not needed by the server, since it does not need the CA cert of its own CA but only of the clients that we want to trust. Right?

 

To reproduce your test #8, I did the above and used the attached rabbitmq.config file with RabbitMQ 3.7.17 and Erlang 22.0.7. Notice that I enabled all available ciphers (https://www.rabbitmq.com/ssl.html#cipher-suites)

I then ran this command, which works successfully:

openssl s_client -connect localhost:5671 \

    -cert ./two_shared_intermediates/result/client_certificate.pem \

    -key ./two_shared_intermediates/result/client_key.pem \

    -CAfile ./two_shared_intermediates/result/chained_ca_certificate.pem -verify 16 -verify_hostname shostakovich

Would you mind taking a second to review my test and point out what is different in your #8 test scenario?

There are 2 differences:

 - The cert option points to a file that is the concatenation of client_certificate.pem and chained_ca_certificate.pem. This is following the advice that the client should send the whole chain of certificates.

 - The CAfile option points to basic/result/ca_certificate.pem, because the key and certificate of server are generated with the basic profile.

In a nutshell, what I did differently is using different tls-gen profiles for the client and server, to mimic my situation where the client is certified by Sectigo and the server by Letsencrypt. Here is the stderr output of the s_client command:

verify depth is 16
depth=1 CN = TLSGenSelfSignedtRootCA, L = $$$$
verify return:1
depth=0 CN = seaview, O = server
verify return:1
139978651051672:error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:s3_pkt.c:1487:SSL alert number 40
139978651051672:error:140790E5:SSL routines:ssl23_write:ssl handshake failure:s23_lib.c:177:

I hope this is a step towards final understanding! :)

Romain

[Luke Bakken]

Hi Romain,

What version of Erlang and what operating system are you using? What version of openssl?

 

I have a question about how you set up test #2, 3, 5 and 8 -

2. chained CA PHP       => basic CA rabbit trusting whole chain  : KO!

When you say "basic CA rabbit trusting whole chain" did you concatenate the root certificates from both profiles? Like this:

$ cat basic/result/ca_certificate.pem \

    two_shared_intermediates/result/chained_ca_certificate.pem > ca_certificates.pem

No, by trusting whole chain I mean trusting intermediate certificates, as opposed to only the root.So that's exactly the file two_shared_intermediates/result/chained_ca_certificate.pem and nothing else. I assume the CA cert from basic profile is not needed by the server, since it does not need the CA cert of its own CA but only of the clients that we want to trust. Right?

Thanks for the explanation. The use of "basic CA" confused me. I thought you meant you were using the CA from the "basic" tls-gen profile as well as the ones from "two_shared_intermediates".

If you look at the two_shared_intermediates/result/chained_ca_certificate.pem file, you will see that it includes three certificates - the root cert and the two intermediates from that profile.

There are 2 differences:

 - The cert option points to a file that is the concatenation of client_certificate.pem and chained_ca_certificate.pem. This is following the advice that the client should send the whole chain of certificates.

If I do that, I get an error:

$ cat chained_ca_certificate.pem client_certificate.pem > client_and_chained_certificates.pem

$ cd ../..

$ openssl s_client -connect shostakovich:5671 \

    -cert ./two_shared_intermediates/result/client_and_chained_certificates.pem \

    -key ./two_shared_intermediates/result/client_key.pem \

    -CAfile ./basic/result/ca_certificate.pem -verify 16 -verify_hostname shostakovich

verify depth is 16

error setting private key

139818357163200:error:0B080074:x509 certificate routines:X509_check_private_key:key values mismatch:crypto/x509/x509_cmp.c:294:

 

 - The CAfile option points to basic/result/ca_certificate.pem, because the key and certificate of server are generated with the basic profile.

OK. When I run a test like yours (but using the original client_certificate.pem file), it succeeds. Please see my rabbitmq.config and output attached to this message.

 

openssl s_client -connect shostakovich:5671 \

    -cert ./two_shared_intermediates/result/client_certificate.pem \

    -key ./two_shared_intermediates/result/client_key.pem \

    -CAfile ./basic/result/ca_certificate.pem -verify 16 -verify_hostname shostakovich

The above command works with the default ciphers enabled. In other words, no ciphers section in rabbitmq.config

In your case, please try enabling all cipher suites in your rabbitmq.config file.

Thanks,

Luke

[Romain WeMove]

Hello Luke,

Le lundi 26 août 2019 15:54:36 UTC+2, Luke Bakken a écrit :

Hi Romain,

What version of Erlang and what operating system are you using? What version of openssl?

On the server, I am running RabbitMQ in Docker. It is an image based on rabbitmq:3.7-management-alpine. From the Dockerfile, I get that:

• OS is Alpine Linux 3.10
• OpenSSL version is 1.1.1c
• Erlang/OTP version is 22.0.7

On the client, I am using Ubuntu 16.04.6, which comes with OpenSSL 1.0.2g

[...]

There are 2 differences:

 - The cert option points to a file that is the concatenation of client_certificate.pem and chained_ca_certificate.pem. This is following the advice that the client should send the whole chain of certificates.

If I do that, I get an error:

$ cat chained_ca_certificate.pem client_certificate.pem > client_and_chained_certificates.pem

$ cd ../..

$ openssl s_client -connect shostakovich:5671 \

    -cert ./two_shared_intermediates/result/client_and_chained_certificates.pem \

    -key ./two_shared_intermediates/result/client_key.pem \

    -CAfile ./basic/result/ca_certificate.pem -verify 16 -verify_hostname shostakovich

verify depth is 16

error setting private key

139818357163200:error:0B080074:x509 certificate routines:X509_check_private_key:key values mismatch:crypto/x509/x509_cmp.c:294:

I believe this is because the chain should be ordered by increasing depth, so the client certificate should be at the beginning of the file:

$ cat client_certificate.pem chained_ca_certificate.pem > client_and_chained_certificates.pem

 

 

 - The CAfile option points to basic/result/ca_certificate.pem, because the key and certificate of server are generated with the basic profile.

OK. When I run a test like yours (but using the original client_certificate.pem file), it succeeds. Please see my rabbitmq.config and output attached to this message.

 

openssl s_client -connect shostakovich:5671 \

    -cert ./two_shared_intermediates/result/client_certificate.pem \

    -key ./two_shared_intermediates/result/client_key.pem \

    -CAfile ./basic/result/ca_certificate.pem -verify 16 -verify_hostname shostakovich

The above command works with the default ciphers enabled. In other words, no ciphers section in rabbitmq.config

In the output you attached, I see the following error:

140308505486528:error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca:ssl/record/rec_layer_s3.c:1536:SSL alert number 48

This cannot be ignored, right? I believe the error is because the rabbitmq config should use the CA file from two_shared_intermediate profile, since this is what the client uses.

 

In your case, please try enabling all cipher suites in your rabbitmq.config file.

Tried just now, same result: 140149088355992:error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:s3_pkt.c:1487:SSL alert number 40

 

Thanks,

Romain

[Luke Bakken]

In the output you attached, I see the following error:

140308505486528:error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca:ssl/record/rec_layer_s3.c:1536:SSL alert number 48

This cannot be ignored, right? I believe the error is because the rabbitmq config should use the CA file from two_shared_intermediate profile, since this is what the client uses.

Good catch, I totally missed that. That error means that the server could not verify the trust chain for the presented client certificate, even though openssl should be sending the entire chain with the request.

What does work without error in my environment is to do the following:

lbakken@shostakovich ~/development/michaelklishin/tls-gen (master=)

$ cat basic/result/ca_certificate.pem two_shared_intermediates/result/chained_ca_certificate.pem > ca_certificates.pem

Then, configure RabbitMQ to use ca_certificates.pem as cacertfile. When you run openssl s_client, you can pass either two_shared_intermediates/result/client_certificate.pem or the file containing that cert first followed by the CA certs.

Output:

$ openssl s_client -connect shostakovich:5671 -cert ./two_shared_intermediates/result/client_and_chained_certificates.pem -key ./two_shared_intermediates/result/client_key.pem -CAfile ./basic/result/ca_certif

icate.pem -verify 16 -verify_hostname shostakovich

verify depth is 16

CONNECTED(00000003)

Can't use SSL_get_servername

depth=1 CN = TLSGenSelfSignedtRootCA, L = $$$$

verify return:1

depth=0 CN = shostakovich, O = server

verify return:1

---

Certificate chain

 0 s:CN = shostakovich, O = server

   i:CN = TLSGenSelfSignedtRootCA, L = $$$$

 1 s:CN = TLSGenSelfSignedtRootCA, L = $$$$

   i:CN = TLSGenSelfSignedtRootCA, L = $$$$

---

Server certificate

-----BEGIN CERTIFICATE-----

-----END CERTIFICATE-----

subject=CN = shostakovich, O = server

issuer=CN = TLSGenSelfSignedtRootCA, L = $$$$

---

Acceptable client certificate CA names

CN = TLSGenSelfSignedtRootCA, L = $$$$

CN = shostakovich, O = Intermediate CA 2

CN = TLSGenSelfSignedtRootCA, L = $$$$

CN = shostakovich, O = Intermediate CA 1

Client Certificate Types: ECDSA sign, RSA sign, DSA sign

Requested Signature Algorithms: ECDSA+SHA512:RSA+SHA512:ECDSA+SHA384:RSA+SHA384:ECDSA+SHA256:RSA+SHA256:ECDSA+SHA224:RSA+SHA224:ECDSA+SHA1:RSA+SHA1:DSA+SHA1

Shared Requested Signature Algorithms: ECDSA+SHA512:RSA+SHA512:ECDSA+SHA384:RSA+SHA384:ECDSA+SHA256:RSA+SHA256:ECDSA+SHA224:RSA+SHA224:ECDSA+SHA1:RSA+SHA1:DSA+SHA1

Peer signing digest: SHA256

Peer signature type: RSA

Server Temp Key: ECDH, P-256, 256 bits

---

SSL handshake has read 2506 bytes and written 1480 bytes

Verification: OK

Verified peername: shostakovich

---

New, TLSv1.2, Cipher is ECDHE-RSA-AES256-GCM-SHA384

Server public key is 2048 bit

Secure Renegotiation IS supported

Compression: NONE

Expansion: NONE

No ALPN negotiated

SSL-Session:

    Protocol  : TLSv1.2

    Cipher    : ECDHE-RSA-AES256-GCM-SHA384

    Session-ID: A44F97B0B52281B3578E51C1262915687DBCE980A882E9DF4812E9B49E3DEA53

    Session-ID-ctx:

    Master-Key: 1FA6134EA6A1FC673002287CC4B0F739B34E6303DFACE3CC6C934D3D6BB034A038AAD7852A61B79F6596F65E79FAD2FA

    PSK identity: None

    PSK identity hint: None

    SRP username: None

    Start Time: 1566917973

    Timeout   : 7200 (sec)

    Verify return code: 0 (ok)

    Extended master secret: no

---

^C

[Romain WeMove]

Hey Luke,

Coming back to this after being out a for a few days.

I finally found what was the difference between our set-ups after this almost drove me crazy: I didn't notice the option {depth, 16} in rabbitmq config.
Only after I also added this option I was able to connect a s_client to RabbitMQ without error.

I had no added this option at all since in my understanding, the standard behaviour is that no specified depth means unlimited depth. Is that different for RabbitMQ / Erlang?

With that option, I am also able to connect with the small PHP script I mentioned earlier in the conversation, using either the full chain and the truncated chain for the local_cert option. It looks like this will solve my initial problem, but I will have to spend some time setting up a proper test environment to confirm.

Thanks,

Romain

[Romain WeMove]

I confirm that this solves my initial problem!

I checked again the documentation of RabbitMQ, there is no mention of a default depth. Might be a good idea to add this information to the documentation. Is there a better place than here to report it?

Thanks again Luke for your time spent on this.

Romain

[Luke Bakken]

Hi Romain,

Thank you very much for your diligence in figuring out the root cause of this issue.

TLS verification depth is mentioned here but the default is not - https://www.rabbitmq.com/ssl.html#peer-verification-depth

A user would have to look up the default value here - http://erlang.org/doc/man/ssl.html#type-allowed_cert_chain_length

If you'd like to file an issue, or open a pull request to clarify our docs, you can do so here - https://github.com/rabbitmq/rabbitmq-website/

This is the section that would be updated, I think - https://github.com/rabbitmq/rabbitmq-website/blob/live/site/ssl.md#peer-verification-depth

Thanks!

Luke