SSL client authorization level of trust

[Wout Slakhorst]

We're using SSL client authorization in our rabbitmq config. We're having problems connecting with our Java clients, which is caused by the server certificate send by rabbitmq. Its Certificate is signed by a CA which is not in the default Java truststore. The root of the certifcate/CA, however, is present in the truststore. Is there any way to have rabbitmq send a longer chain of trust when sending the server certificate??

The alternative is to change the client to trust everyone, which is a less secure option IMHO....

[Michael Klishin]

On 27 October 2014 at 12:02:49, Wout Slakhorst (wout.sl...@gmail.com) wrote:
> We're using SSL client authorization in our rabbitmq config.
> We're having problems connecting with our Java clients, which
> is caused by the server certificate send by rabbitmq. Its Certificate
> is signed by a CA which is not in the default Java truststore. The
> root of the certifcate/CA, however, is present in the truststore.
> Is there any way to have rabbitmq send a longer chain of trust when
> sending the server certificate??

Please post what `openssl s_client` outputs with -verify 1 and -verify 8.
Increasing validation depth in the client may be sufficient.

> The alternative is to change the client to trust everyone, which
> is a less secure option IMHO

There is no need to "trust everyone". You can add add the CA which was used to sign the server certificate to the trust store. 1 extra CA (which you trust based on the information provided) is a bit more secure than "everyone". 
--
MK

Staff Software Engineer, Pivotal/RabbitMQ

[Wout Slakhorst]

As you can see, stunnel sends the whole chain which allows the client to find the right ca. Rabbitmq is just sending the server certificate.

Verify 8, rabbitmq

verify depth is 8

CONNECTED(00000003)

depth=0 /C=NL/ST=Gelderland/L=Groenlo/O=N.V. Nederlandsche Apparatenfabriek Nedap/OU=Healthcare/CN=*.ons.io

verify error:num=20:unable to get local issuer certificate

verify return:1

depth=0 /C=NL/ST=Gelderland/L=Groenlo/O=N.V. Nederlandsche Apparatenfabriek Nedap/OU=Healthcare/CN=*.ons.io

verify error:num=27:certificate not trusted

verify return:1

depth=0 /C=NL/ST=Gelderland/L=Groenlo/O=N.V. Nederlandsche Apparatenfabriek Nedap/OU=Healthcare/CN=*.ons.io

verify error:num=21:unable to verify the first certificate

verify return:1

5892:error:14094418:SSL routines:SSL3_READ_BYTES:tlsv1 alert unknown ca:/SourceCache/OpenSSL098/OpenSSL098-52/src/ssl/s3_pkt.c:1125:SSL alert number 48

5892:error:140790E5:SSL routines:SSL23_WRITE:ssl handshake failure:/SourceCache/OpenSSL098/OpenSSL098-52/src/ssl/s23_lib.c:182:

Verify 1, rabbitmq

verify depth is 1

CONNECTED(00000003)

depth=0 /C=NL/ST=Gelderland/L=Groenlo/O=N.V. Nederlandsche Apparatenfabriek Nedap/OU=Healthcare/CN=*.ons.io

verify error:num=20:unable to get local issuer certificate

verify return:1

depth=0 /C=NL/ST=Gelderland/L=Groenlo/O=N.V. Nederlandsche Apparatenfabriek Nedap/OU=Healthcare/CN=*.ons.io

verify error:num=27:certificate not trusted

verify return:1

depth=0 /C=NL/ST=Gelderland/L=Groenlo/O=N.V. Nederlandsche Apparatenfabriek Nedap/OU=Healthcare/CN=*.ons.io

verify error:num=21:unable to verify the first certificate

verify return:1

5897:error:14094418:SSL routines:SSL3_READ_BYTES:tlsv1 alert unknown ca:/SourceCache/OpenSSL098/OpenSSL098-52/src/ssl/s3_pkt.c:1125:SSL alert number 48

5897:error:140790E5:SSL routines:SSL23_WRITE:ssl handshake failure:/SourceCache/OpenSSL098/OpenSSL098-52/src/ssl/s23_lib.c:182:

Verify 8, stunnel

CONNECTED(00000003)

depth=2 /C=US/O=GeoTrust Inc./CN=GeoTrust Global CA

verify error:num=20:unable to get local issuer certificate

verify return:1

depth=2 /C=US/O=GeoTrust Inc./CN=GeoTrust Global CA

verify error:num=27:certificate not trusted

verify return:1

depth=1 /C=US/O=GeoTrust Inc./CN=GeoTrust SSL CA - G2

verify return:1

depth=0 /C=NL/ST=Gelderland/L=Groenlo/O=N.V. Nederlandsche Apparatenfabriek Nedap/OU=Healthcare/CN=*.ons.io

verify return:1

5915:error:14094418:SSL routines:SSL3_READ_BYTES:tlsv1 alert unknown ca:/SourceCache/OpenSSL098/OpenSSL098-52/src/ssl/s3_pkt.c:1125:SSL alert number 48

5915:error:140790E5:SSL routines:SSL23_WRITE:ssl handshake failure:/SourceCache/OpenSSL098/OpenSSL098-52/src/ssl/s23_lib.c:182:

Verify 1, stunnel

CONNECTED(00000003)

depth=2 /C=US/O=GeoTrust Inc./CN=GeoTrust Global CA

verify error:num=20:unable to get local issuer certificate

verify return:0

5919:error:14094418:SSL routines:SSL3_READ_BYTES:tlsv1 alert unknown ca:/SourceCache/OpenSSL098/OpenSSL098-52/src/ssl/s3_pkt.c:1125:SSL alert number 48

5919:error:140790E5:SSL routines:SSL23_WRITE:ssl handshake failure:/SourceCache/OpenSSL098/OpenSSL098-52/src/ssl/s23_lib.c:182:

[Michael Klishin]

 On 28 October 2014 at 11:45:49, Wout Slakhorst (wout.sl...@gmail.com) wrote:
> As you can see, stunnel sends the whole chain which allows the
> client to find the right ca. Rabbitmq is just sending the server
> certificate.

Got it. Then simply concatenate the certificates [1] and use the concatenated
file as CA certificate.

1. cat root_ca.pem rabbitmq_ca.pem > concatenated_ca.pem

[Wout Slakhorst]

We have all the certificates except the root in the pem file, but still rabbitmq only sends the first one?

[Michael Klishin]

On 29 October 2014 at 11:46:32, Wout Slakhorst (wout.sl...@gmail.com) wrote:
> We have all the certificates except the root in the pem file,
> but still rabbitmq only sends the first one?

It's the Erlang ssl app that sends certificates. I see no evidence of
"only 1 certificate being sent" from your s_client output. I also could not find
anything (e.g. bug reports or questions on erlang-questions) that suggests the ssl app does not send the full chain. The docs [1] mention
that

«The server will always send a certificate chain as part of the TLS handshake»

Can you reproduce the issue with a certificate chain generated with tls-gen [2]?

We are a little jammed right now but having a way to reproduce with tls-gen-produced
certificates would be very helpful. 

1. http://www.erlang.org/doc/apps/ssl/ssl_protocol.html
2. https://github.com/michaelklishin/tls-gen/

[Wout Slakhorst]

This is what my java ssl debug output says:

trigger seeding of SecureRandom

done seeding SecureRandom

Ignoring unsupported cipher suite: TLS_DHE_DSS_WITH_AES_128_CBC_SHA256

Ignoring unsupported cipher suite: TLS_DHE_DSS_WITH_AES_256_CBC_SHA256

Ignoring unsupported cipher suite: TLS_DHE_RSA_WITH_AES_128_CBC_SHA256

Ignoring unsupported cipher suite: TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256

Ignoring unsupported cipher suite: TLS_DHE_RSA_WITH_AES_256_CBC_SHA256

Ignoring unsupported cipher suite: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384

Ignoring unsupported cipher suite: TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384

Ignoring unsupported cipher suite: TLS_RSA_WITH_AES_256_CBC_SHA256

Ignoring unsupported cipher suite: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256

Ignoring unsupported cipher suite: TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384

Ignoring unsupported cipher suite: TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384

Ignoring unsupported cipher suite: TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256

Ignoring unsupported cipher suite: TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA256

Ignoring unsupported cipher suite: TLS_RSA_WITH_AES_128_CBC_SHA256

Allow unsafe renegotiation: false

Allow legacy hello messages: true

Is initial handshake: true

Is secure renegotiation: false

main, setSoTimeout(10000) called

%% No cached client session

*** ClientHello, TLSv1

RandomCookie:  GMT: 1414515210 bytes = { 188, 187, 174, 169, 1, 204, 187, 153, 242, 64, 211, 239, 198, 34, 198, 165, 106, 206, 66, 37, 104, 1, 204, 141, 216, 152, 113, 105 }

Session ID:  {}

Cipher Suites: [TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_256_CBC_SHA, TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDH_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_DSS_WITH_AES_256_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDH_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_DSS_WITH_AES_128_CBC_SHA, TLS_ECDHE_ECDSA_WITH_RC4_128_SHA, TLS_ECDHE_RSA_WITH_RC4_128_SHA, SSL_RSA_WITH_RC4_128_SHA, TLS_ECDH_ECDSA_WITH_RC4_128_SHA, TLS_ECDH_RSA_WITH_RC4_128_SHA, TLS_ECDHE_ECDSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA, SSL_RSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDH_ECDSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDH_RSA_WITH_3DES_EDE_CBC_SHA, SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA, SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA, SSL_RSA_WITH_RC4_128_MD5, TLS_EMPTY_RENEGOTIATION_INFO_SCSV]

Compression Methods:  { 0 }

Extension elliptic_curves, curve names: {secp256r1, sect163k1, sect163r2, secp192r1, secp224r1, sect233k1, sect233r1, sect283k1, sect283r1, secp384r1, sect409k1, sect409r1, secp521r1, sect571k1, sect571r1, secp160k1, secp160r1, secp160r2, sect163r1, secp192k1, sect193r1, sect193r2, secp224k1, sect239k1, secp256k1}

Extension ec_point_formats, formats: [uncompressed]

***

main, WRITE: TLSv1 Handshake, length = 163

main, READ: TLSv1 Handshake, length = 87

*** ServerHello, TLSv1

RandomCookie:  GMT: 1414515210 bytes = { 55, 227, 15, 230, 51, 0, 106, 45, 33, 4, 247, 217, 195, 169, 21, 24, 164, 142, 30, 118, 31, 246, 252, 186, 99, 221, 244, 238 }

Session ID:  {86, 252, 24, 169, 166, 111, 142, 184, 108, 134, 44, 251, 188, 239, 11, 13, 148, 217, 243, 86, 111, 14, 182, 120, 14, 130, 252, 173, 17, 98, 72, 108}

Cipher Suite: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA

Compression Method: 0

Extension ec_point_formats, formats: [uncompressed]

Extension renegotiation_info, renegotiated_connection: <empty>

***

%% Initialized:  [Session-2, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA]

** TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA

main, READ: TLSv1 Handshake, length = 1257

*** Certificate chain

chain [0] = [

[

  Version: V3

  Subject: CN=*.ons.io, OU=Healthcare, O=N.V. Nederlandsche Apparatenfabriek Nedap, L=Groenlo, ST=Gelderland, C=NL

  Signature Algorithm: SHA1withRSA, OID = 1.2.840.113549.1.1.5

  Key:  Sun RSA public key, 2048 bits

  modulus: 24733279816742425603140427742696885392892135048316280125381939536289707399188129076157843361930561639830698918690765836462442216790860434246660604536576624607976153761714908532811944611094767939749097886722901186444369544016031083003762612368583317332708234727700197913395678317621402335890347143490926717692644607097154921679627197707522080697182056609561900409021135127707701181016354606806416417596677155946323755729607921427065759450949394608227954619480013504425809820523520444350268184045738489421604852715001224952775442963071099700321954789025452277849110491227045082802901230247483878206128272701472048146539

  public exponent: 65537

  Validity: [From: Tue Mar 11 01:00:00 CET 2014,

               To: Sun Apr 10 01:59:59 CEST 2016]

  Issuer: CN=GeoTrust SSL CA - G2, O=GeoTrust Inc., C=US

  SerialNumber: [    521421a0 207159a6 96bd1384 98da1a5d]

Certificate Extensions: 8

[1]: ObjectId: 1.3.6.1.5.5.7.1.1 Criticality=false

AuthorityInfoAccess [

  [

   accessMethod: ocsp

   accessLocation: URIName: http://gtssl2-ocsp.geotrust.com

, 

   accessMethod: caIssuers

   accessLocation: URIName: http://gtssl2-aia.geotrust.com/gtssl2.cer

]

]

[2]: ObjectId: 2.5.29.35 Criticality=false

AuthorityKeyIdentifier [

KeyIdentifier [

0000: 11 4A D0 73 39 D5 5B 69   08 5C BA 3D BF 64 9A A8  .J.s9.[i.\.=.d..

0010: 8B 1C 55 BC                                        ..U.

]

]

[3]: ObjectId: 2.5.29.19 Criticality=false

BasicConstraints:[

  CA:false

  PathLen: undefined

]

[4]: ObjectId: 2.5.29.31 Criticality=false

CRLDistributionPoints [

  [DistributionPoint:

     [URIName: http://gtssl2-crl.geotrust.com/gtssl2.crl]

]]

[5]: ObjectId: 2.5.29.32 Criticality=false

CertificatePolicies [

  [CertificatePolicyId: [2.16.840.1.113733.1.7.54]

[PolicyQualifierInfo: [

  qualifierID: 1.3.6.1.5.5.7.2.1

  qualifier: 0000: 16 25 68 74 74 70 3A 2F   2F 77 77 77 2E 67 65 6F  .%http://www.geo

0010: 74 72 75 73 74 2E 63 6F   6D 2F 72 65 73 6F 75 72  trust.com/resour

0020: 63 65 73 2F 63 70 73                               ces/cps

]]  ]

]

[6]: ObjectId: 2.5.29.37 Criticality=false

ExtendedKeyUsages [

  serverAuth

  clientAuth

]

[7]: ObjectId: 2.5.29.15 Criticality=true

KeyUsage [

  DigitalSignature

  Key_Encipherment

]

[8]: ObjectId: 2.5.29.17 Criticality=false

SubjectAlternativeName [

  DNSName: *.ons.io

  DNSName: ons.io

]

]

  Algorithm: [SHA1withRSA]

  Signature:

0000: 19 50 E3 CF 4C 3D 37 A0   76 B6 D8 FB 81 23 61 A1  .P..L=7.v....#a.

0010: 1C 7D BB AE E2 9C 0B 62   CA 73 A2 D7 3D 06 BC 8C  .......b.s..=...

0020: 8C 80 E7 7E 40 7C 58 0D   23 F8 E7 C1 4F 41 8F F7  ....@.X.#...OA..

0030: CC 65 44 70 90 DA BE 32   45 7C 05 4D B3 98 2E 42  .eDp...2E..M...B

0040: 6A C7 E2 38 42 C0 50 7F   C0 AD CD 9E E3 A0 FE F6  j..8B.P.........

0050: CC 55 B1 B4 69 A8 8F 93   8A FD B6 69 D8 C0 05 F3  .U..i......i....

0060: 4A 2F 1E 86 B0 A7 51 3F   8A CF D2 06 43 A5 5D B8  J/....Q?....C.].

0070: 1C 64 C7 02 32 BD 93 FF   B1 99 95 9D 4F 8D BB 06  .d..2.......O...

0080: 0E 49 D8 CE DD E9 D2 D7   4A B3 00 AA A8 87 24 74  .I......J.....$t

0090: BA 00 96 CD 61 77 E5 DF   75 F0 23 C0 87 89 BA 8A  ....aw..u.#.....

00A0: 8F D3 32 56 43 F2 AD 8F   33 F5 1A 03 44 0C FF 8F  ..2VC...3...D...

00B0: AE 46 4B 29 FF 6D F0 6A   2A E6 5A 10 6D 33 18 FA  .FK).m.j*.Z.m3..

00C0: 5F 5E 0D 11 D2 AE 77 C3   6C DA 15 F7 F8 43 FC 10  _^....w.l....C..

00D0: 77 35 44 5A 88 DD 38 2B   74 29 FE 5D 0E 59 E2 87  w5DZ..8+t).].Y..

00E0: 85 77 45 D5 86 DA 6A E0   F8 BB C7 58 0F B0 AD 56  .wE...j....X...V

00F0: DD 83 55 3D 84 6C 94 38   F7 98 C3 FC A5 A4 80 AD  ..U=.l.8........

]

***

%% Invalidated:  [Session-2, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA]

main, SEND TLSv1 ALERT:  fatal, description = certificate_unknown

main, WRITE: TLSv1 Alert, length = 2

main, called closeSocket()

main, handling exception: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: No trusted certificate found

main, called close()

main, called closeInternal(true)

[Michael Klishin]

On 29 October 2014 at 14:08:04, Wout Slakhorst (wout.sl...@gmail.com) wrote:
> chain [0] = [
> [
> Version: V3
> Subject: CN=*.ons.io, OU=Healthcare, O=N.V. Nederlandsche
> Apparatenfabriek Nedap, L=Groenlo, ST=Gelderland, C=NL
> Signature Algorithm: SHA1withRSA, OID = 1.2.840.113549.1.1.5
>

> ]

OK, thanks. This does look like a single certificate.
What Erlang version do you use? What JDK version?

I'll try reproducing with tls-gen soon. 

[Wout Slakhorst]

java version "1.7.0_67"

Java(TM) SE Runtime Environment (build 1.7.0_67-b01)

Java HotSpot(TM) 64-Bit Server VM (build 24.65-b04, mixed mode)

And for Erlang

Erlang/OTP 17 [RELEASE CANDIDATE 2] [erts-6.0] [source] [64-bit] [smp:2:2] [async-threads:10] [hipe] [kernel-poll:false]

[Michael Klishin]

On 29 October 2014 at 15:10:35, Wout Slakhorst (wout.sl...@gmail.com) wrote:
> Erlang/OTP 17 [RELEASE CANDIDATE 2] [erts-6.0] [source] [64-bit]
> [smp:2:2] [async-threads:10] [hipe] [kernel-poll:false]

I don't see anything related in the change logs of 17.0, 17.1 or 17.3:

http://www.erlang.org/download/otp_src_17.0.readme
http://www.erlang.org/download/otp_src_17.1.readme
http://www.erlang.org/download/otp_src_17.3.readme

but I highly recommend using 17.1 over 17 RCs.

Also, it would be very helpful to know if you observe the same results
with a different certificate chain, specifically that generated by tls-gen [1].

1. https://github.com/michaelklishin/tls-gen

[Wout Slakhorst]

We updated to the latest released version avaiable in the debian packages. We still only get the first certificate from the chain.

[Wout Slakhorst]

ok, I got a bit wiser. First of all I did something wrong. I didn't realize that the intermediate CA needed to be in the cacertfile and not the crt file. This does raise another question though: ow both the intermediate chain of the server as well as the chain for the client certificates are in the same file, doesn't this mean that anyone with a certificate from the same intermediate CA can connect to our rabbitmq instance? (Although I'm pretty sure that CA only hands out server certificates and not client certificates)

This takes our Java client a couple of steps further untill it fails with a 'handshake failure'. The same can be seen in the rabbitmq logs:

=ERROR REPORT==== 4-Nov-2014::20:17:01 ===

SSL: certify: ssl_handshake.erl:1403:Fatal error: handshake failure

=ERROR REPORT==== 4-Nov-2014::20:17:06 ===

Error on AMQP connection <0.829.0>:

{ssl_upgrade_error,{tls_alert,"handshake failure"}}

And Java ends with:

*** CertificateVerify

main, WRITE: TLSv1 Handshake, length = 262

main, WRITE: TLSv1 Change Cipher Spec, length = 1

*** Finished

verify_data:  { 172, 199, 125, 119, 200, 134, 128, 228, 109, 91, 203, 17 }

***

main, WRITE: TLSv1 Handshake, length = 48

main, READ: TLSv1 Alert, length = 2

main, RECV TLSv1 ALERT:  fatal, handshake_failure

%% Invalidated:  [Session-3, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA]

main, called closeSocket()

main, handling exception: javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure

[Wout Slakhorst]

Ok solved it now, I feel a bit stupid, but it's also a bit cumbersome.

I had to set the depth to 3 and now it works.

I was thrown off by the fact that the cacert file needs to hold both the client certificate chain as well as part of the server certificate chain. This is done differently in all other languages where the server chain is in the server certificate.

I got it working now for Java. Let's see how Ruby/Openssl copes.

[Michael Klishin]

 On 4 November 2014 at 22:35:52, Wout Slakhorst (wout.sl...@gmail.com) wrote:
> I was thrown off by the fact that the cacert file needs to hold
> both the client certificate chain as well as part of the server
> certificate chain. This is done differently in all other languages
> where the server chain is in the server certificate.

I'm confused. It should not different in Erlang. Why would a chained server certificate contain a part of the client certificate? (unless they share an intermediate CA).

Server certificate chaining in RabbitMQ is no different from Nginx, for example.

In any case, happy that you've finally solved it. We now have a new section in the TLS guide about verification depth, based on your feedback!

[Wout Slakhorst]

Thank you for your assistance, it's great to see that feedback is appreciated.

What i meant with the cacert file:

Our server certificate has intermediate CA's not trusted by default. To have Erlang send them with the ServerHello they have to be placed in the cacertfile, having them in the certfile does not make any difference.

To have our server verify the client certificates which use a self-signed root certificate, the entire chain has to be placed in cacertfile as well. Otherwise rabbitmq will respond with an 'unknown ca' answer.

Hope this clarifies it