RabbitMQ Certificate Trust Store seems to allow all connections to pass through

1,380 views
Skip to first unread message

NZ Ma

unread,
Jun 23, 2022, 4:18:00 AM6/23/22
to rabbitmq-users
Hi Community and Team, 

I am trying to achieve CRL-like function and plugin RabbitMQ Certificate Trust Store offers great solution whereby only the client certificate inside the white list will be allowed to connect to the broker. 

I enabled the plugin and added the following lines to the rabbitmq.conf as per the readme
trust_store.directory        = /path/to/white-list
trust_store.refresh_interval = 30
ssl_options.reuse_sessions = false 
# timely certificate removal is important, for testing purposes

I also have the following configuration enabled for peer-verification: 
listeners.ssl.default = 5671
ssl_options.cacertfile = /path/to/cacert.crt
ssl_options.certfile   =  /path/to/server-cert.crt
ssl_options.keyfile    =  /path/to/server-key.key
ssl_options.verify     = verify_peer
ssl_options.fail_if_no_peer_cert = true

I ensure the plugin is running with rabbitmq-plugins list command: 
[E*] rabbitmq_trust_store

I can see my certificates inside the whitelist recognized via command
rabbitmqctl eval 'io:format(rabbit_trust_store:list()).'

However, connection with certificate that is not inside the white list is still able to pass through via a mutual authentication. The client configuration is as per the pika documentation here (tls_example.py). 

The log for RabbitMQ for the connection is: 
2022-06-23 07:31:54.001798+00:00 [info] <0.6630.0> accepting AMQP connection <0.6630.0> (172.22.0.1:46908 -> 172.22.0.2:5671)
2022-06-23 07:31:54.016780+00:00 [info] <0.6630.0> connection <0.6630.0> (172.22.0.1:46908 -> 172.22.0.2:5671): user 'user' authenticated and granted access to vhost '/'

The log for client for the connection is: 
INFO:pika.adapters.utils.connection_workflow:Pika version 1.2.1 connecting to ('::1', 5671, 0, 0)
INFO:pika.adapters.utils.io_services_utils:Socket connected: <socket.socket fd=6, family=AddressFamily.AF_INET6, type=SocketKind.SOCK_STREAM, proto=6, laddr=('::1', 33316, 0, 0), raddr=('::1', 5671, 0, 0)>
INFO:pika.adapters.utils.io_services_utils:SSL handshake completed successfully: <ssl.SSLSocket fd=6, family=AddressFamily.AF_INET6, type=SocketKind.SOCK_STREAM, proto=6, laddr=('::1', 33316, 0, 0), raddr=('::1', 5671, 0, 0)>
INFO:pika.adapters.utils.connection_workflow:Streaming transport linked up: (<pika.adapters.utils.io_services_utils._AsyncSSLTransport object at 0x7f5eb45d8eb0>, _StreamingProtocolShim: <SelectConnection PROTOCOL transport=<pika.adapters.utils.io_services_utils._AsyncSSLTransport object at 0x7f5eb45d8eb0> params=<ConnectionParameters host=localhost port=5671 virtual_host=/ ssl=True>>).
INFO:pika.adapters.utils.connection_workflow:AMQPConnector - reporting success: <SelectConnection OPEN transport=<pika.adapters.utils.io_services_utils._AsyncSSLTransport object at 0x7f5eb45d8eb0> params=<ConnectionParameters host=localhost port=5671 virtual_host=/ ssl=True>>
INFO:pika.adapters.utils.connection_workflow:AMQPConnectionWorkflow - reporting success: <SelectConnection OPEN transport=<pika.adapters.utils.io_services_utils._AsyncSSLTransport object at 0x7f5eb45d8eb0> params=<ConnectionParameters host=localhost port=5671 virtual_host=/ ssl=True>>
INFO:pika.adapters.blocking_connection:Connection workflow succeeded: <SelectConnection OPEN transport=<pika.adapters.utils.io_services_utils._AsyncSSLTransport object at 0x7f5eb45d8eb0> params=<ConnectionParameters host=localhost port=5671 virtual_host=/ ssl=True>>
INFO:pika.adapters.blocking_connection:Created channel=1

I do have a user credential for PLAIN authentication. The user is set to have permission read all only. 
rabbitmqctl set_permissions -p "/" "user" "." "." ".*"

Both the RabbitMQ server certificate and Client certificate are signed under same CA. 

I have attempted the following to troubleshoot, but the issue persists. 

1. reset/restart the node, restart the container, rebuild the container with empty white list folder then connect. 
2. different combinations to set / comment out the following configurations: 
ssl_options.verify     = verify_peer
ssl_options.fail_if_no_peer_cert = true/false
3. run different versions of RabbitMQ (3.8.9, 3.9.2 and 3.10.5) 
4. use different sets of certificates (self-signed / issued by other CAs) 

Unfortunately I was still not able to resolve the issue. I am quite new for both TLS, as well as for RabbitMQ. 

I would really appreciate your insights. Thank you! 

Ma

Luke Bakken

unread,
Jun 23, 2022, 1:02:44 PM6/23/22
to rabbitmq-users
Hello -

Thanks for the very complete report. I'm taking a look into this today and will get back to you at least by the end of next week.

Luke Bakken

unread,
Jun 23, 2022, 3:55:17 PM6/23/22
to rabbitmq-users
Hello,

Please see this complete example that demonstrates using the cert trust store with a Pika client using x509 client certificate authentication:


I believe your issue stems from the fact that the trust_store.directory should contain Certificate Authority (CA) certificates, not the client certificates themselves.

In my example, I have two independent sets of certificates, each signed by a different CA. If you have docker available, you should be able to run these commands to see the client successfully connect:

cd rabbitmq-users-sLXfiBGaKfQ
./init.sh
docker compose up

In this case, the client successfully connects because BOTH CA files are in the trust store:


If you wish to see the client connection be disallowed, comment out these lines...


...un-comment this line:


...then run the following commands:

docker compose stop
./init.sh
docker compose up

The docker log output should now be something like this:

RabbitMQ log:

[notice] <0.661.0> TLS server: In state wait_cert at ssl_handshake.erl:2084 generated SERVER ALERT: Fatal - Handshake Failure
[notice] <0.661.0>  - "CA not known AND certificate not whitelisted"

Pika client log:

ERROR:pika.adapters.utils.io_services_utils:_AsyncBaseTransport._consume() failed, aborting connection: error=SSLError(1, '[SSL: SSLV3_ALERT_HANDSHAKE_FAILURE] sslv3 alert handshake failure (_ssl.c:2548)'); sock=<ssl.SSLSocket fd=6, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('172.19.0.3', 42494)>;

On Thursday, June 23, 2022 at 1:18:00 AM UTC-7 ning...@gmail.com wrote:

NZ Ma

unread,
Jun 27, 2022, 5:51:06 AM6/27/22
to rabbitmq-users

Hello! 


Thank you so much for taking time off for this comprehensive demonstration and detailed instruction. I am so amazed by how everything is well packaged. I really appreciate it! 


I am sorry for the late reply. I took a while to study and understand the many advanced settings in this demonstration. Especially with the use of x509 certificate based authentication (EXTERNAL). 


I have again performed extensive testing on different cases to explore the behavior of RabbitMQ Trust Store in this case. In my use case, using just CA to whitelist can be quite challenging and less robust as most likely, all the clients will be signed with the same CA. 


Here are my assumption/conditions: 

  1. rmq-0 set of certificates are unrelated to rmq-1 set of certificates. ca_certs.pem in the certs directory is the combination of rmq-0 and rmq-1 CA certificates. 

  2. The rabbitmq server is using rmq-0 server certificate


In short, this is what I did: 

  1. Created a user with CN=rmq-0 so that I can test a client that is signed by rmq-0 CA. (I did not do that initially, so I was not able to connect with rmq-0 client, but this is not due to the TrustStore)

  2. Place/remove rmq-0 and rmq-1 client certificates into the trust store directory

  3. Using Pika client to connect to the rabbitMQ server with rmq-0 and rmq-1 client certificate respectively. 


An interesting behavior I found is that, with just the client certificate, I am able to whitelist / block access to the client with rmq-1 certificate with just the client certificate inside/removed from the trust store directory, provided that the rabbitMQ server is using rmq-0 certificate (issued by different CA). 

White Listing: 

rabbitmq-users-slxfibgakfq-tc3-rmq-0-1        | 2022-06-27 08:20:31.714458+00:00 [debug] <0.754.0> Trust store listed certificates: [{{"client_rmq-0_certificate.pem",1656051228,

rabbitmq-users-slxfibgakfq-tc3-rmq-0-1        | 2022-06-27 08:20:31.714458+00:00 [debug] <0.754.0>                                     62543353},

rabbitmq-users-slxfibgakfq-tc3-rmq-0-1        | 2022-06-27 08:20:31.714458+00:00 [debug] <0.754.0>                                    [{name,"client_rmq-0_certificate.pem"}]},

rabbitmq-users-slxfibgakfq-tc3-rmq-0-1        | 2022-06-27 08:20:31.714458+00:00 [debug] <0.754.0>                                   {{"client_rmq-1_certificate.pem",1656051228,

rabbitmq-users-slxfibgakfq-tc3-rmq-0-1        | 2022-06-27 08:20:31.714458+00:00 [debug] <0.754.0>                                     102077184},

rabbitmq-users-slxfibgakfq-tc3-rmq-0-1        | 2022-06-27 08:20:31.714458+00:00 [debug] <0.754.0>                                    [{name,"client_rmq-1_certificate.pem"}]}]

rabbitmq-users-slxfibgakfq-tc3-rmq-0-1        | 2022-06-27 08:20:31.714788+00:00 [debug] <0.754.0> Updating 2 fetched trust store certificates

rabbitmq-users-slxfibgakfq-tc3-rmq-0-1        | 2022-06-27 08:20:57.257337+00:00 [info] <0.1089.0> accepting AMQP connection <0.1089.0> (172.26.0.1:55042 -> 172.26.0.2:5671)

rabbitmq-users-slxfibgakfq-tc3-rmq-0-1        | 2022-06-27 08:20:57.261471+00:00 [debug] <0.1089.0> auth mechanism TLS extracted username 'O=client,CN=rmq-1' from peer certificate

rabbitmq-users-slxfibgakfq-tc3-rmq-0-1        | 2022-06-27 08:20:57.261613+00:00 [debug] <0.1089.0> Raw client connection hostname during authN phase: {0,0,0,0,0,65535,44058,2}

rabbitmq-users-slxfibgakfq-tc3-rmq-0-1        | 2022-06-27 08:20:57.261738+00:00 [debug] <0.1089.0> Resolved client hostname during authN phase: ::ffff:172.26.0.2

rabbitmq-users-slxfibgakfq-tc3-rmq-0-1        | 2022-06-27 08:20:57.261798+00:00 [debug] <0.1089.0> User 'O=client,CN=rmq-1' authenticated successfully by backend rabbit_auth_backend_internal

rabbitmq-users-slxfibgakfq-tc3-rmq-0-1        | 2022-06-27 08:20:57.268351+00:00 [info] <0.1089.0> connection <0.1089.0> (172.26.0.1:55042 -> 172.26.0.2:5671): user 'O=client,CN=rmq-1' authenticated and granted access to vhost '/'


Blocking: 

rabbitmq-users-slxfibgakfq-tc3-rmq-0-1        | 2022-06-27 08:33:28.396335+00:00 [debug] <0.754.0> Trust store listed certificates: []

rabbitmq-users-slxfibgakfq-tc3-rmq-0-1        | 2022-06-27 08:33:28.396507+00:00 [debug] <0.754.0> Updating 0 fetched trust store certificates

rabbitmq-users-slxfibgakfq-tc3-rmq-0-1        | 2022-06-27 08:33:46.151157+00:00 [notice] <0.993.0> TLS server: In state wait_cert at ssl_handshake.erl:2084 generated SERVER ALERT: Fatal - Handshake Failure

rabbitmq-users-slxfibgakfq-tc3-rmq-0-1        | 2022-06-27 08:33:46.151157+00:00 [notice] <0.993.0>  - "CA not known AND certificate not whitelisted"

rabbitmq-users-slxfibgakfq-tc3-rmq-0-1        | 2022-06-27 08:33:46.177202+00:00 [notice] <0.999.0> TLS server: In state wait_cert at ssl_handshake.erl:2084 generated SERVER ALERT: Fatal - Handshake Failure

rabbitmq-users-slxfibgakfq-tc3-rmq-0-1        | 2022-06-27 08:33:46.177202+00:00 [notice] <0.999.0>  - "CA not known AND certificate not whitelisted"

However, if the client also uses rmq-0 client certificate, it will by-pass the the trust store and connect to the server, even with an empty trust store. 

Bypassed:

rabbitmq-users-slxfibgakfq-tc3-rmq-0-1        | 2022-06-27 08:30:28.363402+00:00 [debug] <0.754.0> Trust store listed certificates: []

rabbitmq-users-slxfibgakfq-tc3-rmq-0-1        | 2022-06-27 08:30:28.363484+00:00 [debug] <0.754.0> Updating 0 fetched trust store certificates

rabbitmq-users-slxfibgakfq-tc3-rmq-0-1        | 2022-06-27 08:30:50.216529+00:00 [info] <0.898.0> accepting AMQP connection <0.898.0> (172.27.0.1:55120 -> 172.27.0.2:5671)

rabbitmq-users-slxfibgakfq-tc3-rmq-0-1        | 2022-06-27 08:30:50.221797+00:00 [debug] <0.898.0> auth mechanism TLS extracted username 'O=client,CN=rmq-0' from peer certificate

rabbitmq-users-slxfibgakfq-tc3-rmq-0-1        | 2022-06-27 08:30:50.221896+00:00 [debug] <0.898.0> Raw client connection hostname during authN phase: {0,0,0,0,0,65535,44059,2}

rabbitmq-users-slxfibgakfq-tc3-rmq-0-1        | 2022-06-27 08:30:50.222091+00:00 [debug] <0.898.0> Resolved client hostname during authN phase: ::ffff:172.27.0.2

rabbitmq-users-slxfibgakfq-tc3-rmq-0-1        | 2022-06-27 08:30:50.222151+00:00 [debug] <0.898.0> User 'O=client,CN=rmq-0' authenticated successfully by backend rabbit_auth_backend_internal

rabbitmq-users-slxfibgakfq-tc3-rmq-0-1        | 2022-06-27 08:30:50.229725+00:00 [info] <0.898.0> connection <0.898.0> (172.27.0.1:55120 -> 172.27.0.2:5671): user 'O=client,CN=rmq-0' authenticated and granted access to vhost '/'

rabbitmq-users-slxfibgakfq-tc3-rmq-0-1        | 2022-06-27 08:30:53.371718+00:00 [info] <0.898.0> closing AMQP connection <0.898.0> (172.27.0.1:55120 -> 172.27.0.2:5671, vhost: '/', user: 'O=client,CN=rmq-0')

I wonder why I was experiencing such behavior in my first post is because my server cert and client cert are always signed with the same CA. 


I would like to confirm: 

  1. This is the behavior of the Trust Store plugin, and the only way to block some certificates with the same CA is through CRL with erlang and Advanced Config. 

  2. Another wonder I have is if rmq-0 and rmq-1 are intermediate CAs signed by the same root CA, is this behavior still holds true. 


I will make a pull request soon if you are interested in my code. 


Once again, thank you for your kind guidance!

Luke Bakken

unread,
Jun 27, 2022, 9:22:14 AM6/27/22
to rabbitmq-users
Hello again -

Thanks for following up.

 I am so amazed by how everything is well packaged. I really appreciate it! 

Thanks. I try to set an example so that other people can hopefully see the best way to report and discuss issues. 

I would like to confirm: 

  1. This is the behavior of the Trust Store plugin, and the only way to block some certificates with the same CA is through CRL with erlang and Advanced Config. 

Yes, that is correct, the server CA acts as though it is part of the trust store. My guess is that when the trust store is empty the server's CA is used as a fallback. I'll double-check that in the code.

If you use CRLs that are fetched by HTTP, note that they will not be cached by Erlang and thus fetched on every new connection. I recommend CRLs on disk. Also note that if you have a huge CRL it probably will affect the performance of new connections.

  1. Another wonder I have is if rmq-0 and rmq-1 are intermediate CAs signed by the same root CA, is this behavior still holds true.

I have not tested the case with intermediate CAs and don't really know the behavior there.

Thanks -
Luke

NZ Ma

unread,
Jun 28, 2022, 3:58:21 AM6/28/22
to rabbitmq-users

Hello again! 

Thank you so much for your insight!

I conducted another round of tests to see if the Trust Store is able to block clients with certificates signed by an Intermediate CA that has the same root CA as the RabbitMQ server. 

Certificate: 

I generated Chained (With Separate Intermediates) Certificates with tls-gen repo, in the following hierarchy: 

  • Chain 1: root CA => client intermediate => client certificate/key pair

  • Chain 2: root CA => server intermediate => server certificate/key pair

The CN were overwritten as rmq, with new user rmq created in the RabbitMQ. 

Configuration: 

RabbitMQ: 

  • CA cert file: CA Chain (root CA cert + server intermediate CA cert)

  • Cert file & Key file: server certificate/key pair

Pika Client 1: 

  • CA cert file: CA Chain (root CA cert + client intermediate CA cert)

  • Cert file & Key file: client certificate/key pair

Result: 

Unfortunately, with an empty trust store directory, client 1 is still able to connect to the server. 


RabbitMQ: 

2022-06-28 06:10:37.506007+00:00 [debug] <0.754.0> Trust store listed certificates: []

2022-06-28 06:10:37.506329+00:00 [debug] <0.754.0> Updating 0 fetched trust store certificates

2022-06-28 06:10:51.092364+00:00 [info] <0.1156.0> accepting AMQP connection <0.1156.0> (172.29.0.1:56078 -> 172.29.0.2:5671)

2022-06-28 06:10:51.096298+00:00 [debug] <0.1156.0> auth mechanism TLS extracted username 'O=client,CN=rmq' from peer certificate

2022-06-28 06:10:51.096376+00:00 [debug] <0.1156.0> Raw client connection hostname during authN phase: {0,0,0,0,0,65535,44061,2}

2022-06-28 06:10:51.096467+00:00 [debug] <0.1156.0> Resolved client hostname during authN phase: ::ffff:172.29.0.2

2022-06-28 06:10:51.096520+00:00 [debug] <0.1156.0> User 'O=client,CN=rmq' authenticated successfully by backend rabbit_auth_backend_internal

2022-06-28 06:10:51.100525+00:00 [info] <0.1156.0> connection <0.1156.0> (172.29.0.1:56078 -> 172.29.0.2:5671): user 'O=client,CN=rmq' authenticated and granted access to vhost '/'

2022-06-28 06:10:51.119506+00:00 [info] <0.1156.0> closing AMQP connection <0.1156.0> (172.29.0.1:56078 -> 172.29.0.2:5671, vhost: '/', user: 'O=client,CN=rmq')

2022-06-28 06:10:51.122134+00:00 [debug] <0.1171.0> Closing all channels from connection '172.29.0.1:56078 -> 172.29.0.2:5671' because it has been closed


Pika Client 1: 

INFO:__main__:ssl.HAS_SNI: True

INFO:pika.adapters.utils.connection_workflow:Pika version 1.2.0 connecting to ('::1', 5671, 0, 0)

INFO:pika.adapters.utils.io_services_utils:Socket connected: <socket.socket fd=8, family=AddressFamily.AF_INET6, type=SocketKind.SOCK_STREAM, proto=6, laddr=('::1', 54602, 0, 0), raddr=('::1', 5671, 0, 0)>

INFO:pika.adapters.utils.io_services_utils:SSL handshake completed successfully: <ssl.SSLSocket fd=8, family=AddressFamily.AF_INET6, type=SocketKind.SOCK_STREAM, proto=0, laddr=('::1', 54602, 0, 0), raddr=('::1', 5671, 0, 0)>

INFO:pika.adapters.utils.connection_workflow:Streaming transport linked up: (<pika.adapters.utils.io_services_utils._AsyncSSLTransport object at 0x7fe2b179b8b0>, _StreamingProtocolShim: <SelectConnection PROTOCOL transport=<pika.adapters.utils.io_services_utils._AsyncSSLTransport object at 0x7fe2b179b8b0> params=<ConnectionParameters host=localhost port=5671 virtual_host=/ ssl=True>>).

INFO:pika.adapters.utils.connection_workflow:AMQPConnector - reporting success: <SelectConnection OPEN transport=<pika.adapters.utils.io_services_utils._AsyncSSLTransport object at 0x7fe2b179b8b0> params=<ConnectionParameters host=localhost port=5671 virtual_host=/ ssl=True>>

INFO:pika.adapters.utils.connection_workflow:AMQPConnectionWorkflow - reporting success: <SelectConnection OPEN transport=<pika.adapters.utils.io_services_utils._AsyncSSLTransport object at 0x7fe2b179b8b0> params=<ConnectionParameters host=localhost port=5671 virtual_host=/ ssl=True>>

INFO:pika.adapters.blocking_connection:Connection workflow succeeded: <SelectConnection OPEN transport=<pika.adapters.utils.io_services_utils._AsyncSSLTransport object at 0x7fe2b179b8b0> params=<ConnectionParameters host=localhost port=5671 virtual_host=/ ssl=True>>

INFO:pika.adapters.blocking_connection:Created channel=1

INFO:__main__:(<Basic.GetOk(['delivery_tag=1', 'exchange=', 'message_count=2', 'redelivered=True', 'routing_key=foobar'])>, <BasicProperties>, b'Hello, world!')



Conclusion: 

The client will still be able to pass through the trust store as ultimately, the root CA cert needs to be examined to verify the validity of the intermediate CAs to form a complete chain of trust. 

(I also tried to use just intermediate CAs in the server and client respectively, but the connection could not be established. )



I was met with another interesting behavior when I tried to connect to RabbitMQ with the current CA Chain setup from clients using rmq-0 or rmq-1 client certificates. 

RabbitMQ: 

rabbitmq-users-slxfibgakfq-rmq-0-1        | 2022-06-28 07:43:08.497405+00:00 [debug] <0.754.0> Trust store listed certificates: []

rabbitmq-users-slxfibgakfq-rmq-0-1        | 2022-06-28 07:43:08.497513+00:00 [debug] <0.754.0> Updating 0 fetched trust store certificates

rabbitmq-users-slxfibgakfq-rmq-0-1        | 2022-06-28 07:43:25.168371+00:00 [notice] <0.4944.0> TLS server: In state wait_cert at tls_record_1_3.erl:204 generated SERVER ALERT: Fatal - Bad Record MAC

rabbitmq-users-slxfibgakfq-rmq-0-1        | 2022-06-28 07:43:25.168371+00:00 [notice] <0.4944.0>  - {record_type_mismatch,21}

rabbitmq-users-slxfibgakfq-rmq-0-1        | 2022-06-28 07:43:25.183309+00:00 [notice] <0.4950.0> TLS server: In state wait_cert at tls_record_1_3.erl:204 generated SERVER ALERT: Fatal - Bad Record MAC

rabbitmq-users-slxfibgakfq-rmq-0-1        | 2022-06-28 07:43:25.183309+00:00 [notice] <0.4950.0>  - {record_type_mismatch,21}



Pika Client 2 (with rmq-0 client certificate): 

INFO:__main__:ssl.HAS_SNI: True

INFO:pika.adapters.utils.connection_workflow:Pika version 1.2.0 connecting to ('::1', 5671, 0, 0)

INFO:pika.adapters.utils.io_services_utils:Socket connected: <socket.socket fd=8, family=AddressFamily.AF_INET6, type=SocketKind.SOCK_STREAM, proto=6, laddr=('::1', 55210, 0, 0), raddr=('::1', 5671, 0, 0)>

ERROR:pika.adapters.utils.io_services_utils:SSL do_handshake failed: error=SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1125)'); <ssl.SSLSocket fd=8, family=AddressFamily.AF_INET6, type=SocketKind.SOCK_STREAM, proto=0, laddr=('::1', 55210, 0, 0), raddr=('::1', 5671, 0, 0)>

Traceback (most recent call last):

  File "/Users/user/opt/anaconda3/lib/python3.8/site-packages/pika/adapters/utils/io_services_utils.py", line 636, in _do_ssl_handshake

    self._sock.do_handshake()

  File "/Users/user/opt/anaconda3/lib/python3.8/ssl.py", line 1309, in do_handshake

    self._sslobj.do_handshake()

ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1125)

ERROR:pika.adapters.utils.connection_workflow:Attempt to create the streaming transport failed: SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1125)'); 'localhost'/(<AddressFamily.AF_INET6: 30>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('::1', 5671, 0, 0)); ssl=True

ERROR:pika.adapters.utils.connection_workflow:AMQPConnector - reporting failure: AMQPConnectorTransportSetupError: SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1125)')

INFO:pika.adapters.utils.connection_workflow:Pika version 1.2.0 connecting to ('127.0.0.1', 5671)

INFO:pika.adapters.utils.io_services_utils:Socket connected: <socket.socket fd=9, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('127.0.0.1', 55211), raddr=('127.0.0.1', 5671)>

ERROR:pika.adapters.utils.io_services_utils:SSL do_handshake failed: error=SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1125)'); <ssl.SSLSocket fd=9, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('127.0.0.1', 55211), raddr=('127.0.0.1', 5671)>

Traceback (most recent call last):

  File "/Users/user/opt/anaconda3/lib/python3.8/site-packages/pika/adapters/utils/io_services_utils.py", line 636, in _do_ssl_handshake

    self._sock.do_handshake()

  File "/Users/user/opt/anaconda3/lib/python3.8/ssl.py", line 1309, in do_handshake

    self._sslobj.do_handshake()

ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1125)


I was surprised by this behavior such that last time, we could use rmq-0 and rmq-1’s certificates to connect, even as they are self-signed certificate, however, in this case, as soon as we introduce CA chain in the server side, and when we tried to connect with a client cert signed by a different CA, we get this error. I wonder if I made some mistakes in my configuration, and what is the implication here.

Thank you for your kind support! 

Luke Bakken

unread,
Jun 28, 2022, 4:38:41 PM6/28/22
to rabbitmq-users
Hello,

Thanks for the detailed description of your test and outcome.

Reading this, I do remember testing scenarios with intermediate certificates quite a while ago. I believe in most cases the server must have all intermediate CAs in it's CA bundle file. It also can't hurt for the client to have these intermediaries as well in its CA bundle. Try to experiment with each. To be honest I have no idea how the trust store plugin plays into all of this, either, but my guess is that it is not sufficient to have just intermediate CA certs in trust store bundles but the entire CA chain.

Another thing to try would be to add the intermediate certificates into the certificate bundle presented by the client as its client certificate.

If you'd like to open another PR with your intermediate certificate tests I would find that interesting.

Thanks -
Luke

NZ Ma

unread,
Aug 1, 2022, 1:15:38 AM8/1/22
to rabbitmq-users

Hello! 

I am sorry for this late reply. 

I did a thorough investigation on both the Trust Store and Erlang CRL. I prepared 2 repositories for you.

1. White-listing through Trust Store

This repo is a branched out from your repo. I tried to deepen the chain of trust while limiting the Trust Store’s SSL Depth. 

Unfortunately, this method does not seem to work in my use case. As long as the Root CA is the same for the broker and the clients, the client will always be able to connect to the broker, regardless of if the client’s certificate is inside the truststore or not.

2.  Black-Listing through Erlang CRL (ssl_crl_has_dir) and advanced.config


Due to the lack of documentation, this is indeed a challenging process that took me many attempts and troubleshooting. I have included all the automation script in the set up of the broker, certificate and CRL generation, as well as detailed step-by-step guide in the readme file. 

While I am able to enable the CRL checking module, I do face issues in the hashing of the CRL files. 

I tried to use the chained CRL file and rehash it with “openssl crl -hash” command, but this leads to {bad_crls,no_relevant_crls} errors. 

I also tried to use individual CRL files generated at each CA level, rehash with  “openssl rehash crl_dir” command, but this leads to {bad_crl,invalid_signature} 

Please refer to the "Current Issue" Section in the readme file for more information. 

I would appreciate it if you can give me some insights in resolving this issue. Which would be the correct CRL form, and how should I re-hash it. 

Once we resolved the issue, I would like to make a pull request on rabbitmq-website to include the documentation on how to enable CRL check on RabbitMQ into the TLS Support Page for others to take reference as well. 

Thank you! 

Luke Bakken

unread,
Aug 3, 2022, 9:23:47 AM8/3/22
to rabbitmq-users
Hello,

Thanks for all of your hard work. I have just returned from vacation. I will find time to look into this again this month.

Luke

NZ Ma

unread,
Aug 4, 2022, 2:06:33 AM8/4/22
to rabbitmq-users
Sure, no problem! 

Thank you so much! 

NZ Ma

unread,
Aug 4, 2022, 6:40:41 AM8/4/22
to rabbitmq-users
Hello! 

It seems that I did something with the previous repo such that it is giving a new error: 

[SSL: SSLV3_ALERT_CERTIFICATE_UNKNOWN] sslv3 alert certificate unknown (_ssl.c:2548)')

If you have similar issue, I have restored the repo to a previous copy in this repository: 

This test fine on macOS. 

Thank you! 

Luke Bakken

unread,
Aug 9, 2022, 4:24:48 PM8/9/22
to rabbitmq-users
Hello -

On Sunday, July 31, 2022 at 10:15:38 PM UTC-7 ning...@gmail.com wrote:

1. White-listing through Trust Store

This repo is a branched out from your repo. I tried to deepen the chain of trust while limiting the Trust Store’s SSL Depth. 

Unfortunately, this method does not seem to work in my use case. As long as the Root CA is the same for the broker and the clients, the client will always be able to connect to the broker, regardless of if the client’s certificate is inside the truststore or not.


Yes, this is how RabbitMQ / Erlang currently behaves. It appears that the server certificate's CA is used as a "fallback" when the client can't be validated by the trust store.
 

2.  Black-Listing through Erlang CRL (ssl_crl_has_dir) and advanced.config


Due to the lack of documentation, this is indeed a challenging process that took me many attempts and troubleshooting. I have included all the automation script in the set up of the broker, certificate and CRL generation, as well as detailed step-by-step guide in the readme file. 

While I am able to enable the CRL checking module, I do face issues in the hashing of the CRL files. 

I tried to use the chained CRL file and rehash it with “openssl crl -hash” command, but this leads to {bad_crls,no_relevant_crls} errors. 

I also tried to use individual CRL files generated at each CA level, rehash with  “openssl rehash crl_dir” command, but this leads to {bad_crl,invalid_signature} 

Please refer to the "Current Issue" Section in the readme file for more information. 

I would appreciate it if you can give me some insights in resolving this issue. Which would be the correct CRL form, and how should I re-hash it. 

Once we resolved the issue, I would like to make a pull request on rabbitmq-website to include the documentation on how to enable CRL check on RabbitMQ into the TLS Support Page for others to take reference as well.


I will look into this. I don't have a lot of experience with CRLs so this will be a good learning opportunity.

Luke 
Reply all
Reply to author
Forward
0 new messages