RabbitMQ Shovels sends messages to other RabbitMQ Server despite certificates expired

93 views
Skip to first unread message

Marc Peter

unread,
Oct 21, 2022, 4:25:13 AM10/21/22
to rabbitmq-users

Hello everyone, 

We are currently testing different states of our ssl encryption with RabbitMQ an ran into a quite confusion behaviour,  which does not match our expectations. 

We have two computers with RabbitMQ installed. One, lets call it client, is not encrypted locally, but the shovels communicate with the second, lets call it server, which is using ssl.
Everything on the client computer is not using ssl locally but once the messages leave the client via the shovels, this connection should be encrypted using ssl, since the server computer runs with ssl enabled. 

Here is a png of the shovel status to get a better picture of the setup:
Shovels.png

Lets talk about the behaviour which confuses us. If our used certificates expires and is not valid anymore. The client shovels still sends the messages to the server computer. Message queues are still empty because they are send immediatly: 
Queues.png

 Our expectation here would be, that the messages should not be send but instead should be stacked in the in the shovels, when the certificates are not valid anymore.

In the log are warnings about missing configurations, but we declared these in the config. log with warnings.png

I will append the used config to this.

Is there something wrong with the setup or the config file? Or are our expectations regarding the behavior wrong? What must be changed to achive our expectations?

Thank you in advance.

Greetings,

Marc

rabbitmq.conf
Connections.png
certs in explorer.png

Marc Peter

unread,
Oct 21, 2022, 4:31:18 AM10/21/22
to rabbitmq-users
I forgot to mention that the version of RabbitMQ we are using is the version 3.10.6 with Erlang 25

Greetings,
Marc

Luke Bakken

unread,
Oct 21, 2022, 11:02:34 AM10/21/22
to rabbitmq-users
Hi Marc,

Unfortunately some of your screenshots are too small to see.

I would start by exporting your definitions and attaching them to your response. My guess is that you are not enabling verification in your shovel URIs.

Thanks,
Luke

Marc Peter

unread,
Oct 24, 2022, 4:18:52 AM10/24/22
to rabbitmq-users
Hi Luke,

thanks for your response. I reattachted the screenshots i used. 

I will look into my definitions again and attach them.

Thanks,
Marc
log with warnings.png
Shovels.png

Marc Peter

unread,
Oct 24, 2022, 7:22:49 AM10/24/22
to rabbitmq-users
Hello,

as requested i attatched the exported definitions.

Greetings,
Marc

rabbit_MAFBSPQAFE01_2022-10-24.json

Luke Bakken

unread,
Oct 24, 2022, 2:57:15 PM10/24/22
to rabbitmq-users
Yes, you will have to enable server verification for your amqps URIs, like this - https://www.rabbitmq.com/uri-query-parameters.html#basics

Marc Peter

unread,
Oct 25, 2022, 7:14:10 AM10/25/22
to rabbitmq-users
Hi Luke,

sadly I don't get your suggested fix to work.

When add the  amqp_client.ssl_options to the rabbitmq.conf, the service wont start at all. 

It might just be the syntax since i have not found an example.

I attached the conf file.

Thanks

rabbitmq.conf

Luke Bakken

unread,
Oct 25, 2022, 11:24:29 AM10/25/22
to rabbitmq-users
Hello,

RabbitMQ probably logs why it won't start. Did you check the log file?

I meant for you to change your shovel URI strings instead, but your method should work. However, the documentation states that you must specify amqp_client settings in the advanced.config file, not in the rabbitmq.conf file!

Create a file named advanced.config in the same directory as your rabbitmq.conf file with this content, remove the invalid settings from rabbitmq.conf, and restart:

[
    {amqp_client, [
        {ssl_options, [
            {cacertfile, "c:/certificates/converted/testdevelopment_0000_root.pem"},
            {certfile, "c:/certificates/converted/public-testdevelopment_0000.pem"},
            {keyfile, "c:/certificates/converted/private-testdevelopment_0000.pem"},
            {verify, verify_peer},
            {fail_if_no_peer_cert, true}
        ]}
    ]}
].


You should use these settings on the servers that are using amqps:// URIs for their shovels.

Thanks,
Luke

Marc Peter

unread,
Oct 26, 2022, 6:41:16 AM10/26/22
to rabbitmq-users
Hi Luke,

Yes i checked the log-file but it did not why it wouldn't start.

I have implemented your suggestion but unfortunaly I didnt quite work. With the configs attached, the two rabbitmq client instance didnt send any of the messages but instead they were all stacked in the queue besides valid certificates on both rabbitmq client and rabbitmq server computers.
 shovel_stacking.png


I also tried it with ssl_options.fail_if_no_peer_cert = false on the rabbitmq server instance but then it seems to be always sending the messages. Even when the certificates provides are invalid or expired.

The rabbitmq client instance is using the amgp shovels and the rabbitmq server instance is using the amgps shovels.

I have attached all four config files.

Is the configuration on both instances fine?

Thanks,
Marc

client_advanced.config
client_rabbitmq.conf
shovel_stacking.png
SERVER_rabbitmq.conf
SERVER_advanced.config

Luke Bakken

unread,
Oct 26, 2022, 9:47:39 AM10/26/22
to rabbitmq-users
Thanks for the information. I will return to this question on Friday.

Marc Peter

unread,
Oct 31, 2022, 10:48:28 AM10/31/22
to rabbitmq-users
Hi Luke,

did you have the chance to look at the given information by now?

Thanks,
Marc

Luke Bakken

unread,
Oct 31, 2022, 12:06:23 PM10/31/22
to rabbitmq-users
No I haven't, I have been working on higher priority items, some of which are for paying customers.

I do have this starred for follow-up.

Luke Bakken

unread,
Oct 31, 2022, 5:01:04 PM10/31/22
to rabbitmq-users
Hi Marc,

I will be putting together a docker compose environment to test out your issue. If you'd like to follow along, here's my repo -


Starting work on this now

Luke Bakken

unread,
Oct 31, 2022, 7:40:10 PM10/31/22
to rabbitmq-users
Hi Marc,

This branch contains expired certificates for the rmq1 host in the docker compose environment.


As expected, the shovel does not start and the rmq0 host logs show this error:

[error] <0.1054.0> Shovel 'shoveled-rmq1' failed to connect (URI: amqps://rmq1:5671): {tls_alert,{certificate_expired,"TLS client: In state wait_cert at ssl_handshake.erl:2081 generated CLIENT ALERT: Fatal - Certificate Expired\n"}}
[error] <0.1054.0> Shovel 'shoveled-rmq1' has no more URIs to try for connection
[error] <0.1054.0> Shovel 'shoveled-rmq1' could not connect to destination


NOTE: check out the shovel definition here, especially the source and destination URI:


Each server is configured to verify its peer and request a client certificate. The fail_if_no_peer_certificate setting is NOT be necessary to detect an expired cert.

I tried the advanced.config method and it did not work, so I'm going to be filing a bug for that.

In your environment I'd double-check that the certs are actually expired, and that you have set up your shovel URIs in the same way as I have.

Thanks,
Luke

Luke Bakken

unread,
Dec 1, 2022, 5:23:08 PM12/1/22
to rabbitmq-users
Hi again Marc,

For what it's worth, using advanced.config to specify certificates DOES work correctly. See this project:


It will start up two RabbitMQ nodes, and a producer and consumer that send messages via a Shovel. All authentication takes place using client certificates.

Thanks,
Luke

On Monday, October 31, 2022 at 4:40:10 PM UTC-7 Luke Bakken wrote:
Hi Marc,

Luke Bakken

unread,
Dec 1, 2022, 5:23:52 PM12/1/22
to rabbitmq-users
Reply all
Reply to author
Forward
0 new messages