RabbitMQ Shovels sends messages to other RabbitMQ Server despite certificates expired

[Marc Peter]

Hello everyone, 

We are currently testing different states of our ssl encryption with RabbitMQ an ran into a quite confusion behaviour,  which does not match our expectations. 

We have two computers with RabbitMQ installed. One, lets call it client, is not encrypted locally, but the shovels communicate with the second, lets call it server, which is using ssl.
Everything on the client computer is not using ssl locally but once the messages leave the client via the shovels, this connection should be encrypted using ssl, since the server computer runs with ssl enabled. 

Here is a png of the shovel status to get a better picture of the setup:

Lets talk about the behaviour which confuses us. If our used certificates expires and is not valid anymore. The client shovels still sends the messages to the server computer. Message queues are still empty because they are send immediatly: 

 Our expectation here would be, that the messages should not be send but instead should be stacked in the in the shovels, when the certificates are not valid anymore.

In the log are warnings about missing configurations, but we declared these in the config. 

I will append the used config to this.

Is there something wrong with the setup or the config file? Or are our expectations regarding the behavior wrong? What must be changed to achive our expectations?

Thank you in advance.

Greetings,

Marc

[Marc Peter]

I forgot to mention that the version of RabbitMQ we are using is the version 3.10.6 with Erlang 25

Greetings,
Marc

[Luke Bakken]

Hi Marc,

Unfortunately some of your screenshots are too small to see.

I would start by exporting your definitions and attaching them to your response. My guess is that you are not enabling verification in your shovel URIs.

Thanks,

Luke

[Marc Peter]

Hi Luke,

thanks for your response. I reattachted the screenshots i used. 

I will look into my definitions again and attach them.

Thanks,
Marc

[Marc Peter]

Hello,

as requested i attatched the exported definitions.

Greetings,

Marc

[Luke Bakken]

Yes, you will have to enable server verification for your amqps URIs, like this - https://www.rabbitmq.com/uri-query-parameters.html#basics

[Marc Peter]

Hi Luke,

sadly I don't get your suggested fix to work.

When add the  amqp_client.ssl_options to the rabbitmq.conf, the service wont start at all. 

It might just be the syntax since i have not found an example.

I attached the conf file.

Thanks

[Luke Bakken]

Hello,

RabbitMQ probably logs why it won't start. Did you check the log file?

I meant for you to change your shovel URI strings instead, but your method should work. However, the documentation states that you must specify amqp_client settings in the advanced.config file, not in the rabbitmq.conf file!

Create a file named advanced.config in the same directory as your rabbitmq.conf file with this content, remove the invalid settings from rabbitmq.conf, and restart:

[
    {amqp_client, [
        {ssl_options, [
            {cacertfile, "c:/certificates/converted/testdevelopment_0000_root.pem"},
            {certfile, "c:/certificates/converted/public-testdevelopment_0000.pem"},
            {keyfile, "c:/certificates/converted/private-testdevelopment_0000.pem"},
            {verify, verify_peer},
            {fail_if_no_peer_cert, true}
        ]}
    ]}
].

You should use these settings on the servers that are using amqps:// URIs for their shovels.

Thanks,

Luke

[Marc Peter]

Hi Luke,

Yes i checked the log-file but it did not why it wouldn't start.

I have implemented your suggestion but unfortunaly I didnt quite work. With the configs attached, the two rabbitmq client instance didnt send any of the messages but instead they were all stacked in the queue besides valid certificates on both rabbitmq client and rabbitmq server computers.

 

I also tried it with ssl_options.fail_if_no_peer_cert = false on the rabbitmq server instance but then it seems to be always sending the messages. Even when the certificates provides are invalid or expired.

The rabbitmq client instance is using the amgp shovels and the rabbitmq server instance is using the amgps shovels.

I have attached all four config files.

Is the configuration on both instances fine?

Thanks,

Marc

[Luke Bakken]

Thanks for the information. I will return to this question on Friday.

[Marc Peter]

Hi Luke,

did you have the chance to look at the given information by now?

Thanks,
Marc

[Luke Bakken]

No I haven't, I have been working on higher priority items, some of which are for paying customers.

I do have this starred for follow-up.

[Luke Bakken]

Hi Marc,

I will be putting together a docker compose environment to test out your issue. If you'd like to follow along, here's my repo -

https://github.com/lukebakken/rabbitmq-users-4zRHj0C2ONM

Starting work on this now

[Luke Bakken]

Hi Marc,

This branch contains expired certificates for the rmq1 host in the docker compose environment.

https://github.com/lukebakken/rabbitmq-users-4zRHj0C2ONM/tree/lukebakken/expired-rmq1-certificates

As expected, the shovel does not start and the rmq0 host logs show this error:

[error] <0.1054.0> Shovel 'shoveled-rmq1' failed to connect (URI: amqps://rmq1:5671): {tls_alert,{certificate_expired,"TLS client: In state wait_cert at ssl_handshake.erl:2081 generated CLIENT ALERT: Fatal - Certificate Expired\n"}}
[error] <0.1054.0> Shovel 'shoveled-rmq1' has no more URIs to try for connection
[error] <0.1054.0> Shovel 'shoveled-rmq1' could not connect to destination

NOTE: check out the shovel definition here, especially the source and destination URI:

https://github.com/lukebakken/rabbitmq-users-4zRHj0C2ONM/blob/lukebakken/expired-rmq1-certificates/rmq0-defs.json#L35-L43

Each server is configured to verify its peer and request a client certificate. The fail_if_no_peer_certificate setting is NOT be necessary to detect an expired cert.

I tried the advanced.config method and it did not work, so I'm going to be filing a bug for that.

In your environment I'd double-check that the certs are actually expired, and that you have set up your shovel URIs in the same way as I have.

Thanks,

Luke

[Luke Bakken]

Hi again Marc,

For what it's worth, using advanced.config to specify certificates DOES work correctly. See this project:

https://groups.google.com/g/rabbitmq-users/c/4zRHj0C2ONM

It will start up two RabbitMQ nodes, and a producer and consumer that send messages via a Shovel. All authentication takes place using client certificates.

Thanks,

Luke

On Monday, October 31, 2022 at 4:40:10 PM UTC-7 Luke Bakken wrote:

Hi Marc,

[Luke Bakken]

Here is the correct link

https://github.com/lukebakken/rabbitmq-users-4zRHj0C2ONM