Re-enabling logging of TLS errors

[Woon Yung Liu]

Hi,

Back in 2019, I recall that RabbitMQ would log TLS-related errors (e.g. handshake failure, certificate revoked etc), which are useful for auditing.

Towards the end of 2019, this functionality disappeared and I have not successfully identified what brought about this change.

Since it was a POC's demo that was being completed, I simply rolled back the Docker image to the version with RabbitMQ 3.7.14 and completed the demo.

Does anyone have any idea as to how I might be able to re-enable such logging? Or better yet, have any suggestions for auditing such errors with RabbitMQ?

I've been looking through the Erlang documentation and RabbitMQ's source code, but Erlang is very much Greek to me. :(

I guess it disappeared with OTP 22.0, when they "improved" logging for the SSL app.

I've tried to set the log_level option of ssl_options to debug, as well as configuring the log categories for "default" and "connection" to "debug" (I also tried "all" too). But nothing about TLS seems to appear.

Thanks in advance.

[Luke Bakken]

Hello,

We haven't made specific changes to reduce TLS logging. The Erlang VM is what implements TLS as well as logging, so something may have changed there.

Can you provide a very specific example of something that has changed?

Thanks -

Luke

[Woon Yung Liu]

Hi again,

I understand that it likely lies at the Erlang side of things, but I haven't figured out how to make things work.

I'm hoping to find folks here who have tried achieving this before, who can share their experience.

From what I saved, some of the TLS alert messages used to look like these:

2019-10-17 13:48:01.586 [info] <0.1917.0> TLS server: In state certify at ssl_handshake.erl:1370 generated SERVER ALERT: Fatal - Unknown CA

2019-10-17 13:47:06.409 [info] <0.920.0> TLS server: In state certify at ssl_handshake.erl:1363 generated SERVER ALERT: Fatal - Certificate Revoked

They would appear without any special configuration to the logging settings. I just needed to enable TLS and such messages would appear, if somebody deliberately/accidentally used a misconfigured TLS client to connect.

Once again, thank you.

Best Regards,

Woon Yung

[Luke Bakken]

Hello,

Yep, you will still see those messages if they happen with the most recent RabbitMQ and Erlang versions.

If you can provide a scenario in which you expect to see them but they don't appear, that would be interesting. Providing the certificates as well as code if necessary would allow me to try and diagnose it.

Thanks,

Luke

[Woon Yung Liu]

Hi again,

That's odd. I don't recall having seen any of those messages for a really long time. I'm not looking at a specific scenario, but logging of all TLS errors in general. For testing, I've been just connecting using openssl s_client, with a keypair issued by a CA that is not listed within the cacert file that RabbitMQ has.

On a side note: after struggling with Erlang, I've finally figured out how to see the OTP 22+ SSL app's debug messages; they appear only in the console, thus I had to start RabbitMQ manually instead of starting the Windows service. Unfortunately, those "TLS Server" alert messages still don't seem to appear anywhere, so I don't see anything about the connection's acceptance having failed (i.e. it silently gets dropped).

Since I'm now using RabbitMQ 3.8.21 and OTP 24.0, I'll see if upgrading again would help. I'll continue looking around after that.

Once again, thank you.

Best Regards,

Woon Yung

[Luke Bakken]

Ok, this is definitely worth looking into.

I will try a test like you describe on my Windows machine using Erlang 24.3.1 and RabbitMQ 3.9.13

I will test with openssl s_client and with a Python client app using the Pika library.

Would you mind attaching your complete RabbitMQ config files as well as provide any other config info? Did you set any environment vars, for example?

Thanks,

Luke

--
You received this message because you are subscribed to a topic in the Google Groups "rabbitmq-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/rabbitmq-users/VMURp2izHfE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to rabbitmq-user...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/a81b56c7-b8de-4505-9ffa-733aede05af9n%40googlegroups.com.

[Woon Yung Liu]

Hi again,

Your responses give me the impression that the lack of alerts is unexpected.

So I repeated this test, but back in Docker again. I tested various images from 3.7.x to 3.9.x, from Dockerhub. I have attached the docker-compose.yml and relevant configuration files, in case you would like them. My certificates are private and I could generate a new set for testing if you would prefer that done, but there is nothing special about the certificates I use.

Here is a list of some of the images tested, as well as the RabbitMQ & Erlang OTP versions within:

-------------------------------------------------------

rabbitmq:3.9.13-management-alpine

RabbitMQ 3.9.8 Erlang 24.1.4

-------------------------------------------------------

rabbitmq:3.9.0-management-alpine
RabbitMQ 3.9.0 Erlang 24.0.4
-------------------------------------------------------

rabbitmq:3.8.27-management-alpine
RabbitMQ 3.8.27 Erlang 24.3.2

-------------------------------------------------------

rabbitmq:3.8.6-management-alpine
RabbitMQ 3.8.6 Erlang 23.0.3
-------------------------------------------------------

rabbitmq:3.8.5-management-alpine
RabbitMQ 3.8.5 Erlang 23.0.3

-------------------------------------------------------

rabbitmq:3.8.0-management-alpine
3.8.0 Erlang 22.1.5
-------------------------------------------------------

rabbitmq:3.7.28-management-alpine
3.7.28 Erlang 22.3.4.7
-------------------------------------------------------

Throughout the whole 3.9.x series, I could see the TLS alerts:

    rabbitmq_1  | 2022-03-17 01:26:52.658902+00:00 [noti] <0.931.0> TLS server: In state wait_cert at ssl_handshake.erl:2015 generated SERVER ALERT: Fatal - Unknown CA

From 3.8.6 to 3.8.27, this appears:

    rabbitmq_1  | Logger - error: {removed_failing_handler,rabbit_log}

Before 3.8.5, nothing is logged. When I tested 3.7.28, nothing is logged either.

----

As a final test, I upgraded the 3.8.21 installation on my laptop to 3.9.13, and the alerts started appearing too. I didn't have to change the Erlang runtime.

I'm not sure whether this means there is an issue with 3.8.x. But for my project's purposes, I suppose moving up to 3.9.x is the direction I need.

Once again, thank you for your time.

Best Regards,

Woon Yung

[Luke Bakken]

Hello,

I tested RabbitMQ 3.7.28 / Erlang 21.3.8.24 using the attached Python test program and RabbitMQ configuration file. When I use two sets of non-related certificates, I see the following logged to both the RabbitMQ console as well as the log file:

2022-03-21 11:03:50.261 [info] <0.753.0> TLS server: In state certify received CLIENT ALERT: Fatal - Unknown CA

Note that I do not see the above message when testing with openssl s_client

Perhaps there was an issue with Docker but it sounds like everything is resolved with the latest RabbitMQ and Erlang.

Thanks,

Luke

On Wednesday, March 16, 2022 at 7:25:23 PM UTC-7 piror...@gmail.com wrote:

Hi again,