RabbitMQ Certificate Authentication issue

[Santosh Mohapatra]

Dear All,

We are trying to Implement Certificate based authentication in RabbitMQ , we have followed below steps and getting error as mentioned below, Please guide us on this, is this the Certificate issue or due to any Config issue.

- Deployed Certificate and keys in RabbitMQ server

- Using Java trust store connecting to RabbitMQ server and getting below error:

Config File:

[
  {ssl, [{versions, ['tlsv1.3', 'tlsv1.2','tlsv1.1']}]},
  {rabbit, [{ssl_listeners, [5671]},
  {auth_mechanisms, ['PLAIN', 'AMQPLAIN', 'EXTERNAL']},
  {ssl_options, [{cacertfile,           "/etc/pki/tls/_.picouser.int.sap.hana.ondemand.com_DigiCert_TLS_RSA_SHA256_2020_CA1_"},
                           {certfile,   "/etc/pki/tls/publickey.pem"},
                           {keyfile,    "/etc/pki/tls/privatekey.pem"},
                           {verify,     verify_peer},
                           {versions, ['tlsv1.2', 'tlsv1.1']},
                           {password,  "changeit"},
                           {fail_if_no_peer_cert, true}]},
    {log, [{file, [{level, debug}]}]}]}
].

Error:

Exception in thread "main" javax.net.ssl.SSLException: Couldn't kickstart handshaking
    at sun.security.ssl.Alert.createSSLException(Alert.java:127)
    at sun.security.ssl.TransportContext.fatal(TransportContext.java:370)
    at sun.security.ssl.TransportContext.fatal(TransportContext.java:313)
    at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:411)
    at sun.security.ssl.SSLSocketImpl.ensureNegotiated(SSLSocketImpl.java:819)
    at sun.security.ssl.SSLSocketImpl.access$200(SSLSocketImpl.java:75)
    at sun.security.ssl.SSLSocketImpl$AppOutputStream.write(SSLSocketImpl.java:1104)
    at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
    at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
    at java.io.DataOutputStream.flush(DataOutputStream.java:123)
    at com.rabbitmq.client.impl.SocketFrameHandler.sendHeader(SocketFrameHandler.java:160)
    at com.rabbitmq.client.impl.SocketFrameHandler.sendHeader(SocketFrameHandler.java:170)
    at com.rabbitmq.client.impl.AMQConnection.start(AMQConnection.java:314)
    at com.rabbitmq.client.impl.recovery.RecoveryAwareAMQConnectionFactory.newConnection(RecoveryAwareAMQConnectionFactory.java:65)
    at com.rabbitmq.client.impl.recovery.AutorecoveringConnection.init(AutorecoveringConnection.java:160)
    at com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:1216)
    at com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:1173)
    at com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:1131)
    at com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:1294)
    at CertificateAuth.main(CertificateAuth.java:41)
    Suppressed: java.net.SocketException: Connection reset by peer: socket write error
        at java.net.SocketOutputStream.socketWrite0(Native Method)
        at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:111)
        at java.net.SocketOutputStream.write(SocketOutputStream.java:155)
        at sun.security.ssl.SSLSocketOutputRecord.encodeAlert(SSLSocketOutputRecord.java:83)
        at sun.security.ssl.TransportContext.fatal(TransportContext.java:401)
        ... 18 more
Caused by: java.net.SocketException: Connection reset by peer: socket write error
    at java.net.SocketOutputStream.socketWrite0(Native Method)
    at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:111)
    at java.net.SocketOutputStream.write(SocketOutputStream.java:155)
    at sun.security.ssl.SSLSocketOutputRecord.flush(SSLSocketOutputRecord.java:268)
    at sun.security.ssl.HandshakeOutStream.flush(HandshakeOutStream.java:89)
    at sun.security.ssl.ClientHello$ClientHelloKickstartProducer.produce(ClientHello.java:572)
    at sun.security.ssl.SSLHandshake.kickstart(SSLHandshake.java:509)
    at sun.security.ssl.ClientHandshakeContext.kickstart(ClientHandshakeContext.java:115)
    at sun.security.ssl.TransportContext.kickstart(TransportContext.java:250)
    at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:394)
    ... 16 more

Thanks and Regards

Santosh

[Luke Bakken]

Hi Santosh,

Please see our TLS troubleshooting guide to first determine if your certificates were created correctly: https://www.rabbitmq.com/troubleshooting-ssl.html

You should be able to use the s_server and s_client commands to validate everything, including client certificates. When you verify your certs, please capture all of the output from all commands into a file and attach the file for review. Note that certificates will be present but they will display public information only. You can redact some of the information if necessary.

Questions -

• What version of RabbitMQ and Erlang are you using?
• What version of Java and the Java AMQP client are you using?
• Are you using wildcard certificates?
• Are you trying to use the X509 certificate to authenticate client applications? If so, have you done the following -
  • Enabled the  rabbitmq_auth_mechanism_ssl plugin
  • Create an X509 cert where the CN value contains the username you wish to use
  • Create a password-less user in RabbitMQ with the CN / username value
• What is logged by RabbitMQ when your application logs " Couldn't kickstart handshaking"?

Thanks,

Luke

[Chetan Chauhan]

Hello,

Please help us to connect the local rabbitMQ server using the certificate based authentication.

• What version of RabbitMQ and Erlang are you using?  RabbitMQ 3.10.5 and  Erlang 25.0.1
• What version of Java and the Java AMQP client are you using? JDK 1.8 and  AMQP client 5.14
• Are you using wildcard certificates?  yes

• Are you trying to use the X509 certificate to authenticate client applications? If so, have you done the following -

• • Enabled the  rabbitmq_auth_mechanism_ssl plugin : YES
  • Create an X509 cert where the CN value contains the username you wish to use : Yes, i tried that part as well.
  • Create a password-less user in RabbitMQ with the CN / username value : Yes done that
• What is logged by RabbitMQ when your application logs " Couldn't kickstart handshaking"?  RabbitMQ log file not having any entry regarding the " Couldn't kickstart handshaking  ". 

please let me know. do i post more information ?

Thank You,

Chetan Kumar Chauhan.

[Luke Bakken]

Hello,

Since you are using wildcard certificates your setup will be slightly more complicated than usual.

You misunderstood my question below. When your application tries to connect, but fails, what does RabbitMQ log in its log file? I'm guessing it is something like "TLS handshake failed". You can attach a compressed archive of your RabbitMQ logs in your response and I can check it.

Can you also provide the commands you are running to create your Java cert trust store and add your certs to it?

Thanks,

Luke

[Chetan Chauhan]

Hello,

PFB the RabbitMQ logs:

2022-09-23 18:13:37.267000+05:30 [notice] <0.698.0> TLS server: In state hello at tls_handshake.erl:249 generated SERVER ALERT: Fatal - Handshake Failure
2022-09-23 18:13:37.267000+05:30 [notice] <0.698.0>  - malformed_handshake_data
2022-09-23 18:13:51.464000+05:30 [error] <0.641.0> Command timed out: 'powershell.exe -NoLogo -NoProfile -NonInteractive -Command (Get-PSDrive c).Free'
2022-09-23 18:13:51.464000+05:30 [error] <0.641.0> ** Generic server rabbit_disk_monitor terminating
2022-09-23 18:13:51.464000+05:30 [error] <0.641.0> ** Last message in was update
2022-09-23 18:13:51.464000+05:30 [error] <0.641.0> ** When Server state == {state,"c:/Users/chetankumar.chauhan/AppData/Roaming/RabbitMQ/db/rabbit@LP-5CD138BK05-mnesia",
2022-09-23 18:13:51.464000+05:30 [error] <0.641.0>                                50000000,418641043456,100,10000,
2022-09-23 18:13:51.464000+05:30 [error] <0.641.0>                                #Ref<0.2290225655.1637351426.164775>,false,
2022-09-23 18:13:51.464000+05:30 [error] <0.641.0>                                true,10,120000}
2022-09-23 18:13:51.464000+05:30 [error] <0.641.0> ** Reason for termination ==
2022-09-23 18:13:51.464000+05:30 [error] <0.641.0> ** {could_not_determine_disk_free,
2022-09-23 18:13:51.464000+05:30 [error] <0.641.0>        [{rabbit_disk_monitor,get_disk_free,2,
2022-09-23 18:13:51.464000+05:30 [error] <0.641.0>             [{file,"rabbit_disk_monitor.erl"},{line,254}]},
2022-09-23 18:13:51.464000+05:30 [error] <0.641.0>         {rabbit_disk_monitor,internal_update,1,
2022-09-23 18:13:51.464000+05:30 [error] <0.641.0>             [{file,"rabbit_disk_monitor.erl"},{line,216}]},
2022-09-23 18:13:51.464000+05:30 [error] <0.641.0>         {rabbit_disk_monitor,handle_info,2,
2022-09-23 18:13:51.464000+05:30 [error] <0.641.0>             [{file,"rabbit_disk_monitor.erl"},{line,166}]},
2022-09-23 18:13:51.464000+05:30 [error] <0.641.0>         {gen_server,try_dispatch,4,[{file,"gen_server.erl"},{line,1120}]},
2022-09-23 18:13:51.464000+05:30 [error] <0.641.0>         {gen_server,handle_msg,6,[{file,"gen_server.erl"},{line,1197}]},
2022-09-23 18:13:51.464000+05:30 [error] <0.641.0>         {proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]}
2022-09-23 18:13:51.464000+05:30 [error] <0.641.0>
2022-09-23 18:13:51.464000+05:30 [error] <0.641.0>   crasher:
2022-09-23 18:13:51.464000+05:30 [error] <0.641.0>     initial call: rabbit_disk_monitor:init/1
2022-09-23 18:13:51.464000+05:30 [error] <0.641.0>     pid: <0.641.0>
2022-09-23 18:13:51.464000+05:30 [error] <0.641.0>     registered_name: rabbit_disk_monitor
2022-09-23 18:13:51.464000+05:30 [error] <0.641.0>     exception exit: could_not_determine_disk_free
2022-09-23 18:13:51.464000+05:30 [error] <0.641.0>       in function  rabbit_disk_monitor:get_disk_free/2 (rabbit_disk_monitor.erl, line 254)
2022-09-23 18:13:51.464000+05:30 [error] <0.641.0>       in call from rabbit_disk_monitor:internal_update/1 (rabbit_disk_monitor.erl, line 216)
2022-09-23 18:13:51.464000+05:30 [error] <0.641.0>       in call from rabbit_disk_monitor:handle_info/2 (rabbit_disk_monitor.erl, line 166)
2022-09-23 18:13:51.464000+05:30 [error] <0.641.0>       in call from gen_server:try_dispatch/4 (gen_server.erl, line 1120)
2022-09-23 18:13:51.464000+05:30 [error] <0.641.0>       in call from gen_server:handle_msg/6 (gen_server.erl, line 1197)
2022-09-23 18:13:51.464000+05:30 [error] <0.641.0>     ancestors: [rabbit_disk_monitor_sup,rabbit_sup,<0.220.0>]
2022-09-23 18:13:51.464000+05:30 [error] <0.641.0>     message_queue_len: 0
2022-09-23 18:13:51.464000+05:30 [error] <0.641.0>     messages: []
2022-09-23 18:13:51.464000+05:30 [error] <0.641.0>     links: [<0.351.0>]
2022-09-23 18:13:51.464000+05:30 [error] <0.641.0>     dictionary: []
2022-09-23 18:13:51.464000+05:30 [error] <0.641.0>     trap_exit: false
2022-09-23 18:13:51.464000+05:30 [error] <0.641.0>     status: running
2022-09-23 18:13:51.464000+05:30 [error] <0.641.0>     heap_size: 4185
2022-09-23 18:13:51.464000+05:30 [error] <0.641.0>     stack_size: 28
2022-09-23 18:13:51.464000+05:30 [error] <0.641.0>     reductions: 27847
2022-09-23 18:13:51.464000+05:30 [error] <0.641.0>   neighbours:
2022-09-23 18:13:51.464000+05:30 [error] <0.641.0>
2022-09-23 18:13:51.464000+05:30 [error] <0.351.0>     supervisor: {local,rabbit_disk_monitor_sup}
2022-09-23 18:13:51.464000+05:30 [error] <0.351.0>     errorContext: child_terminated
2022-09-23 18:13:51.464000+05:30 [error] <0.351.0>     reason: could_not_determine_disk_free
2022-09-23 18:13:51.464000+05:30 [error] <0.351.0>     offender: [{pid,<0.641.0>},
2022-09-23 18:13:51.464000+05:30 [error] <0.351.0>                {id,rabbit_disk_monitor},
2022-09-23 18:13:51.464000+05:30 [error] <0.351.0>                {mfargs,{rabbit_disk_monitor,start_link,[50000000]}},
2022-09-23 18:13:51.464000+05:30 [error] <0.351.0>                {restart_type,{transient,1}},
2022-09-23 18:13:51.464000+05:30 [error] <0.351.0>                {shutdown,300000},
2022-09-23 18:13:51.464000+05:30 [error] <0.351.0>                {child_type,worker}]
2022-09-23 18:13:55.047000+05:30 [info] <0.703.0> Enabling free disk space monitoring

Command to add the cert:

keytool -import  -trustcacerts -alias picouser_2023 -file "\cert\_.picouser.int.sap.hana.ondemand.com_DigiCert_TLS_RSA_SHA256_2020_CA1_.cer" -keystore "C:\Program Files\Java\jdk1.8.0_311\jre\lib\security\cacerts"

Please let me know if you want more information.

Thank You,

Chetan Kumar Chauhan.

[Luke Bakken]

Thank you! I will try to find time to assist within the next week.

[Chetan Chauhan]

Hello,

May, I know did you find anything ?

Thank You,

Chetan Kumar Chauhan.

[Luke Bakken]

Please be patient. It hasn't been a week and I'm on vacation this week.

[Chetan Chauhan]

Hello,

Did you get a chance to check this issue ?

Please sorry to ping you again as its urgent.

Thank You,

Chetan Kumar Chauhan.

[Luke Bakken]

Hi Chetan,

I have some time to look at this today. Please remember that the support you receive here (and all over the internet) is free so patience is appreciated.

If this is truly urgent, paid support for RabbitMQ is available - https://www.rabbitmq.com/#support

I forgot to ask - please let me know what version of RabbitMQ and Erlang you are using. I will be testing with RabbitMQ 3.11.1 and Erlang 25.1.1

Luke

[Luke Bakken]

Hi Chetan,

Please see this complete example that shows how to create certificates for use with Java key stores -

https://github.com/lukebakken/rabbitmq-users-CzgiqOUso2k

It uses our tls-gen project to create an initial set of certificates - https://github.com/rabbitmq/tls-gen

Then, look at the setup-certs.sh script to see how I add them to Java key stores.

Note the custom Dockerfiles in the consumer and producer directories that pass the correct arguments to the JVM to specify the key stores.

Please carefully review the repository and let me know if you have any questions.

I have not yet tried this with wildcard certificates but I'm going to do that next and will follow up.

Thanks,

Luke

[Luke Bakken]

This branch demonstrates that using a wildcard server certificate works correctly:

https://github.com/lukebakken/rabbitmq-users-CzgiqOUso2k/tree/lukebakken/wildcard

Note that you can't use wildcard client certificates.