Shovel Error with TLS

676 views
Skip to first unread message

Dave Murphy

unread,
Sep 3, 2019, 6:38:41 PM9/3/19
to rabbitmq-users
Having a heck of a time with TLS within the RabbitMQ environment. I've enclosed the client server-a and the log from that server.

The shovel is basically failing to connect to a remote rabbitmq server. Locally, it connects through ampqs, but not remotely. The config on the remote server is the same as the client system. Any guidance would be appreciated.

It's unclear to me if I need to take further steps with Erlang, switch to the advanced config file for TLS or what the deal is.
RABBIT~1.LOG
rabbitmq.config

Luke Bakken

unread,
Sep 6, 2019, 8:35:44 AM9/6/19
to rabbitmq-users
Hi Dave,

We've got a TLS troubleshooting guide here - https://www.rabbitmq.com/troubleshooting-ssl.html

Also, can you send your shovel definitions?

Thanks -
Luke

Dave Murphy

unread,
Sep 9, 2019, 3:01:39 PM9/9/19
to rabbitm...@googlegroups.com
I used the format similar to here with no luck, but perhaps I'm not interpreting this correctly. for "myhost" is that the destination I'm trying to connect to or the local host? Second, am I supposed to load certificates into the JAVA keystore as well or anything to be performed in Erlang files?
amqps://myhost?cacertfile=/path/to/ca_certificate.pem
  &certfile=/path/to/client_certificate.pem
  &keyfile=/path/to/client_key.pem
  &verify=verify_peer
  &server_name_indication=myhost

--
You received this message because you are subscribed to a topic in the Google Groups "rabbitmq-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/rabbitmq-users/LuQ4b6T_VgQ/unsubscribe.
To unsubscribe from this group and all its topics, send an email to rabbitmq-user...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/236a038a-f743-4532-8b4b-5e61feec7db9%40googlegroups.com.

Dave Murphy

unread,
Sep 10, 2019, 4:10:36 PM9/10/19
to rabbitm...@googlegroups.com
Looing into this issue today, things may have gotten further with the new connection string format. Seeing firewall errors, so port may just be blocked at this point. Wasn't getting that far to send connection traffic. Hopefully that's all it is now.

Luke Bakken

unread,
Sep 10, 2019, 6:39:07 PM9/10/19
to rabbitmq-users
Hi Dave,

If you have a moment, would you mind noting what you changed with your amqps:// definition that worked?

I intended to write back to make sure that the paths to your certs were readable by RabbitMQ but it appears you found a different solution.

Thanks,
Luke
To unsubscribe from this group and all its topics, send an email to rabbitmq-users+unsubscribe@googlegroups.com.

Dave Murphy

unread,
Sep 12, 2019, 2:55:59 PM9/12/19
to rabbitm...@googlegroups.com
Firewall issue looks to be resolved, but connection is resetting/failing. I ran wireshark on the target host and saw connection attempts which then get reset. On the shovel client I'm getting this:

2019-09-12 11:25:46.166 [error] <0.1040.0> Shovel 'my_first_shovel' failed to connect (URI: amqps://target.server:5671): closed
2019-09-12 11:25:46.166 [error] <0.1040.0> Shovel 'my_first_shovel' has no more URIs to try for connection and will terminate
2019-09-12 11:25:46.166 [info] <0.1040.0> terminating static worker with {failed_to_connect_using_provided_uris,[{rabbit_amqp091_shovel,make_conn_and_chan,2,[{file,"src/rabbit_amqp091_shovel.erl"},{line,324}]},{rabbit_amqp091_shovel,connect_source,1,[{file,"src/rabbit_amqp091_shovel.erl"},{line,78}]},{rabbit_shovel_worker,handle_cast,2,[{file,"src/rabbit_shovel_worker.erl"},{line,64}]},{gen_server2,handle_msg,2,[{file,"src/gen_server2.erl"},{line,1050}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,249}]}]}
2019-09-12 11:25:46.166 [error] <0.1040.0> ** Generic server <0.1040.0> terminating
** Last message in was {'$gen_cast',init}
** When Server state == {state,undefined,undefined,undefined,undefined,my_first_shovel,static,#{ack_mode => on_confirm,dest => #{add_forward_headers => false,add_timestamp_header => false,fields_fun => #Fun<rabbit_amqp091_shovel.17.14964843>,module => rabbit_amqp091_shovel,props_fun => #Fun<rabbit_amqp091_shovel.17.14964843>,resource_decl => #Fun<rabbit_amqp091_shovel.22.14964843>,uris => ["amqps://administrator:Password1@localhost"]},name => my_first_shovel,reconnect_delay => 5,shovel_type => static,source => #{delete_after => never,module => rabbit_amqp091_shovel,prefetch_count => 10,queue => <<"Store.1.3470">>,resource_decl => #Fun<rabbit_amqp091_shovel.22.14964843>,uris => ["amqps:// target.server :5671?cacertfile=C:\\RabbitMQ\\domain-ca.crt&certfile=C:\\RabbitMQ\\client.crt&keyfile=C:\\RabbitMQ\\client.unc.key&verify=verify_peer&server_name_indication= target.server "]}},undefined,undefined,undefined,undefined,undefined}
** Reason for termination ==
** {failed_to_connect_using_provided_uris,[{rabbit_amqp091_shovel,make_conn_and_chan,2,[{file,"src/rabbit_amqp091_shovel.erl"},{line,324}]},{rabbit_amqp091_shovel,connect_source,1,[{file,"src/rabbit_amqp091_shovel.erl"},{line,78}]},{rabbit_shovel_worker,handle_cast,2,[{file,"src/rabbit_shovel_worker.erl"},{line,64}]},{gen_server2,handle_msg,2,[{file,"src/gen_server2.erl"},{line,1050}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,249}]}]}
2019-09-12 11:25:46.166 [error] <0.1040.0> CRASH REPORT Process <0.1040.0> with 0 neighbours exited with reason: failed_to_connect_using_provided_uris in rabbit_amqp091_shovel:make_conn_and_chan/2 line 324 in gen_server2:terminate/3 line 1166
2019-09-12 11:25:46.166 [error] <0.1036.0> Supervisor {<0.1036.0>,mirrored_supervisor_sups} had child my_first_shovel started with rabbit_shovel_worker:start_link(static, my_first_shovel, #{ack_mode => on_confirm,dest => #{add_forward_headers => false,add_timestamp_header => false,fields_fun => ...,...},...}) at <0.1040.0> exit with reason failed_to_connect_using_provided_uris in rabbit_amqp091_shovel:make_conn_and_chan/2 line 324 in context child_terminated

**************************************************************************************************************************  

Here is the shovel connection config:

{rabbitmq_shovel,
  [{shovels,
    [
     {my_first_shovel,
      [

       {sources,
         [{brokers,  ["amqps://target.server:5671?cacertfile=C:\\RabbitMQ\\domain-ca.crt&certfile=C:\\RabbitMQ\\client.crt&keyfile=C:\\RabbitMQ\\client.key&verify=verify_peer&server_name_indication= target.server "]},
         {declarations, []}]
        },

       {destinations,
         [{broker, "amqps://admin:pwd@localhost"},
         {declarations, []}]
        },

     {queue, <<"Store.1.3470">>},

     {prefetch_count, 10},

     {ack_mode, on_confirm},

     {publish_fields, [{exchange,<<"Ncr.Retail.Store">>}]},

     {publish_properties, [{delivery_mode, 2}]},

     {reconnect_delay, 5}

**************************************************************************************************************************

The certs were converted with OpenSSL, just without the pem extension but are in proper form. I concatenated the domain certs as the hosts are in different domains with different CA's. The key was saved without a password, so no password needed in the connection.


To unsubscribe from this group and all its topics, send an email to rabbitmq-user...@googlegroups.com.

--
You received this message because you are subscribed to a topic in the Google Groups "rabbitmq-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/rabbitmq-users/LuQ4b6T_VgQ/unsubscribe.
To unsubscribe from this group and all its topics, send an email to rabbitmq-user...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/2fd103da-1b27-4acc-a2ee-78e6ac1611f3%40googlegroups.com.

Michael Klishin

unread,
Sep 17, 2019, 2:14:54 PM9/17/19
to rabbitmq-users
The exception says that a Shovel failed to connect. Target node logs would potentially have clues [1]. [2][3] are also relevant.

Dave Murphy

unread,
Sep 17, 2019, 6:20:45 PM9/17/19
to rabbitm...@googlegroups.com
Maybe I need to take a step back. The existing shovel uses the non-secure method, sending the username and password in the ampq:// connection string. That works no problem.

In transitioning to a TLS based connection, there seems to be no username/password portion. I've read through the docs but I cannot for the life of me make sense of what needs to be on both client and server. Everything appears to be there. Do I have to allow a specific host access on the server now that I'm using a certificate, versus a user?

I checked the target server log, no connection attempts logged for port 5671. It's like the handshake never happens or completes and with no connection, no log.

--
You received this message because you are subscribed to a topic in the Google Groups "rabbitmq-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/rabbitmq-users/LuQ4b6T_VgQ/unsubscribe.
To unsubscribe from this group and all its topics, send an email to rabbitmq-user...@googlegroups.com.

Dave Murphy

unread,
Sep 17, 2019, 7:07:16 PM9/17/19
to rabbitm...@googlegroups.com
Went through the troubleshooting steps and all appears to check out.

The steps recommended in this guide are:

Verify effective configuration - Configuration works with non SSL on 5672

Verify that the node listens for TLS connections
 Run rabbitmq-diagnostics listeners (doesn't work in windows)
 Run rabbitmqctl status - Seems to be listening on target server
{listeners,
     [{clustering,25672,"::"},
      {amqp,5672,"::"},
      {amqp,5672,"0.0.0.0"},
      {'amqp/ssl',5671,"::"},
      {'amqp/ssl',5671,"0.0.0.0"
      {http,15672,"::"},
      {http,15672,"0.0.0.0"}]},
Verify file permissions - Successful load on service startup

Verify TLS support in Erlang/OTP
 rabbitmq-diagnostics --silent tls_versions (doesn't work in windows)
C:\>rabbitmqctl eval ssl:versions().
[{ssl_app,"9.1"},
{supported,['tlsv1.2','tlsv1.1',tlsv1]},
{supported_dtls,['dtlsv1.2',dtlsv1]},
{available,['tlsv1.2','tlsv1.1',tlsv1,sslv3]},
{available_dtls,['dtlsv1.2',dtlsv1]}]

Verify certificate/key pairs and test with alternative TLS client or server using OpenSSL command line tools (successful)
---
No client certificate CA names sent
Peer signing digest: SHA512
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 3209 bytes and written 433 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES256-GCM-SHA384
    Session-ID: 901C10F619EEFD184235299238C4F64AF0B4978A7F11B14DC79A5C06F9D6DA61

    Session-ID-ctx:
    Master-Key: B906536F451B636F05D2A1BE2D21E3E1C50411D2C3CB3D1FFBE2A26AFC3F7C25
AA071C3EE22B26EC301F1D041BEF6F44
    Key-Arg   : None
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 300 (seconds)
    TLS session ticket:
    0000 - 86 a8 06 f0 97 a8 62 53-43 27 99 00 6d 93 5e 30   ......bSC'..m.^0
    0010 - 61 64 61 56 59 e2 a5 a3-0e ed b8 23 a7 70 78 fd   adaVY......#.px.
    0020 - 8f 89 25 bf 8e cf a6 d9-fb 36 dc 88 2a 34 d5 25   ..%......6..*4.%
    0030 - 08 9d b9 cd fb 98 bc 23-7f 82 18 1c 3f 30 b0 45   .......#....?0.E
    0040 - 29 85 a3 6f da ec e0 d5-9b 3b 72 c3 85 7f 4c a0   )..o.....;r...L.
    0050 - ce 17 ed 23 b3 1a d2 00-1c a4 12 a5 04 f2 21 73   ...#..........!s
    0060 - 05 c9 d0 28 4f 1d 24 69-24 80 80 6a 01 eb 17 b1   ...(O.$i$..j....
    0070 - 9e 72 f4 f1 97 9a 13 c3-37 94 8f 9c 45 2d 40 cd   .r......7...E-@.
    0080 - 8c 49 c5 94 1b 8f 18 44-62 9e 23 40 27 b7 0d 57   .I.....Db.#@'..W
    0090 - b1 d6 64 9a 92 7a 20 ec-ac cc 27 a9 80 0b 3c 35   ..d..z ...'...<5

    Start Time: 1568760835
    Timeout   : 300 (sec)
    Verify return code: 0 (ok)
---

Verify available and configured cipher suites and certificate key usage options - Large list, seems to be ok

Verify client connections with a TLS-terminating proxy

And finally, test a real client connection against a real server connection again
openssl s_client -connect targetserver:5671 -cert c:\\RabbitMQ\\client.crt -key C:\\RabbitMQ\\client.unc.key -CAfile C:\\RabbitMQ\\domain-ca.crt
---
No client certificate CA names sent
Peer signing digest: SHA512
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 3055 bytes and written 302 bytes
Verification: OK
---
New, TLSv1.2, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES256-GCM-SHA384
    Session-ID: 61B9C8F97617A8E594537AA63D7138CF9A5E29128299D6F44D0D1074B85ADCC0
    Session-ID-ctx:
    Master-Key: 536494781ED1DA1A843DBD07BDB7828C875ECE835901EA7AF04DF1D8BC889C53E4D74A7231DC728A3C79D9735F3A1796
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: 1568761225
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
    Extended master secret: no
---
read:errno=0

On Tue, Sep 17, 2019 at 11:15 AM Michael Klishin <mkli...@pivotal.io> wrote:
--
You received this message because you are subscribed to a topic in the Google Groups "rabbitmq-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/rabbitmq-users/LuQ4b6T_VgQ/unsubscribe.
To unsubscribe from this group and all its topics, send an email to rabbitmq-user...@googlegroups.com.

Dave Murphy

unread,
Sep 17, 2019, 7:56:10 PM9/17/19
to rabbitm...@googlegroups.com
Ok, this may shed some light on things.
I reconstructed the shovel connection from the ground up using the formatting from here: https://www.rabbitmq.com/shovel-static.html

Here is what I see now: Login was refused using authentication mechanism PLAIN

On Tue, Sep 17, 2019 at 11:15 AM Michael Klishin <mkli...@pivotal.io> wrote:
--
You received this message because you are subscribed to a topic in the Google Groups "rabbitmq-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/rabbitmq-users/LuQ4b6T_VgQ/unsubscribe.
To unsubscribe from this group and all its topics, send an email to rabbitmq-user...@googlegroups.com.

Michael Klishin

unread,
Sep 18, 2019, 1:39:06 PM9/18/19
to rabbitmq-users
To unsubscribe from this group and all its topics, send an email to rabbitmq-users+unsubscribe@googlegroups.com.

Dave Murphy

unread,
Sep 18, 2019, 11:55:49 PM9/18/19
to rabbitm...@googlegroups.com
Was not pre-pending the username and password with the connection string. I believe it's fixed now.... that was a pain.

To unsubscribe from this group and all its topics, send an email to rabbitmq-user...@googlegroups.com.

--
You received this message because you are subscribed to a topic in the Google Groups "rabbitmq-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/rabbitmq-users/LuQ4b6T_VgQ/unsubscribe.
To unsubscribe from this group and all its topics, send an email to rabbitmq-user...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/fddbdd96-345b-4ada-9304-85f9bbb5369e%40googlegroups.com.
Reply all
Reply to author
Forward
0 new messages