RabbitMQ using SSL and client certificate validation failure
3,199 views
Skip to first unread message
Jeremy Gardner
Jun 13, 2019, 1:27:06 PM6/13/19
to rabbitmq-users
Good Afternoon,
I am currently prototyping RabbitMQ to use SSL and client certificates on a Windows 7 system. The certificates being used are all installed to the proper stores in windows (e.g. the root is installed to Trusted Root, the intermediate is in Intermediate, and the client and server certificates are installed to Personal). The certificates are in both the Local Computer and Current User stores. When I start the server, this is what the log looks like:
2019-06-13 12:15:49.469 [warning] <0.361.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent": rebuilding indices from scratch
2019-06-13 12:15:49.471 [info] <0.354.0> Started message store of type persistent for vhost '/'
2019-06-13 12:15:49.482 [warning] <0.388.0> Setting Ranch options together with socket options is deprecated. Please use the new map syntax that allows specifying socket options separately from other options.
2019-06-13 12:15:49.483 [info] <0.402.0> started TCP listener on [::]:5672
2019-06-13 12:15:49.483 [warning] <0.403.0> Setting Ranch options together with socket options is deprecated. Please use the new map syntax that allows specifying socket options separately from other options.
2019-06-13 12:15:49.490 [info] <0.417.0> started TCP listener on 0.0.0.0:5672
2019-06-13 12:15:49.492 [warning] <0.418.0> Setting Ranch options together with socket options is deprecated. Please use the new map syntax that allows specifying socket options separately from other options.
2019-06-13 12:15:49.493 [info] <0.433.0> started TLS (SSL) listener on [::]:5671
2019-06-13 12:15:49.494 [warning] <0.434.0> Setting Ranch options together with socket options is deprecated. Please use the new map syntax that allows specifying socket options separately from other options.
2019-06-13 12:15:49.500 [info] <0.449.0> started TLS (SSL) listener on 0.0.0.0:5671
2019-06-13 12:15:49.501 [info] <0.279.0> Setting up a table for connection tracking on this node: tracked_connection_on_node_rabbit@AL079SB3R52W
2019-06-13 12:15:49.501 [info] <0.279.0> Setting up a table for per-vhost connection counting on this node: tracked_connection_per_vhost_on_node_rabbit@AL079SB3R52W
2019-06-13 12:15:49.535 [warning] <0.475.0> Could not find handle.exe, please install from sysinternals
2019-06-13 12:15:49.575 [info] <0.528.0> Management plugin: HTTPS listener started on port 15672
2019-06-13 12:15:49.575 [info] <0.635.0> Statistics database started.
2019-06-13 12:15:49.575 [notice] <0.104.0> Changed loghwm of C:/Users/user/AppData/Roaming/RabbitMQ/log/RABBIT~1.LOG to 50
2019-06-13 12:15:49.808 [info] <0.645.0> TLS server: In state certify received CLIENT ALERT: Fatal - Certificate Unknown
2019-06-13 12:15:49.471 [info] <0.354.0> Started message store of type persistent for vhost '/'
2019-06-13 12:15:49.482 [warning] <0.388.0> Setting Ranch options together with socket options is deprecated. Please use the new map syntax that allows specifying socket options separately from other options.
2019-06-13 12:15:49.483 [info] <0.402.0> started TCP listener on [::]:5672
2019-06-13 12:15:49.483 [warning] <0.403.0> Setting Ranch options together with socket options is deprecated. Please use the new map syntax that allows specifying socket options separately from other options.
2019-06-13 12:15:49.490 [info] <0.417.0> started TCP listener on 0.0.0.0:5672
2019-06-13 12:15:49.492 [warning] <0.418.0> Setting Ranch options together with socket options is deprecated. Please use the new map syntax that allows specifying socket options separately from other options.
2019-06-13 12:15:49.493 [info] <0.433.0> started TLS (SSL) listener on [::]:5671
2019-06-13 12:15:49.494 [warning] <0.434.0> Setting Ranch options together with socket options is deprecated. Please use the new map syntax that allows specifying socket options separately from other options.
2019-06-13 12:15:49.500 [info] <0.449.0> started TLS (SSL) listener on 0.0.0.0:5671
2019-06-13 12:15:49.501 [info] <0.279.0> Setting up a table for connection tracking on this node: tracked_connection_on_node_rabbit@AL079SB3R52W
2019-06-13 12:15:49.501 [info] <0.279.0> Setting up a table for per-vhost connection counting on this node: tracked_connection_per_vhost_on_node_rabbit@AL079SB3R52W
2019-06-13 12:15:49.535 [warning] <0.475.0> Could not find handle.exe, please install from sysinternals
2019-06-13 12:15:49.575 [info] <0.528.0> Management plugin: HTTPS listener started on port 15672
2019-06-13 12:15:49.575 [info] <0.635.0> Statistics database started.
2019-06-13 12:15:49.575 [notice] <0.104.0> Changed loghwm of C:/Users/user/AppData/Roaming/RabbitMQ/log/RABBIT~1.LOG to 50
2019-06-13 12:15:49.808 [info] <0.645.0> TLS server: In state certify received CLIENT ALERT: Fatal - Certificate Unknown
2019-06-13 12:15:49.818 [info] <0.7.0> Server startup complete; 7 plugins started.
* rabbitmq_shovel_management
* rabbitmq_management
* rabbitmq_web_dispatch
* rabbitmq_shovel
* rabbitmq_management_agent
* rabbitmq_auth_mechanism_ssl
* rabbitmq_event_exchange
2019-06-13 12:15:49.872 [info] <0.649.0> TLS server: In state certify received CLIENT ALERT: Fatal - Certificate Unknown
* rabbitmq_shovel_management
* rabbitmq_management
* rabbitmq_web_dispatch
* rabbitmq_shovel
* rabbitmq_management_agent
* rabbitmq_auth_mechanism_ssl
* rabbitmq_event_exchange
2019-06-13 12:15:49.872 [info] <0.649.0> TLS server: In state certify received CLIENT ALERT: Fatal - Certificate Unknown
I'm getting errors regarding the certificates. The same certificates that are installed to windows are also in the directory referenced by the config file. The config file appears as follows:
## Default Listener
listeners.ssl.default = 5671
listeners.ssl.default = 5671
## SSL Options
ssl_options.versions.1 = tlsv1.2
ssl_options.verify = verify_peer
ssl_options.fail_if_no_peer_cert = false
ssl_options.honor_cipher_order = true
ssl_options.honor_ecc_order = true
ssl_options.client_renegotiation = false
ssl_options.secure_renegotiate = true
ssl_options.verify = verify_peer
ssl_options.fail_if_no_peer_cert = false
ssl_options.honor_cipher_order = true
ssl_options.honor_ecc_order = true
ssl_options.client_renegotiation = false
ssl_options.secure_renegotiate = true
ssl_options.ciphers.1 = ECDHE-ECDSA-AES256-GCM-SHA384
ssl_options.ciphers.2 = ECDHE-RSA-AES256-GCM-SHA384
ssl_options.ciphers.3 = ECDH-ECDSA-AES256-GCM-SHA384
ssl_options.ciphers.4 = ECDH-RSA-AES256-GCM-SHA384
ssl_options.ciphers.5 = DHE-RSA-AES256-GCM-SHA384
ssl_options.ciphers.6 = DHE-DSS-AES256-GCM-SHA384
ssl_options.ciphers.7 = ECDHE-ECDSA-AES128-GCM-SHA256
ssl_options.ciphers.8 = ECDHE-RSA-AES128-GCM-SHA256
ssl_options.ciphers.9 = ECDH-ECDSA-AES128-GCM-SHA256
ssl_options.ciphers.10 = ECDH-RSA-AES128-GCM-SHA256
ssl_options.ciphers.11 = DHE-RSA-AES128-GCM-SHA256
ssl_options.ciphers.12 = DHE-DSS-AES128-GCM-SHA256
ssl_options.ciphers.2 = ECDHE-RSA-AES256-GCM-SHA384
ssl_options.ciphers.3 = ECDH-ECDSA-AES256-GCM-SHA384
ssl_options.ciphers.4 = ECDH-RSA-AES256-GCM-SHA384
ssl_options.ciphers.5 = DHE-RSA-AES256-GCM-SHA384
ssl_options.ciphers.6 = DHE-DSS-AES256-GCM-SHA384
ssl_options.ciphers.7 = ECDHE-ECDSA-AES128-GCM-SHA256
ssl_options.ciphers.8 = ECDHE-RSA-AES128-GCM-SHA256
ssl_options.ciphers.9 = ECDH-ECDSA-AES128-GCM-SHA256
ssl_options.ciphers.10 = ECDH-RSA-AES128-GCM-SHA256
ssl_options.ciphers.11 = DHE-RSA-AES128-GCM-SHA256
ssl_options.ciphers.12 = DHE-DSS-AES128-GCM-SHA256
ssl_options.cacertfile = C:/users/user/documents/rabbitmq/cacert.pem
ssl_options.certfile = C:/users/user/documents/rabbitmq/Server.pem
ssl_options.keyfile = C:/users/user/documents/rabbitmq/Server.key.pem
ssl_options.depth = 8
ssl_options.certfile = C:/users/user/documents/rabbitmq/Server.pem
ssl_options.keyfile = C:/users/user/documents/rabbitmq/Server.key.pem
ssl_options.depth = 8
auth_mechanisms.1 = PLAIN
auth_mechanisms.2 = AMQPLAIN
auth_mechanisms.3 = EXTERNAL
auth_mechanisms.2 = AMQPLAIN
auth_mechanisms.3 = EXTERNAL
ssl_cert_login_from = common_name
## Management
management.listener.port = 15672
management.listener.ip = 0.0.0.0
management.listener.ssl = true
management.listener.ssl_opts.cacertfile = C:/users/user/documents/rabbitmq/cacert.pem
management.listener.ssl_opts.certfile = C:/users/user/documents/rabbitmq/Server.pem
management.listener.ssl_opts.keyfile = C:/users/user/documents/rabbitmq/Server.key.pem
management.listener.ssl_opts.verify = verify_peer
management.listener.ssl_opts.fail_if_no_peer_cert = false
management.listener.ssl_opts.certfile = C:/users/user/documents/rabbitmq/Server.pem
management.listener.ssl_opts.keyfile = C:/users/user/documents/rabbitmq/Server.key.pem
management.listener.ssl_opts.verify = verify_peer
management.listener.ssl_opts.fail_if_no_peer_cert = false
The second issue is that I'm having trouble creating shovels via the HTTP API, but I don't know if the issue stems from the first problem or if they are wholly unrelated. When creating the shovel, I'm seeing this failure in the log:
2019-06-13 12:20:53.491 [error] <0.926.0> Shovel 'LocalTo-127.0.0.1' failed to connect (URI: amqps://127.0.0.1): {tls_alert,"handshake failure"}
2019-06-13 12:20:53.491 [error] <0.926.0> Shovel 'LocalTo-127.0.0.1' has no more URIs to try for connection and will terminate
2019-06-13 12:20:53.491 [info] <0.931.0> TLS client: In state certify at ssl_handshake.erl:1373 generated CLIENT ALERT: Fatal - Handshake Failure - {bad_cert,hostname_check_failed}
2019-06-13 12:20:53.491 [error] <0.926.0> Shovel 'LocalTo-127.0.0.1' has no more URIs to try for connection and will terminate
2019-06-13 12:20:53.491 [info] <0.931.0> TLS client: In state certify at ssl_handshake.erl:1373 generated CLIENT ALERT: Fatal - Handshake Failure - {bad_cert,hostname_check_failed}
2019-06-13 12:20:53.495 [info] <0.926.0> terminating static worker with {failed_to_connect_using_provided_uris,[{rabbit_amqp091_shovel,make_conn_and_chan,2,[{file,"src/rabbit_amqp091_shovel.erl"},{line,324}]},{rabbit_amqp091_shovel,connect_source,1,[{file,"src/rabbit_amqp091_shovel.erl"},{line,78}]},{rabbit_shovel_worker,handle_cast,2,[{file,"src/rabbit_shovel_worker.erl"},{line,64}]},{gen_server2,handle_msg,2,[{file,"src/gen_server2.erl"},{line,1050}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,249}]}]}
2019-06-13 12:20:53.496 [error] <0.926.0> ** Generic server <0.926.0> terminating
** Last message in was {'$gen_cast',init}
** When Server state == {state,undefined,undefined,undefined,undefined,{<<"/">>,<<"LocalTo-127.0.0.1">>},dynamic,#{ack_mode => on_publish,dest => #{dest_queue => <<"Local-TAISSubscriptionQueue">>,fields_fun => #Fun<rabbit_shovel_parameters.11.26683091>,module => rabbit_amqp091_shovel,props_fun => #Fun<rabbit_shovel_parameters.12.26683091>,resource_decl => #Fun<rabbit_shovel_parameters.10.26683091>,uris => ["amqps://user:f...@127.0.0.1?cacertfile=C:/users/user/documents/rabbitmq/comboca.pem&certfile=C:/users/user/documents/rabbitmq/PHClient.pem&keyfile=C:/users/user/documents/rabbitmq/PHClient.key.pem&verify=verify_peer&fail_if_no_peer_cert=false&auth_mechanism=external&"]},name => <<"LocalTo-127.0.0.1">>,reconnect_delay => 15,shovel_type => dynamic,source => #{delete_after => never,module => rabbit_amqp091_shovel,prefetch_count => 1000,queue => <<"MySource">>,resource_decl => #Fun<rabbit_shovel_parameters.14.26683091>,source_exchange_key => <<>>,uris => ["amqps://user:f...@127.0.0.1?cacertfile=C:/users/user/documents/rabbitmq/comboca.pem&certfile=C:/users/user/documents/rabbitmq/PHClient.pem&keyfile=C:/users/user/documents/rabbitmq/PHClient.key.pem&verify=verify_peer&fail_if_no_peer_cert=false&auth_mechanism=external"]}},undefined,undefined,undefined,undefined,undefined}
** Reason for termination ==
** {failed_to_connect_using_provided_uris,[{rabbit_amqp091_shovel,make_conn_and_chan,2,[{file,"src/rabbit_amqp091_shovel.erl"},{line,324}]},{rabbit_amqp091_shovel,connect_source,1,[{file,"src/rabbit_amqp091_shovel.erl"},{line,78}]},{rabbit_shovel_worker,handle_cast,2,[{file,"src/rabbit_shovel_worker.erl"},{line,64}]},{gen_server2,handle_msg,2,[{file,"src/gen_server2.erl"},{line,1050}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,249}]}]}
2019-06-13 12:20:53.496 [error] <0.926.0> CRASH REPORT Process <0.926.0> with 0 neighbours exited with reason: failed_to_connect_using_provided_uris in rabbit_amqp091_shovel:make_conn_and_chan/2 line 324 in gen_server2:terminate/3 line 1166
2019-06-13 12:20:53.496 [error] <0.925.0> Supervisor {<0.925.0>,rabbit_shovel_dyn_worker_sup} had child {<<"/">>,<<"LocalTo-127.0.0.1">>} started with rabbit_shovel_worker:start_link(dynamic, {<<"/">>,<<"LocalTo-127.0.0.1">>}, [{<<"ack-mode">>,<<"on-publish">>},{<<"dest-protocol">>,<<"amqp091">>},{<<"dest-queue">>,<<"Local-...">>},...]) at <0.926.0> exit with reason failed_to_connect_using_provided_uris in rabbit_amqp091_shovel:make_conn_and_chan/2 line 324 in context child_terminated
2019-06-13 12:20:53.512 [info] <0.945.0> TLS client: In state certify at ssl_handshake.erl:1373 generated CLIENT ALERT: Fatal - Handshake Failure - {bad_cert,hostname_check_failed}
2019-06-13 12:20:53.496 [error] <0.926.0> ** Generic server <0.926.0> terminating
** Last message in was {'$gen_cast',init}
** When Server state == {state,undefined,undefined,undefined,undefined,{<<"/">>,<<"LocalTo-127.0.0.1">>},dynamic,#{ack_mode => on_publish,dest => #{dest_queue => <<"Local-TAISSubscriptionQueue">>,fields_fun => #Fun<rabbit_shovel_parameters.11.26683091>,module => rabbit_amqp091_shovel,props_fun => #Fun<rabbit_shovel_parameters.12.26683091>,resource_decl => #Fun<rabbit_shovel_parameters.10.26683091>,uris => ["amqps://user:f...@127.0.0.1?cacertfile=C:/users/user/documents/rabbitmq/comboca.pem&certfile=C:/users/user/documents/rabbitmq/PHClient.pem&keyfile=C:/users/user/documents/rabbitmq/PHClient.key.pem&verify=verify_peer&fail_if_no_peer_cert=false&auth_mechanism=external&"]},name => <<"LocalTo-127.0.0.1">>,reconnect_delay => 15,shovel_type => dynamic,source => #{delete_after => never,module => rabbit_amqp091_shovel,prefetch_count => 1000,queue => <<"MySource">>,resource_decl => #Fun<rabbit_shovel_parameters.14.26683091>,source_exchange_key => <<>>,uris => ["amqps://user:f...@127.0.0.1?cacertfile=C:/users/user/documents/rabbitmq/comboca.pem&certfile=C:/users/user/documents/rabbitmq/PHClient.pem&keyfile=C:/users/user/documents/rabbitmq/PHClient.key.pem&verify=verify_peer&fail_if_no_peer_cert=false&auth_mechanism=external"]}},undefined,undefined,undefined,undefined,undefined}
** Reason for termination ==
** {failed_to_connect_using_provided_uris,[{rabbit_amqp091_shovel,make_conn_and_chan,2,[{file,"src/rabbit_amqp091_shovel.erl"},{line,324}]},{rabbit_amqp091_shovel,connect_source,1,[{file,"src/rabbit_amqp091_shovel.erl"},{line,78}]},{rabbit_shovel_worker,handle_cast,2,[{file,"src/rabbit_shovel_worker.erl"},{line,64}]},{gen_server2,handle_msg,2,[{file,"src/gen_server2.erl"},{line,1050}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,249}]}]}
2019-06-13 12:20:53.496 [error] <0.926.0> CRASH REPORT Process <0.926.0> with 0 neighbours exited with reason: failed_to_connect_using_provided_uris in rabbit_amqp091_shovel:make_conn_and_chan/2 line 324 in gen_server2:terminate/3 line 1166
2019-06-13 12:20:53.496 [error] <0.925.0> Supervisor {<0.925.0>,rabbit_shovel_dyn_worker_sup} had child {<<"/">>,<<"LocalTo-127.0.0.1">>} started with rabbit_shovel_worker:start_link(dynamic, {<<"/">>,<<"LocalTo-127.0.0.1">>}, [{<<"ack-mode">>,<<"on-publish">>},{<<"dest-protocol">>,<<"amqp091">>},{<<"dest-queue">>,<<"Local-...">>},...]) at <0.926.0> exit with reason failed_to_connect_using_provided_uris in rabbit_amqp091_shovel:make_conn_and_chan/2 line 324 in context child_terminated
2019-06-13 12:20:53.512 [info] <0.945.0> TLS client: In state certify at ssl_handshake.erl:1373 generated CLIENT ALERT: Fatal - Handshake Failure - {bad_cert,hostname_check_failed}
2019-06-13 12:20:53.513 [error] <0.939.0> Shovel 'LocalTo-127.0.0.1' failed to connect (URI: amqps://127.0.0.1): {tls_alert,"handshake failure"}
2019-06-13 12:20:53.513 [error] <0.939.0> Shovel 'LocalTo-127.0.0.1' has no more URIs to try for connection and will terminate
2019-06-13 12:20:53.513 [info] <0.939.0> terminating static worker with {failed_to_connect_using_provided_uris,[{rabbit_amqp091_shovel,make_conn_and_chan,2,[{file,"src/rabbit_amqp091_shovel.erl"},{line,324}]},{rabbit_amqp091_shovel,connect_source,1,[{file,"src/rabbit_amqp091_shovel.erl"},{line,78}]},{rabbit_shovel_worker,handle_cast,2,[{file,"src/rabbit_shovel_worker.erl"},{line,64}]},{gen_server2,handle_msg,2,[{file,"src/gen_server2.erl"},{line,1050}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,249}]}]}
2019-06-13 12:20:53.513 [error] <0.939.0> ** Generic server <0.939.0> terminating
** Last message in was {'$gen_cast',init}
** When Server state == {state,undefined,undefined,undefined,undefined,{<<"/">>,<<"LocalTo-127.0.0.1">>},dynamic,#{ack_mode => on_publish,dest => #{dest_queue => <<"Local-TAISSubscriptionQueue">>,fields_fun => #Fun<rabbit_shovel_parameters.11.26683091>,module => rabbit_amqp091_shovel,props_fun => #Fun<rabbit_shovel_parameters.12.26683091>,resource_decl => #Fun<rabbit_shovel_parameters.10.26683091>,uris => ["amqps://user:f...@127.0.0.1?cacertfile=C:/users/user/documents/rabbitmq/comboca.pem&certfile=C:/users/user/documents/rabbitmq/PHClient.pem&keyfile=C:/users/user/documents/rabbitmq/PHClient.key.pem&verify=verify_peer&fail_if_no_peer_cert=false&auth_mechanism=external&"]},name => <<"LocalTo-127.0.0.1">>,reconnect_delay => 15,shovel_type => dynamic,source => #{delete_after => never,module => rabbit_amqp091_shovel,prefetch_count => 1000,queue => <<"MySource">>,resource_decl => #Fun<rabbit_shovel_parameters.14.26683091>,source_exchange_key => <<>>,uris => ["amqps://user:f...@127.0.0.1?cacertfile=C:/users/user/documents/rabbitmq/comboca.pem&certfile=C:/users/user/documents/rabbitmq/PHClient.pem&keyfile=C:/users/user/documents/rabbitmq/PHClient.key.pem&verify=verify_peer&fail_if_no_peer_cert=false&auth_mechanism=external"]}},undefined,undefined,undefined,undefined,undefined}
** Reason for termination ==
** {failed_to_connect_using_provided_uris,[{rabbit_amqp091_shovel,make_conn_and_chan,2,[{file,"src/rabbit_amqp091_shovel.erl"},{line,324}]},{rabbit_amqp091_shovel,connect_source,1,[{file,"src/rabbit_amqp091_shovel.erl"},{line,78}]},{rabbit_shovel_worker,handle_cast,2,[{file,"src/rabbit_shovel_worker.erl"},{line,64}]},{gen_server2,handle_msg,2,[{file,"src/gen_server2.erl"},{line,1050}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,249}]}]}
2019-06-13 12:20:53.513 [error] <0.939.0> CRASH REPORT Process <0.939.0> with 0 neighbours exited with reason: failed_to_connect_using_provided_uris in rabbit_amqp091_shovel:make_conn_and_chan/2 line 324 in gen_server2:terminate/3 line 1166
2019-06-13 12:20:53.514 [error] <0.925.0> Supervisor {<0.925.0>,rabbit_shovel_dyn_worker_sup} had child {<<"/">>,<<"LocalTo-127.0.0.1">>} started with rabbit_shovel_worker:start_link(dynamic, {<<"/">>,<<"LocalTo-127.0.0.1">>}, [{<<"ack-mode">>,<<"on-publish">>},{<<"dest-protocol">>,<<"amqp091">>},{<<"dest-queue">>,<<"Local-...">>},...]) at <0.939.0> exit with reason failed_to_connect_using_provided_uris in rabbit_amqp091_shovel:make_conn_and_chan/2 line 324 in context child_terminated
2019-06-13 12:20:58.899 [info] <0.958.0> TLS server: In state abbreviated received CLIENT ALERT: Fatal - Certificate Unknown
2019-06-13 12:20:53.513 [error] <0.939.0> Shovel 'LocalTo-127.0.0.1' has no more URIs to try for connection and will terminate
2019-06-13 12:20:53.513 [info] <0.939.0> terminating static worker with {failed_to_connect_using_provided_uris,[{rabbit_amqp091_shovel,make_conn_and_chan,2,[{file,"src/rabbit_amqp091_shovel.erl"},{line,324}]},{rabbit_amqp091_shovel,connect_source,1,[{file,"src/rabbit_amqp091_shovel.erl"},{line,78}]},{rabbit_shovel_worker,handle_cast,2,[{file,"src/rabbit_shovel_worker.erl"},{line,64}]},{gen_server2,handle_msg,2,[{file,"src/gen_server2.erl"},{line,1050}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,249}]}]}
2019-06-13 12:20:53.513 [error] <0.939.0> ** Generic server <0.939.0> terminating
** Last message in was {'$gen_cast',init}
** When Server state == {state,undefined,undefined,undefined,undefined,{<<"/">>,<<"LocalTo-127.0.0.1">>},dynamic,#{ack_mode => on_publish,dest => #{dest_queue => <<"Local-TAISSubscriptionQueue">>,fields_fun => #Fun<rabbit_shovel_parameters.11.26683091>,module => rabbit_amqp091_shovel,props_fun => #Fun<rabbit_shovel_parameters.12.26683091>,resource_decl => #Fun<rabbit_shovel_parameters.10.26683091>,uris => ["amqps://user:f...@127.0.0.1?cacertfile=C:/users/user/documents/rabbitmq/comboca.pem&certfile=C:/users/user/documents/rabbitmq/PHClient.pem&keyfile=C:/users/user/documents/rabbitmq/PHClient.key.pem&verify=verify_peer&fail_if_no_peer_cert=false&auth_mechanism=external&"]},name => <<"LocalTo-127.0.0.1">>,reconnect_delay => 15,shovel_type => dynamic,source => #{delete_after => never,module => rabbit_amqp091_shovel,prefetch_count => 1000,queue => <<"MySource">>,resource_decl => #Fun<rabbit_shovel_parameters.14.26683091>,source_exchange_key => <<>>,uris => ["amqps://user:f...@127.0.0.1?cacertfile=C:/users/user/documents/rabbitmq/comboca.pem&certfile=C:/users/user/documents/rabbitmq/PHClient.pem&keyfile=C:/users/user/documents/rabbitmq/PHClient.key.pem&verify=verify_peer&fail_if_no_peer_cert=false&auth_mechanism=external"]}},undefined,undefined,undefined,undefined,undefined}
** Reason for termination ==
** {failed_to_connect_using_provided_uris,[{rabbit_amqp091_shovel,make_conn_and_chan,2,[{file,"src/rabbit_amqp091_shovel.erl"},{line,324}]},{rabbit_amqp091_shovel,connect_source,1,[{file,"src/rabbit_amqp091_shovel.erl"},{line,78}]},{rabbit_shovel_worker,handle_cast,2,[{file,"src/rabbit_shovel_worker.erl"},{line,64}]},{gen_server2,handle_msg,2,[{file,"src/gen_server2.erl"},{line,1050}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,249}]}]}
2019-06-13 12:20:53.513 [error] <0.939.0> CRASH REPORT Process <0.939.0> with 0 neighbours exited with reason: failed_to_connect_using_provided_uris in rabbit_amqp091_shovel:make_conn_and_chan/2 line 324 in gen_server2:terminate/3 line 1166
2019-06-13 12:20:53.514 [error] <0.925.0> Supervisor {<0.925.0>,rabbit_shovel_dyn_worker_sup} had child {<<"/">>,<<"LocalTo-127.0.0.1">>} started with rabbit_shovel_worker:start_link(dynamic, {<<"/">>,<<"LocalTo-127.0.0.1">>}, [{<<"ack-mode">>,<<"on-publish">>},{<<"dest-protocol">>,<<"amqp091">>},{<<"dest-queue">>,<<"Local-...">>},...]) at <0.939.0> exit with reason failed_to_connect_using_provided_uris in rabbit_amqp091_shovel:make_conn_and_chan/2 line 324 in context child_terminated
2019-06-13 12:20:58.899 [info] <0.958.0> TLS server: In state abbreviated received CLIENT ALERT: Fatal - Certificate Unknown
The code I'm using to create the shovel appears as follows:
Value = new Shovel
{
SourceProtocol = "amqp091",
SourceUri = "amqps://user:f@" + details.IPAddress +
"?cacertfile=C:/users/user/documents/rabbitmq/comboca.pem&" +
"certfile=C:/users/user/documents/rabbitmq/PHClient.pem&" +
"keyfile=C:/users/user/documents/rabbitmq/PHClient.key.pem&" +
"verify=verify_peer&" +
"fail_if_no_peer_cert=false&" +
"auth_mechanism=external",
SourceQueue = queueName,
DestinationProtocol = "amqp091",
DestinationUri = "amqps://user:f@" + details.IPAddress +
"?cacertfile=C:/users/user/documents/rabbitmq/comboca.pem&" +
"certfile=C:/users/user/documents/rabbitmq/PHClient.pem&" +
"keyfile=C:/users/user/documents/rabbitmq/PHClient.key.pem&" +
"verify=verify_peer&" +
"fail_if_no_peer_cert=false&" +
"auth_mechanism=external&",
DestinationQueue = "Local-TAISSubscriptionQueue",
AckMode = "on-publish",
ReconnectDelayInSeconds = 15
},
Any ideas as to where this is failing and how to address it?
Thanks,
Jeremy Gardner
Luke Bakken
Jun 13, 2019, 1:40:56 PM6/13/19
to rabbitmq-users
Hi Jeremy,
To simulate the connection that your shovel is making, you will want to run OpenSSl in client mode like this. Substitute HOST with the hostname or IP you're using to connect, and substitue CN_NAME with the CN value from C:/users/user/documents/rabbitmq/Server.pem:
openssl s_client -connect HOST:5671 -cert C:/users/user/documents/rabbitmq/PHClient.pem -key keyfile=C:/users/user/documents/rabbitmq/PHClient.key.pem -CAfile C:/users/user/documents/rabbitmq/comboca.pem -verify 8 -verify_hostname CN_NAME
Please redirect the output of the above command to a file and attach it with your response.
Would you mind letting us know what version of RabbitMQ and Erlang you're using?
RabbitMQ depends on the Erlang VM for TLS support, and as such has no knowledge of the Windows certificate stores. It will read certificates from files only.
The best place for you to start will be to ensure that the TLS handshake can be established to port 5671 of your RabbitMQ server. We have a troubleshooting guide for that:
To simulate the connection that your shovel is making, you will want to run OpenSSl in client mode like this. Substitute HOST with the hostname or IP you're using to connect, and substitue CN_NAME with the CN value from C:/users/user/documents/rabbitmq/Server.pem:
openssl s_client -connect HOST:5671 -cert C:/users/user/documents/rabbitmq/PHClient.pem -key keyfile=C:/users/user/documents/rabbitmq/PHClient.key.pem -CAfile C:/users/user/documents/rabbitmq/comboca.pem -verify 8 -verify_hostname CN_NAME
Please redirect the output of the above command to a file and attach it with your response.
Thanks,
Luke
Jeremy Gardner
Jun 13, 2019, 1:45:03 PM6/13/19
to rabbitmq-users
I'm currently running RabbitMQ 3.7.9 and Erlang 21.1.
I'll attach the output shortly.
Jeremy Gardner
Jun 13, 2019, 6:17:02 PM6/13/19
to rabbitmq-users
This is the error I'm seeing:
Error opening client certificate private key file keyfile=C:/users/user/docume
nts/rabbitmq/PHClient.key.pem
7808:error:0200107B:system library:fopen:Unknown error:.\crypto\bio\bss_file.c:4
02:fopen('keyfile=C:/users/user/documents/rabbitmq/PHClient.key.pem','rb')
7808:error:20074002:BIO routines:FILE_CTRL:system lib:.\crypto\bio\bss_file.c:40
4:
unable to load client certificate private key file
nts/rabbitmq/PHClient.key.pem
7808:error:0200107B:system library:fopen:Unknown error:.\crypto\bio\bss_file.c:4
02:fopen('keyfile=C:/users/user/documents/rabbitmq/PHClient.key.pem','rb')
7808:error:20074002:BIO routines:FILE_CTRL:system lib:.\crypto\bio\bss_file.c:40
4:
unable to load client certificate private key file
I'm using certificates that were generated on another machine. Is this part of the problem?
Thanks,
Jeremy
Luke Bakken
Jun 14, 2019, 9:50:57 AM6/14/19
to rabbitmq-users
Hi Jeremy,
I did a web search with the following text ...
fopen pem "\crypto\bio\bss_file.c"
fopen pem "\crypto\bio\bss_file.c"
The results lead me to believe that C:/users/user/documents/rabbitmq/PHClient.key.pem doesn't exist or is zero-sized. Could you double-check that?
It shouldn't matter on which machine the file was generated.
Luke
Jeremy Gardner
Jun 14, 2019, 11:29:56 AM6/14/19
to rabbitmq-users
Luke,
The file does exist, and it does contain data. The first and last lines:
-----BEGIN RSA PRIVATE KEY-----
MIIEpQIBAAKCAQEA2KcBrO5T/hA/a98BCpFPkDu5fGECV5m0o7h5iiiZTQkeXesj
MIIEpQIBAAKCAQEA2KcBrO5T/hA/a98BCpFPkDu5fGECV5m0o7h5iiiZTQkeXesj
...
uSD0jn6zXRJpizDIUAZ0bV+zD84X9tP4HeiMC8YdjDmCu00KYsAYC38=
-----END RSA PRIVATE KEY-----
-----END RSA PRIVATE KEY-----
Thanks,
Jeremy
Luke Bakken
Jun 14, 2019, 11:35:37 AM6/14/19
to rabbitmq-users
Hi Jeremy -
How did you install OpenSSL to run the test commands? I'm wondering if a different binary release of OpenSSL could resolve that.
I'll fire up my Windows 8.1 VM and will install OpenSSL via chocolatey to see if I can reproduce this.
Thanks -
Luke
Jeremy Gardner
Jun 14, 2019, 2:15:53 PM6/14/19
to rabbitmq-users
Hi Luke,
I think I have it figured out, and it had nothing to do with the certificates at all. When I was making the URI for the shovel, I was missing an ampersand in the middle of the string. Instead of providing me back an error about an invalid URI, it gave me an error regarding the certificate.
I appreciate your help.
Thanks,
Jeremy
Reply all
Reply to author
Forward
0 new messages