"probable authentication error" on ssl with remote client *non guest user*

[Sebastien Roy]

I am trying to figure out how to get rabbitMQ + pika working with TLS/SSL. I can't figure out if I have an error with pika code or with rabbitMQ configuration.

0) erlang version 20, rabbitMQ version 3.5.7, pika version 0.11.2 on an Ubuntu 16.04 VM machine from a popular cloud service provider. Firewall is DOWN for those tests.

1) The rabbitMQ server runs physically on the same machine as the client for testing purposes. I get the same problem regardless of using "localhost" as host or the domain name. As a result, the client and server certificates are coincidentially the same.

2) certificate created with letsencrypt.

3) rabbitMQ config:

[

  {rabbit, [

     {tcp_listeners, []},

     {ssl_listeners, [5671]},

     {ssl_options, [{cacertfile,"/etc/rabbitmq/<domain>/<domain>.chain+root.crt"},

                    {certfile,"/etc/rabbitmq/<domain>/cert.pem"},

                    {keyfile,"/etc/rabbitmq/<domain>/privkey.pem"},

                    {verify,verify_none},

                    {fail_if_no_peer_cert,false}]}

   ]}

].

Hoping to use verify_peer and true once I get it to work, Certificates files ownership is set to rabbitmq:rabbitmq

4) sample python code:

def params(creds):

    ssl_opts = dict(

                  #ssl_version=ssl.PROTOCOL_TLSv1,

                  ca_certs=creds["ca"],

                  keyfile=creds["key"],

                  certfile=creds["cert"],

                  #cert_reqs=ssl.CERT_REQUIRED)

                  )

    print(ssl_opts)

    credentials = pika.PlainCredentials(creds["user"], creds["pw"])

    return pika.ConnectionParameters(

                                port=5671,

                                virtual_host=creds["vhost"],

                                host=creds["hosts"],

                                credentials=credentials,

                                ssl=True,

                                ssl_options=ssl_opts

                                          )

def consumer(context, creds):

   connection = pika.BlockingConnection(params(creds))

   channel = connection.channel()

   channel.exchange_declare(exchange=XNAME,

                            exchange_type='direct')

    result = channel.queue_declare(exclusive=True)

   queue_name = result.method.queue

    for n_mode in NOTE_CONFIG:

       channel.queue_bind(exchange=XNAME,

                          queue=queue_name,

                          routing_key=n_mode)

    def callback(ch, method, properties, body):

       print(body)

    channel.basic_consume(callback,

                         queue=queue_name,

                         no_ack=True)

   channel.start_consuming()

Now here is all the things I tested:

- if I use openssl s_client/s_server setup, it works. I can establish a functionning connection using the cert/key/CAfile either though localhost or the domain name.

- If I use openssl s_client to connect to the rabbitMQ port 5671, I can establish a connection.

- If I remove the `{tcp_listeners, []}` line from the config (to reactivate non TLS/SSL port 5672) I can launch the consumer (with ssl=False) and use a publisher to succesfully send messages.

Now my problem is when I try to use the above with TLS/SSL on port 5671. Here is the pika backtrace:

Traceback (most recent call last):

  File "/home/logger/lf-tools/venv/Linux/bin/notifier", line 6, in <module>

    exec(compile(open(__file__).read(), __file__, 'exec'))

  File "/home/logger/lf-tools/src/lsnotify/bin/notifier", line 6, in <module>

    main()

  File "/home/logger/lf-tools/src/lsnotify/lsnotify/__main__.py", line 25, in main

    notify.consumer(options)

  File "/home/logger/lf-tools/src/lsnotify/lsnotify/notify.py", line 111, in consumer

    connection = pika.BlockingConnection(params(CREDS))

  File "/home/logger/lf-tools/venv/Linux/lib/python3.5/site-packages/pika/adapters/blocking_connection.py", line 374, in __init__

    self._process_io_for_connection_setup()

  File "/home/logger/lf-tools/venv/Linux/lib/python3.5/site-packages/pika/adapters/blocking_connection.py", line 414, in _process_io_for_connection_setup

    self._open_error_result.is_ready)

  File "/home/logger/lf-tools/venv/Linux/lib/python3.5/site-packages/pika/adapters/blocking_connection.py", line 466, in _flush_output

    raise maybe_exception

pika.exceptions.ProbableAuthenticationError: (-1, 'EOF')

And here is the rabbitMQ log:

=INFO REPORT==== 2-Feb-2018::21:04:12 ===

accepting AMQP connection <0.408.0> (159.203.11.42:57616 -> 159.203.11.42:5671)

=ERROR REPORT==== 2-Feb-2018::21:04:15 ===

closing AMQP connection <0.408.0> (159.203.11.42:57616 -> 159.203.11.42:5671):

{handshake_error,starting,0,

    {error,undef,'connection.start_ok',

        [{ssl,connection_info,

             [{sslsocket,

                  {gen_tcp,#Port<0.3264>,tls_connection,undefined},

                  <0.409.0>}],

             []},

         {rabbit_reader,ssl_info,2,

             [{file,"src/rabbit_reader.erl"},{line,1240}]},

         {rabbit_reader,'-notify_auth_result/4-lc$^0/1-0-',3,

             [{file,"src/rabbit_reader.erl"},{line,1171}]},

         {rabbit_reader,'-notify_auth_result/4-lc$^0/1-0-',3,

             [{file,"src/rabbit_reader.erl"},{line,1172}]},

         {rabbit_reader,notify_auth_result,4,

             [{file,"src/rabbit_reader.erl"},{line,1169}]},

         {rabbit_reader,auth_phase,2,

             [{file,"src/rabbit_reader.erl"},{line,1123}]},

         {rabbit_reader,handle_method0,3,

             [{file,"src/rabbit_reader.erl"},{line,933}]},

         {rabbit_reader,handle_input,3,

             [{file,"src/rabbit_reader.erl"},{line,847}]}]}}

My understanding is that during the connection, we fail to authenticate on the rabbitMQ server although the username and password were provided. What is wrong? Thanks!

[lba...@pivotal.io]

Hello Sebastien -

I will try to find some time to test this out. Thanks. What version of Python are you using?

[lba...@pivotal.io]

One other item I noticed - Erlang 20 is not supported with RabbitMQ 3.5.7. You should try this out with RabbitMQ 3.7.3.

[lba...@pivotal.io]

Hello -

I can't reproduce what you are reporting. I have created a gist that has my Python code as well as the RabbitMQ configuration I used:

https://gist.github.com/lukebakken/6d091c1452b01c8bec24db2360fecfb7

I traced the interaction and attached it as trace.pcapng. This document has a good explanation of how to trace traffic using Wireshark. Please note the bit at the end about setting RSA ciphers if you'd like to decode the SSL traffic:

https://www.rabbitmq.com/amqp-wireshark.html