RabbitMQ Management Over HTTPS and Nginx

[Dario Zadro]

I'm having trouble with this topic as posted on Stackoverflow here:
https://stackoverflow.com/questions/49742269/rabbitmq-management-over-https-and-nginx

Can anyone help? Thanks in advance.

[Luke Bakken]

Hi Dario -

Do you have any logs from nginx to share?

If you modify your RabbitMQ and nginx configuration to use plain HTTP, does it work?

Thanks -

Luke

[Dario Zadro]

Hey Luke -

Thanks so much for your reply!

RabbitMQ works great on HTTP over 15672

I changed my nginx config, based on this:

https://stackoverflow.com/questions/48881248/rabbitmq-nginx-config

And it looks like this now (changing actual hostname to domain.com);

    location ~* /rabbitmq/api/(.*?)/(.*) {

      proxy_pass https://domain.com:15671/api/$1/%2F/$2?$query_string;

      proxy_buffering                    off;

      proxy_set_header Host              $http_host;

      proxy_set_header X-Real-IP         $remote_addr;

      proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;

      proxy_set_header X-Forwarded-Proto $scheme;

    }

    location ~* /rabbitmq/(.*) {

      rewrite ^/rabbitmq/(.*)$ /$1 break;

      proxy_pass https://domain.com:15671;

      proxy_buffering                    off;

      proxy_set_header Host              $http_host;

      proxy_set_header X-Real-IP         $remote_addr;

      proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;

      proxy_set_header X-Forwarded-Proto $scheme;

    }

Then nginx error log:

2018/04/11 10:43:02 [error] 28001#28001: *81 upstream timed out (110: Connection timed out) while SSL handshaking to upstream, client: 10.17.25.174, server: , request: "GET /rabbitmq/ HTTP/1.1", upstream: "https://10.17.25.130:15671/", host: “domain.com"

Thoughts? I’m guessing my nginx conf doesn’t know how to “listen” on 15671 since it only has server blocks for 80 and 443

Dario

--
You received this message because you are subscribed to a topic in the Google Groups "rabbitmq-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/rabbitmq-users/l_IxtiXeZC8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Luke Bakken]

Hi Dario -

> RabbitMQ works great on HTTP over 15672

Does this mean that you have tested nginx as a reverse proxy for RabbitMQ using plain HTTP?

> *81 upstream timed out (110: Connection timed out) while SSL handshaking to upstream

That error suggests that something is wrong with the TLS/SSL configuration for RabbitMQ management, or perhaps with your certificates. Can you connect directly to https://domain.com:15671 using a browser? What is the output of the following command?

openssl s_client -connect domain.com:15671

If I have time I will try to get a reverse proxy set up on my workstation to diagnose.

Thanks,

Luke

Hey Luke -

[Dario Zadro]

Hi Luke -

I’m only able to access the management GUI using http://10.17.25.130:15761 which is the local IP of the machine. Any attempt to use my nginx config is causing issues. Would be great to have you test/show a working case. I have an UpWork job open, if that’s of interest to take on? I guess I can also past in my entire nginx config here?

The openssl command just hangs at:

root@web:/home/user/_logs# openssl s_client -connect domain.com:15671

CONNECTED(00000003)

There is more more output after that.

Dario

On Apr 11, 2018, at 11:13 AM, Luke Bakken <lba...@pivotal.io> wrote:

Hi Dario -

> RabbitMQ works great on HTTP over 15672

Does this mean that you have tested nginx as a reverse proxy for RabbitMQ using plain HTTP?

> *81 upstream timed out (110: Connection timed out) while SSL handshaking to upstream

That error suggests that something is wrong with the TLS/SSL configuration for RabbitMQ management, or perhaps with your certificates. Can you connect directly to https://domain.com:15671using a browser? What is the output of the following command?

[Luke Bakken]

Hi Dario,

Let's take a step back and do some examination.

The fact that http://10.17.25.130:15761 works (note that you are using http://) means that the management plugin/GUI is not configured correctly to use TLS/SSL. The output from the openssl command backs this up.

You should review your configuration to ensure that the cacertfile, certfile, and keyfile settings point to the correct files and that the files are readable by the rabbitmq user.

You will know that you have TLS/SSL configured correctly when the openssl s_client command produces output similar to running it as openssl s_client -connect google.com:443

After that, you should be able to browse to https://10.17.25.130:15761. Once that works, the nginx configuration can be re-examined.

You may wish to follow along with the recommendations here: https://www.rabbitmq.com/troubleshooting-ssl.html

Thanks,

Luke

[Dario Zadro]

Hi Luke -

Following the example on https://www.rabbitmq.com/troubleshooting-ssl.html shows that my CA/cert/key is valid.

I’m not using HTTP over 15671 but rather over 15672.

Doing openssl s_client -connect domain.com:443 <— also shows return code of 0 (ok)

I doesn’t seem I can test openssl s_client -connect 10.17.25.130:15671 as the cert is tied to “domain.com” and the rabbitmq.config references domain.com as well (attaching here again below)

Continuing down the troubleshooting doc, I see the listener in the log file:

=INFO REPORT==== 11-Apr-2018::15:28:06 ===

started SSL Listener on [::]:5671

Really stuck here! Any more thoughts?

[

  {ssl, [{versions, ['tlsv1.2', 'tlsv1.1']}]},

  {rabbit, [

      {reverse_dns_lookups, true},

      {tcp_listeners, [5672]},

      {ssl_listeners, [5671]},

      {ssl_options, [

        {cacertfile, "/etc/ssl/certs/NameCheap.pem"},

        {certfile,   "/etc/nginx/ssl/domain.crt"},

        {keyfile,    "/etc/nginx/ssl/domain.key"},

        {versions, ['tlsv1.2', 'tlsv1.1']}

      ]}

    ]

  },

  {rabbitmq_management, [

    {listener, [

     {port, 15671},

     {ssl,  true},

     {ssl_opts, [

        {cacertfile, "/etc/ssl/certs/NameCheap.pem"},

        {certfile,   "/etc/nginx/ssl/domain.crt"},

        {keyfile,    "/etc/nginx/ssl/domain.key"},

        {versions, ['tlsv1.2', 'tlsv1.1']}

      ]}

    ]}

  ]}

].

[Luke Bakken]

Hi Dario -

The listener you see in the log file is for the AMQP protocol port - 5671.

> I’m not using HTTP over 15671 but rather over 15672.

In an earlier message, you wrote this:

"I’m only able to access the management GUI using http://10.17.25.130:15761 which is the local IP of the machine"

Did you mis-type the URL? Notice that you wrote http://10.17.25.130:15761 which is why I thought you accessed the management UI over http:// and port 15671.

Can you use your web browser to visit https://10.17.25.130:15761 ? If you receive an error, what is it?

What is the complete output of running openssl s_client -connect 10.17.25.130:15671 ? Please redirect the output to a file and attach it.

It's important to establish that TLS is configured correctly for RabbitMQ before attempting to do anything with nginx. I am working on a complete example for this using certificates generated by this project: https://github.com/michaelklishin/tls-gen/

Thanks,

Luke

To unsubscribe from this group and all its topics, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.

[Dario Zadro]

Hi Luke -

Yes, sorry about that - it was me mis-typing.

Browsing http://10.17.25.130:15671 shows a blank screen “unable to connect”

openssl s_client -connect 10.17.25.130:15671 only shows one line:

root@domain:/home/user# openssl s_client -connect 10.17.25.130:15671

CONNECTED(00000003)

It just hangs from there, and I have to end the process.

Here’s more output on the cert below. I have a cert with and without the chain added (tried both with rabbit config):

root@domain:/home/user# openssl x509 -noout -modulus -in /etc/nginx/ssl/domain_no_chain.crt  | openssl md5

(stdin)= 025501ca554cb55b650ee52f09ceb039

root@domain:/home/user# openssl rsa -noout -modulus -in /etc/nginx/ssl/domain.key  | openssl md5

(stdin)= 025501ca554cb55b650ee52f09ceb039

root@domain:/home/user# openssl x509 -noout -modulus -in /etc/nginx/ssl/domain.crt  | openssl md5

(stdin)= 025501ca554cb55b650ee52f09ceb039

Is my rabbitmq.config file not setup properly? I shared that in my last reply.

Thanks again! I hear you that we need to get the secure connection before moving on to nginx…

Dario

To unsubscribe from this group and all its topics, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.

[Luke Bakken]

Dario -

The openssl output should show a lot more. I suspect that the certificate files are not located where the configuration says they should be, or they don't have the correct permissions.

What is the output of the following commands when run as root?

ls -la /etc
ls -la /etc/nginx
ls -la /etc/nginx/ssl
ls -la /etc/ssl
ls -la /etc/ssl/certs

Thanks,

Luke

On Wednesday, April 11, 2018 at 4:00:31 PM UTC-7, Dario Zadro wrote:

Hi Luke -

[Dario Zadro]

Hey Luke -

All the directories and needed files within them are 644 at a minimum.

I’m giving up on the SSL (for now) and trying to proxy through 127.0.0.1:15672

Seems it’s now an nginx troubleshooting issue with the JS file paths returning 404:

https://web.zadro.co/rabbitmq/

My nginx config:

    location ~* /rabbitmq/api/(.*?)/(.*) {

      proxy_pass http://127.0.0.1:15672/api/$1/%2F/$2?$query_string;

      proxy_buffering                    off;

      proxy_set_header Host              $http_host;

      proxy_set_header X-Real-IP         $remote_addr;

      proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;

      proxy_set_header X-Forwarded-Proto $scheme;

    }

    location ~* /rabbitmq/(.*) {

      rewrite ^/rabbitmq/(.*)$ /$1 break;

      proxy_pass http://127.0.0.1:15672;

      proxy_buffering                    off;

      proxy_set_header Host              $http_host;

      proxy_set_header X-Real-IP         $remote_addr;

      proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;

      proxy_set_header X-Forwarded-Proto $scheme;

    }

Any thoughts on this?

I’ll circle back around to the SSL once the proxy is configured.

Thanks again!

Dario

[Luke Bakken]

Hi Dario,

It's very helpful to me to provide the full output of commands when requested, rather than summarizing what you saw. Frequently when I am assisting people on this mailing list what they think they see is not exactly correct, and a second pair of eyes is useful.

Could you please provide your complete, unedited nginix configuration file as an attachment rather than just part of the file? I can use that locally to try to reproduce what you are seeing.

Thanks,

Luke

[Dario Zadro]

Hi Luke -

Appreciate all your help! 

Here’s the full nginx:

server {

        gzip off;

        open_file_cache off;

        listen 80 default_server;

        listen [::]:80 default_server ipv6only=on;

        return 301 https://web.zadro.co$request_uri;

}

server {

gzip off;

open_file_cache off;

listen 10.17.25.130:443 default_server ssl http2;

  root /home/user/www;

index index.php index.html index.htm;

access_log /home/user/_logs/access.log;

error_log /home/user/_logs/error.log;

include browsercache;

include security;

        include blacklist;

include errorpages;

include sslconfig;

        ssl_certificate /etc/nginx/ssl/web_zadro_co.crt;

        ssl_certificate_key /etc/nginx/ssl/web_zadro_co.key;

    location ~* /rabbitmq/api/(.*?)/(.*) {

      proxy_pass http://127.0.0.1:15672/api/$1/%2F/$2?$query_string;

      proxy_buffering                    off;

      proxy_set_header Host              $http_host;

      proxy_set_header X-Real-IP         $remote_addr;

      proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;

      proxy_set_header X-Forwarded-Proto $scheme;

    }

    location ~* /rabbitmq/(.*) {

      rewrite ^/rabbitmq/(.*)$ /$1 break;

      proxy_pass http://127.0.0.1:15672;

      proxy_buffering                    off;

      proxy_set_header Host              $http_host;

      proxy_set_header X-Real-IP         $remote_addr;

      proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;

      proxy_set_header X-Forwarded-Proto $scheme;

    }

location / {

                try_files $uri $uri/ $uri.php?$args;

        }

location ~ \.php$ {

try_files $uri =404;

        fastcgi_split_path_info ^(.+\.php)(.*)$;

            fastcgi_pass unix:/var/run/user_fpm.sock;

        fastcgi_index index.php;

include fastcgi_params;

    }

}

[Dario Zadro]

Hi Luke -

I got it working with proxy pass-through to 15762

After troubleshooting, it was my browser cache file causing issues! Ugh.

Thanks for all your help. I’m going to start re-enabling all SSL functionality piece-by-piece and see how it goes.

Dario

[Luke Bakken]

Thank you Dario for the update. Have a good weekend.