SSL Termination via HAProxy to vernemq TCP 1883

[Helder Costa]

Hello all,

Is it possible to have HAProxy doing SSL termination between client and vernemq and then proxy then client request to vernemq 1883 TCP ?

Thanks in advance

[André Fatton]

Hello Helder,

yes, MQTT Client <-> HAProxy TLS termination <-> Verne is possible (and used quite often). Traffic between HAProxy and Verne will not be encrypted, if this goes to a TCP port (1883) at the broker.

You can use this setup with or without requiring Client certs at the HAProxy. You can also deploy this with Proxy Protocol v2 enabled between HAProxy and Verne, if you need original source ip information and/or CN name information (CN as MQTT username) at the broker.

Cheers,

André

[Helder Costa]

Thank you Andre for your quick answer.

But i still haven't figured out why does this doesn't work.

So i have a very simple haproxy configuration file:

global
    log         /dev/log local0 info
    chroot      /var/lib/haproxy
    pidfile     /var/run/haproxy.pid
    maxconn     5000
    user        haproxy
    group       haproxy
    daemon
    stats socket /var/lib/haproxy/stats mode 644 level admin

    ssl-default-bind-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
    ssl-default-bind-ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
    ssl-default-bind-options prefer-client-ciphers no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets

    ssl-default-server-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
    ssl-default-server-ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
    ssl-default-server-options no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets
    ssl-dh-param-file /etc/haproxy/dhparams.pem

defaults
    mode                    http
    log                        global
    option                  httplog
    option                  dontlognull
    option                  http-server-close
    option                  redispatch
    retries                 3
    timeout http-request    10s
    timeout http-keep-alive 5s

... Http stuff

listen srv-vernemq
    bind 1xxx.xxx.xxx.xxx:1883
    mode tcp
    server srv-vernemq-01 xx.xx.xx.xx:1883 check send-proxy-v2

When using this configuration i am able to connect to vernemq via haproxy

eg:

$ mosquitto_sub -h <FQDN> -p 1883 -t '#' -u <USER> -P <PASSWORD> -d
Client (null) sending CONNECT
Client (null) received CONNACK (0)
Client (null) sending SUBSCRIBE (Mid: 1, Topic: #, QoS: 0, Options: 0x00)
Client (null) received SUBACK
Subscribed (mid: 1): 0

If i change the haproxy to do SSL termination:

HAPROXY:

listen srv-vernemq
    bind 1xxx.xxx.xxx.xxx:1883 ssl crt /etc/certs/cert.pem
    mode tcp
    server srv-vernemq-01 xx.xx.xx.xx:1883 check send-proxy-v2

Trying with mosquitto:

$ mosquitto_sub -h <FQDN> -p 1883 -t '#' -u <USER> -P <PASSWORD> -d

Client (null) sending CONNECT
Client (null) sending CONNECT
Client (null) sending CONNECT
Client (null) sending CONNECT

Stays like this until i get a timeout.

I think that i am missing something maybe in HAProxy or how i use mosquitto to validate the setup, which also leaves to ask another question, how or which tool do the vernemq community recommends to try/validate/access the vernemq server that supports TCP, SSL and Websockets, i am new to this mqtt world so knowledge is null regarding this subject, but in the documentation i also dident find any examples or references in how to interact with vernemq from a client perspective.

Thank you all in advance.

Op donderdag 10 juni 2021 om 15:10:27 UTC+2 schreef a...@erl.io:

[André Fatton]

I think you need to point mosquitto_sub to a --cafile to connect to a TLS endpoint.

I'm surprised that your second mosquitto_sub attempt doesn't give you any TLS error though. If you suspect an issue with your server cert, you might want to use "openssl s_client" to try to connect to the port and see what this tells you.

-André

[Helder Costa]

Hi Andre,

Thank you for your help, you where correct, after adding lets encrypt root certificate as --cafile i could successfully connect throughout haproxy to vernemq.

mosquitto_sub -h xxx.xxx.xxx.xxx -p 1883 -t '#' -u <USER> -P <PASS> -d --cafile isrgrootx1.pem

Client (null) sending CONNECT
Client (null) received CONNACK (0)
Client (null) sending SUBSCRIBE (Mid: 1, Topic: #, QoS: 0, Options: 0x00)
Client (null) received SUBACK
Subscribed (mid: 1): 0

Client (null) sending PINGREQ
Client (null) received PINGRESP

By they way do you know if its possible to use mosquitto to test an websocket connection ?

Thank you very much for your help.

Op donderdag 10 juni 2021 om 22:40:23 UTC+2 schreef a...@erl.io:

[Eduardo Hering]

Helder,

I suggest you make haproxy.cfg simpler, then add stuff to it.

global

  log stdout format raw local0

defaults

frontend f_mqtt

  bind 0.0.0.0:8883 crt ./ssl.crt ssl

  use_backend b_mqtt

  mode tcp

  maxconn 100

backend b_mqtt

  balance roundrobin

  server mqtt0 10.0.0.x:1883 check send-proxy-v2 maxconn 50

  mode tcp

  log global

Best regards,

Eduardo N Hering

————————

http://enhering.com 

-- 
You received this message because you are subscribed to the Google Groups "vernemq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vernemq-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/vernemq-users/45ab566d-06e9-41d5-aae5-4545565d6630n%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Eduardo Hering]

I also suggest you use MQTT Explorer to help debugging MQTT related stuff.

Best regards,

Eduardo N Hering

————————

http://enhering.com 

On 10 Jun 2021, at 16:56, Helder Costa <hjtgco...@gmail.com> wrote: