MQTT TLS connection via ESP8266/Arduino

[Federico D'Amico]

Hello,
we're trying to connect via ESP8266 to our AMQP via MQTT plugin.

We want to connect to MQTT via certificates.

Actually we have used Mosquitto server and we are able to connect correctly but on AMQP server we are not.

We used tls-gen to generate the server certificates and CA for mosquitto and for amqp.

We don't have any logs on the AMQP server when we tried to connect, and generic error on the Arduino (ESP8266) logs (-2).

This is our Mosquitto conf:

#Protocol mqtt
protocol mqtt
# Uses the common name from the client cert as the username
use_identity_as_username true
# Path the the Certificate Authority Cert.
cafile /home/feddam/tls-gen/basic/result/ca_certificate.pem
# Path to server private key
keyfile /home/feddam/tls-gen/basic/result/server_raspberrypi_key.pem
# Path to server certificate
certfile /home/feddam/tls-gen/basic/result/server_raspberrypi_certificate.pem
# Set TLS version
tls_version tlsv1.2
# Require that clients provide certificates
require_certificate true


This is our AMQP conf:

listeners.ssl.default = 5671
ssl_options.cacertfile=/etc/rabbitmq/federico/ca_certificate.pem
ssl_options.certfile=/etc/rabbitmq/federico/server_raspberrypi_certificate.pem
ssl_options.keyfile=/etc/rabbitmq/federico/server_raspberrypi_key.pem
ssl_options.verify= verify_peer
log.file.level = debug
#ssl_options.password = MySecretPassword
ssl_options.fail_if_no_peer_cert = true
mqtt.listeners.tcp.default = 1883
mqtt.listeners.ssl.default = 8883
ssl_options.versions.1 = tlsv1.1
ssl_options.versions.2 = tlsv1.2
ssl_options.versions.3 = tlsv1.3
mqtt.allow_anonymous  = false
mqtt.vhost            = TEST
mqtt.exchange         = test_exchange
mqtt.ssl_cert_login = true
ssl_options.depth                = 0
auth_mechanisms.3 = EXTERNAL
auth_mechanisms.1 = PLAIN
auth_mechanisms.2 = AMQPLAIN
ssl_cert_login_from = common_name


tcp_listen_options.backlog = 128
tcp_listen_options.nodelay = true
tcp_listen_options.linger.on      = true
tcp_listen_options.linger.timeout = 0
tcp_listen_options.sndbuf  = 32768
tcp_listen_options.recbuf  = 32768
mqtt.tcp_listen_options.backlog = 128
mqtt.tcp_listen_options.nodelay = true
mqtt.tcp_listen_options.linger.on      = true
mqtt.tcp_listen_options.linger.timeout = 0
mqtt.tcp_listen_options.sndbuf  = 32768
mqtt.tcp_listen_options.recbuf  = 32768
raft.wal_max_size_bytes = 67108864


#ssl_options.honor_cipher_order = true
#ssl_options.honor_ecc_order    = true

vm_memory_high_watermark.relative=0.7


ssl_options.ciphers.1 = TLS_AES_256_GCM_SHA384
ssl_options.ciphers.2 = TLS_AES_128_GCM_SHA256
ssl_options.ciphers.3 = TLS_CHACHA20_POLY1305_SHA256
ssl_options.ciphers.4 = TLS_AES_128_CCM_SHA256
ssl_options.ciphers.5 = TLS_AES_128_CCM_8_SHA256
ssl_options.ciphers.6 = ECDHE-ECDSA-AES256-GCM-SHA384
ssl_options.ciphers.7 = ECDHE-RSA-AES256-GCM-SHA384
ssl_options.ciphers.8 = ECDHE-ECDSA-AES256-CCM
ssl_options.ciphers.9 = ECDHE-ECDSA-AES256-CCM8
ssl_options.ciphers.10 = ECDHE-ECDSA-AES256-SHA384
ssl_options.ciphers.11 = ECDHE-RSA-AES256-SHA384
ssl_options.ciphers.12 = ECDHE-ECDSA-CHACHA20-POLY1305
ssl_options.ciphers.13 = ECDHE-RSA-CHACHA20-POLY1305
ssl_options.ciphers.14 = ECDHE-ECDSA-AES128-GCM-SHA256
ssl_options.ciphers.15 = ECDHE-RSA-AES128-GCM-SHA256
ssl_options.ciphers.16 = ECDHE-ECDSA-AES128-CCM
ssl_options.ciphers.17 = ECDHE-ECDSA-AES128-CCM8
ssl_options.ciphers.18 = ECDH-ECDSA-AES256-GCM-SHA384
ssl_options.ciphers.19 = ECDH-RSA-AES256-GCM-SHA384
ssl_options.ciphers.20 = ECDH-ECDSA-AES256-SHA384
ssl_options.ciphers.21 = ECDH-RSA-AES256-SHA384
ssl_options.ciphers.22 = ECDH-ECDSA-AES128-GCM-SHA256
ssl_options.ciphers.23 = ECDH-RSA-AES128-GCM-SHA256
ssl_options.ciphers.24 = ECDHE-ECDSA-AES128-SHA256
ssl_options.ciphers.25 = ECDHE-RSA-AES128-SHA256
ssl_options.ciphers.26 = ECDH-ECDSA-AES128-SHA256
ssl_options.ciphers.27 = ECDH-RSA-AES128-SHA256
ssl_options.ciphers.28 = DHE-RSA-AES256-GCM-SHA384
ssl_options.ciphers.29 = DHE-DSS-AES256-GCM-SHA384
ssl_options.ciphers.30 = DHE-RSA-AES256-SHA256
ssl_options.ciphers.31 = DHE-DSS-AES256-SHA256
ssl_options.ciphers.32 = DHE-RSA-AES128-GCM-SHA256
ssl_options.ciphers.33 = DHE-DSS-AES128-GCM-SHA256
ssl_options.ciphers.34 = DHE-RSA-CHACHA20-POLY1305
ssl_options.ciphers.35 = DHE-RSA-AES128-SHA256
ssl_options.ciphers.36 = DHE-DSS-AES128-SHA256
ssl_options.ciphers.37 = ECDHE-ECDSA-AES256-SHA
ssl_options.ciphers.38 = ECDHE-RSA-AES256-SHA
ssl_options.ciphers.39 = ECDH-ECDSA-AES256-SHA
ssl_options.ciphers.40 = ECDH-RSA-AES256-SHA
ssl_options.ciphers.41 = ECDHE-ECDSA-AES128-SHA
ssl_options.ciphers.42 = ECDHE-RSA-AES128-SHA
ssl_options.ciphers.43 = ECDH-ECDSA-AES128-SHA
ssl_options.ciphers.44 = ECDH-RSA-AES128-SHA
ssl_options.ciphers.45 = DHE-RSA-AES256-SHA
ssl_options.ciphers.46 = DHE-DSS-AES256-SHA
ssl_options.ciphers.47 = DHE-RSA-AES128-SHA
ssl_options.ciphers.48 = DHE-DSS-AES128-SHA




This is the Arduino code:



void connectMQTT() {
  delay(3000);
  WiFi.mode(WIFI_STA);
  WiFi.begin(WIFI_SSID, WIFI_PASSWORD);

  Serial.println(String("Attempting to connect to SSID: ") + String(WIFI_SSID));

  while (WiFi.status() != WL_CONNECTED) {
    Serial.print(".");
    delay(1000);
  }

  Serial.println("");  
  Serial.print("Local IP: ");
  Serial.println(WiFi.localIP());

  NTPConnect();

  net.allowSelfSignedCerts();
  net.setTrustAnchors(&cert);
  net.setClientRSACert(&client_crt, &key);
  net.allowSelfSignedCerts();
  net.setInsecure();

  client.setServer(MQTT_HOST, 8883);
  client.setCallback(messageReceived);


  Serial.println("Connecting to MQTT IOT");

  while (!client.connect(THINGNAME)) {
    Serial.print(".");
    delay(1000);
  }

  if (!client.connected()) {
    Serial.println("MQTT IoT Timeout!");
    return;
  }
  // Subscribe to a topic
  client.subscribe(AWS_IOT_SUBSCRIBE_TOPIC);

  Serial.println("MQTT IoT Connected!");
}



AS you can see we used the same certificates, and with the same Arduino Firmware we're able to connect to Mosquitto but not to AMQP.

We used also MQTTX client and we're able to connect with the client certificates to AMQP and Mosquitto.

[Luke Bakken]

Hello,

• Did you create a user in RabbitMQ whose username matches the CN= value in your client certificate?
• Did you enable the correct RabbitMQ plugin for X509 certificate authentication? rabbitmq_auth_mechanism_ssl
• Does your environment work with username/password auth over TLS? (in other words, verify_none instead of verify_peer)

Please also see our troubleshooting guide: https://www.rabbitmq.com/docs/troubleshooting-ssl

[Federico D'Amico]

Hello, yes, we have configured correct CN in client certificate, enabled the plugin authentication SSL and disabled the verification of peer.
As said we are able to connect via MQTTx client to Rabbitmq without any issue using the same certificates used with Mosquitto server and Rabbitmq server but if we use with Arduino esp8266 or esp idf the same certificates we are not able to connect to Rabbitmq but we are able to Mosquitto

Same certificates for server and client:
MQTTx<->Mosquitto: OK
MQTTx<->Rabbitmq: OK
Mosquitto client <-> Rabbitmq: OK
mosquitto client <-> Mosquitto: OK
Arduino Esp8266 <-> Mosquitto: OK
Esp-idf <-> Rabbitmq: KO
Arduino Esp8266 <-> Mosquitto: OK
Esp-idf Esp8266 <-> RabbitMq: KO

We don't have any logs on the Rabbitmq server, can you suggest any way to debug what is going on?

[Luke Bakken]

Your RabbitMQ TLS configuration is quite a bit different than your configuration for Mosquitto

• You list specific ciphers - why?
• Your Mosquitto config only allows TLS1.2, but your RabbitMQ configuration lists three versions
• Your RabbitMQ configuration has a lot of un-related TCP configuration that is not present in Mosquitto
• You haven't told us the versions of all of the software you are using. Erlang, RabbitMQ, Mosquitto, MQTTx, etc

I suggest making your RabbitMQ config as close to your Mosquitto one as possible.

The most likely issue here is that the TLS library used by the "Esp8266" device has an issue with the Erlang TLS implementation. RabbitMQ does not implement TLS, the Erlang VM does.

But, we have no idea what TLS library is being used by either at the moment because we don't know what software versions you're using.

If you can reproduce this issue using an Arduino emulator, or in some other way, we might be able to assist you. My only other suggestion is to use Wireshark or tcpdump to get a capture of a working TLS handshake vs a non-working one, and compare.

[Federico D'Amico]

Hi,

• we've specified all supported ciphers to test if it was a cipher issue 
• we've disabled tls1.3 on rabbitmq but nothing changed
• which one?
• RabbitMQ 3.13.0 - Erlang 26.2.2 - Mqttx v1.9.10 - Mosquitto 2.0.11 
  
  Attached the wireshark captures

[Luke Bakken]

Sorry, I wasn't offering to read your TLS handshake captures for you. You should take the time to do some diagnosis yourself, since this issue is specific to something in your environment.

Your only other option is to instruct me on how to reproduce the issue, though I do not have access to an " Arduino esp8266" device.

[Federico D'Amico]

We're setted a test server. We've replicated with this esp32 online simulator: https://wokwi.com/projects/394776844086894593

if you use other client with same certificate you'll be able to connect without any issue

[Luke Bakken]

That's a cool simulator. I will try to reproduce your issue when I have time. Please note that I have to prioritize other work before I can provide free support.

[Luke Bakken]

Hello,

I just spent about an hour getting a local environment setup for MQTT client cert authentication with RabbitMQ -

https://github.com/lukebakken/rabbitmq-users-VT4GKJVJT7s-mqtt-tls

...only to find out that I must pay to use the ESP simulator with my local environment.

Is the "private gateway" something you'd be willing to pay for so I can continue debugging?

https://docs.wokwi.com/guides/esp32-wifi#the-private-gateway

https://wokwi.com/club?ref=docs_club

Thanks,

Luke

[Federico D'Amico]

Hello, i can buy you a esp8266 or esp32 dev kit.

Thanks,

Federico

[Luke Bakken]

It turns out the root cause of this issue is due to TLS handshake timeouts when client cert auth / verification is enabled. RabbitMQ does not log this timeout (yet), which made the problem difficult to diagnose.

Many thanks to Federico for providing the ESP8266 hardware I used to figure this out 🥇

https://github.com/rabbitmq/rabbitmq-server/issues/11171