Mqtt weird behavior under load test

[Kilic Ali-Firat]

Hi everyone, 

We are using MQTT in our company and we want to find the max load in number of clients with a fixed rate of publication of one mosquitto node. 

Our current architecture is to have one AWS server with two mosquitto (one getting data from publisher, other one getting data to consumers) like below : 

Publisher -> mosquitto_in -> mosquitto_bridge -> mosquitto_out -> Consumer

In publisher side, we are sending 70 msg / sec with a payload of 1,3 KB per message and qos used is 1. 

In consumer side, it's classical java paho reading message. 

I did a first test with 100 devices publishing in parallel and my test has been successful. 

I got good performances from AWS service : 20 % - 40 % CPU used, only 1 GB used on 16 GB max. 

But when I did the same test with 150 devices, I have a weird behavior of mosquitto_out process : sometimes it reach 100 % and decrease after few minutes until the process crash and disconnects my consumers. 

I cannot understand what happening inside mosquitto to have a such behavior and if someone has an idea about what I'm missing, I will take it. 

Below my mosquittos configurations. 

mosquitto_in : 

pid_file /var/run/mosquitto_bridge_in.pid


persistence true
persistence_location /var/lib/mosquitto/in/


log_dest file /var/log/mosquitto/mosquitto_bridge_in.log


include_dir /etc/mosquitto/conf.d


# Bridge options
connection bridge


# Broker address to forward messages coming in mosquitto_in
address 127.0.0.1:1884


# Define the topic to share with other mosquitto. Note that the qos below is
# maximum qos, it means that you can only forward to other brokers with a
# qos 0.
topic streaming_batch/+/+ out 0 "" ""
cleansession false

# Mqtt load options
max_inflight_messages 128
max_queued_messages 0

mosquitto_out : 

pid_file /var/run/mosquitto_bridge_out.pid


persistence true
persistence_location /var/lib/mosquitto/out


log_dest file /var/log/mosquitto/mosquitto_bridge_out.log


include_dir /etc/mosquitto/conf.d


max_inflight_messages 128
max_queued_messages 0


port 1884


# No standand of mosquitto. Read more here : https://mosquitto.org/man/mosquitto-conf-5.html
upgrade_outgoing_qos true

[toast-uz]

Hi Killic,

To clarify the issue, could you change the Consumer QoS to 0?

As my experience, QoS>0 of Consumer cannot offer enough performance in the IoT model (many producer & single consumer) even under setting max_inflight_messages 0.

Regards,

Tatsuzo (toast-uz)

[Kilic Ali-Firat]

In qos 0, it will work but my use case doesn't allow to use a qos < 0. 

The thing is that I need all messages once and I need to use the qos 1 .. 

[toast-uz]

Seems a performance bottleneck by protocol overhead between mosquitto_out and Consumer.

The candidate solutions are:

1) Use QoS=0 on your consumer.

2) Create many consumers subscribing each topic.

     ex)  Consumer1 subscribes a/#, Consumer2 subscribes b/#, ...

[toast-uz]

p.s.

Setting max_inflight_messages 0 (unlimited) may improve a little.

[Kilic Ali-Firat]

First, I see that I was using an old version of mosquitto (1.4.14). 

Second one, with this version, I have an another behavior : cpu increases to 100 % sometimes and then my consumer throughput decreases but producers and consumer are not disconnected. 

When CPU increases to 100 %, I get less messages than expected and during 20/30 seconds, consumer didn't get any messages.

When CPU decreases, consumer received again messages from the broker .. really I cannot understand what it is happening ..