MQTT plugin performance

[Atul Kshirsagar]

We have been doing performance testing on RabbitMQ primarily focused on MQTT protocol. Our use case is to support MQTT publishers with QoS 1 using two way TLS and need to target throughput rate of more than 50K msgs/sec. However, our performance tests yielded us not so encouraging throughput numbers. Below given is our test setup and results that we got:

Setup

14 node RabbitMQ (version 3.5.7) cluster on AWS

Instance type of RabbitMQ node: m4.xlarge

Persistent disk attached to node: 8GB general purpose SSD

AWS ELB configured to load balance between all 14 RabbitMQ nodes.

Test

Tests were performed for both MQTT and AMQP protocols. Following two scenarios were tested:

1. 1 topic 
2. Multiple topics

In each case there was only one subscriber per topic and multiple publishers. Also, the qos for MQTT for both publisher and subscriber was set to 1. For AMQP, acknowledgements were enabled for both publishers and subscribers (with durable queues for subscribers) to simulate qos 1 behavior of MQTT.

Both publishers and subscribers were Java clients using paho client library for MQTT and rabbitmq amqp java library for AMQP.

Both publishers and subscribers were using two-way TLS (client cert based auth).

Results

1. MQTT
1. 1 topic: With 30 concurrent publishers we got about 2.5K msgs/sec with around 80-85K msgs getting queued.
2. 14 topics: With 10 concurrent publishers per topic we got about 11K msgs/sec with around 200 msgs getting queued.
2. AMQP
1. 1 topic: With 20 concurrent publishers we got about 20K msgs/sec with around 49K msgs getting queued.
2. 14 topics: With about 1 publisher per topic we got about 47K msgs/sec with around 3K msgs getting queued.

We are seeing a major decrease in performance (throughput) of RabbitMQ  when we switch to using MQTT. 

• Increasing number of topics does seem to increase performance but beyond 14 topics we don’t see any change in throughput. Somehow it seems that highest throughput tops at 11K msgs/s.
• Increasing or decreasing the cluster size doesn’t seem to make any difference. 
• We also tried using Golang based publishers (https://github.com/takanorig/mqtt-bench) to eliminate if it is client side issue but we saw the same numbers with this client too.
• When we changed to use Qos 0 for MQTT publisher then we found that the throughput increased significantly (comparable to AMQP). However, our use case requires use of Qos1 and this possibly looks like MQTT plugin issue.

Questions:

1. Can anyone share any data about MQTT performance on Rabbit? Is MQTT plugin known to cause performance  degradation?
2. Are there any configuration changes that we can do to get better performance from MQTT plugin of Rabbit? 
3. Is there anything wrong with our setup/tests?

Regards,

Atul Kshirsagar

[Michael Klishin]

There is some overhead but it's not where the time is spent. The time is spent transferring messages between cluster nodes.

 Mirroring has significant effect on throughput. Having publishers and consumers connected to different nodes

means more messages must be transferred between nodes first before they can be delivered to the consumer(s).

When data locality is taken care of, you can go well above 50 or 100K messages per second:

https://blog.pivotal.io/pivotal/products/rabbitmq-hits-one-million-messages-per-second-on-google-compute-engine

This has been discussed many times on this list before.

1. Are there any configuration changes that we can do to get better performance from MQTT plugin of Rabbit? 

Some hints are available at http://www.rabbitmq.com/networking.html. Using multiple queues is another

suggestion but with multiple publishers and subscribers with MQTT that happens anyway.

 

1. Is there anything wrong with our setup/tests?

Regards,

Atul Kshirsagar

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ

[Atul Kshirsagar]

What we are trying to make sense of is why MQTT performance drops by around 4x compared to AMQP. Data locality (or lack thereof) applies to both AMQP and MQTT in this case but MQTT somehow gets impacted more. Unfortunately switching to using AMQP is not an option in our case. 

--Atul

[Laing, Michael]

For AMQP, are you using persistent messages & durable queues?

[Atul Kshirsagar]

Yes, we are using persistent messages and durable queues for AMQP:

AMQPPublisher.java

channel.basicPublish(EXCHANGE_NAME, routingKey, MessageProperties.PERSISTENT_TEXT_PLAIN, message.getBytes("UTF-8"));

AMQPSubscriber.java

channel.queueDeclare("AMQPSubscriber-"+bindingKey.replace(".", "-"), true, false, cleanSession, null).getQueue();

--Atul

[Laing, Michael]

Are any queues mirrored by policy for either MQTT or AMQP?

We are interested in your experiences because we are also preparing to test MQTT under load.

ml

PS We never persist messages in our AWS clusters as we found it led to partitions and relatively poor performance. In fact, to proceed with MQTT, we will have to have the ability to remove persistence for QoS 1.

[V Z]

Is there any update to this from either side?

I tried the following setup:

- MQTT QoS=0 and AMQP non-persistent/not confirmed 

- single publisher and single consumer

- Rabbit 3.5.7 on local Linux VMs (2 node cluster but I connected publisher and consumer to the same VM) 

- consumer prefetch=1000 (both MQTT and AMQP)

and observed:

AMQP: ~40K/sec publish and deliver

MQTT: ~8K/sec publish and ~1K/sec deliver

My MQTT Java app (using Paho client) had to pause frequently because the max in-flight buffer was full, whereas my AMQP Java client was flow-controlled by Rabbit server. This seems to suggest that ingestion rate is hurt in case of MQTT, doesn't it?

I used other MQTT brokers, and had no such effect -- 40K/sec to publish and to deliver, which means I was basically saturating my network interface.

Is there an MQTT PerfTest like there is for AMQP?

[Aviv Rosenfeld]

Hi Atul,

Did you find a solution for this one? Are you still using RabbitMQ mqtt plugin?

I have a similar use case, targeting to handle +50K msg/s using RabbitMQ mqtt plugin.

Regards,

Aviv