Evaluating MQTT Clients

[idca...@gmail.com]

Hi,

    I would like to evaluate several MQTT clients (NOT brokers). For example, I am interested to know how many messages per second can be published by a client before getting throttled. I know that there are tools for evaluating brokers like Jmeter, mqtt-malaria (I have used Jmeter for evaluating brokers) but I have  not found any tool for evaluating clients. Some info on this would be very helpful.

Thank you.

[Hans Jespersen]

What do you mean by "getting throttled"? Are you just trying to determine the maximum throughput of various client implementations on a given hardware and OS platform or is there a specific use case or network architecture that you are targeting? Which clients and which programming language (s) do you want to test? What broker are you going to test them against because clients don't run by themselves without connecting to some broker?

[Ananth Bala]

Thank you for your email.   
                                                                              
Yes, I would like to determine the maximum throughput of various client implementations. I was thinking of using "Mosquitto" as the broker and checking the efficiency of client implementations such as 'Eclipse Paho", 'libmosquitto', 'emqtd' etc. What I would like to determine is whether there is a limit on the publishing rate of client. For example, if I want to publish a lot of messages (say 100000) on several topics, does the client experience some throttling. By throttling, I mean the inability of clients to publish a lot of messages within a certain time period, and as a result breaks. 

Thanks in advance for your inputs.

Thanks,
Ananth                                                                                                                                                         

On Tue, Sep 22, 2015 at 4:47 PM, Hans Jespersen <hans.je...@gmail.com> wrote:

What do you mean by "getting throttled"? Are you just trying to determine the maximum throughput of various client implementations on a given hardware and OS platform or is there a specific use case or network architecture that you are targeting? Which clients and which programming language (s) do you want to test? What broker are you going to test them against because clients don't run by themselves without connecting to some broker?

--
To learn more about MQTT please visit http://mqtt.org
---
You received this message because you are subscribed to a topic in the Google Groups "MQTT" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/mqtt/17rsdjj3-xc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to mqtt+uns...@googlegroups.com.
To post to this group, send email to mq...@googlegroups.com.
Visit this group at http://groups.google.com/group/mqtt.
For more options, visit https://groups.google.com/d/optout.

[Hans Jespersen]

In real production pub/sub systems, publishing is usually the easiest part of the system. You are most likely to hit a bottleneck on a brokers ability to ingest messages (particularly at QoS 1 or higher). If you are doing the complete pub/sub round trip then you might also find the throttling comes from a slow consumers inability to receive and process messages.

You can of course just publish messages and discard them at the broker which really proves nothing if the publishing client is not the bottleneck in the real world.

If you are interested to compare various clients I do have access to a very high speed MQTT broker as a comparison to whatever your results are with mosquitto. If the same client can get higher numbers with a different broker then the bottleneck is not the client.

[Ananth Bala]

Thank you so much for the reply. I see your reasoning. Could you please share your broker?

Thanks again,

[Hans Jespersen]

The Solace MQTT Broker is available for free 30 day trial download from http://dev.solacesystems.com/downloads. It's called the Virtual Message Router (VMR) and it also supports REST, JMS, WebSockets, and some other wire protocols and client APIs.

On the same site you will see an MQTT benchmarking tool call sdkperf_mqtt. This is a command line tool that includes the Paho Java MQTT client and which can generate messages at various configurable rates and sizes.

For example running sdkperf_mqtt against mosquito 1.3.5 on my macbook pro, I get the following results:

$ sdkperf_mqtt -cip localhost -mpq 0 -ptl mytopic -mr 30000 -mn 3000000 -msa 16

---------------

Aggregate Stats (Total # clients: 1):

-------------------------------------------------

Total Messages transmitted = 3000000

Computed publish rate (msg/sec) = 23460.0

-------------------------------------------------

running the same test against a Solace VMR also running on my mac (in virtual box) I get these results: 

sdkperf_mqtt -cip 10.0.0.91 -mpq 0 -ptl mytopic -mr 30000 -mn 3000000 -msa 16

---------------

Aggregate Stats (Total # clients: 1):

-------------------------------------------------

Total Messages transmitted = 3000000

Computed publish rate (msg/sec) = 26182.0

-------------------------------------------------

In both cases the broker fails to get to the target message rate of 30,000 msg/sec (the -mr 30000 parameter . These are 16 byte messages (-msa 16) and the test ran over 300,000 total messages (-mn 30000) using QoS 0 ( -mpq 0). I could run these benchmarks against a hardware appliance that would yield much high rates but I think you can see already that the Paho client libraries are not the bottleneck when running full out using mosquitto as the broker.

For a full list of command line options for sdkperf_mqtt see "sdkperf_mqtt -hm". Some options make more sense for the JMS or REST versions of sdkperf so if you see references to queues or other non-mqtt messaging artifacts its just a carryover from the other versions of the tool and the fact that the Solace broker supports more than just mqtt clients,

-hans

[Roger Light]

Hi Hans,

Thanks for sharing, that looks like an interesting tool. I'm not sure
I agree with your assessment though - I do not think the broker is the
limit in the test you show.

I wrote a simple C program that listens on a socket, accepts a
connection on that socket, blindly sends a CONNACK response indicating
success, then reads as fast as it can. Using that test, I get very
similar results to you - a publish rate in the range of 23-27k/s.
During the test it only uses about 14% CPU. I can run it under
callgrind and still get the same performance, which is not what I
would expect if the server was the limiting factor.

The code is here:

https://gist.github.com/ralight/0d1d6cb4e353c075b152

Cheers,

Roger

> --
> To learn more about MQTT please visit http://mqtt.org
> ---

> You received this message because you are subscribed to the Google Groups
> "MQTT" group.
> To unsubscribe from this group and stop receiving emails from it, send an

[Hans Jespersen]

Thank for calling me out on this one Roger. In retrospect, I posted exactly the test that I most dislike because it's the pub/sub equivalent of a no-op. Benchmarking how fast a client can publish to no subscribers is not that useful. You have rightly highlighted that it amounts to a test of how fast a broker can throw away messages.

My original point was that in the real world, a single client's ability to publish one stream of data is less likely to be a bottleneck. It's more likely to be the round-trip time, or the I/O rates in a persistent storage layer, or network bandwidth, or the maximum number of concurrent client connections, or something else.

A better comparison would be to show the same broker and client producing varied results by modifying some of these other parameters (like switching QoS from 0 to 1, enabling persistence, moving the broker from a local host to a remote host, or increasing the number of concurrent client connections). My intent was not to disparage mosquitto in any way. Thank you for your tremendous contributions to the MQTT community. 

-hans

[Roger Light]

Hi Hans,

> My original point was that in the real world, a single client's ability to
> publish one stream of data is less likely to be a bottleneck. It's more
> likely to be the round-trip time, or the I/O rates in a persistent storage
> layer, or network bandwidth, or the maximum number of concurrent client
> connections, or something else.

Yes, very much agreed.

> My intent was not to disparage mosquitto in any way. Thank you
> for your tremendous contributions to the MQTT community.

Don't worry, I didn't take it as that - I could hardly not take a look though :)

Cheers,

Roger