MQTT Scaling

[Nguyen Tien Dat]

Hi everyone,

I'm implementing a mobile chat app. Some of my requirements are Clustered and QoS 2. Currently my choice is MQTT over ActiveMQ. It works but I'm not very happy with it's performance. Does anyone have experience with scaling with any kind of MQTT Brokers? My question is quite general so feel free to have more discussion.

Thanks.

[Hans Jespersen]

You do not say what you are trying to scale (throughput, connect counts, # of topics, # of users, amount of persisted messages stored, etc). It would be helpful to know some of your design goals in order to make recommendations. There are a very wide variety of MQTT brokers out there and many of the offerings specialize in different dimensions of the scalability problem. At the very highest end of scalability wrt connect counts and message throughput there are even dedicated purpose built appliances like the IBM MessageSight and Solace Message Router boxes. These boxes can handle 1 million or more concurrent connections and 1 million or more message/sec of throughput (even at QoS 2). However if you are trying to run this all in a public cloud like Amazon AWS or it has to run in your own private cloud on OpenStack or vSphere then that's important to know these constraints because then the list of available choices would be quite different.

[Nguyen Tien Dat]

Thanks for your response.
My design goal is to scale number of connections. As I said I'm creating a chat app so my current design is one topic per chat group (chat between 2 people also create 1 topic). Along with that, I also have topics for notification and some topics for mission critical business. Therefore I also need to scale number of topics but I'm not very sure if my approach is correct or not. FYI, I'm having about 5 million users and expect to grow more.

For your suggested solutions, It's very helpful and I will do some more researches on that. Anyway, I prefer open source so that I have fully control over it and can have an in-house maintenance.

Thanks.

--
To learn more about MQTT please visit http://mqtt.org
---
You received this message because you are subscribed to a topic in the Google Groups "MQTT" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/mqtt/sFjCBovLavw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to mqtt+uns...@googlegroups.com.
To post to this group, send email to mq...@googlegroups.com.
Visit this group at http://groups.google.com/group/mqtt.
For more options, visit https://groups.google.com/d/optout.

[Hans Jespersen]

For Open Source I typically recommend Mosquitto but based on some quick research on Stack Overflow it looks like Mosquitto would have to scale horizontally in units of ~20,000 connections per instance which would mean running 250 instances to get to 5 Million connections. 

Mosquitto Connection Limit - http://stackoverflow.com/questions/10087396/mqtt-mosquitto-linux-connection-limit

You mentioned not being happy with the performance of Apache ActiveMQ. Have you tried Apache Apollo? Apollo is a long running attempt at building a next-gen ActiveMQ. I can't speak to it's performance or stability but according to the link below at least one person has managed to get ~1M MQTT connections to work.

Apache Apollo Connection Limit - http://stackoverflow.com/questions/29358313/max-mqtt-connections

[Nguyen Tien Dat]

Mosquitto performance is great but I don't know how people scale out Mosquitto brokers in production. It has bridge feature but not a "real" clustering. So it would be great if someone give me an example on how to config Mosquitto to work in real life.

For Apollo, I haven't look seriously on that since it looked like abandoned (I also tried Apache Artemis). But the links you provided really shed some light on me. Many thanks :)

[Hans Jespersen]

You never said if your connections require SSL/TLS or not. That makes a huge difference in the max number of connections as well as the CPU and memory requirements of the broker. You may also get better scalability at QoS 0 or QoS 1. I'm not sure why you feel you need QoS 2 for a chat app. 

[Nguyen Tien Dat]

Actually I'm expecting to use MQTT for multiple purposes. For chat, yes I can accept QoS 1 and no SSL/TLS. I also have some flow which process critical transactions and it should be QoS 2 and requires SSL/TLS.

On Tue, Nov 24, 2015 at 9:42 AM, Hans Jespersen <hans.je...@gmail.com> wrote:

You never said if your connections require SSL/TLS or not. That makes a huge difference in the max number of connections as well as the CPU and memory requirements of the broker. You may also get better scalability at QoS 0 or QoS 1. I'm not sure why you feel you need QoS 2 for a chat app. 

--

[Karl Palsson]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

> On Tue, Nov 24, 2015 at 9:42 AM, Hans Jespersen
> <hans.je...@gmail.com> wrote:
>
> > You never said if your connections require SSL/TLS or not. That makes a
> > huge difference in the max number of connections as well as the CPU and
> > memory requirements of the broker.

Says who? This is a commonly quoted claim but rarely backed up,
and it needs to be refuted more often.

The cpu overhead of a tls connection is primarily at connection
opening, and then is fairly proportional to the traffic size.
There is a memory overhead, but it's not nearly as big as people
claim.

In my own testing with mosquitto, with qos1 and large numbers of
permanent clients with small messages ~<1k, enabling TLS is more
like a 2-3% cpu increase. You run into plain connection limits
_well_ before you run into problems because of TLS. If you are
rapidly cycling clients, (as in the world of a web server, and
HTTP vs HTTPS where these claims typically originate) then yes,
you'll see of an impact on CPU.

Sincerely,
Karl Palsson

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)

iQIcBAEBAgAGBQJWVGukAAoJEBmotQ/U1cr2dEAQAIIuA+T25FpBGU1UXdXg4jv1
eOjpWBjyHe1iEb/LR15JOX9UK/Gu78fOISFYpD1/ut2WSgJc541VkAQyU1CiQbO7
4cjRfzU5NtOGasG2vTBftJawISfcgLVymOPWVu7HDk0NPt7XBgLzpse9TovWFv3O
HFdCTydu9vQ77evagIk9gnq3K+o82yAj/AYCDOIGxCJtHENToRigNwjEs1ddCzxS
ZlXj+Qg1r77fDQzp96+ZXq8quXERWWVIRRJ/7iKL+4JFGQuT2mdORUOchNCEei86
1bZyMB02oin7c8TOfOu8WAil8J1Wz1JSzDPWUbjieLgDEvPy6YNHB4LIPbT9qCZe
cY5a2mFpiotlUHx0cie6sBQny9akuatGGDNkw7CIw+6Jj5pMz6BMYAeLqjyT9ip/
bAWLdEyQym6/56e8zG7oq7DtexeJyNyjv11Aa88Hna40j0UwfH8sWxBawcPyA9zO
R8hiGt7Ajyq6Cefn6bWsUs7ETEy8vNprTDAPgeBKJmUHkPsbNgKwNdrvIJnpFe/i
fJ5fWXnidUmnUu7Jf8P2WPhdSxh3pa1qEmahDqCCPfnO0VHKIWvZtDPXWIOeQkLk
xdMVTCDdE4i+pp0WIhtngW9WUh9vJInv4H81LLBSGjpqHPLVhal2wXnvErF79S5Z
NAbY6l2xCYzF1kL3SFUF
=Rir6
-----END PGP SIGNATURE-----

[Hans Jespersen]

I understand that the memory overhead for OpenSSL with SSL_OP_NO_COMPRESSION and SSL_MODE_RELEASE_BUFFERS is about 128KB per connection.

At 5 million connections that's 640GB. 

 

[Karl]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

(TL;DR: don't believe everything you read on the internet...)

sure, that math adds up, that sounds totally serious and
reasonable. Just relies on two assumptions....

a) it actually takes 128kB/connection
b) you can actually get to 5 million concurrent connections

Both of those sound nice, but a*b is only a valid number if "a"
is reasonable and if "b" is relevant.

There's a longstanding story out there that ssl takes 128KB per
connection. see http://stackoverflow.com/a/14037828/600217 and
many many other places, was probably very true at one time, at
least for openssl; note that the answer was provided in 2012.
Without question, there _were_ absolutely, _real_ memory issues
with TLS in the past. People have made those complaints for a
long time now, and no matter what we might think of even the
openssl project, most of these concerns have actually been
addressed. (Time flies when you're having fun)

Unless you're the sort of person who thinks, "I should run centos
5.x and openssl 0.9.8, those are proven long term 'stable'!"
these sorts of historic claims are largely irrelevant.

Measure first, make claims second!

Don't want to measure? Sure, measurements are hard, and tedious,
and pretty boring really. Here's some references from other
people. Please be aware, they're generally discussing HTTP, but
if it's ok for HTTP, it's ok for mqtt with lots of short lived
clients, and if you lots of long lived clients, you're _better_
than HTTP.

"good" stories
* Google runs a few servers: https://istlsfastyet.com/
* Presenting at the IETF doesn't make it true, but it helps: "For everything but very short connections, TLS is not inducing any major traffic overhead (nor CPU or memory overhead)." https://www.ietf.org/proceedings/90/slides/slides-90-uta-5.pdf
* We've been here for a while, the _real_ complaints are that cert management is complicated: Let's go back to 2010, when it was _already_ claimed that SSL was efficient enough https://www.imperialviolet.org/2010/06/25/overclocking-ssl.html

"bad" stories (that actually become somewhat good
https://journal.paul.querna.org/articles/2011/04/05/openssl-memory-use/

Cheers,
Karl P

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)

iQIcBAEBAgAGBQJWVSfCAAoJEBmotQ/U1cr2K88P/AvF8Fed/ER7jFGBDix44rXw
kQqXOr/7iuwkENyLDecbi9E7PlnVf+Hs3QBGo27dN3RcoisEPMXSZCjBrN5Oqf8Z
MC1oGXhECRRYI0uc2DITmJbUjPjhVnHXV2DggolRwVDDFk+u1FzgPUgp68XgO4Vg
+eiL9CjZ/LH+pUN+vnFj8Gh1IeDks2MLN7IkeqQQVGSlvP6NSNijlm4KQyYHN34z
RTqMKwrjj1HMqo1c1yrNxe1Uay7dpuuu1BmTGYSNw8DdvkMgQX9fgdT0FFJnhxmO
kIar20A3o7fYMyFLLNSRM2lgcvgFHhUrWiGDNQ04j3NAGJO4WjLZeogPkWIaBQoK
IZ0kIT+LwuwutPVZCHpw9pnksUQ2zeeTiQWNEhFRjkkRt80UVsDIFJHtDd9NKYgw
YgabooK2d17vMHf+30jOSKgGohjpWCdytfSfkuSlo/rQGyVBVUzwjrBxb1JSTGqs
6deNs7XW85GYdNiBps3UWWzJCLeuSraWpnAeBD/8ljPrU6th0TdAgfFB3688ukrz
oTmTpiTkGHj+EGfbwyNkq1X9svJWim985I+Z015dbAx/21WdJlS+H9ikdjlKXKl+
IwZAEQeaV6a9yE3gHqwckGA8roYbK7TdxbSsRaAvq/4Pwu49CZ7kXRBVRxF5RFkd
eceAE3n0s+3ScVGs98Nf
=ta6t
-----END PGP SIGNATURE-----

[Roger Light]

Hi,

There's also https://www.mail-archive.com/mosquit...@lists.launchpad.net/msg00115.html

Crude, but reckons ~10kB per mqtt connection as a start.

Cheers,

Roger

--
To learn more about MQTT please visit http://mqtt.org
---

You received this message because you are subscribed to the Google Groups "MQTT" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mqtt+uns...@googlegroups.com.

[Hans Jespersen]

10KB per connection is much more encouraging. It's incredible hard to get good information on this topic so real and up to date measurements are much appreciated. I am just kicking off some testing with HAProxy in front of a cluster of MQTT brokers and had planned to use it's built-in support for SSL termination. If it's just 10KB per connection I might be able to just use a pass-through all the way back to the brokers for added security. Either way I will have measured results with the most recent openssl libs.

[Karl Palsson]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Old results, but also this:
https://github.com/remakeelectric/mqtt-malaria/blob/master/results.500bytes.xClients.0.166mps.pdf

I'd forgotten that file was in the repo, I was going to pull it
out today back in the office. Those are all with SSL turned on,
and it's only hinted at in the section labelled "500 bytes / 3000
clients at regular 1/6 Hz rate" that turning it off had no
impact. The "perf" report shows that it's not spending any time
in SSL code.

Note that those tests were done with and old mosquitto 1.2.2 and
later a 1.3 preview build that included some performance fixes.
(These were eventually released as mosuqitto 1.2.3)

Cheers,
Karl P

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)

iQIcBAEBAgAGBQJWVYo2AAoJEBmotQ/U1cr2ziQQAIPdrmVAT9ttaNAxrcYRNL85
HHAdDArCyJ5oWFJtxjNEjCfL5iRF0OSm4fnbhgzqcOuT8LU/HjFk+VhmIp4WwwEr
NeDnz0MJeaZVl3X/RbfV1/8sw8xK+9BxyBdudET2aO27FvaC0997pxpmsljmlVwU
VBIAQaUKlutWKEo6sddXDNRSIFLQzBU43I7DfTUvJcDcQZOr4LBKOQ+DBtxYfM5s
k/SweHdDcn+M4aFsMJ+PWcKfyt044BEQibX9SK5rrV6NX76lNVGQW9oFVnoJeIXR
X8cId6J+gA6BS+O/z01n8LQAJEHEwWHzVKu+8Tv3NbLRe7Lv0Gf/0GB9Zte1ozyx
OHUIW2S0XxZZ1NoHuVK8QOSZMgHitvVpSHpcLk5cKI2oJXcxTOwY5T5nL6HNNPLV
aAgYSAWqGGYLjcyULLuGgrGr6SVuKxZaB163blVhMb5W2YNbUoNr+gyPSL6KIt6M
cif9fp9mjfOQBBFUP18Pdg0ZbrH3Ysy1XN3zl5FXpFVescDunKrfDNYlKSASwWmY
OeZoyxIESWJYyGfKh6B8kqkDehJ4g228mX0ht8oUa4HpdlSNRDmeW7KeQ5o5C33T
MsPRoo0aIMmT7S6yTa3tEoZ9n0NIyjLZ4VusOvKXMawQwalB2WBV5D6meS7wIzlh
0XkhZcox8saGCiEe2rlx
=/xT/
-----END PGP SIGNATURE-----

[Sudhama Bhatia]

Nguyen, Was reading this article but havent even gotten close to trying it. https://medium.com/@lelylan/how-to-build-an-high-availability-mqtt-cluster-for-the-internet-of-things-8011a06bd000#.l1fkgyrf3

Also, Nguyen, did you have any luck ? We are currently using socket.io and node.js for chat but were contemplating moving to mqtt and were interested in this question as well

[Nguyen Tien Dat]

I read the article from the beginning of our research but since mosca didn't have QoS 2 support then we go with other approach. Basically we follow design from it and able to run with ActiveMQ - acceptable performance.
Not related to MQTT, but currently trying this: http://actor.im (akka and custom built protocol, plus great apps). Maybe you will interested in that too.

On Dec 20, 2015 6:06 AM, "Sudhama Bhatia" <sudhama...@gmail.com> wrote:

Nguyen, Was reading this article but havent even gotten close to trying it. https://medium.com/@lelylan/how-to-build-an-high-availability-mqtt-cluster-for-the-internet-of-things-8011a06bd000#.l1fkgyrf3

Also, Nguyen, did you have any luck ? We are currently using socket.io and node.js for chat but were contemplating moving to mqtt and were interested in this question as well

--
To learn more about MQTT please visit http://mqtt.org
---

You received this message because you are subscribed to a topic in the Google Groups "MQTT" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/mqtt/sFjCBovLavw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to mqtt+uns...@googlegroups.com.

To post to this group, send email to mq...@googlegroups.com.

Visit this group at https://groups.google.com/group/mqtt.

[Joan Llopis]

Hi,
I think https://verne.mq/ is worth a try.

From Its web page:

What is VerneMQ?

VerneMQ is a high-performance, distributed MQTT message broker. It scales horizontally and vertically on commodity hardware to support a high number of concurrent publishers and consumers while maintaining low latency and fault tolerance.

[mj]

Hi, 

Very interesting thread on MQTT performance and scaling. 

Is there any conclusion? We want to build an IoT gateway and are exploring mqtt scalability. 

[Lee Wong]

Hi Nguyan,

What is your experience on actor.im? It seems inactive since late last year. 

Thank you.

Cheers,

Lee