Using another MQTT broker with Mainflux.

[Hashin Jithu]

First of all, congrats and a big thank you to Drasco and team for the amazing work you do with Mainflux. I have deployed mainflux for an experimental set up (I used docket to install). I want to know how I can configure (install) mainflux to use a third party MQTT like Mosca or HiveMQ. How to go forward with just replacing the mainflux-mqtt with another MQTT broker?

Thanks in advance for the response!

- Hashin

[Drasko DRASKOVIC]

Hello Hashin,

On Fri, Jan 13, 2017 at 10:31 AM, Hashin Jithu <hash...@gmail.com> wrote:
> First of all, congrats and a big thank you to Drasco and team for the
> amazing work you do with Mainflux. I have deployed mainflux for an
> experimental set up (I used docket to install).

Thanks for the kind words.

> I want to know how I can
> configure (install) mainflux to use a third party MQTT like Mosca or HiveMQ.
> How to go forward with just replacing the mainflux-mqtt with another MQTT
> broker?

Mainflux MQTT server adaptation is located here:
https://github.com/mainflux/mainflux-mqtt. There you will see that we
took Aedes, NodeJS based MQTT server which is basically evolution of
Mosca. Aedes comes from Matteo Collina, author of Mosca server and is
rewrite from the scratch to make Mosca faster
(https://github.com/mcollina/aedes) and more scalable.

We have been evaluating several MQTT brokers and we ended up with
Aedes. Before that we were using Erlang based EMQTT broker:
https://github.com/mainflux/emqttd-docker, but EMQTT project had a lot
of unstability and breaking changes, so we switched to Aedes.

MQTT broker is decoupled from rest of Mainflux components via NATS
broker (https://nats.io/). So - in order to forward messages coming
from MQTT sensors to the Mainflux Core (where Mainflux database is),
MQTT broker pushes messages to NATS, on the topic mainflux/mqtt/core
(which should be remembered as "mqtt -> core"):
https://github.com/mainflux/mainflux-mqtt/blob/master/mainflux-mqtt.js#L104

Equivalent for receive - when someone pushes message to the Mianflux
channel via HTTP, then Core will:
1) Write this message into database and
2) Forward this message towards MQTT broker by publishing it on NATS,
topic " mainflux/core/mqtt" :
https://github.com/mainflux/mainflux-mqtt/blob/master/mainflux-mqtt.js#L77

Thus, whatever broker you take you must be capable to do 2 things:
1) Subscribe to NATS topic "mainflux/core/mqtt" in order to recieve
messages from the Core and the re-publish to MQTT clients over MQTT
PUB
2) Intercept all messages coming from MQTT sensors and publish them to
NATS topic "mainflux/mqtt/core", so that Core also can recieve this
message and write it into database

When you are sending message from MQTT to Core, it wuld be important
to fill-in the msg{} struct:
https://github.com/mainflux/mainflux-mqtt/blob/master/mainflux-mqtt.js#L97-L103
msg.publisher = client.id
msg.payload = packet.payload.toJSON().data
msg.topic = packet.topic

This is the information that Core needs in order to properly store
MQTT message in the database.

I am not sure how this can be done with HiveMQ, but I guess they have
Java plugins that can do exactly the same thing. For Mosca - you just
copy what we did with Aedes, it is very similar :).

Can you please tell me why do you really need to change MQTT broker? I
can understand for HiveMQ - you want to use cloud-based broker. But I
think that changing Aedes with Mosca does not make a lot of sense,
especially that Aedes is new and improved version of Mosca.

Best regards,
Drasko

[Hashin Jithu]

Thank you so much for the detailed reply. I am sure this is exactly the information I was looking for!

I was playing around VerneMQ for a while and is impressed by their credentials (I am yet to empirically verify their claims). We actually did set up a system with lelylan with VerneMQ and it is working fine. It is at this time I stumbled upon mainflux and I am totally hooked up with the platform (thank you once again!). I was sceptical about a NodeJS based MQTT against an Erlang based one. 

Also, during initial phases, we did setup a lelylan system with Mosca and I do know that they don't support MQTT QoS 2. And I am anticipating that we will need QoS 2 at some point of time. So was not sure if mainflux-mqtt will support QoS 2. I am afraid I posted this question before checking it out. For me, the gut feeling for Erlang was a bit strong and I was enquiring about the chances of using VerneMQ with mainflux. Hope you are clear with my position.

And any insights will be highly appreciated. :) Thank you once again! 

[Drasko DRASKOVIC]

Hello Hashin,

On Mon, Jan 16, 2017 at 12:35 PM, Hashin Jithu <hash...@gmail.com> wrote:
> First of all, thank you so much for the detailed reply. I am sure this is

> exactly the information I was looking for!
>
> I was playing around VerneMQ for a while and is impressed by their
> credentials (I am yet to empirically verify their claims). We actually did
> set up a system with lelylan with VerneMQ and it is working fine. It is at
> this time I stumbled upon mainflux and I am totally hooked up with the
> platform (thank you once again!). I was sceptical about a NodeJS based MQTT
> against an Erlang based one.

We did a lot of research on brokers as we felt the same... EMQ looked
promising, but turned out to be very unstable and not suitable for
production.

VerneMQ looks sharper, but I've spotted serious probelm with it that
is a deal-breaker for our set-up:
https://github.com/erlio/vernemq/issues/197
It is my opinion that this will not scale well, and we need a broker
to which we have an access and hooks to leak out messages to some
external queue (Kafka or similar) before they are published from that
particular host to other MQTT brokers to which this host is bridged
to.

VerneMQ does not allow this for the moment (neither is RabbitMQ), but
Aedes does.

>
> Also, during initial phases, we did setup a lelylan system with Mosca and I
> do know that they don't support MQTT QoS 2. And I am anticipating that we
> will need QoS 2 at some point of time. So was not sure if mainflux-mqtt will
> support QoS 2. I am afraid I posted this question before checking it out.
> For me, the gut feeling for Erlang was a bit strong and I was enquiring
> about the chances of using VerneMQ with mainflux. Hope you are clear with my
> position.

Aedes supports QoS2:
https://github.com/mcollina/aedes#clientpublishmessage-callback

I am not against supporting VerneMQ in Mainflux - they have Lua
inteface for plugins, and all one has to do is to write a small plugin
for auth forwarding, similar to what we currently did with Aedes
(there is just a few lines of code).

But I must say that I am not convinced that this is a good choice (yet) because:
- I expect NodeJS optimized Aedes to be much faster (Erlang shines in
concurrent connections due to BEAM light processes, not speed)
- Erlang deployments are more difficult tahn NodeJS
- It will be hadred for people to handle Erlang module, as NodeJS is
more current in the community
- Issue with VerneMQ scaling I was mentioning

Keep in mind that all Mainflux microservices - MQTT broker included -
are behind the NGINX reverse proxy, which will do the load balancing.
This in combination with Kubernetes should give you HA, no mather
which MQTT broker you run behind. So - Erlang clustering does not make
much sense neither - as we already have Redis in the system anyway,
and we have NGINX for HA, and so on... If there was ONLY MQTT broker
in the system - then I understand, you use Erlang to do all this jobs
(distributed mem via Mnesia, clustering, fault tolerance via OTP,
...). But since we have all these component in the system
independently of MQTT broker (Redis for distributed mem and
clustering, NGINX and Kubernetes for fault-tolerance/monitoring/HA,
...) then it does not makes to much sense to go through Erlang
deployment pains and probably obtain lower performance.

What is really missing in my opinion is good and robust Golang MQTT
broker. InfluxData stopped work on SurgeMQ
(https://github.com/influxdata/surgemq), although it was very
promising... I guess one day Mainflux team will have to do it :).

BR,
Drasko

[Drasko DRASKOVIC]

On Mon, Jan 16, 2017 at 1:06 PM, Drasko DRASKOVIC
<drasko.d...@gmail.com> wrote:
> VerneMQ looks sharper, but I've spotted serious probelm with it that
> is a deal-breaker for our set-up:
> https://github.com/erlio/vernemq/issues/197
> It is my opinion that this will not scale well, and we need a broker
> to which we have an access and hooks to leak out messages to some
> external queue (Kafka or similar) before they are published from that
> particular host to other MQTT brokers to which this host is bridged
> to.

Giving here a second look, it looks like taking data of every
individual MQTT broker in the cluster can be done via
`auth_on_publish` hook as described here:
https://vernemq.com/docs/plugindevelopment/publishflow.html

From what I see here:
https://github.com/erlio/vmq_diversity#your-first-lua-plugin it is
supported even in VMQ_Diversity, so it should be relatively easy to
implement a Lua plugin that connects to NATS
(https://github.com/dawnangel/lua-nats) and/or does communication with
mainflux-auth.

In my opinion VerneMQ can be integrated relatively easy in the place of Aedes.

Only question is - do we need this, as this introduces more complexity
and I can not see real benefits for now...

BR,
Drasko

[Hashin Jithu]

Hi Drasko,

Thank you so much for the detailed reply. I was behind this very issue for last two days trying to make a benchmark to see if VerneMQ really make any difference. But having gone through your answer, it is pretty clear to me that the choice won't make much of a difference. As you said, there *might* be small gains, but the complexity of erlang deployment and the other things that comes with it will outweigh all the advantages we might get.

Also, my preference for the erlang was due to it's ability in handling concurrent connections, but since we are using NGINX I think that might not be an issue for individual brokers (I was totally missing this one!). Thank you once again for the kind reply. I am pretty new to this and was simply thinking out loud with an open mind. Your detailed reply was so helpful!

I am going through the code and *will* come back if I have anymore doubts. Thank you so much!

[Drasko DRASKOVIC]

Hi Hashin,

On Tue, Jan 17, 2017 at 11:27 AM, Hashin Jithu <hash...@gmail.com> wrote:
> Hi Drasko,
>
> Thank you so much for the detailed reply. I was behind this very issue for
> last two days trying to make a benchmark to see if VerneMQ really make any
> difference. But having gone through your answer, it is pretty clear to me
> that the choice won't make much of a difference. As you said, there *might*
> be small gains, but the complexity of erlang deployment and the other things
> that comes with it will outweigh all the advantages we might get.

Yes - I think it is difficult to see at this point if Erlang solution
will bring us benefits in the system arch we have now.
As I pointed out, I think Erlang really shines when you have "Erlang
only" server, because Erlang/OTP ecosystem can handle everything -
from clustering to distributed hashes, from HA to fault tolerance. But
since Mainflux already adds for example Redis (for Auth server) or
NGINX for load balancing, and since we have Docker containers that can
be monitored by Kubernetes and restarted quickly whenever one Docker
dies (this resembles to OTP supervisor) - adding Erlang-based MQTT
broker will not bring much benefit.

Now, if we had **only** MQTT broker - this is other story. Then I
would probably look for Erlang solution.

What also is a bit downside with Erlang is that plugins/modifications
are a bit harder to implement for many people. Go is easy to grasp,
learn and read. Erlang is still cryptic to a big number of people.

My proposition is that we start with simplest solution - and in this
case it is Aedes - until we build all parts of Mainflux system to be
robust (there are now bigger priority tasks, like security, etc...).
And once we put Mainflux in production and start scaling it we can
discover issues and determine requirements more precisely. Then we can
see if it is worth switching to VerneMQ or something else - because as
I pointed out in previous mail it will be trivial to do this.

>
> Also, my preference for the erlang was due to it's ability in handling
> concurrent connections, but since we are using NGINX I think that might not
> be an issue for individual brokers (I was totally missing this one!). Thank
> you once again for the kind reply. I am pretty new to this and was simply
> thinking out loud with an open mind. Your detailed reply was so helpful!

From my understanding well-crafted SW in Go can probably come very
close. In the end it is all about SW architecture, not the language
you are using (although it surely helps when one language has built-in
mechanism for IO and concurrency handling). But look at the example of
NATS broker we are using - it is a beast! For me this is a perfect
example of great Golang engineering. Here is an interesting article
from real-life experience:
https://blog.greta.io/a-signaling-system-for-10m-concurrent-connections-10d327fd6837#.pwlv83x9k,
proving that NATS can handle C10M problem
(http://c10m.robertgraham.com/p/manifesto.html). You can read more
about NATS impressive benchmarks here:
http://bravenewgeek.com/benchmarking-message-queue-latency/

It is my impression that if we can somehow leverage on this we could
have a killer MQTT broker.

But of course, in Mainflux system (and probably in any bigger system)
there is (NGINX) proxy, there is Auth server to which all requests are
forwarded for inspection, it probably has to do lookups in the Redis,
and so on... things are more complex than just pure server that can
accept a lot of simultaneous connections... I think we are doing
things for now in good direction, but it is still to early to optimize
at this point.

BR,
Drasko

[Drasko DRASKOVIC]

Hello Andre,
thanks a lot for the response. I am keeping your response below as
Google Groups deleted it for some reason (allowing post does not
re-create the message... hm...)

Please note that question here is not Erlang vs Golang (or NodeJS),
but Erlang in Mainflux system - does it make sense behind NGINX that
does SSL termination and TCP proxying and Mainflux Auth server that
checks every message (by authenticating client SSL certificate).

I know VerneMQ does not use Mnesia, but if I understand correctly
solution is somewhat similar - a distributed hash table in RAM that
would allow clustering.

I would love that we include one day VerneMQ support in Mainflux, and
I think it should be really easy and can be done even via Lua plugin
as I mention before (and notably via "auth_on_publish_hook" -
https://vernemq.com/docs/plugindevelopment/publishflow.html) and
Erlang NATS client:https://nats.io/download/yuce/teacup_nats/. This is
done today in Aedes MQTT Javascript code:
https://github.com/mainflux/mainflux-mqtt/blob/master/mainflux-mqtt.js#L47-L83
to allow MQTT broker to communicate with Mainflux Core service via
NATS, and should be trivial to re-create in VerneMQ.

As a side note, one more thing that bothers me with VerneMQ is MIT
license. I would like to see Apache-2.0 license there, as I was bitten
in the past by patents hiding behind MIT-licensed SW.

A propos: https://blog.chef.io/2009/08/11/why-we-chose-the-apache-license/

BR,
Drasko

On Tue, Jan 17, 2017 at 3:46 PM, André Graf <an...@dergraf.org> wrote:
> Hi Drasko, hi Hashin,
>
> Thanks for checking out VerneMQ in combination with the Mainflux platform.
>
> I am one of the core VerneMQ developers and would like to point out some
> clarifications to what has been mentioned above.
>
> The scalability issue https://github.com/erlio/vernemq/issues/197 has been
> resolved by the introduction of the new subscriber groups feature which is
> available in the current nightly build and will be part of the upcoming
> release.
>
> VerneMQ doesn't use the Mnesia database (RabbitMQ and EMQ) do.
>
> Let us know if you need help setting up VerneMQ (https://vernemq.com). We're
> also on Slack btw. http://slack-invite.vernemq.com/ and IRC #vernemq on
> freenode.
>
> Cheers,
> Andre
>
> --
> You received this message because you are subscribed to the Google Groups
> "Mainflux" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to mainflux+u...@googlegroups.com.
> To post to this group, send email to main...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/mainflux/1269c047-15d4-4171-839c-8090079b9e98%40googlegroups.com.
>
> For more options, visit https://groups.google.com/d/optout.

[André Graf]

In some scenarios it makes sense to use an external proxy, either for load balancing or just as an TLS terminator.

VerneMQ is released under Apache 2.0

[Drasko DRASKOVIC]

On Tue, Jan 17, 2017 at 5:11 PM, André Graf <an...@dergraf.org> wrote:
> In some scenarios it makes sense to use an external proxy, either for load
> balancing or just as an TLS terminator.

In Mainflux project reverse proxy - in our case NGINX - has multiple roles:
- Load balancing
- TLS termination
- Auth Request Forwarding
- HTTP, TCP (for MQTT) and UDP (for CoAP) proxy-ing
- UI (static) server
- Probably rate limiting or some other things (via Lua plugins,
OpenResty, Kong, don't know now...)

Probably most important for now is Auth Request Forwarding, where
reverse proxy acts similar as XACML PEP
(https://docs.oracle.com/cd/E27515_01/common/tutorials/authz_xacml_pep.html)
- it intercepts all requests and redirects them to mainflux-auth
(https://github.com/mainflux/mainflux-auth) server which does AtuhX
and AuthZ. This keeps architecture clean and microservices well
decoupled.

So - we need proxy. And we need Auth server. And MQTT broker gets
request only after these two (or in between, if MQTT broker has to
process something - like `client_id` before re-sending req to Auth
server).

> VerneMQ is released under Apache 2.0

Great! I don't know why I thought it was MIT (I am following too many
projects, things start to mix in my head ;)).

BR,
Drasko