Huge IOT Project - Is MQTT right choice?

[Gourav Goyal]

Hi,

I'm working for a big electronics company which is getting involved in IOT.

Our goal is to build a single app (Android/iOS) which can handle all thermostats, security devices, water leak detectors etc installed in the home.

These devices would be connected to the cloud.

Additionally, We'll be supporting other company's devices also.

1. Is the MQTT right choice for this scenario? Any good alternatives?

2. Our stack is .NET so is there any active MQTT broker library for C# which can handle millions of devices connected to cloud?

[Paul Fremantle]

Gourav

The question you are asking is a big one. There are certainly large deployments of MQTT (like FB) that show it can work at your scale. The details are where the real challenges are in any protocol, not just MQTT. The honest answer is that MQTT will work well in many ways, but you will have to work closely with the broker to support millions of devices. 

I don't know of an MQTT broker for .NET that scales to millions of devices. There is this https://github.com/ppatierno/gnatmq but I don't know about scaling.

 Does the broker itself need to be in .NET? For example, IBM has an appliance that scales very large, or you could have a broker written in another model but everything connecting to it on the server side in .NET.

Paul

--
To learn more about MQTT please visit http://mqtt.org
---
You received this message because you are subscribed to the Google Groups "MQTT" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mqtt+unsubscribe@googlegroups.com.
To post to this group, send email to mq...@googlegroups.com.
Visit this group at https://groups.google.com/group/mqtt.
For more options, visit https://groups.google.com/d/optout.

--

Paul Fremantle

Doctoral Researcher, University of Portsmouth, School of Computing

Visiting Scientist, Institute of the Architecture of Application Systems, Stuttgart

Visiting Lecturer, Software Engineering Programme, Oxford University

Co-Founder, WSO2
Apache Member and Committer

email: paul.fr...@port.ac.uk, pa...@fremantle.org

twitter: pzfreo / skype: paulfremantle / blog: http://pzf.fremantle.org

[Ranjith Kumar Diraviyam]

Hi Gourav , 

MQTT is a right choice and you can use the SN clients for few of your devices which needs intermitted management. 

There are a lot of parameters beyond your application that are needed to set your mqtt broker to scale beyond 1M.   The hardware you use and the OS tweaks you use plays a major role here beyond the application.   

I personally love more linux than Windows and have scaled applications on linux.   We run our broker and have connected upto 100K nodes on a i5 2.5 GHz 8 GB RAM Ubuntu Machine with the Ulimit parameters tweaked.  This numbers can be increased as the Hardware capacity is increased. We have a developed a small distribution back end server which acts as a pivot for all the brokers.   

Your requirement does have things to do with more of user security on the mobile application side as each user will have to act only on their devices connected to the cloud. 

A Million devices or more is possible with a distributed set up. 

Good Luck. 

Cheers , 

Ranjith 

CEO , Bevywise Networks

www.bevywise.com

[Brahmaiah Doma]

Hi Ranjith,

  I would like to know how to scale the IOT topics to handle million devices at a time and managing security ?

Regards,

Brahmaiah

[Sumeet Puri]

MQTT is as good a protocol for scale as any other and has multiple advantages. It’s IP based, is very lightweight (say relative to HTTP) and has both queuing and topic (pub sub) constructs which make it ideal for IOT implementation. It’s popular too, which has it’s advantages. 

From a scale perspective, it will come down to the choice of the broker. 

You also need to think end to end. It’s not just getting devices connected. Can you address them at scale. For example, if a 10 million devices are connected, and you want John from NYC 6th Av’s water leak detector to be restarted, how do you ensure that the message only gets pushed from your .NET backend to that device? And how does this happen securely? Ideally, you just want to send a message to a topic (via .NET and probably using AMQP in the backend or even REST via micro services) and this should automatically get to the desired device, rather than maintaining a cache based device registry, having multiple bridges and brokers for front end/back end connectivity etc , which complicates things.

So given the above scenarios, at scale, you need: end to end device addressing, bidirectional messaging pipes (and not just capture one way), security, and deployment flexibly (cloud, on prem, edge). 

In my experience, you would need a tiered architecture for this. 

1. The connection layer - A wide and shallow connection layer, where each broker will do thousands/hundreds of thousands of connections, but at a relatively low throughout (say a few messages per second per broker). 

2. The core layer - Then you need a narrow and deep back end tier, which allows multiple back and apps to talk to the devices via topics. Queuing is key here, as back end apps have to cope with a firehose. Load balancing is also critical as you would have multiple instances of the applications. Multi protocol support help as well, as you might want to stream data into various insights systems going forward. 

3. The edge layer - when required, this is the layer for edge connectivity and aggregation. For example, 10s or 100s of sensors in a home or a factory floor can connect to a gateway, which then connects to the connection layer over MQTT. devices connecting to the GW could be over MQTT or other protocols - depending on what the device and sensor supports. You transform and standardise at the gateway layer to deal with legacy and innovation at the edge. 

There are different brokers to choose from. For example Solace (I work for Solace) provides multi protocol messaging/event streaming brokers which talk MQTT/AMQP/REST and more with .NET/Java APIs - paho and qpid are open source and popular for all languages including .NET. Edge nodes can scale upto 200K concurrent streaming connections. Core could be software or even hardware, which can scale upto 30 million messages per second. Solace has multiple production deployment with millions of streaming connections with the above architecture.  

You can read more details here: https://solace.com/internet-of-things

You might also find the QCon video usefu articulating the IOT at scale conceptsl: https://www.youtube.com/watch?v=cNKyGgtflLQ

There are other obviously providers as well, and all have their pros and cons. The good thing is that you have choice. 

Thanks and Regards, 

Sumeet. 

--
To learn more about MQTT please visit http://mqtt.org
---
You received this message because you are subscribed to the Google Groups "MQTT" group.

To unsubscribe from this group and stop receiving emails from it, send an email to mqtt+uns...@googlegroups.com.

[ran...@bevywise.com]

Hi Brahmaiah, 

Sorry.  I missed your reply. 

Every hardware has a limit. First we should understand what is the productive limit for our software in a hardware we wish to use for the production (say i5 with 4 GB RAM).  Based on that you should build your micro services and check productive limit for each of your software. 

As you have to go with the distributed model, you need to focus on fail over and self healing as well.    

Security should be thought into every process we do in every component. 

The list goes long and we can have a private communication on the same. 

Cheers, 

Ranjith 

Bevywise Networks Inc.

www.bevywise.com