Open-source multi-tenant monitoring system, Monasca

419 views
Skip to first unread message

Roland Hochmuth

unread,
Dec 16, 2016, 8:37:06 AM12/16/16
to MQTT
Hi, I recently attended at talk by Jan-Piet Mens at the Open Source Monitoring Conference (OSMC 2016) where I was also presenting. JP presented on several areas, such as MQTT and the ESP8266. It was a fascinating talk for me, as I've been wondering for sometime about the suitability of the open-source monitoring-as-a-service project that I work on, Monasca, and JP's talk got me thinking and tinkering a little bit about that again. 

Monasca is a open-source Monitoring-as-a-Service project that has it's root in OpenStack. You can think of it as offering features similar to AWS CloudWatch or DataDog. It is a push-based metrics system that supports authentication and multi-tenancy.


Prometheus would be another example of an application that provides similar capabilities. Although, Prometheus is focused on more on providing an operators tool only and doesn't support multi-tenancy. Monasca provides the ability to store and query metrics, evaluate functions on metrics, such as average, calculate thresholds, create alarms and send notifications. It also supports Grafana, the time-series dashboard. It is scalable, performant and very extensible via micro-services that consume and publish to Apache Kafka.


In terms of IoT, my thoughts are that there are a lot of devices, such as sensors, that are associated with an account or tenant. For example, a household or business might want to monitor their home or building, but they normally wouldn't want to share that information with others households/businesses. Monasca could be used for doing this.


I started to work on a mqtt subscriber to handle metrics sent from clients and publishing/sending them to Monasca. It seems relatively straight-forward, although there are details to work out, such as authentication and what the exact schema is. What Monasca would allow, assuming it is deployed in the cloud somewhere, is for a bunch of IOT devices running in separate households or businesses, to send data. The user would then be able to login to Monasca/Grafana, using the same account that the IoT data is being stored in, visualize time-series graphs, as well as create alarms/alerts and notifications. Other web portals can be "easily" developed on top of the Monasca API.


Monasca is completely open-source. There are several large companies involved in it's development within the OpenStack community, such as HPE, Cisco, Fujitsu, Charter Communications, SAP and others and Monasca is deployed in production environments.


If anyone has any thoughts on the applicability of Monasca in the IoT space as well as potential development of a mqtt broker for Monasca I would very much appreciate the feedback. For example, if someone said I don't see how this would be applicable, then I would very much like to get that feedback. On the other hand, if this seems like an interesting project with some potential to mqtt/IoT as I've described, that feedback would also be appreciated. I would also be happy discussing the project in detail if anyone is interested in learning more.


Regards --Roland

Ranjith Kumar Diraviyam

unread,
Dec 19, 2016, 10:31:38 AM12/19/16
to MQTT
Hi, 

We were doing some work trying to put the MQTT messages at the back end of our Broker into Kafka and then process it with the Spark streaming engine and I believe we would end up doing something similar. Thanks for the timely post on Monasca.  I had a quick look at the videos and also read a few documentation and I believe this would fit perfectly for the mqtt data analysis. 

I will write back  my complete feedback once we deploy Monasca in our local set up and do a test run. 

Cheers, 
Ranjith 
Bevywise Networks.

Roland Hochmuth

unread,
Dec 19, 2016, 4:11:18 PM12/19/16
to MQTT
Hi Ranjith, I look forward to the feedback. Probably the one more difficult areas of Monasca is the initial deployment as several technologies are involved.

If you are familiar with Vagrant we have a complete environment that can be deployed as simple as "vagrant up" at, https://github.com/openstack/monasca-api/tree/master/devstack. If you clone the monasca-api repo, cd .../monasca-api/devstack, then "vagrant up", you should get a completely working local deployment of Monasca in a Vagrant ubuntu xenial64 image.

In that same directory is the file "plugin.sh" which contains all the bash scripts for deploying Monasca. On the HPE team doing development, we primarily use 16GB MacBook's, so that Vagrant deployment is known to work there. There are other teams using Linux.

Another source of potential information is the Puppet Monasca modules at, https://github.com/openstack/puppet-monasca and the Ansible Monasca modules at, https://github.com/openstack/openstack-ansible-os_monasca.

We've also been working on Docker deployments of Monasca recently at, https://github.com/hpcloud-mon/monasca-docker as well as Kubernetes deployment at, https://github.com/hpcloud-mon/monasca-docker/tree/master/k8s. The Kubernetes deployment is known to work using, kube-cluster-osx at, https://github.com/TheNewNormal/kube-cluster-osx. If you are familiar with Docker and Kubernetes this might actually be a better way to get started than the Vagrant image listed up above.

There were also some blogs that are a little dated at this point, but could still prove to be useful at, https://blog.zhaw.ch/icclab/how-to-install-and-setup-monasca-1-3/.

What you were thinking about doing sounds along the lines of what I was thinking. 

Let me know if you run into problems and I can try and help out as well as answer any questions around Monasca.

Regards --Roland

ran...@bevywise.com

unread,
Mar 31, 2017, 2:49:43 AM3/31/17
to MQTT
Hi Ronald , 

I want to give you an update on what we have done.  We tried using the Monasca and a few more distributed data storage services available around. But most of them felt heavy and not in control. We decided to go on our own keep our broker inside the distributed architecture. We have done an initial version


The implementation currently runs on our own servers - http://mqttserver.com 

I believe we are reinventing the wheel. But our own implementation provides a feel of more control than Monasca and others. 

Best Regards,
Ranjith 
CEO, Bevywise Networks.
Reply all
Reply to author
Forward
0 new messages