[GSoC 20] OpenWISP Monitoring Module

[Hardik Jain]

11 May, 2020 - 15 May, 2020

PRs Merged :

• [monitoring] Allow to store IP addresses of interfaces
• [lua-monitoring] Find out ip address information of each interface
  
• [tests] Pushing up coverage
• [feature] Listen to DeviceConnection.is_working changes (to increase frequency of device recovery)

PRs Open for Review:

• [enhancement] Add disk usage to resources (lua-monitoring)
• [feature] Collect and display memory, CPU and flash usage of devices

Working next on issues:

• [enhancement] Implement Swappable models
• [enhancement] Optimize metric model storage

On a personal note, I learnt many new and exciting things so thanks all the mentors for reviewing my work, pointing to useful resources.

Hope, to keep learning and improving :)

Best Regards,

Hardik Jain

:)

[Hardik Jain]

18 May, 2020 - 22 May, 2020

PRs Merged :

• [docs] Add available features section #84
• [enhancement] Allow Threshold values to be float

PRs Open for Review: 

• [enhancement] Implement Swappable models

• [enhancement] Introduce the concept of metric configuration

• [enhancement] Add disk usage to resources (lua-monitoring)

• [feature] Collect and display memory, CPU and flash usage of devices

Blocked PRs:

• [docs] Document metrics stored in influxdb (blocked by #91, #76)
  

Working next on issues:

• [docs] Update README
• [enhancement] Abstract code communicating with the time series DB
• [enhancement] Add support for another timeseries DB

I will focus on these issues currently as other issues might might lead to conflicts with existing PRs.

[Hardik Jain]

PRs Merged :

• [feature] Swappable models

Open PRs:

• [enhancement] Add disk usage to resources
• [enhancement] Introduce the concept of metric configuration

• [docs] Document metrics stored in influxdb

• [feature] Collect and display memory, CPU and flash usage of device

• [enhancement] Abstract code communicating with the time series DB

Bugs to be dealt:

• [urgent] Checks stopped working

[Hardik Jain]

1 June, 2020 - 5 June, 2020

PRs Merged

• [enhancement] Add disk usage to resources (lua-monitoring)

PRs Open for Review:

• [enhancement] Show cpu, disk_usage and available_memory in device status
• [swappable] Further improvements
• Rename models Threshold, Graph to AlertThreshold, Chart

• [enhancement] Abstract code communicating with the time series DB

• [docs] Document metrics stored in influxdb
• [enhancement] Introduce the concept of metric configuration

• [feature] Collect and display memory, CPU and flash usage of devices

Working next on Issue:

• [charts] Add way to register and unregister new chart configurations

Best Regards

Hardik Jain

:)

[Hardik Jain]

8 June, 2020 - 12 June, 2020

PRs Merged:

• Rename models Threshold, Graph to AlertThreshold, Chart (deployed in staging env)

• [enhancement] Show cpu, disk_usage and available_memory in device status

• [feature] Collect and display memory, CPU and flash usage of devices

• [minor/chores] Correct disk usage data units (lua-monitoring)
  

PRs open for review:

• [charts] Add possibility to export histogram charts

• [docs] Document metrics stored in influxdb
• [enhancement] Abstract code communicating with the time series DB
• [enhancement] Introduce the concept of metric configuration

• [charts] Add way to register and unregister new chart configurations

• [bug] Fix alerts sent for new metrics within threshold and improvements to #76

Currently Working on:

• [feature] Add a check which inspects device configuration status periodically

Working next on issues:

• [bug] admin: Metric > Graphs: chart JS is broken
• Optimize unit test runtime speed

[Hardik Jain]

14 June, 2020 - 19, June, 2020

PRs Merged:

• [minor] Replace ugettext_lazy with gettext_lazy

PRs open for review:

• [charts] Add possibility to export histogram charts
  

• [enhancement] Introduce the concept of metric configuration

• [docs] Document metrics stored in influxdb

• [fix] Avoid log propagation when using root logger

• [charts] Add way to register and unregister new chart configurations

• [feature] Add a check which inspects device configuration status periodically

• [bug] Fix alerts sent for new metrics within threshold

• [tests] Optimize unit test runtime speed
• [test-utils] Added TimeLoggingTestRunner to detect slow tests (openwisp-utils)
  

Currently working on:

• [enhancement] Abstract code communicating with the time series DB

[Hardik Jain]

22 June, 2020 - 26 June, 2020

PRs Merged:

• [test-utils] Added TimeLoggingTestRunner to detect slow tests (openwisp-utils)
• [tests] Optimize unit test runtime speed

• [timeseries] Refactor order parameter to only allow 'time' / '-time'
• [fix] Fixed backward incompatible change in query configuration
• [enhancement] Abstract code communicating with the time series DB (in staging env)
  

• [bug] Fix alerts sent for new metrics within threshold

• [charts] Add possibility to export histogram charts

• [models] Use ContentType.objects.get_for_model

• [fix] Avoid log propagation when using root logger

PRs open for review:

• [feature] Add Monitoring Checks section in Device Admin

• [enhancement] Introduce the concept of metric configuration

• [docs] Document metrics stored in influxdb #81 #92
• [fix] Fixed broken chart js in metric admin change form
• [change] Added 'failure_reason' to is_working_changed signal (openwisp-controller)
  
• [monitoring] Do not send notifications on is_working changes when failure reasons were related to connectivity issues

• [charts] Add way to register and unregister new chart configurations
• [feature] Add a check which inspects device configuration status periodically

Currently working on:

• [metrics] Merge traffic metrics (will iterate after metric configuration gets merged as it might be helpful for validation ;) )

Working next on issues:

• [feature] Add an Alert Settings section in device admin

[Hardik Jain]

29 June, 2020 - 3 July, 2020

PRs Merged:

• [fix] Fixed broken TemplateAdmin (openwisp-controller)

• [change] Added 'failure_reason' to is_working_changed signal (openwisp-controller)

PRs open for review:

• [feature] Add Monitoring Checks section in Device Admin

• [check] Refactor Check classes #152

• [enhancement] Introduce the concept of metric configuration
• [docs] Document metrics stored in influxdb

• [fix] Fixed broken chart js in metric admin change form

• [monitoring] Do not send notifications on is_working changes when failure reasons were related to connectivity issues
• [charts] Add way to register and unregister new chart configurations
• [feature] Add a check which inspects device configuration status periodically

• [metrics] Merge traffic metrics (minor work remains)

Currently working on:

• [timeseries] Add initial support for second timeseries db (I have sent my findings and for the time am improving basic structure for both)
• [feature] Add a Check which inspects links and nodes & creates relevant charts for the same (awaiting response to query in this  comment)

Best regards

Hardik Jain

:)

[Federico Capoano]

I reviewed most PRs, merged what I could.

In the next week we should focus on two things:

1. get what is in the "in-progress" column of the board merged in the dev branch

2. have a call to discuss what is in the "next" column, if you have something ready to show regarding the timeseries DB we can take advantage so you can demo the prototypes by sharing your screen
    we can do this in the second half of the week, please make sure to have something to show, the charts for the ping metric for example.
    at the moment both me and Pablo are leaning towards Elasticsearch. I also have an actual real world installation where we could start using it very soon. 

Best regards

Federico Capoano

--
You received this message because you are subscribed to the Google Groups "OpenWISP" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openwisp+u...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/openwisp/b68223dd-2bfe-4608-84bd-cb9267285df3o%40googlegroups.com.

[Hardik Jain]

6 July, 2020 - 10 July, 2020

PRs Merged:

• [enhancement] Introduce the concept of metric configuration (dev)
  
• [monitoring] Do not send notifications on is_working changes when failure due to connectivity issues (dev)

• [fix] Fixed broken chart js in metric admin change form

• [check] Refactor Check classes
• [metrics] Merge traffic metrics (dev)
• [charts] Add way to register and unregister new chart configurations (dev)
• [metric-config] Add fill metric configuration migration (dev)
• [feature] Add Monitoring Checks section in Device Admin (dev)

PRs open for review:

• [feature] Add a check which inspects device configuration status periodically

• [fix] Create new models Chart, AlertSettings, fixes migrations #161
• [feature] Add an Alert Settings section in device admin (Blocked by above PR, migration issues)
  

• [docs] Document metrics stored in influxdb

Currently working on:

• [timeseries] Add initial support for second timeseries db

[Hardik Jain]

13 July, 2020 - 17 July, 2020

PRs open for review:

• [feature] Add Monitoring Checks section in Device Admin

• [fix] Create new models Chart, AlertSettings, fixes migrations

• [feature] Add an Alert Settings section in device admin (blocked by above PR)
• [docs] Document metrics stored in influxdb (images need to be updated)

Currently working on:

• [timeseries] Add initial support for elasticsearch

        Progress so far:

1. Index Lifecycle Management works perfectly and is much more efficient than Delete by Query as it's handled by elasticsearch and we don't need to run a task in the background. Retention can be done not only on the basis of time but also size of data (and of course their combo). We also get multiple options like shrinking shards, making index read-only, force merge and so on. I have tested it and it works great.
2. We also chose elasticsearch-dsl over elasticsearch-sql for querying due to current limitations of the latter (noting for future reference :) ).
   
3. The elasticsearch client that can be used in openwisp-monitoring is almost prepared what remains is top fields and improving queries.
4. Besides that some changes would be needed to avoid redundancies in code thus reducing the number of requests made, new tests and docs.

Working next on issue:

• [alert settings] Add way to specify global defaults

Best regards,

Hardik Jain

:)

[Federico Capoano]

Great report Hardik, informative and brief, thanks.

For the record on this list, since we didn't share our thought process here:

We definitely want to see Elasticsearch supported as a timeseries DB for openwisp-monitoring, as it should be more practical to use in high-availability setups in which OpenWISP needs to scale horizontally to support more load than influxdb.

At the moment influxdb does not have an open source implementation of HA but only a commercial one and we need to find a good open source alternative for the OpenWISP Community to use.

F.

--

You received this message because you are subscribed to the Google Groups "OpenWISP" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openwisp+u...@googlegroups.com.

To view this discussion on the web, visit https://groups.google.com/d/msgid/openwisp/0d7ecaa4-76c3-4530-9a9b-85b8226ffd36n%40googlegroups.com.

[Hardik Jain]

20 July, 2020 - 24 July, 2020

PRs open for review:

• [feature] Add Monitoring Checks section in Device Admin
• [fix] Create new models Chart, AlertSettings, fixes migrations
• [feature] Add an Alert Settings section in device admin (blocked by above PR)
• [docs] Document metrics stored in influxdb (images need to be updated)

Currently working on:

• [timeseries] Add initial support for elasticsearch

Besides tests the elasticsearch client is now fully ready with abstraction complete (optimizations needed).

* Aggregations have been used prominently to generate chart data.

* Support for top_fields has been added, had to modify existing structure to make it abstract enough.

* Tests on travis is currently taking around 80s (corresponding tests on influxdb take place in 25s, so improvement needed).

Optimizations we can do:

1) disable _source and _all as we are primarily using aggregations so might not actually need it. Benefits include lesser space being used thus lesser resources consumption as mentioned here (https://www.elastic.co/blog/elasticsearch-as-a-time-series-data-store)
2) Refresh indices after every write (by default index is refreshed every second, works well in practical case but not during tests.)
3) Don't delete indices between tests but rather only the data (might save time taken in creation of indexes, putting mappings and templates)

I did find few more optimizations but we might not be able to use it.

Maybe we can create a separate issue for this to keep things simple and easier to review :)

Also, as I put it up yesterday on IM, if mentors are free then we can have a meet sometime early next week, so I can explain the structure  better (Queries, Mappings, Settings, Aggregations, etc. used) and display the prototype (fully integrated now ;) )

[Hardik Jain]

27 July, 2020 - 31 July, 2020

PRs Merged:

• [feature] Add Monitoring Checks section in Device Admin (dev)
  

• [fix] Create new models Chart, AlertSettings, fixes migrations
  

• [feature] Add an Alert Settings section in device admin (dev)

• [docs] Document metrics stored in influxdb

• [fix] Avoid register_chart, unregister chart import while apps are being loaded (dev)

PRs Ready for review:

• [timeseries] Add initial support for elasticsearch (wifi_clients chart needs to be fixed)
• [admin] Improve device checks tab (will iterate in a short while)

Currently working on:

• [feature] Add a check which inspects device configuration status periodically

• [alert settings] Add way to specify global defaults

[Hardik Jain]

3rd August, 2020 - 7th August, 2020

PRs Merged:

• [feature] Add a check which inspects device configuration status periodically

• [admin] Improve device checks tab

PRs Ready for review:

• Merge dev branch in master

• [timeseries] Add initial support for elasticsearch

• [enhancement] Add way to specify global defaults

Currently working on:

• [feature] Metric collection resilient to failures

[Hardik Jain]

10th August, 2020 - 14th August, 2020

PRs Merged:

• [celery] Make tasks resilient to failures

• Merge dev branch in master

PRs Ready for Review:

• [timeseries] Make metric collection resilient to failures

• [enhancement] Add way to specify global defaults
• [timeseries] Add initial support for elasticsearch

Working next on issues:

• [ux] Help user to find out what's wrong
• [ux] Threshold value for resource metrics is weird

[Hardik Jain]

17th August, 2020 - 21st August, 2020

PRs Merged:

• [timeseries] Make metric collection resilient to failures
• [enhancement] Add way to specify global defaults

• [chores] Update docs for #133
• [chores] Improvements to PR #185

• [ux] Help user to find out what's wrong

PRs Ready for Review:

• [ux] Improve threshold values of resource metrics

• [timeseries] Add initial support for elasticsearch

Working next on issues:

• [docs] Update README
• [docs] Add to NetJSON implementations page

[Hardik Jain]

24th August, 2020 - 28th August, 2020

PRs Merged:

• [ux] Improve threshold values of resource metrics

• Issues/212 health checks
• [fix] Config applied checks will now work only on ok and problem health status
• [fix] Fixed create_ping and create_config migrations
• [docs] Update Readme
  

PRs Ready for Review:

• [fix] Fixed tolerance to be checked in minutes and not seconds
• [chores] Correct use of swappable in migrations and include migrations in qa-checks
• [fix] AlertSettings tolerance field should not have any default value
• Merge dev in master (there are currently two blockers before we merge it which have been mentioned in opening comment).

Next:

Google Summer of Code wraps up this week and it was a very great learning experience for me.

I think we added some great features to the monitoring module which should be helpful to multiple users.

Though few improvements as listed in the backlog column and resolving an issue with Elasticsearch (which will be a big add-on)

remained and I hope to work upon them and continue contributing to OpenWISP and open source in general.

Thanks a lot Federico, Pablo, Noumbissi, Ajay, Gagan and whoever helped me to improve knowing or unknowingly.

It wouldn't had been this great a journey without you all 😊.

Thanks & Regards

Hardik Jain

:)

[Hardik Jain]

Hi everyone,

I recently shared my experience of working this summer on OpenWISP Monitoring (as a project included in Google Summer of Code) in a medium blog.

Besides sharing my experience, the blog can act us a guide to all the cool features contained in openwisp-monitoring.

Regards,

Hardik Jain

:)