Extending weewx - collecting processor and OS statistics and graphing the data

[William Phelps]

I am running weewx on a Raspberry Pi, and geek that I am, I wanted to collect some performance data. I read the doc and decided to add an extension to do this. Others may find this useful so I'm posting it here.

I am currently collecting the following (field name, contents):

 "cpuUsagePercent" - OS Usage Percent - sum of all states except idle

 "cpuWaitPercent" - OS Wait Percent - time spent in wait

 "cpuLoadFactor" - OS Load Factor

 "cpuTemperature" - R-Pi Processor temperature

 "logErrors" - Log file errors

 "logOPErrors" - Log file Operational errors

 "logFTPErrors" - Log file FTP errors

 "logRFErrors" - Log file Restful errors

With the exception of processor temperature (specific to the Raspberry Pi) the code should work on any hardware.

The OS usage data comes from /proc/stat - see http://www.linuxhowtos.org/System/procstat.htm for an explanation.

To install this extension:

1) shutdown weewx

2) decide which fields you want to record, and add them to the schemas.py file, following the instructions in the "customizing weewx" documentation, section 5.1, "Adding a new observation type"

3) copy "cpustats.py" and "procstat.py" to your weewx bin/user folder and modify as needed

4) edit your "weewx.conf" file and add "user.cpustats.CpuStats" to the front of the list of services - unless you've added other services, "service_list" should look like this

        service_list = user.cpustats.CpuStats, weewx.wxengine.StdConvert, weewx.wxengine.StdCalibrate, weewx.wxengine.StdQC, weewx.wxengine.StdArchive, weewx.wxengine.StdTimeSynch, weewx.wxengine.StdPrint, weewx.wxengine.StdRESTful, weewx.wxengine.StdReport

5) restart weewx

Once you've got it running and the data is being collected, the next step is to add graphs in your skin.conf file. I can post examples of how to do this if people want.

You can see actual live data at http://www.meier-phelps.com/weather and http://www.meier-phelps.com/OhloneFarm .

I have found this to be very helpful in monitoring and tuning these 2 systems. In particular the IO Wait percent is a good indicator of SD card health on the R-Pi.

William

[Oz Greg]

That is great work..  I am also considering writing some kind of monitoring function that runs via cron to verify that I have received data in the last five minutes and if not push a message to my phone (via the android intent) to alert me..

[vds]

On Tuesday, May 7, 2013 10:13:25 AM UTC-7, William Phelps wrote:

Once you've got it running and the data is being collected, the next step is to add graphs in your skin.conf file. I can post examples of how to do this if people want.

ummmm yes - knowing what to add to the templates would be helpful, as would knowing how to check from the commandline that it's doing something.

couple other things I found so far:

 - cpu temperature is Pi-centric.  A variable or something to make it optional would be helpful (I commented it out)

 - I think you had at least one tabs-vs-spaces issue in the cpustats.py file. Looked wrong 

[William Phelps]

Greg,

I've been thinking along the same lines, but leaning more toward an external service for the monitoring. I use SiteUpTime.com to monitor a couple of club web sites and was thinking of trying to use it to monitor my weather stations as well.

William

[Oz Greg]

As I have a QNAP which also acts as mysql server (thus reducing the requirements for lot of writes on my PI SD Card).. I will write a service on the qnap (most likely in php) and send a message via pushover if the archive DB is over x minutes old...  I already got some code in place to do this but I not extended it yet to handle my pi box.... (I ready have this code on my external website and look at the touch timestamps of the clientraw files and send a message if they are not being updated so it is not like I am going to be inventing the wheel.)

The trouble with siteuptime is I externally host my website so this service would only work out if my external host is up or down not if the information is being updated or not which does happen..

[William Phelps]

Now that you've got "cpustats" installed and running, how do you add the graphs?

1) add new labels to the "Generic" sub-section of the "Labels" section of "skin.conf":

[Labels]

    (...)

    [[Generic]]

        (...)

        windvec        = Wind Vector

        cpuUsagePercent = CPU Usage

        cpuWaitPercent = IO Wait

        cpuLoadFactor  = CPU Load Factor

        cpuTemperature = CPU Temperature

        logErrors      = Log Errors

        logIOErrors    = IO Errors

        logFTPErrors   = FTP Errors

        logRFErrors    = RF Errors

2) add images to the "ImageGenerator" section of the same file (skin.conf):

There are sections for "day_images", "week_images", etc. Add graphs as you see fit. 

Here's what I used for my "day_images" section:    

[[day_images]]

        x_label_format = %H:%M

        bottom_label_format = %m/%d/%y %H:%M

        time_length = 86400    # == 27 hours

#        time_length = 75600    # == 24 hours

        (...)

        [[[dayradiation]]]

            [[[[radiation]]]]

        [[[daycpupercent]]]

            chart_line_colors = 0xce02020, 0x2020e0, 0x20a020

            y_label = %

            aggregate_type = avg

            aggregate_interval = 300  # 5 minute average

            [[[[cpuUsagePercent]]]]

            [[[[cpuWaitPercent]]]]

        [[[daycpuload]]]

            [[[[cpuLoadFactor]]]]

                aggregate_type = avg

                aggregate_interval = 300  # 5 minute average

        [[[daycputemperature]]]

            y_label =   °C

            chart_line_colors = 0x2020c0, 0x20a020, 0xc02020

            yscale = 20, 50, 5

            [[[[cpuTemperature]]]]

                aggregate_type = avg

                aggregate_interval = 300  # 5 minute average

        [[[daylogerrors]]]

            chart_line_colors = 0x2020c0, 0x20a020, 0xc02020, 0xc0c020

#            plot_type = bar (broken, don't use)

            [[[[logErrors]]]]

            [[[[logIOErrors]]]]

            [[[[logFTPErrors]]]]

            [[[[logRFErrors]]]]

and in the "week_images" section I added

    [[week_images]]

        x_label_format = %d

        bottom_label_format = %m/%d/%y %H:%M

#        time_length = 604800    # == 8 days

        time_length = 518400    # == 7 days

        aggregate_type = avg

        aggregate_interval = 3600

        image_width = 600

        image_height = 200

        (...)

        [[[weekradiation]]]

            [[[[radiation]]]]

        [[[weekcpupercent]]]

            [[[[cpuUsagePercent]]]]

        [[[weekcpuload]]]

            [[[[cpuLoadFactor]]]]

        [[[weekcputemperature]]]

            [[[[cpuTemperature]]]]

        [[[weeklogerrors]]]

            chart_line_colors = 0x2020c0, 0x20a020, 0xc02020, 0xc0c020

            [[[[logErrors]]]]

            [[[[logIOErrors]]]]

            [[[[logFTPErrors]]]]

            [[[[logRFErrors]]]]

I added similar entries to the "month_images" and "year_images" - decide for yourself what plots you want.  Note that I used different widths for each one to make the longer period graphs easier to read.

3) Finally, edit the html templates to add the generated graphs. There are four of these, one each for "current", "week" (last 7 or 8 days), "month" (last 30 days), and "year" (last 365 days).  (the labels say "Calendar" but that's not really true)

You could add the new data to the "Current Conditions" and "Since Midnight" sections but I chose not to do that, I just added the graphs.

Changes to "index.html.tmpl" - look for "<div id="plots">" and add the new images where you want them:

        <div id="plots">

        (...)

          <img src="daycpupercent.png"    alt="CPU Percent" />

          <img src="daycpuload.png"    alt="CPU Load Factor" />

          <img src="daycputemperature.png"    alt="CPU Temperature" />

          <img src="daylogerrors.png"    alt="LOG Errors" />

        (...)

        </div> <!-- End id "plots" -->

You can add similar entries to the other templates as you see fit, for example, in week.html.tmpl or month.html.tmpl...

William

[Derek Sedunary]

William

Thanks for you excellent work.

I am running Weewx on a headless HP thin client I bought for $20 on eBay and have been looking for a way to monitor it and your Weewx extension has added the functions I needed.

The CPU usage graph is possibly showing a problem that I certainly would not have seen otherwise. The CPU usage is steadily increasing. I am not sure if it is a problem just with the graph it it is an actual problem with the CPU. Rebooting the computer does not reset the CPU usage which, to me, suggests it may be a problem with how the value displayed is calculate.

Has anyone else seen a similar plot for CPU usage?

Derek...

[William Phelps]

Derek,

Glad you like it & that it is helping. That's why I wrote it, to be able to monitor my RPi's for problems.

the plot you posted shows a very small increase over time - wondering if this might be due to normal o/s overhead.

William

[Derek Sedunary]

William

I hope you can help me. I am trying to add cpu temperatures to my plots but I cannot use your code as I gave up using a Raspberry Pi a few months ago and moved to a HP T5530 thin client that I have substituted a 20 gig laptop drive for the 64 meg flash drive the thin client op ran on.

I found lm-sensor would display my cpu temp and adding PySensors would allow me to display the cpu temp on the command line.

This short program:
import sensors

sensors.init()
try:
      for chip in sensors.iter_detected_chips():
         for feature in chip:
          if (feature.label) == ("Core 0"):
           cpu_temp = (feature.get_value())
           print cpu_temp
finally:
    sensors.cleanup()

would print 46.0 on the command line.

I tried substituting this into your cpustats.py replacing the raspberry pi section:

# get RPi processor temperature
#         try:
#             cmd = "sensor -u"
#             p = Popen(cmd, shell=True, stdout=PIPE)
#             o = p.communicate()[0]
#             cpu_temp = float(o.replace("'C\n", '').partition('=')[2])
#         except (IOError, KeyError):
#             cpu_temp = 0.0

#import sensors  <- Moved to the decelerations at the top of cpustats.py

sensors.init()
        try:
             for chip in sensors.iter_detected_chips():
                for feature in chip:
                  if (feature.label) == ("Core 0"):
                   cpu_temp = (feature.get_value())
        finally:
             sensors.cleanup()

Unluckily after stopping and restarting weewx I get the following errors:

Oct  9 17:50:48 debian weewx[11290]: wxengine: Caught unrecoverable exception in wxengine:
Oct  9 17:50:48 debian weewx[11290]:     ****  unexpected indent (cpustats.py, line 95)
Oct  9 17:50:48 debian weewx[11290]:     ****  Traceback (most recent call last):
Oct  9 17:50:48 debian weewx[11290]:     ****    File "/home/weewx/bin/weewx/wxengine.py", line 878, in main
Oct  9 17:50:48 debian weewx[11290]:     ****      engine = EngineClass(config_dict)
Oct  9 17:50:48 debian weewx[11290]:     ****    File "/home/weewx/bin/weewx/wxengine.py", line 70, in __init__
Oct  9 17:50:48 debian weewx[11290]:     ****      self.loadServices(config_dict)
Oct  9 17:50:48 debian weewx[11290]:     ****    File "/home/weewx/bin/weewx/wxengine.py", line 124, in loadServices
Oct  9 17:50:48 debian weewx[11290]:     ****      self.service_obj.append(weeutil.weeutil._get_object(svc)(self, config_dict))
Oct  9 17:50:48 debian weewx[11290]:     ****    File "/home/weewx/bin/weeutil/weeutil.py", line 716, in _get_object
Oct  9 17:50:48 debian weewx[11290]:     ****      mod = __import__(module)
Oct  9 17:50:48 debian weewx[11290]:     ****    File "/home/weewx/bin/user/cpustats.py", line 95
Oct  9 17:50:48 debian weewx[11290]:     ****      try:
Oct  9 17:50:48 debian weewx[11290]:     ****     ^
Oct  9 17:50:48 debian weewx[11290]:     ****  IndentationError: unexpected indent
Oct  9 17:50:48 debian weewx[11290]:     ****  Exiting.

The IndentationError: unexpected indent has got me stumped. I have match the indent of "try" to the indent of "try" in the section above. I am way out of my with this and had supprised myself in get my little python app to display just the cpu temp and not all the other details pysensor was displaying.

Can you give me any clues for a way forward?

Derek...

[Derek Sedunary]

The problem was the line above the error.

My addition should have been:

#import sensors  <- Moved to the decelerations at the top of cpustats.py

        sensors.init() #<----- this line needs 8 white space characters before the "s"

        try:
             for chip in sensors.iter_detected_chips():
                for feature in chip:
                  if (feature.label) == ("Core 0"):
                   cpu_temp = (feature.get_value())
        finally:
             sensors.cleanup(

)

And now it works.

William, thanks for your work on cpuststs.py, it is a very useful addition to weewx.

Derek...

[vigilancewx]

I have followed Matthew Wall’s cpu monitoring tutorial on Weewx Wiki and succeeded in getting the extra graphs running.

A nice well structured tutorial

What I would like to achieve are William Phelps  graphs but I am having some problems

https://groups.google.com/forum/#!searchin/weewx-user/cpu$20temp/weewx-user/sk7kenddddQ/83oWjA1CtpUJ

 

 

(decide which fields you want to record, and add them to the schemas.py file, following the instructions in the "customizing weewx" documentation, section 5.1, "Adding a new observation type")

What sections of schemas.py do I need to change?

 

 

4) edit your "weewx.conf" file and add "user.cpustats.CpuStats" to the front of the list of services - unless you've added other services, "service_list" should look like this……..

Is it only the service list I need to alter?

 

The remaining mods seem similar to additional graphs I created using the wiki tutorial

 

Thanks for your help

[Meteo Oberwallis]

Hello.

I know it's been a while since this thread, but it's still interesting to me. Hence my question. Does this still work with the current weewx version? I would like to expand this, but I keep getting an error message. What I also don't quite understand is what "decide which fields you want to record, and add them to the schemas.py file, following the instructions in the "customizing weewx" documentation, section 5.1, "Adding a new observation type" means. Can someone please explain to me what exactly I should do to make it work?

Thanks and Greetings

[vince]

Given this is really ancient dead code, I'd suggest you look into using https://github.com/weewx/weewx/wiki/cmon which is in the weewx core repos.

[Meteo Oberwallis]

Hello.

Thank you. i have this Problem:

pi@Wetterstation:~ $ sudo wee_reports
Using configuration file /etc/weewx/weewx.conf
Generating for all time

Traceback (most recent call last):

  File "/usr/share/weewx/weewx/reportengine.py", line 197, in run
    obj.start()
  File "/usr/share/weewx/weewx/reportengine.py", line 378, in start
    self.run()
  File "/usr/share/weewx/weewx/imagegenerator.py", line 42, in run
    self.gen_images(self.gen_ts)
  File "/usr/share/weewx/weewx/imagegenerator.py", line 107, in gen_images
    plot = self.gen_plot(plotgen_ts,
  File "/usr/share/weewx/weewx/imagegenerator.py", line 270, in gen_plot
    new_data_vec_t = self.converter.convert(data_vec_t)
  File "/usr/share/weewx/weewx/units.py", line 872, in convert
    new_unit_type = self.group_unit_dict.get(val_t[2], USUnits[val_t[2]])
  File "/usr/lib/python3.9/collections/__init__.py", line 941, in __getitem__
    return self.__missing__(key)            # support subclasses that define __missing__
  File "/usr/lib/python3.9/collections/__init__.py", line 933, in __missing__
    raise KeyError(key)
KeyError: 'group_cpu_temp'

Can you Help me?

Thx

[Meteo Oberwallis]

Good Morning.

Is it possible to adjust the sizes? As you can see in the picture, the resolution is displayed much too large. For example, if you could display the kB in MB? You can see in my graphic that there is no space for the axis to display everything.

Many thanks for your help

Greetings

Stefan