collecting processor usage and load on a Raspberry Pi - interesting hack

[William Phelps]

I wanted to collect some data from the Raspberry Pi I'm using to run weewx, so I decided to tackle writing an extension and adding a couple of fields to the database. More on that in another thread

 I was interested in collecting CPU load and percentage (MCU?). I figured I could just call "top" and parse the results, not very difficult in Python. Most people doing this use (100 - idle) for CPU percent. I ran into something interesting though - every time I ran my test program I got the same numbers. Idle was always 31.2. This was a bit of a puzzle until it dawned on me that the processing necessary to set up the command must be putting a fixed load on the system. So how do you get around that? The solution is to run "top" for 2 iterations, with a small delay, then parse the results from the second pass. I don't know if this is common knowledge, but it sure felt like a bulb lighting up when I figured it out, so I thought I'd post it here.

The code looks like this:

        try:

            cmd = "top -bn2d.5" # two samples 1/2 second apart

            p = Popen(cmd, shell=True, stdout=PIPE)

            o = p.communicate()[0].split()

            i = 1+o[1:].index('top') # find second page

            o = o[i:] # delete first page

            cpu_load = float(o[o.index('average:')+1].rstrip(',')) # find & extract load1

            cpu_user = float(o[o.index('us,')-1]) # cpu usage in User

            cpu_sys = float(o[o.index('sy,')-1]) # cpu usage in System

            cpu_idle = float(o[o.index('id,')-1]) # cpu Idle

            cpu_percent = 100 - cpu_idle

        except (IOError, KeyError):

            cpu_load = 0.0

            cpu_percent = 0.0

William

[gjr80]

Hi William,

Funnily enough I was trying to put some load figures for my RPi onto my site today. I just put a couple of lines of code in one of my templates:

#import os
#set $systemload = os.getloadavg()

Just gives me a string with the 3 load figures in brackets. Afraid that parsing something like the results from 'top' is beyond me at present! Top does give a deal more info though.

regards,

Gary

[William Phelps]

Gary,

I did not know about the "os" module - thanks! That's a much simpler way to get the load averages. I also just found another way to get cpu usage, by simply reading /proc/stat - http://stackoverflow.com/questions/1296703/getting-system-status-in-python . Rather than call 'top', why not use the numbers from /proc/stat directly?

If you are interested in keeping track of the RPI's processor temperature, here's the code for that:

# get RPi processor temperature

        try:

            cmd = "/opt/vc/bin/vcgencmd measure_temp"

            p = Popen(cmd, shell=True, stdout=PIPE)

            o = p.communicate()[0]

            cpu_temp = re.sub(r'[^0-9.]', '', o.partition('=')[2])

        except (IOError, KeyError):

            cpu_temp = 0.0

I am now archiving the load average (1 minute only, the first one), the cpu percentage, and the temperature.

William

[gjr80]

William,
Yes I saw that link when I was googling today. I have some php code in my pages so am thinking of displaying the load info using php - there is a similar function in php - that way I can refresh the load figures with a page refresh. Doing it as I have at the moment makes it largely static info with updates every archive period. Handy info on the CPU temp, I am concerned that I might start overloading the RPi as I keep adding to it (hence the foray into load averages) and I expect CPU temp is just another indicator.

Gary

[William Phelps]

I decided to see if I could do this without using 'top'.  Wrote the following module "procstat.py"

# get processor times as list

def get():

    """

    Fetches a list of time units the cpu has spent in various modes

    Detailed explanation at http://www.linuxhowtos.org/System/procstat.htm

    """

    cpuStats = file("/proc/stat", "r").readline()

    columns = cpuStats.replace("cpu", "").split(" ")

    return map(int, filter(None, columns))

Now cpustats.py can use this to get the cpu percentage:

            cpulist2 = procstat.get() # get current processor stats

            dt = list((t2-t1) for t1, t2 in zip(self.cpulist1, cpulist2)) # delta since last call

            idle_time = float(dt[3])

            total_time = sum(dt)

            cpu_percent = ((total_time-idle_time)/total_time)*100

            self.cpulist1 = cpulist2

This is cool because there is no need for time.sleep - each call to the "cpustats.py" extension computes the usage for the interval since the previous call.

William

[William Phelps]

You can see the plots at http://www.meier-phelps.com/weather

[Thomas Keffer]

Nice feature. Looks like you've got that RPi working hard! :-)

-tk

On Tue, Apr 23, 2013 at 10:31 AM, William Phelps <wbph...@gmail.com> wrote:

You can see the plots at http://www.meier-phelps.com/weather

--
You received this message because you are subscribed to the Google Groups "Weewx user's group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--

Tom Keffer

kef...@threefools.org
+1 541-386-8891 (h)
+1 541-490-9507 (c)

Skype: tkeffer

[William Phelps]

A second R-Pi running weewx came online yesterday, serving the weather station on the school farm at Ohlone Elementary:

http://www.meier-phelps.com/OhloneWeather/index.html

This one is in the goat stable/chicken coop, so it gets a bit warmer, as you can see. The Davis Envoy is right next to the R-Pi. So far I've not done anything to help cool the Pi's processor, it will be interesting to see if that becomes a problem when we get to summer and it gets hotter.

William

[vds]

pretty cool....I'd be interested in seeing the whole set of scripts etc. that you cooked up for cpu and temperature.

I tried to use lm_sensors on my Dockstar a few months ago, not googling the fact that it causes a system lockup....and there went my 435 days of uptime.  Sigh.  Up 70 days after the fact at this point.

[William Phelps]

I will post full instructions with code soon. just added counts of errors in syslog

[Neil Trimboy]

Nice day/night colours. Can u post them also.

[William Phelps]

the day/night/edge colors:

 

    daynight_day_color   = 0xf0ffff

    daynight_night_color = 0xffe0e0

    daynight_edge_color  = 0xe0e0ff

[gjr80]

Hi William,

Did you ever get your processor load code finished? Interested in seeing what you came up with.

regards,

Gary

[William Phelps]

Yes, I did. You can see some charts on my web page, http://www.meier-phelps.com/weather

I will be posting the code in a new topic since it's applicable to more than just the R-Pi

William