Clarification on Adding a new observation type & custom service (I've got > 1 measurement)

[Messy Potamia]

I have read "create a custom service" and "add a new type to the database" and am ready to start but I'd like you to clarify for me a couple things. When you wrote the example using electricity from a meter as the example, did you intend for this to pertain to only one parameter? Reason I ask is because you use the word Type, of which there could be multiple instances of that type, i.e. kilowatthours used to date (the 'meter reading'), and possibly other parameters like supply voltage rms, etc.  So does the example here infer that only one parameter is being added? Then when you get to the Add a new type to our existing schema, are you adding a type which can be implemented in more than one field in the weewx.sdb, or, is it only to add a single parameter (i.e. 'field') and I have to do it again to add two separate fields (am I using the right term 'field'?) 

Let me explain what I need to do. I have my maxbotix distance sensor working with some python code to read the serial stream, do some simple calculations, and save the results to a file. I noticed Matt has a package on Github for this however I cannot use that method, I need to take about 20 seconds worth of rapidfire measurements into a buffer, sort the buffer and average it so I can get Average and Difference which will be used to obtain Lake Height Elevation (known precise elevation of sensor minus distance, then converted to decimal feet), and to obtain Wave height (simply the max minus min of that reading sequence), theoretically dealing with wind and boat waves on the lake.  All this is done independent of weewx so it doesn't slow weewx down.

In the electricity example it states where "(code that downloads the consumption data from the connection to the meter goes here)" the execution of that code should take no more than a couple seconds max. So I run my mb.py code via cron every xx minutes and it writes out a 2-parameter file mbdata.dat.  At this point in the example:

def new_archive_packet(self, event):

        (code that downloads the consumption data from the connection to the meter)

        event.record['electricity'] = retrieved_value

I will have simple python code that reads those two values from mbdata.dat and puts them into two variables for later use in weewx, i.e. writing it in the archive and the reports, and that open/read/close should be very quick.

However here is where I am not clear. I'll have two measurements, LakeElevation (decimal feet) and WaveHeight (decimal inches).  

In the next steps in what is shown in the Customization guide, I have to create two new fields (why are they called 'types'?) in the schema? I want to do this as smoothly as possible, and haven't done it before. 

Hopefully the rest should go smoothly but first I want to make sure I'm getting those two added to the schema and archived properly.

Thank you for clarifying. 

My code for my maxbotix read of NUM_READS from ttyAMA0 isn't pretty but it works quite well and a minimum amount of time is wasted between each read so the sensor's output is as continuous as possible. If anyone needs it I'll share.

Phil

[Messy Potamia]

After some thought (i.e. staring at, and re-reading many times) about the instructions and what I need to do, here is the tentative "lakedata.py" I came up with, all based on the example in Customization guide. Please tell me if this is correct:

#!/usr/bin/env python

#file user/lakedata.py

import weewx

from weewx.engine import StdService

        

class AddlakeElevation(StdService):

    def __init__(self, engine, config_dict):

      # Initialize my superclass first:

      super(AddlakeElevation, self).__init__(engine, config_dict)

      # Bind to any new archive record events:

      self.bind(weewx.NEW_ARCHIVE_RECORD, self.new_archive_packet)

    def new_archive_packet(self, event):

        #(code that reads one of the two measurements from a file)

        event.record['lakeElevation'] = retrieved_value

class AddlakeWaveheight(StdService):

    def __init__(self, engine, config_dict):

      # Initialize my superclass first:

      super(AddlakeWaveheight, self).__init__(engine, config_dict)

      # Bind to any new archive record events:

      self.bind(weewx.NEW_ARCHIVE_RECORD, self.new_archive_packet)

    def new_archive_packet(self, event):

        #(code that reads 2nd of the two measurements from a file)

        event.record['lakeWaveheight'] = retrieved_value

[Messy Potamia]

OR is this more correct? I just realized that when I add the service(s) in [Engine] [[Services]] process_services, one would prefer to add only one additional service, rather than a separate service for every measurement? Someone please tell me if this is correct:

#!/usr/bin/env python

#file user/lakedata.py

import weewx

from weewx.engine import StdService

        

class AddLakedata(StdService):

    def __init__(self, engine, config_dict):

      # Initialize my superclass first:

      super(AddLakedata, self).__init__(engine, config_dict)

      # Bind to any new archive record events:

      self.bind(weewx.NEW_ARCHIVE_RECORD, self.new_archive_packet)

    def new_archive_packet(self, event):

        #(code that reads two measurements from a file)

        event.record['lakeElevation'] = retrieved_value1

        event.record['lakeWaveheight'] = retrieved_value2

[Thomas Keffer]

This is exactly right. You want one service that handles all of your data.

If you add columns to the 'archive' table of your database and give them names 'lakeElevation' and 'lakeWaveheight', then the data will be saved in the database along with everything else.

Alternatively, you can set up a separate database, reserved exclusively for your lake data, and put the data in there. But, that's a more advanced method.

-tk

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Messy Potamia]

Okay but now that I'm ready to modify the schema,

​​

the 'electricity' example doesn't seem to explain where the full name comes from. Here's what I've got so far, my lakedata.py looks like

​​

this:

#file user/lakedata.py
import weewx
from weewx.engine import StdService
       
class AddLakedata(StdService):

    def __init__(self, engine, config_dict):
      # Initialize my superclass first:
      super(AddLakedata, self).__init__(engine, config_dict)
      # Bind to any new archive record events:
      self.bind(weewx.NEW_ARCHIVE_RECORD, self.new_archive_packet)

    def new_archive_packet(self, event):

        #(code that reads two measurements from a file)
       

        retrieved_value1 = 594.0  #dummy values for now
        retrieved_value2 = 1.0

        event.record['lakeElevation'] = retrieved_value1
        event.record['lakeWaveheight'] = retrieved_value2
       

import schemas.wview
schema_with_lakedata = schemas.wview.schema + [('lakeElevation', 'REAL')] + [('lakeWaveheight', 'REAL')]

​     

What do I put in
[DataBindings]
    [[wx_binding]]  

       .... schema = user.[what goes here, and where does it come from?].schema_with_lakedata                           

​

​             ...​

--
You received this message because you are subscribed to a topic in the Google Groups "weewx-user" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/weewx-user/ton1GlQ0oZE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to weewx-user+...@googlegroups.com.

[Thomas Keffer]

If the file is user/lakedata.py, then what you want is:

[DataBindings]

  [[wx_binding]]

    ...

    schema = user.lakedata.schema_with_lakedata

-tk

[Messy Potamia]

Ohhhh kay... that got it for me, thanks.

[Messy Potamia]

Tom, the files in user/*.py are "live" files in that if I want to
make a change to the code in there in how it processes data I simply
edit it, and then restart weewx, right?

[Thomas Keffer]

Right. A Python is not aware of changes in its sources unless you restart the program.

-tk

[Messy Potamia]

The next steps, Specify Units for the new type, I made this entry in
extensions.py and it didn't work, most likely because I don't
understand what the guide wanted me to do:
import weewx.units
#weewx.units.obs_group_dict['lakeElevation'] = 'group_lake' #WHERE
ARE THESE DEFINED? Caused generator to stop. So I commented them out.
#weewx.units.obs_group_dict['lakeWaveheight'] = 'group_lake' #IS THIS
CORRECT? OR NEEDS ITS OWN GROUP?

Then in skin.conf I added the following in each day_, week_, and
month_images section:
[[[daylake]]]
[[[[lakeElevation]]]]
label = Lake Elevation (feet MSL)
The example showed adding electricity in the month section, I did the
three; guide says then a .png will be generated. I stopped/restarted
(after commenting out the not-working entries in extensions.py), and
next cycle, no plots were created.
I'm wondering if the guide is leaving some things out that I'm not
picking up on.

[Messy Potamia]

Also the guide's example calls the example parameter 'electricity'
early in the example, but down next in the specify units & use the new
type it refers to 'electric'.
Something I'm not picking up, relating to my measurements
'lakeElevation' and 'lakeWaveheight'.

[Thomas Keffer]

Unit groups are just a convenient way of collecting together like measurement types, giving them all the same unit system. For example, all temperatures are in "group_temperature".

I don't see why lake elevation needs its own group. There's already the very similar group_altitude, which offers unit choices of "foot" or "meter". Lake wave height could probably use this same group (group_altitude). However, if that's not going to work, it's possible to add new groups, but it's not documented.

Then it's just a matter of following the instructions in the Customizing Guide. Go into user/extensions.py and add the following:

import weewx.units

weewx.units.obs_group_dict['lakeElevation'] = 'group_altitude'

weewx.units.obs_group_dict['lakeWaveheight'] = 'group_altitude'

Then your plots should work.

-tk

[Thomas Keffer]

I'm not seeing the reference to 'electric.' Can you cut and paste it?

-tk

[Messy Potamia]

Use the new type
Now you've added a new type. How do you use it?
Pretty much like any other type. For example, to do a plot of the month's electric consumption, totaled by day, add this section to the [[month_images]] section of skin.conf:

[[[monthelectric]]]
    [[[[electricity]]]]
        aggregate_type = sum
        aggregate_interval = 86400
        label = Electric consumption (daily total)
This will cause the generation of an image monthelectric.png, showing a plot of each day's consumption for the past month.

I don't see the correlation between the examples and 'monthelectric.png'.

[Messy Potamia]

OK I made that substitution you sent above into extensions.py.

Here's a summary of my changes to skin.conf. Not sure if it matters
but I need lake elevation to display as %.2f, not %.0f as the 'feet'
is listed, so I made a 'foot2', like you did with 'knot' and 'knot2'
.

My Additions to skin.conf:

[units][[groups]] added:

group_lakeelev = foot2

group_lakewave = inch2

[[StringFormats]]
added:
foot2 = %.2f

inch2 = %.1f


[[Labels]] added:
foot2 = " ft MSL"

inch2 = " in"
(same as inch but using my *2 labels for consistency)

Then in
[Labels]
[[Generic]] added:
lakeElevation = Lake Elev MSL

lakeWaveheight = Lake Waves


Then in
[ImageGenerator]
[[day_images]]

[[[daylake]]]
[[[[lakeElevation]]]]
label = Lake Elevation (feet MSL)

[[week_images]]

[[[weeklake]]]

[[[[lakeElevation]]]]
label = Lake Elevation (feet MSL)

[[month_images]]

[[[monthlake]]]

[[[[lakeElevation]]]]
label = Lake Elevation (feet MSL)

[Messy Potamia]

​I really need to have custom units, i.e. inch2, feet2, so I can show the results more clearly; right now wave height is coming as feet when it's really decimal inches (this is a lake with ripples, not swells...). Plus I've altered the 'foot' from %.0f to %.2f just to get lake elevation in decimal feet, and of course that makes my station elevation display as 610.00 feet. It's philosophically wrong to alter the baseline code of yours to make it fit something I've added, so I need to get the units right, I'm just not sure what I've done wrong. If you or anyone has a chance please give me a clue.
Thanks. Shutting a few things down here, tornado warnings just sounded.​

[Thomas Keffer]

Why not just override the formatting? 

$current.lakeElevation.format(".2f")

See the formatting section of the Customizing Guide.

For the wave height, you could just put it in group_rain. Then the height would be in inches, mm, or cm, depending on the unit system (US, METRICWX, or METRIC, respectively). There's nothing magical about the groups. Believe me, this is simpler than adding a new unit group.

-tk

[Messy Potamia]

wave height: yes I put it in group_rain first, but it gave me .2f and I wanted .1f.

Yes I should have overridden the formatting!  It's working the way I want it and will leave it as is for now. I just finished editing all the skins (skin.conf & *.html.tmpl), then figuring out where my typos were and reediting, etc. Learned a lot.

You probably don't want to know what I did to get it the way I wanted it... suffice it to say I'm now including /home/weewx/bin/weewx/units.py in my daily backup.

Know it's been said before but this is a great piece of work, I admire it, and thank you.

Thanks for corresponding!

Phil