Re: [weewx-user] weewx customisations - heating_degrees, sunshine hours and THSW

[Thomas Keffer]

Hello, Alan, and welcome to our group!

At some level, all these projects can be done. Starting with the first

1. Calculating sunshine hours is best done by writing a custom service, rather than modifying VantagePro.py. The reason is exactly as you suspected: it is much easier to preserve your changes across upgrades. I'm not sure what the role is of your variable min_limit_Wm2, but I'm assuming the radiation must be greater than this value in order for the "sunshine hours" to be calculated. If so, your service would look something like this (NOT TESTED):

import syslog

import weewx

from weewx.wxengine import StdService

class AlansClass(StdService):

    

    def __init__(self, engine, config_dict):

        # Pass the initialization information on to my superclass:

        super(AlansClass, self).__init__(engine, config_dict)

        

        try:

            # Dig the needed options out of the configuration dictionary.

            # If a critical option is missing, an exception will be thrown and

            # the calculation will not be done:

            self.min_sunshine = config_dict['Alan'].get('min_sunshine', 2.0)

            

            # If we got this far, it's ok to start intercepting events:

            self.bind(weewx.NEW_ARCHIVE_RECORD, self.newArchiveRecord)

            

        except Exception, e:

            syslog.syslog(syslog.LOG_INFO, "Sunshine hours will not be calculated. Reason: %s" % e)

            

    def newArchiveRecord(self, event):

        """Gets called on a new archive record event."""

        

        radiation = event.record.get('radiation')

        if radiation is not None and radiation > self.min_sunshine:

            event.record['sunshine_hours'] = radiation * event.record['interval'] / 60.0

        else:

            event.record['sunshine_hours'] = 0.0

Put this in a file alan.py in the directory bin/user. You then add this new service to the list of services to be run. Put it at the beginning, so its generated values can be used by the other services:

service_list = user.alan.AlansClass, weewx.wxengine.StdConvert, weewx.wxengine.StdCalibrate, weewx.wxengine.StdQC, weewx.wxengine.StdArchive, weewx.wxengine.StdTimeSynch, weewx.wxengine.StdPrint, weewx.wxengine.StdRESTful, weewx.wxengine.StdReport

    

The final step is to add the new type to the database. If you are willing to throw away whatever data you already have, then this is easy: just add your new type to the end of the default schema. So, it would look like (look in bin/user/schemas.py):

defaultArchiveSchema = [('dateTime',             'INTEGER NOT NULL UNIQUE PRIMARY KEY'),

                        ('usUnits',              'INTEGER NOT NULL'),

                        ('interval',             'INTEGER NOT NULL'),

                        .

                        . (values elided)

                        .

                        ('rainBatteryStatus',    'REAL'),

                        ('outTempBatteryStatus', 'REAL'),

                        ('inTempBatteryStatus',  'REAL'),

                        ('sunshine_hours',       'REAL')]

Now delete your old archive and stats files (/home/weewx/archive/weewx.sdb and /home/weewx/archive/stats.sdb, respectively) and you're good to go.

There is just one small glitch: while the unit Watt/m2 exists in weewx, you've created a new one, Watt*hours/m2, which does not. The automatic formatting and labeling will not be done for you. The results will be reasonable, but if you want to refine them, that will require some work. We'll save that for a later lesson!

2. As for the UK definition of heating days, the easiest is to follow their simpler calculation: the sum of the number of degrees C above 22 degrees. This is super easy. Just go into the file skin/Standard/skin.conf and change the lines that look like

          # Base temperature for heating days, with unit:

          heating_base = 65, degree_F

          # Base temperature for cooling days, with unit:

          cooling_base = 65, degree_F

to:

          # Base temperature for heating days, with unit:

          heating_base = 22, degree_C

          # Base temperature for cooling days, with unit:

          cooling_base = 22, degree_C

If you want to do their more sophisticated 'weighted' calculation, then that's a lot more complicated and will require specialized SQL queries to the stats database, best done with a custom search list in a custom generator.

3. As for THSW, switching from LOOP to LPS commands to the VantagePro is a lot harder. You'd basically have to write your own device driver, recycling large parts from the existing driver. I don't mean to discourage you if you want to try your hand, but expect it to be a challenge. If you go this route, just plug in your driver name for option "driver" in the [Vantage] section, to get it to run instead of the standard driver.

An alternative approach is to find a formula you like and then calculate it in a custom service, much like we've done above with "sunshine hours." This would be a much simpler approach!

Let us know how it goes!

-tk

On Mon, Dec 24, 2012 at 9:32 AM, Alan Major <aland...@gmail.com> wrote:

Hi all

Well after researching the various alternative software packages I decided on weewx to go with my newly purchased Davis Vantage Pro 2 and have been trying hard over the last two weeks to make my way along a very steep learning curve. Linux, Python and the Raspberry Pi, I decided to run the software on, are all new to me. My only programming experience was writing some programs in BASIC around 30 years ago and since then dabbling with spreadsheets. However I have so far managed to get Debian Wheezy up and running on the Pi, installed weewx and config all the basics to get an experimental web site working at http://thecorner.me.uk/weewx (the wind speeds are slightly low until I can get a taller pole for the annometer).

Now that I can understand a little more about the capabilities of the software and the weather station I have three customisations I would like to make to suit my needs:

1. Add data for sunshine hours (my weather station includes a solar radiation monitor) for use in the templates and to produce plots
   

2. Calculate heating-degrees using the UKCPO9 formula so as to be compatible with UK sourced data

3. Make the THSW index shown on my console available for the plots

So far, having read the documentation and poked around in the various files, my thoughts on tackling these customisations are that:

1. sunshine hours would be best tackled by adding an extra value to the derived values section in Vantage Pro.py using a formula along the lines of (radiation > min_limit_Wm2) * archive_interval where min_limit_Wm2 is a value stored in the weewx.conf file
2. heating-degrees needs the formula in wxformulas.py changing and the calls to that formula from stats.py changing to pass the required variables
3. the THSW index is available in LOOP2 packet available from the console if the LPS command is used instead of LOOP so a change will be needed to the command issued to the console. the definitions of the vp2loop changing together with the format used to parse the data in VantagePro.py and that some how the value captured in the archive database
   
4. both the databases would need new types adding to store the new dataas
   

However what I have not been able to work out so far is how best to make these changes without editing the files directly. I have not been able to get my head around custom services and generators which if I understand are they key if I want to preserve as many of my changes as possible across future upgrades. Is this correct?

Would any of you care to comment on my thoughts at this conceptual level and provide some pointers the best way to tackle the customisations I would like to make to my installation?

Any help would be much appreciated. Thank you.

Alan Major
Birdham, UK

--
 
 

--

Tom Keffer

kef...@threefools.org
+1 541-386-8891 (h)
+1 541-490-9507 (c)

Skype: tkeffer

[Alan Major]

Tom

Thanks for your guidance and code. I will develop this further and implement once the holiday period is over.

Alan Major
Birdham, UK

[Alan Major]

Hi

Well after a much 'trial and effort' I have managed to incorporate data on sunshine hours into my database and display it on my web pages. In the end, after further research I decided to measure sunshine hours when actual solar radiation observed exceeded 40% of the potential solar radiation on a horizontal surface given by the formula:

solar constant * SIN (elevation angle of the sun)

using 1373 as my solar constant and cutting off when the elevation angle of the sun is less than 6 degrees to avoid issues with potential errors at low angles

To do this I have used the following code:

import syslog
import ephem
from math import sin
from datetime import datetime

import weewx
from weewx.wxengine import StdService

class sunHoursClass(StdService):
   
    def __init__(self, engine, config_dict):

        super(sunHoursClass, self).__init__(engine, config_dict)
   
        self.bind(weewx.NEW_ARCHIVE_RECORD, self.newArchiveRecord)

           
    def newArchiveRecord(self, event):
        """Gets called on a new archive record event."""

        # Step 1 - Calculate Sun Elevation Angle as gS in radians

        loc = ephem.Observer()
        loc.lon = '-0.835'
        loc.lat = '50.795'
        loc.pressure = 0
        loc.date = datetime.fromtimestamp(event.record.get('dateTime'))
        s = ephem.Sun()
        s.compute(loc)
        gS = s.alt

        # Step 2 - Calculate predicted solar radiation level as pR and record
       
        pR = 1373 * sin(gS) * 0.4
        if pR > 0:
            event.record['predRadiation'] = pR
        else:
            event.record['predRadiation'] = 0.0

        # Step - 3 Calculate if any sunshine hours and record

        radiation = event.record.get('radiation')

        if radiation is not None and gS > 0.104719755 and radiation > pR:
            event.record['sunshineHours'] = event.record['interval'] / 60.0
        else:
            event.record['sunshineHours'] = 0.0

Now to see how this measures up to reality - hopefully I have got all the coding correct, so it remains to be seen is whether this approach can produce a reason approximation of when the sun is shining! If the code is good and anyone else wants to use it they are very welcome.

As for the tracking of heating-degrees I have simply adjusted the base temperature in the weewx.conf file and will run with the data collected on that basis. When the time comes to compare with UKCP9 data all the base data is in the database to restate heat-degrees within whatever spreadsheet I use to undertake my comparison. So that one was easily sorted

This just leaves THSW. I have not been able to find any suitable formula to calculate this index - it seems that Davis guard this information well! I would really like to be able to include THSW on my current web page and it seems so frustrating that it can't easily be accessed.

Thanks Tom, for your pointers that have enabled me to get this far. It has been a very steep uphill climb ...

Alan Major
Birdham, UK

[Robin]

This looks interesting. I see from the dates that this precedes v3. I have two questions.

1. Are instructions for adding the service the same in v3?
   
2. Can I add the new type to the v3 database without destroying existing data (if so how)?

Thank you.

Robin

[Thomas Keffer]

Hello, Robin

Answering your questions

1. Adding a weewx service is the same in V3 as in older versions.

2. The V3 way of adding a new type is to put it in its own database. That way, the regular database, with its wview schema, remains untouched. See the sections Customizing the database and Using multiple bindings in the Customizing Guide. However, if you wish, you could add to the wview schema, but then you'd have to dump the database, change the schema, then reload it.

-tk

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Robin]

Tom,

Thank you for your help. I have the new service up and running and a I am now checking the collected data to check everything in working as I think it should.

One last question that my addled brain can't get around. How do I set 'sunshinehours' units to hours?

Robin

On Tuesday, 25 December 2012 01:59:52 UTC+2, Tom Keffer wrote:

[Thomas Keffer]

See the section Specify units for the new type in the Customizing Guide.

One thing to note: think of the observation type as "length of sunshine." It could be in any unit that measures time. So, if your database uses any of the weewx unit systems (US, METRIC, or METRICWX), then it would probably be in group_elapsed and should be stored in seconds.

The actual display unit is independent of how it is stored.

By default, when members of group_elapsed are displayed, such as $current.sunshine, the results will be in seconds. This can be overridden by adding an appropriate suffix, such as $current.sunshine.hour

-tk

--

[Robin]

I've been struggling to get this working properly. I have just realised it is probably a time issue.

Am I correct that weewx stored records in UTC? If so this would explain my issue perfectly.

Can anybody help with the simplest way to modify Alan's code so that it works on local time (for me that is Athens)?

Thank you.

Robin

On Tuesday, 25 December 2012 01:59:52 UTC+2, Tom Keffer wrote:

[gjr80]

Hi,

For what it is worth date/time fields in weewx are stored in Unix time ie a lapsed seconds since 00:00:00 UTC 1 January 1970. How they are displayed depends on how you format them. Unless you are doing something from left field dateTimes/time stamps should really look after themselves. What exactly is your problem? It would be helpful if you posted the code concerned as well as the symptoms you are seeing.

Gary

[Robin]

Code was originally posted by Alan Major. It uses the time to calculate the sun angle and then uses sin to do a rough maximum radiation value for that time of day. Compare this to current radiation and voilà you can see if the sun is shinning.

The code is 

import syslog
import ephem
from math import sin
from datetime import datetime

import weewx
from weewx.wxengine import StdService

class sunHoursClass(StdService):
   
    def __init__(self, engine, config_dict):

        super(sunHoursClass, self).__init__(engine, config_dict)
   
        self.bind(weewx.NEW_ARCHIVE_RECORD, self.newArchiveRecord)

           
    def newArchiveRecord(self, event):

        """Gets called on a new archive record event."""

        # Step 1 - Calculate Sun Elevation Angle as gS in radians

        loc = ephem.Observer()
        loc.lon = '-0.835'
        loc.lat = '50.795'
        loc.pressure = 0
        loc.date = datetime.fromtimestamp(event.record.get('dateTime'))
        s = ephem.Sun()
        s.compute(loc)
        gS = s.alt

        # Step 2 - Calculate predicted solar radiation level as pR and record
       
        pR = 1373 * sin(gS) * 0.4
        if pR > 0:
            event.record['predRadiation'] = pR
        else:
            event.record['predRadiation'] = 0.0

        # Step - 3 Calculate if any sunshine hours and record

        radiation = event.record.get('radiation')

        if radiation is not None and gS > 0.104719755 and radiation > pR:
            event.record['sunshineHours'] = event.record['interval'] / 60.0
        else:
            event.record['sunshineHours'] = 0.0

I have used my lat/lon but the code does not give the same values as a manual calculation. The only thing I could come up with was time.

Regards

Robin

On Tuesday, 25 December 2012 01:59:52 UTC+2, Tom Keffer wrote:

[mwall]

On Monday, June 29, 2015 at 8:54:05 AM UTC-4, Robin wrote:

Code was originally posted by Alan Major. It uses the time to calculate the sun angle and then uses sin to do a rough maximum radiation value for that time of day. Compare this to current radiation and voilà you can see if the sun is shinning.

robin,

the theoretical maximum solar radiation is calculated by the StdWXCalculate service starting in weewx 3.2.  look at the maxsolarrad field in every LOOP packet and archive record.  it uses the ryan-stolzenbach algorithm.  the bras algorithm is also included, by by default not enabled.  weewx 3.2 also adds windrun, evapotranspiration, humidex, apparent temperature, beaufort, and cloudbase.

m
 

[Robin]

Thank you Matthew.

When will 3.2 be released?

Robin

On Tuesday, 25 December 2012 01:59:52 UTC+2, Tom Keffer wrote:

[Thomas Keffer]

Real Soon Now. Coding is done, documentation is done. We are just doing the final testing. If you want to try it (and we can use all the testers we can get), you can download it from GitHub https://github.com/weewx/weewx/archive/master.zip

-tk

--

[gjr80]

Notwithstanding Matthew's post the issue with the Sun's altitude calculation is indeed one of time zones. Pyephem expects UTC but the datetime.fromtimestamp will be providing a time using your systems timezone. Change .fromtimestamp to .utcfromtimestamp and it shoudl work. I expect the original code worked OK as the author was in the UK and local time was probably the same as or very close to UTC.

Gary

...