event.record['ET'] throws me a KeyError. Why?

[Maarten van der Hoeven]

Hi group,

Having programming experience, but new to python. Want to write some piece of code, that calculates the amount of precipitation shortage (rain minus evaotranspiration). It's a measure of drought.

Using some example coding, and extended those.

See the piece of code below. event.record['barometer'] and event.record['outTemp'] is working (used as an example, no drought-relation ship); they give real values.

However, event.record['ET'] gives me a KeyError. I dont understand it. It is in my wview_extended schema (which I extended), its also populated in my database.

Why the KeyError?

import schemas.wview_extended

schema = {

    'table': schemas.wview_extended.table + [('neerslagtekort', 'REAL')],

    'day_summaries' : schemas.wview_extended.day_summaries + [('neerslagtekort', 'SCALAR')]

}

import weewx.units

weewx.units.obs_group_dict['neerslagtekort'] = 'group_rain'

neerslag = 0

verdamping = 0

neerslagtekort = 0

class AddElectricity(StdService):

    def __init__(self, engine, config_dict):

      # Initialize my superclass first:

      super(AddElectricity, self).__init__(engine, config_dict)

      # Bind to any new archive record events:

      self.bind(weewx.NEW_ARCHIVE_RECORD, self.new_archive_record)

      self.last_total = None

    def new_archive_record(self, event):

       barometer = event.record['barometer']

       outTemp = event.record['outTemp']

       ET = event.record['ET']

       shortage = precipitation - evaporation

       if self.last_total:

          event.record['neerslagtekort'] = shortage

       self.last_total = shortage

       print("**** NEERSLAGTEKORT **** ")

       print(shortage)

Traceback (most recent call last):

  File "/usr/share/weewx/weewxd", line 261, in <module>

    main()

  File "/usr/share/weewx/weewxd", line 154, in main

    engine.run()

  File "/usr/share/weewx/weewx/engine.py", line 202, in run

    self.dispatchEvent(weewx.Event(weewx.POST_LOOP))

  File "/usr/share/weewx/weewx/engine.py", line 224, in dispatchEvent

    callback(event)

  File "/usr/share/weewx/weewx/engine.py", line 596, in post_loop

    self._software_catchup()

  File "/usr/share/weewx/weewx/engine.py", line 656, in _software_catchup

    self.engine.dispatchEvent(weewx.Event(weewx.NEW_ARCHIVE_RECORD,

  File "/usr/share/weewx/weewx/engine.py", line 224, in dispatchEvent

    callback(event)

  File "/usr/share/weewx/user/electricity.py", line 64, in new_archive_record

    ET = event.record['ET']

KeyError: 'ET'

mysql> select from_unixtime(datetime), neerslagtekort,ET,rain from archive order by datetime desc limit 5;

+-------------------------+---------------------+--------------------------+------+

| from_unixtime(datetime) | neerslagtekort      | ET                       | rain |

+-------------------------+---------------------+--------------------------+------+

| 2020-05-26 23:26:00     | -1.5559202862588535 |  0.000005153290200925555 |    0 |

| 2020-05-26 23:25:00     |                NULL |  0.000005156477243511403 |    0 |

| 2020-05-26 23:24:00     |                  20 |  0.000005159745765396156 |    0 |

| 2020-05-26 23:23:00     |                  20 | 0.0000051629007161015734 |    0 |

| 2020-05-26 23:22:00     |                NULL |  0.000005165840624070325 |    0 |

+-------------------------+---------------------+--------------------------+------+

5 rows in set (0.00 sec)

[gjr80]

Hi,

Where does your service sit in the service order in relation to StdWXCalculate in weewx.conf under [Engine] [[Services]]? One possible reason is that if your station does not provide field ET (not many (any?) provide a numeric value in every archive record) and the StdWXCalculate service then calculate ET if it can. If your service that references ET is executed before StdWXCalculate then field ET will not (yet) exist in the archive record. In this case the solution is put your service after StdWXCalculate.

Gary

[Maarten van der Hoeven]

Hi,

thanks for the pointer. While coding and testing, I use the Simulator to generate loops and archive records. When I connect my real Vantage Pro to WeeWX, the error disappears. The station is now providing all the value, including ET. Simulator is not providing ET, thats for sure now.

When hooking up the Simulator again, I followed your suggestion, and moved my service completely to the bottom (to be sure), see below. Still no success, ET is still a KeyError when using the Simulator.

Is there a way to enhance the Simulator with the ET-field (so to speak), that you know of?

[Engine]

    

    [[Services]]

        # This section specifies the services that should be run. They are

        # grouped by type, and the order of services within each group

        # determines the order in which the services will be run.

        prep_services = weewx.engine.StdTimeSynch

        process_services = weewx.engine.StdConvert, weewx.engine.StdCalibrate, weewx.engine.StdQC, weewx.wxservices.StdWXCalculate

        archive_services = weewx.engine.StdArchive

        restful_services = weewx.restx.StdStationRegistry, weewx.restx.StdWunderground, weewx.restx.StdPWSweather, weewx.restx.StdCWOP, weewx.restx.StdWOW, weewx.restx.StdAWEKAS, user.mqtt.MQTT

        report_services = weewx.engine.StdPrint, weewx.engine.StdReport

        data_services = user.electricity.AddElectricity

Op woensdag 27 mei 2020 14:33:08 UTC+2 schreef gjr80:

[Maarten van der Hoeven]

Hmmm, maybe adding ET to Simulator is not such a bad idea... Just adding the ET-observation to this code:

        self.observations = {
            'outTemp'    : Observation(magnitude=20.0,  average= 50.0, period=24.0, phase_lag=14.0, start=start_ts),
            'inTemp'     : Observation(magnitude=5.0,   average= 68.0, period=24.0, phase_lag=12.0, start=start_ts),
            'barometer'  : Observation(magnitude=1.0,   average= 30.1, period=48.0, phase_lag= 0.0, start=start_ts),
            'pressure'   : Observation(magnitude=1.0,   average= 30.1, period=48.0, phase_lag= 0.0, start=start_ts),
            'windSpeed'  : Observation(magnitude=5.0,   average=  5.0, period=48.0, phase_lag=24.0, start=start_ts),
            'windDir'    : Observation(magnitude=180.0, average=180.0, period=48.0, phase_lag= 0.0, start=start_ts),
            'windGust'   : Observation(magnitude=6.0,   average=  6.0, period=48.0, phase_lag=24.0, start=start_ts),
            'windGustDir': Observation(magnitude=180.0, average=180.0, period=48.0, phase_lag= 0.0, start=start_ts),
            'outHumidity': Observation(magnitude=30.0,  average= 50.0, period=48.0, phase_lag= 0.0, start=start_ts),
            'inHumidity' : Observation(magnitude=10.0,  average= 20.0, period=24.0, phase_lag= 0.0, start=start_ts),
            'radiation'  : Solar(magnitude=1000, solar_start=6, solar_length=12),
            'UV'         : Solar(magnitude=14,   solar_start=6, solar_length=12),
            'rain'       : Rain(rain_start=0, rain_length=3, total_rain=0.2, loop_interval=self.loop_interval),
            'txBatteryStatus': BatteryStatus(),
            'windBatteryStatus': BatteryStatus(),
            'rainBatteryStatus': BatteryStatus(),
            'outTempBatteryStatus': BatteryStatus(),
            'inTempBatteryStatus': BatteryStatus(),
            'consBatteryVoltage': BatteryVoltage(),

Op dinsdag 26 mei 2020 23:38:26 UTC+2 schreef Maarten van der Hoeven:

[gjr80]

As you say the use of simulator/VP2 explains it, the simulator driver does not emit ET so it must rely on StdWXCalculate when using the vantage driver it does (in your case) emit ET so it is there for your service.

Regards the order I apologise, I mislead you, you do indeed need to change the order but not by changing the order of the service lines (it will have no effect by itself) but rather moving your service to another service line such that it is processed after StdWXCalculate but before the record is saved to archive by StdArchive, something like this should work (untested):

[Engine]

[[Services]]
# This section specifies the services that should be run. They are
# grouped by type, and the order of services within each group
# determines the order in which the services will be run.
prep_services = weewx.engine.StdTimeSynch

data_services = ,
process_services = weewx.engine.StdConvert, weewx.engine.StdCalibrate, weewx.engine.StdQC, weewx.wxservices.StdWXCalculate, user.electricity.AddElectricity

archive_services = weewx.engine.StdArchive
restful_services = weewx.restx.StdStationRegistry, weewx.restx.StdWunderground, weewx.restx.StdPWSweather, weewx.restx.StdCWOP, weewx.restx.StdWOW, weewx.restx.StdAWEKAS

report_services = weewx.engine.StdPrint, weewx.engine.StdReport

Equally you could add it to the archive_services line before StdArchive, but that is a little misleading in terms of what your service does.

[Tom Keffer]

1. The ordering of your services is still not quite right. The group "data_services" is run before "process_services." You want

data_services = ,

process_services = weewx.engine.StdConvert, weewx.engine.StdCalibrate, weewx.engine.StdQC, weewx.wxservices.StdWXCalculate, user.electricity.AddElectricity

2. In the interest of realism, the simulator occasionally emits a None value for a datum. This is what is causing your ET to be None. You need to get in the habit of checking for None. Something like:

ET = event.record.get('ET')

if ET is not None:

  event.record['neerslagtekort'] = ...

3. Incidentally, you really should change the names to something more meaningful than "electricity.py" and "AddElectricity".

-tk

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/a91ef05e-b076-47e4-ba5a-6d460eaeeb44%40googlegroups.com.

[Maarten van der Hoeven]

Yes, the simulator doesnt provide ET, whereas the Vantage driver does provide ET.

But, bingo, your suggestion did the trick. Now I've placed my service just after StdWxCalculate, and now there is an ET popping up

Thanks!

Op woensdag 27 mei 2020 22:26:49 UTC+2 schreef gjr80:

[Maarten van der Hoeven]

Thanks Tom! I've put suggestion 1 and 2 in place.

About 3, yeah, you are right. Picking up and using some working code doesnt mean it should remain called AddElectricity. CalcDrought should be more in place.

Again, thank you both!

Op woensdag 27 mei 2020 22:27:33 UTC+2 schreef Tom Keffer:

To unsubscribe from this group and stop receiving emails from it, send an email to weewx...@googlegroups.com.