Weewx crashing

[Thomas Carlin]

Hey Tom et al.,

I have an issue with Weewx, which has been running well until yesterday.  I am running it on Debian 6.3.0-18, fully up to date with a Vantage Pro 2.  Weewx was installed through apt-get, and is fully up to date.  I do have several modifications, including adation sensors that I wrote a small driver for, but they have not caused me any issues in the past, and in the event that a sensor is unavailable, fail gracefully.  I have not modified any 'original' scripts, but have added fields to the database, tags, the mentioned driver, and am using the cmon plugin.  I would be happy to provide more details if necessary.

The error in syslog is below.  Please note that this is the first occurrence of this (that I can find in my logs).

Nov 15 19:30:20 Ds-1 weewx[27964]: manager: Added record 2017-11-15 19:30:20 MST (1510799420) to database 'cmon.sdb'

Nov 15 19:30:20 Ds-1 weewx[27964]: manager: Added record 2017-11-15 19:30:20 MST (1510799420) to daily summary in 'cmon.sdb'

Nov 15 19:30:20 Ds-1 weewx[27964]: manager: Added record 2017-11-15 19:30:00 MST (1510799400) to database 'weewx.sdb'

Nov 15 19:30:20 Ds-1 weewx[27964]: engine: Main loop exiting. Shutting engine down.

Nov 15 19:30:20 Ds-1 weewx[27964]: engine: Shutting down StdReport thread

Nov 15 19:30:20 Ds-1 weewx[27964]: engine: StdReport thread has been terminated

Nov 15 19:30:21 Ds-1 weewx[27964]: engine: Caught unrecoverable exception in engine:

Nov 15 19:30:21 Ds-1 weewx[27964]:     ****  unsupported operand type(s) for +=: 'NoneType' and 'float'

Nov 15 19:30:21 Ds-1 weewx[27964]:     ****  Traceback (most recent call last):

Nov 15 19:30:21 Ds-1 weewx[27964]:     ****    File "/usr/share/weewx/weewx/engine.py", line 871, in main

Nov 15 19:30:21 Ds-1 weewx[27964]:     ****      engine.run()

Nov 15 19:30:21 Ds-1 weewx[27964]:     ****    File "/usr/share/weewx/weewx/engine.py", line 201, in run

Nov 15 19:30:21 Ds-1 weewx[27964]:     ****      self.dispatchEvent(weewx.Event(weewx.POST_LOOP))

Nov 15 19:30:21 Ds-1 weewx[27964]:     ****    File "/usr/share/weewx/weewx/engine.py", line 223, in dispatchEvent

Nov 15 19:30:21 Ds-1 weewx[27964]:     ****      callback(event)

Nov 15 19:30:21 Ds-1 weewx[27964]:     ****    File "/usr/share/weewx/weewx/engine.py", line 561, in post_loop

Nov 15 19:30:21 Ds-1 weewx[27964]:     ****      self._software_catchup()

Nov 15 19:30:21 Ds-1 weewx[27964]:     ****    File "/usr/share/weewx/weewx/engine.py", line 633, in _software_catchup

Nov 15 19:30:21 Ds-1 weewx[27964]:     ****      self.engine.dispatchEvent(weewx.Event(weewx.NEW_ARCHIVE_RECORD, record=record, origin='software'))

Nov 15 19:30:21 Ds-1 weewx[27964]:     ****    File "/usr/share/weewx/weewx/engine.py", line 223, in dispatchEvent

Nov 15 19:30:21 Ds-1 weewx[27964]:     ****      callback(event)

Nov 15 19:30:21 Ds-1 weewx[27964]:     ****    File "/usr/share/weewx/weewx/engine.py", line 588, in new_archive_record

Nov 15 19:30:21 Ds-1 weewx[27964]:     ****      dbmanager.addRecord(event.record, accumulator=self.old_accumulator)

Nov 15 19:30:21 Ds-1 weewx[27964]:     ****    File "/usr/share/weewx/weewx/manager.py", line 245, in addRecord

Nov 15 19:30:21 Ds-1 weewx[27964]:     ****      self._addSingleRecord(record, cursor, log_level)

Nov 15 19:30:21 Ds-1 weewx[27964]:     ****    File "/usr/share/weewx/weewx/manager.py", line 1183, in _addSingleRecord

Nov 15 19:30:21 Ds-1 weewx[27964]:     ****      _day_summary.addRecord(record, weight=_weight)

Nov 15 19:30:21 Ds-1 weewx[27964]:     ****    File "/usr/share/weewx/weewx/accum.py", line 242, in addRecord

Nov 15 19:30:21 Ds-1 weewx[27964]:     ****      func(self, record, obs_type, add_hilo, weight)

Nov 15 19:30:21 Ds-1 weewx[27964]:     ****    File "/usr/share/weewx/weewx/accum.py", line 301, in add_value

Nov 15 19:30:21 Ds-1 weewx[27964]:     ****      self[obs_type].addSum(val, weight=weight)

Nov 15 19:30:21 Ds-1 weewx[27964]:     ****    File "/usr/share/weewx/weewx/accum.py", line 88, in addSum

Nov 15 19:30:21 Ds-1 weewx[27964]:     ****      self.sum     += val

Nov 15 19:30:21 Ds-1 weewx[27964]:     ****  TypeError: unsupported operand type(s) for +=: 'NoneType' and 'float'

Nov 15 19:30:21 Ds-1 weewx[27964]:     ****  Exiting.

I do have a watchdog on the process that tries to re-launch weewx if it detects that it is not running.  Most recently it crashed at 11:00 A, and the watchdog tried to re-launch every minute until 19:30, when it started successfully, and the system threw the same error every time.  

Do you need any more information from me, and/or is there something else that I can do to help troubleshoot this issue?

Thank you in advance, and thank you for this fantastic software that you have written!

[Thomas Carlin]

Sorry, Debian 9.2, Stretch, fully updated.  I misread /proc/version

[gjr80]

Hi,

I suspect the issue will be to do with one of the fields you have added to the weeWX archive. It appears that the 'sum' field for one of the accumulators is being initialised to None instead of 0.0 and consequently when an observation value is added you get the error shown in your logs. I don't think the issue is with the driver you wrote. Might help if we could get a better picture of your setup. Can you please provide a sanitised copy of weewx.conf along with details of the schema you are using for the archive table in your weex database.

Gary

[Thomas Carlin]

Hey Gary,

The config file is below.  I didn't think about it yesterday, but I am using the history generator script (found here). I have modified the schema to include a few extra data sources, but if i understand correctly, I have not tried to add any accumulators.  What would that code like for the initialization you mention?  (Other than = None)?  Could I add a debug line somewhere to get a little more information out of the system to determine more precisely what accumulation is incorrect?  As a side note, since it started running again, it has run without hiccup for a day, with no changes at all.

Thanks again!

Datasources:

schema_with_esp = schemas.wview.schema + [('DoorOpen1', 'REAL'), ('DoorClosed1', 'REAL'), ('DoorOpen2', 'REAL'), ('DoorClosed2', 'REAL'), ('IrrigationFlow', 'REAL'), ('IrrigationPressure', 'REAL'),('esp1Signal', 'REAL'), ('esp2Signal', 'REAL'),  ('esp3Signal', 'REAL'), ('esp3LIPOVoltage', 'REAL'), ('esp3LIPOCurrent', 'REAL'), ('esp3SolarVoltage', 'REAL'), ('esp3SolarCurrent', 'REAL'), ('esp3OutputVoltage', 'REAL'), ('esp3OutputCurrent', 'REAL'), ('extraHumid3', 'REAL'), ('esp4Signal', 'REAL'), ('esp4LIPOVoltage', 'REAL')]

I also have added these to, and created some simple groups for these, following the documentation, which works as expected.  

Weewx.conf (Edited for brevity)

# WEEWX CONFIGURATION FILE

#

# Copyright (c) 2009-2015 Tom Keffer

# See the file LICENSE.txt for your rights.

##############################################################################

# This section is for general configuration information.

# Set to 1 for extra debug info, otherwise comment it out or set to zero

debug = 1

# Root directory of the weewx data file hierarchy for this station

WEEWX_ROOT = /

# How long to wait before timing out a socket (FTP, HTTP) connection

socket_timeout = 20

# Do not modify this. It is used when installing and updating weewx.

version = 3.7.1

##############################################################################

#   This section is for information about the additional ESP8266 Sensors that are in use.

[ESP8266]

    [[http://1.2.3.4]]

        mac = 5C:CF:7F:02:0C:17

        Temp = extraTemp1

        Humid = extraHumid1

        doorOpen = DoorOpen1

        doorClosed = DoorClosed1

Signal = esp1Signal

    

    [[http://1.2.3.4]]

        mac = 5C:CF:7F:07:5B:90

        Temp = extraTemp2

        Humid = extraHumid2

        doorOpen = DoorOpen2

        doorClosed = DoorClosed2

Signal = esp2Signal

#    [[http://1.2.3.4]]

#        mac = 30:AE:A4:30:93:54

#        Temp = extraTemp3

#        Humid = extraHumid3

# Signal = esp4Signal

# Voltage = esp4LIPOVoltage

#    [[http://1.2.3.4]]

#        mac = 5C:CF:7F:12:17:97

#        Flow = IrrigationFlow

#        Pressure = IrrigationPressure

# Signal = esp3Signal

# LIPO_Voltage = esp3LIPOVoltage

# LIPO_Current = esp3LIPOCurrent

# Solar_Voltage = esp3SolarVoltage

# Solar_Current = esp3SolarCurrent

# Output_Voltage = esp3OutputVoltage

# Output_Current = esp3OutputCurrent

##############################################################################

#   This section is for information about the station.

[Station]

    

    # Description of the station location

    location = y

    

    # Latitude and longitude in decimal degrees

    latitude = 

    longitude = 

    

    # Altitude of the station, with unit it is in. This is downloaded from

    # from the station if the hardware supports it.

    altitude = 1234, foot

    

    # Set to type of station hardware. There must be a corresponding stanza

    # in this file with a 'driver' parameter indicating the driver to be used.

    station_type = Vantage

    

    # If you have a website, you may specify an URL

    #station_url = http://www.example.com

    

    # The start of the rain year (1=January; 10=October, etc.). This is

    # downloaded from the station if the hardware supports it.

    rain_year_start = 1

    

    # Start of week (0=Monday, 6=Sunday)

    week_start = 6

##############################################################################

[Vantage]

    # This section is for the Davis Vantage series of weather stations.

    

    # Connection type: serial or ethernet 

    #  serial (the classic VantagePro)

    #  ethernet (the WeatherLinkIP)

    type = ethernet

    

    # If the connection type is serial, a port must be specified:

    #   Debian, Ubuntu, Redhat, Fedora, and SuSE:

    #     /dev/ttyUSB0 is a common USB port name

    #     /dev/ttyS0   is a common serial port name

    #   BSD:

    #     /dev/cuaU0   is a common serial port name

    port = /dev/ttyUSB0

    

    # If the connection type is ethernet, an IP Address/hostname is required:

    host = 1.2.3.4

    

    ######################################################

    # The rest of this section rarely needs any attention. 

    # You can safely leave it "as is."

    ######################################################

    

    # Serial baud rate (usually 19200)

    baudrate = 19200

    

    # TCP port (when using the WeatherLinkIP)

    tcp_port = 23

    

    # TCP send delay (when using the WeatherLinkIP):

    #tcp_send_delay = 0.5

    tcp_send_delay = 1

    

    # The id of your ISS station (usually 1). If you use a wind meter connected

    # to a anemometer transmitter kit, use its id

    iss_id = 1

    

    # How long to wait for a response from the station before giving up (in

    # seconds; must be greater than 2)

    timeout = 4

    

    # How long to wait before trying again (in seconds)

    wait_before_retry = 1.2

    

    # How many times to try before giving up:

    max_tries = 4

    

    # The driver to use:

    driver = weewx.drivers.vantage

##############################################################################

#   This section is for uploading data to Internet sites

##NOTHING HERE ##

##############################################################################

#   This section specifies what reports, using which skins, to generate.

[StdReport]

    

    # Where the skins reside, relative to WEEWX_ROOT

    SKIN_ROOT = /etc/weewx/skins

    

    # Where the generated reports should go, relative to WEEWX_ROOT

    HTML_ROOT = /var/www/html/weewx

    

    # The database binding indicates which data should be used in reports.

    data_binding = wx_binding

    

    # Each of the following subsections defines a report that will be run.

    

    [[StandardReport]]

        # See the customizing guide to change the units, plot types and line

        # colors, modify the fonts, display additional sensor data, and other

        # customizations. Many of those changes can be made here by overriding

        # parameters, or by modifying templates within the skin itself.

        

        # The StandardReport uses the 'Standard' skin, which contains the

        # images, templates and plots for the report.

        skin = Sofaskin

        [[[Units]]]

            [[[[Groups]]]]

                group_altitude = foot

                group_speed2 = mile_per_hour2

                group_pressure = inHg

                group_rain = inch

                group_rainrate = inch_per_hour

                group_temperature = degree_F

                group_degree_day = degree_F_day

                group_speed = mile_per_hour

    #[[cmon]]

    #    HTML_ROOT = /var/www/html/weewx/cmon

    #    skin = cmon

        

    [[FTP]]

 ##NOTHING HERE ##

    

    [[RSYNC]]

##NOTHING HERE ##

##############################################################################

#   This service acts as a filter, converting the unit system coming from

#   the hardware to a unit system in the database.

[StdConvert]

    

    # The target_unit affects only the unit system in the database. Once

    # chosen it cannot be changed without converting the entire database.

    # Modification of target_unit after starting weewx will result in

    # corrupt data - the database will contain a mix of US and METRIC data.

    #

    # The value of target_unit does not affect the unit system for

    # reporting - reports can display US, Metric, or any combination of units.

    #

    # In most cases, target_unit should be left as the default: US

    #

    # In particular, those migrating from a standard wview installation

    # should use US since that is what the wview database contains.

    

    # DO NOT MODIFY THIS VALUE UNLESS YOU KNOW WHAT YOU ARE DOING!

    target_unit = US    # Options are 'US', 'METRICWX', or 'METRIC'

##############################################################################

#   This section can adjust data using calibration expressions.

[StdCalibrate]

    

    [[Corrections]]

        # For each type, an arbitrary calibration expression can be given. 

        # It should be in the units defined in the StdConvert section.

        # Example:

        foo = foo + 0.2

##############################################################################

#   This section is for quality control checks. If units are not specified,

#   values must be in the units defined in the StdConvert section.

[StdQC]

    

    [[MinMax]]

        barometer = 26, 32.5, inHg

        outTemp = -40, 120, degree_F

        inTemp = 10, 120, degree_F

        outHumidity = 0, 100

        inHumidity = 0, 100

        windSpeed = 0, 120, mile_per_hour

        pressure = 24, 34.5, inHg

##############################################################################

#   This section controls the origin of derived values.

[StdWXCalculate]

    

    [[Calculations]]

        # Derived quantities are calculated by this service. Possible values are:

        #  hardware        - use the value provided by hardware

        #  software        - use the value calculated by weewx

        #  prefer_hardware - use value provide by hardware if available,

        #                      otherwise use value calculated by weewx

        

        pressure = prefer_hardware

        barometer = prefer_hardware

        altimeter = prefer_hardware

        windchill = prefer_hardware

        heatindex = prefer_hardware

        dewpoint = prefer_hardware

        inDewpoint = prefer_hardware

        rainRate = prefer_hardware

##############################################################################

#   For hardware that supports it, this section controls how often the

#   onboard clock gets updated.

[StdTimeSynch]

    

    # How often to check the weather station clock for drift (in seconds)

    clock_check = 14400

    

    # How much it can drift before we will correct it (in seconds)

    max_drift = 5

##############################################################################

#   This section is for configuring the archive service.

[StdArchive]

    

    # If the station hardware supports data logging then the archive interval

    # will be downloaded from the station. Otherwise, specify it (in seconds).

    archive_interval = 300

    

    # If possible, new archive records are downloaded from the station

    # hardware. If the hardware does not support this, then new archive

    # records will be generated in software.

    # Set the following to "software" to force software record generation.

    record_generation = software

    

    # Whether to include LOOP data in hi/low statistics

    loop_hilo = True

    

    # The data binding used to save archive records

    data_binding = wx_binding

##############################################################################

#   This section binds a data store to a database.

[DataBindings]

    

    [[wx_binding]]

        # The database must match one of the sections in [Databases].

        # This is likely to be the only option you would want to change.

        database = archive_sqlite

        # The name of the table within the database

        table_name = archive

        # The manager handles aggregation of data for historical summaries

        manager = weewx.wxmanager.WXDaySummaryManager

        # The schema defines the structure of the database.

        # It is *only* used when the database is created.

        schema = user.esp8266.schema_with_esp

    [[cmon_binding]]

        manager = weewx.manager.DaySummaryManager

        schema = user.cmon.schema

        table_name = archive

        database = cmon_sqlite

##############################################################################

#   This section defines various databases.

[Databases]

    

    # A SQLite database is simply a single file

    [[archive_sqlite]]

        database_type = SQLite

        database_name = weewx.sdb

    

    # MySQL

    [[archive_mysql]]

        database_type = MySQL

        database_name = weewx

    [[cmon_sqlite]]

        database_name = cmon.sdb

        database_type = SQLite

##############################################################################

#   This section defines defaults for the different types of databases.

[DatabaseTypes]

    

    # Defaults for SQLite databases

    [[SQLite]]

        driver = weedb.sqlite

        # Directory in which the database files are located

        SQLITE_ROOT = /var/lib/weewx

    

    # Defaults for MySQL databases

    [[MySQL]]

        driver = weedb.mysql

        # The host where the database is located

        host = localhost

        # The user name for logging in to the host

        user = weewx

        # The password for the user name

        password = weewx

##############################################################################

#   This section configures the internal weewx engine.

[Engine]

    

    [[Services]]

        # This section specifies the services that should be run. They are

        # grouped by type, and the order of services within each group

        # determines the order in which the services will be run.

        prep_services = weewx.engine.StdTimeSynch

        data_services = user.esp8266.add_esp_records,

        process_services = weewx.engine.StdConvert, weewx.engine.StdCalibrate, weewx.engine.StdQC, weewx.wxservices.StdWXCalculate, user.cmon.ComputerMonitor

        archive_services = weewx.engine.StdArchive

        restful_services = weewx.restx.StdStationRegistry, weewx.restx.StdWunderground, weewx.restx.StdPWSweather, weewx.restx.StdCWOP, weewx.restx.StdWOW, weewx.restx.StdAWEKAS

        report_services = weewx.engine.StdPrint, weewx.engine.StdReport

##############################################################################

#  Options for extension 'cmon'

[ComputerMonitor]

    data_binding = cmon_binding

[Thomas Carlin]

I did a little more playing with this today, and determined that the problem is with extraTemp3, which I just started using, but is built into the system.  That being the case, I decided to drop and re-build the daily summaries, just for fun.  My theory being that was done right before the issue started occurring to add a battery voltage field, and there could have been an issue there somehow.  It hasn't been running for too long yet, but so far it seems to be working.  I'll keep an eye on it over the next few days, and see how it does, and update the thread.

Thanks,

[gjr80]

Yep, that could explain the symptoms. The accumulator for a given day and for a given observation is at times initialised from that particular day from that particular observation's daily summary table. If the sum field of the day and daily summary concerned was None instead of a numeric then the sum field of that accumulator would be initialised with None instead of 0.0 and you would see the symptoms you experienced. Rebuilding the daily summaries would fix such a problem. It will be interesting to see if the same issue comes back, certainly if you were making changes to the database schema there is potential for a hiccup to occur, if you are relying on weeWX to handle all database interraction then such a hiccup should never occur.

Gary

[Thomas Carlin]

Well, so much for that theory.   About 7:35p it crashed again.  But unplugging the device that generates that data has allowed it to run normally again.  However, based on your response the last record in this output would be the culprit then correct?  If my count is correct, both sum and wsum would be considered None when the accumulator is initialized.  Any idea what could cause this behavior?

sqlite> select * from archive_day_extraTemp3 where datetime >= 1510640000;
dateTime|min|mintime|max|maxtime|sum|count|wsum|sumtime

1510642800|70.52|1510728900|78.26|1510721100|2935.64|40|880692.0|12000

1510729200|65.48|1510746000|78.08|1510779600|16512.56|232|4953768.0|69600

1510815600|67.28|1510842900|71.06|1510842000|2934.08|43|880224.0|12900

1510902000|||||0.0|0|0.0|0

1510988400|74.48|1511058300|83.12|1511039400||45||13500

sqlite>

sqlite> select dateTime, inTemp, extraTemp3 from archive where datetime >= 1511057400;

dateTime|inTemp|extraTemp3

1511057400|70.6000000000002|75.02

1511057700|70.3999999999998|74.66

1511058000|70.3999999999998|74.66

1511058300|70.2767123287672|74.48

1511058600|70.2|

1511069400|69.8092307692308|

A f thought:  

In my driver if the correct data is not supplied, None is sent to event.record, is this correct, or should it simply not be called in that circumstance?

Snippit from my current driver:

for sensor in self.sensors:

                        if sensor != 'mac':

#                            print "DB Entry: " + self.sensors[sensor]

                            try:

                                value = esp_records[sensor]

                            except KeyError:

                                value = None

#                            print "Value: " + str(value)

                            event.record[self.sensors[sensor]] = value

I never modify the database without going through weewx, unless I am purging old data, so it shouldn't be an issue, but you never know.

Thanks Gary

[Thomas Carlin]

The plot thickens.  Right before I went to bed this morning, I checked the current daily summary record for extraTemp3, and it looked as you would expect it to.  It had data in it for both sum and wsum, unlike the one posted above.  Roughly an hour later, Weewx crashed with the same error, and this morning when I went to look into it, the current record, same one as last night, no longer had data in it for the same fields.  

After looking a little closer, I noticed that the extraHumid3, which I created, did exactly the same thing as extraTemp3.  Since both these pieces of data come from the same DHT11 sensor, my current theory is that under the wrong conditions, my sensor is spitting out garbage that is not being properly sanitized, which somehow empties those fields.  Then next time the loop runs, everything falls apart.  I'm setting up a sandbox right now so I can play with it without screwing with my 'production' system.  Just hoping the sandbox crashes like the current system does!

[Thomas Carlin]

I'm pretty sure this is what is happening.  Take a look at the output below:

...

DB Entry: extraTemp3

Value: nan

DB Entry: extraHumid3

Value: nan

...

Unable to access http://192.168.2.105

REC:    2017-11-19 12:38:00 MST (1511120280) ..., extraHumid3: nan, extraTemp1: 55.58, extraTemp3: nan, heatindex: 31.7143461695...

The next attempt to add data to the database resulted in the program crashing due to a type mismatch.  

May I request that a blurb be added to the documentation here: http://www.weewx.com/docs/customizing.htm#Adding_2nd_source about sanitizing the data before it is fed to event.record()?

[Thomas Keffer]

But, how did you create a NaN? They are not returned from any of the Python mathematical functions.

-tk

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Thomas Carlin]

Hey Tom

At this point in the code 'nan' is not equivalent to 'Not a Number', it is simply a string that is pulled out of an http page, not generated by python.  Originally, it is generated on my Arduino when data cannot be pulled from the temperature sensor.  Unfortunately in this case when you run float("nan") in python, it returns nan, and that was the extent of my sanitization.  I'll write a condition into my driver to catch that, and should be good to go.

To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.

[Thomas Keffer]

So, if I'm understanding you correctly, the Arduino produces numbers as strings. It can also produce the string "nan" when no data are available. So, when it came time to convert the string to a floating point number in your driver, you effectively had float("nan"), which is how the Python NaN was produced. Is that it?

If that's the case, that just sounds like a run-of-the-mill programming error. There's no end of things like that which could go wrong. We can't list them all in the Customizing Guide.

-tk

To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+unsubscribe@googlegroups.com.

[gjr80]

Thomas C,

Sorry I did not respond to your earlier posts after mine, the extract you provided does indeed show a daily summary record with sum and wsum fields as None (well the db fields contain nothing but they will end up as python None values within weeWX). At the moment I have still not been able to fathom the circumstances that would cause this.

One thing about your driver though. If you are using the default weeWX schema, or a modified version of the default schema to which you have only added numeric (read SQL type REAL) fields, then your driver must only emit numeric values or the python None value. If you emit strings it will cause an exception to be thrown within weeWX. The default weeWX behaviour is if the sensor is present and emitting values then the driver should emit those values as numeric values in the approriate loop packet/archive record field, if the sensor is present but offline then the driver emits the python None value in the field concerned and if the sensor does not exist then the driver should not emit anything for that sensor ie the field is excluded from the loop packet/archive record.

Note. Technically there is no reason that you cannot add a string to a loop packet or an archive record but if it is placed in a field that weeWX tries to 'accumulate' it will cause an exception.

I still don't see how your driver can cause the exception you first posted (emitting a string will cause a similar but clearly different exception), but it may be worthwhile reworking your driver to conform with the weeWX expected behaviour. It might also be worthwhile instrumenting your driver to drop the emitted loop packet to the log so we can see exactly what is coming out of the driver in the leadup to the exception.

Gary

[Thomas Carlin]

Tom,

Your interpretation is exactly correct, and yes it is a coding error on my side.  My thought was just a single sentence there about the Weewx engine not doing any sanitation, and that being the responsibility of the driver.  If not, no worries, just a thought.

Gary,

No worries on the delay.  You got me going in the right direction, and we got it figured out! It's not like you get paid for doing this.  Until yesterday, the extent of the sanitation that I was doing on the incoming data was running float(data), and catching any ValueError exception.  Now, because nan is a special 'string', when you run float('nan'), it returns NaN, but now it is the Python 'number' NaN.  When you add a number to NaN, your result is always NaN.  When this is stuffed into the database, apparently it is interpreted as SQL NULL.  Next time the loop runs, Python queries the database, and interprets SQL NULL as None, and then we get the type mismatch.  

I have added some more sanitation for this, and we are back up and running, and it *should* conform to expected behavior.  

Does that make sense?

Thank you for the input on the driver.  I have a laundry list of things that I would like to add to make it more robust, and I'll add the loop output to that.  I may be converting to a queue as well, simply due to the number of sensors that are being polled, and the "couple of seconds" note in the Second Datasource section of the docs.  But that is a project for another day!

--
You received this message because you are subscribed to a topic in the Google Groups "weewx-user" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/weewx-user/TE57NMTaIXw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to weewx-user+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Thomas Carlin
970-401-3805

[Thomas Keffer]

Thanks for the explanation. I think I understand the failure mode now.

One thing we could do is put a check in ScalarStats.addSum() to make sure the incoming value is, in fact, a number (and not a String, None, or NaN). I don't think it's a good idea to just swallow and suppress the error, but this way, it would be immediately obvious who and what the culprit is. The patch would look something like this:

diff --git a/bin/weewx/accum.py b/bin/weewx/accum.py

index 26bf243..ee54db3 100644

--- a/bin/weewx/accum.py

+++ b/bin/weewx/accum.py

@@ -85,6 +85,8 @@

     def addSum(self, val, weight=1):

         """Add a scalar value to my running sum and count."""

         if val is not None:

+            if not isinstance(val, (float, int)):

+                raise ValueError("accum: Expected float or int, got %s" % val)

             self.sum     += val

             self.count   += 1

             self.wsum    += val * weight

-tk

To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+unsubscribe@googlegroups.com.

[gjr80]

Well I am embarrased to say that to say I did not know about nan/NaN and python, today's lesson I guess.

Gary

[Thomas Keffer]

Implemented at commit 40d7a2c.

-tk

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.

To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+unsubscribe@googlegroups.com.

[Thomas Carlin]

I don't think this will catch float('nan'), which is how I was bit this time.  

val = float('nan')

>>> isinstance(val, (float, int))
True
>>>

>>> type(val)

<type 'float'>

May I recommend as an alternative, slightly more robust solution:

def addSum(self, val, weight=1):
         """Add a scalar value to my running sum and count."""
         if val is not None:

+            if (not isinstance(val, (float, int))) or (math.isnan(val)):

+                raise ValueError("accum: Expected float or int, got %s" % val)
             self.sum     += val
             self.count   += 1
             self.wsum    += val * weight

To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.

[gjr80]

Might have to be careful we don't break python 2.5 compatibility, I think isnan() was introduced in python 2.6.

Gary

[Thomas Carlin]

True, didn't think about that.  It's dirty, but 

str(val) == 'nan'

would work also.  Aside from isnan(), that is the most readable solution that I am aware of.

[Thomas Keffer]

Good point. I went with

val != val

Commit  0bce71e 

-tk

--

You received this message because you are subscribed to the Google Groups "weewx-user" group.

To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+unsubscribe@googlegroups.com.

[Thomas Carlin]

That should do the trick.  Thanks Tom!

Enjoy Thanksgiving tomorrow all!

On Tuesday, November 21, 2017 at 5:29:12 AM UTC-7, Tom Keffer wrote:

Good point. I went with

val != val

Commit  0bce71e 

-tk

On Mon, Nov 20, 2017 at 9:47 PM, Thomas Carlin <thomas...@gmail.com> wrote:

True, didn't think about that.  It's dirty, but 

str(val) == 'nan'

would work also.  Aside from isnan(), that is the most readable solution that I am aware of.

On Monday, November 20, 2017 at 8:33:28 PM UTC-7, gjr80 wrote:

Might have to be careful we don't break python 2.5 compatibility, I think isnan() was introduced in python 2.6.

Gary

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.

To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.