process.py errors

[Joakim Thulin]

Good morning,

I am sorry to bother you with more of my weather station errors. I threw out the old WH2080-weather station yesterday and replaced it with a brand new WN2080 (http://www.foshk.com/Weather_Professional/WN2080.html). This model has solar sensors, so I changed my weather.ini file to ws type=3080. I have also performed pywws.reprocess -u.

The config section of weather.ini looks like this now:

ws type = 3080

usb activity margin = 0.0

pressure offset = 4

logdata sync = 1

day end hour = 21, False

gnuplot encoding = utf8

template encoding = utf-8

language = sv

rain day threshold = 0.2

gnuplot version = 5.0

frequent writes = False

asynchronous = False

I get one of these two errors directly after reboot:

2019-11-02 10:26:13:pywws.logger:pywws version 19.7.1, build 1661 (05eab44)

2019-11-02 10:26:33:pywws.weatherstation:status {'lost_connection': False, 'rain_overflow': False}

2019-11-02 10:26:33:pywws.process:2019-11-02 08:10:00 rain reset 1.5 -> 0.0

2019-11-02 10:26:33:pywws.process:2019-11-02 08:29:02 rain reset 1.5 -> 0.0

2019-11-02 10:26:33:pywws.process:2019-11-02 08:45:02 rain reset 1.5 -> 0.0

2019-11-02 10:26:33:pywws.process:2019-11-02 09:05:00 rain reset 1.5 -> 0.0

2019-11-02 10:26:33:pywws.process:2019-11-02 09:20:00 rain reset 1.5 -> 0.0

2019-11-02 10:28:57:pywws.livelog:math domain error

Traceback (most recent call last):

  File "/usr/local/lib/python3.7/dist-packages/pywws/livelog.py", line 91, in live_log

    tasks.do_live(data)

  File "/usr/local/lib/python3.7/dist-packages/pywws/regulartasks.py", line 211, in do_live

    self._do_common(now, sections, live_data=calib_data)

  File "/usr/local/lib/python3.7/dist-packages/pywws/regulartasks.py", line 143, in _do_common

    self.do_plot(template)

  File "/usr/local/lib/python3.7/dist-packages/pywws/regulartasks.py", line 234, in do_plot

    self.plotter.do_plot(input_xml, output_file) == 0):

  File "/usr/local/lib/python3.7/dist-packages/pywws/plot.py", line 744, in do_plot

    of.write(self.plot_data(plot_no, plot, source))

  File "/usr/local/lib/python3.7/dist-packages/pywws/plot.py", line 926, in plot_data

    value = eval(subplot.ycalc)

  File "<string>", line 1, in <module>

  File "/usr/local/lib/python3.7/dist-packages/pywws/conversions.py", line 199, in dew_point

    gamma = ((a * temp) / (b + temp)) + math.log(float(hum) / 100.0)

ValueError: math domain error

Or the other type of error:

2019-11-02 10:30:50:pywws.logger:pywws version 19.7.1, build 1661 (05eab44)

2019-11-02 10:31:32:pywws.weatherstation:status {'lost_connection': False, 'rain_overflow': False}

2019-11-02 10:32:57:pywws.process:2019-11-02 09:05:00 rain reset 1.5 -> 0.0

2019-11-02 10:32:57:pywws.process:2019-11-02 09:20:00 rain reset 1.5 -> 0.0

2019-11-02 10:32:57:pywws.livelog:list index out of range

Traceback (most recent call last):

  File "/usr/local/lib/python3.7/dist-packages/pywws/livelog.py", line 81, in live_log

    pywws.process.process_data(context)

  File "/usr/local/lib/python3.7/dist-packages/pywws/process.py", line 761, in process_data

    start = generate_hourly(context.calib_data, context.hourly_data, start)

  File "/usr/local/lib/python3.7/dist-packages/pywws/process.py", line 621, in generate_hourly

    hourly_data.update(hourlygen(calib_data, prev))

  File "/usr/local/lib/python3.7/dist-packages/pywws/filedata.py", line 433, in update

    for k in E:

  File "/usr/local/lib/python3.7/dist-packages/pywws/process.py", line 602, in hourlygen

    acc.add_raw(data)

  File "/usr/local/lib/python3.7/dist-packages/pywws/process.py", line 255, in add_raw

    self.wind_fil.add(data)

  File "/usr/local/lib/python3.7/dist-packages/pywws/process.py", line 218, in add

    self.Ve -= speed * sin_LUT[direction]

IndexError: list index out of range

Do these two process.py errors stop pywws?

Do you have any suggestions how I could get rid of them?

Best regards from Joakim Thulin

[Jim Easterbrook]

On 02/11/2019 09:44, Joakim Thulin wrote:
> Good morning,
> I am sorry to bother you with more of my weather station errors. I threw
> out the old WH2080-weather station yesterday and replaced it with a
> brand new WN2080
> (http://www.foshk.com/Weather_Professional/WN2080.html). This model has
> solar sensors, so I changed my weather.ini file to ws type=3080. I have
> also performed pywws.reprocess -u.
>
> The config section of weather.ini looks like this now:
>
> ws type = 3080
> usb activity margin = 0.0
> pressure offset = 4
> logdata sync = 1
> day end hour = 21, False
> gnuplot encoding = utf8
> template encoding = utf-8
> language = sv
> rain day threshold = 0.2
> gnuplot version = 5.0
> frequent writes = False
> asynchronous = False
>
> I get one of these two errors directly after reboot:
>

>     gamma = ((a * temp) / (b + temp)) + math.log(float(hum) / 100.0)
> ValueError: math domain error

This means the humidity value is zero. The old stations never reported
values below about 20%, but maybe the new ones are different? If your
station is really reporting zero then I probably need to change the
pywws.conversions module.

Meanwhile you could get round the problem with a user calibration
module, replacing zero values with 'None'.

> Or the other type of error:
>

>   File "/usr/local/lib/python3.7/dist-packages/pywws/process.py", line
> 218, in add
>     self.Ve -= speed * sin_LUT[direction]
> IndexError: list index out of range

This is very odd - it suggests you have wind directions outside the
usual 0..15 range. Can you check your data to see what's going on?
--
Jim Easterbrook <http://www.jim-easterbrook.me.uk/>

[Joakim Thulin]

Below is a dump from testweatherstation -d (I believe it is the same as the fixed section of status.ini). Is it possible to get the data you requested from this?

pi@nas:~ $ pywws-testweatherstation -d

12:25:16:pywws.logger:pywws version 19.7.1, build 1661 (05eab44)

{'abs_pressure': 990.5,

 'alarm': {'abs_pressure': {'hi': 1040, 'lo': 960},

           'dewpoint': {'hi': 10, 'lo': -10},

           'hum_in': {'hi': 65, 'lo': 35},

           'hum_out': {'hi': 70, 'lo': 45},

           'illuminance': 0,

           'rain': {'day': 150, 'hour': 3},

           'rel_pressure': {'hi': 1040, 'lo': 960},

           'temp_in': {'hi': 20, 'lo': 0},

           'temp_out': {'hi': 30, 'lo': -10},

           'time': '12:00',

           'uv': 0,

           'wind_ave': {'bft': 0, 'ms': 5},

           'wind_dir': 0,

           'wind_gust': {'bft': 0, 'ms': 10},

           'windchill': {'hi': 20, 'lo': 0}},

 'alarm_1': {'bit0': False,

             'bit3': False,

             'hum_in_hi': False,

             'hum_in_lo': False,

             'hum_out_hi': False,

             'hum_out_lo': False,

             'time': False,

             'wind_dir': False},

 'alarm_2': {'pressure_abs_hi': False,

             'pressure_abs_lo': False,

             'pressure_rel_hi': False,

             'pressure_rel_lo': False,

             'rain_day': False,

             'rain_hour': False,

             'wind_ave': False,

             'wind_gust': False},

 'alarm_3': {'dew_point_hi': False,

             'dew_point_lo': False,

             'temp_in_hi': False,

             'temp_in_lo': False,

             'temp_out_hi': False,

             'temp_out_lo': False,

             'wind_chill_hi': False,

             'wind_chill_lo': False},

 'current_pos': 4064,

 'data_changed': 0,

 'data_count': 239,

 'date_time': '2019-11-02 12:25:00',

 'display_1': {'alarm_time': False,

               'clock_12hr': False,

               'date_mdy': False,

               'pressure_rel': True,

               'show_day_name': False,

               'show_year': True,

               'time_scale_24': True,

               'wind_gust': False},

 'display_2': {'rain_day': False,

               'rain_hour': True,

               'rain_month': False,

               'rain_total': False,

               'rain_week': False,

               'temp_out_chill': False,

               'temp_out_dew': False,

               'temp_out_temp': True},

 'display_3': {'alarm_illuminance_hi': False,

               'alarm_uv_hi': False,

               'bit3': False,

               'bit4': False,

               'bit6': False,

               'bit7': False,

               'illuminance_fc': False,

               'illuminance_wm2': False},

 'lux_wm2_coeff': 0,

 'magic_0': 85,

 'magic_1': 170,

 'max': {'abs_pressure': {'date': '2018-01-01 12:00:00', 'val': 1015.3},

         'dewpoint': {'date': '2018-01-01 12:18:00', 'val': 10.2},

         'hum_in': {'date': '2018-01-01 12:00:00', 'val': 65},

         'hum_out': {'date': '2019-11-02 01:01:00', 'val': 99},

         'illuminance': {'date': None, 'val': 28774.4},

         'rain': {'day': {'date': '2018-01-01 12:06:00', 'val': 4.5},

                  'hour': {'date': '2018-01-01 12:06:00', 'val': 4.5},

                  'month': {'date': '2018-01-01 12:06:00', 'val': 4.5},

                  'total': {'date': '2018-01-01 12:06:00', 'val': 4.5},

                  'week': {'date': '2018-01-01 12:06:00', 'val': 4.5}},

         'rel_pressure': {'date': '2018-01-01 12:00:00', 'val': 1013.2},

         'temp_in': {'date': '2018-01-01 12:00:00', 'val': 28.4},

         'temp_out': {'date': '2018-01-01 12:13:00', 'val': 21.4},

         'uv': {'date': None, 'val': 0},

         'wind_ave': {'date': '2018-01-01 15:47:00', 'val': 0.7},

         'wind_gust': {'date': '2019-11-02 03:13:00', 'val': 2.4},

         'windchill': {'date': '2018-01-01 12:13:00', 'val': 21.4}},

 'min': {'abs_pressure': {'date': '2019-11-02 12:14:00', 'val': 990.5},

         'dewpoint': {'date': '2018-01-01 12:59:00', 'val': -0.3},

         'hum_in': {'date': '2018-01-01 13:55:00', 'val': 24},

         'hum_out': {'date': '2018-01-01 12:00:00', 'val': 1},

         'rel_pressure': {'date': '2019-11-02 12:14:00', 'val': 994.5},

         'temp_in': {'date': '2018-01-01 12:01:00', 'val': 21.5},

         'temp_out': {'date': '2018-01-01 12:00:00', 'val': -40},

         'windchill': {'date': '2018-01-01 12:00:00', 'val': -40}},

 'rain_factor_raw': 4224,

 'read_period': 5,

 'rel_pressure': 994.5,

 'settings_1': {'bit3': False,

                'bit4': False,

                'pressure_hPa': False,

                'pressure_inHg': False,

                'pressure_mmHg': False,

                'rain_in': False,

                'temp_in_F': False,

                'temp_out_F': False},

 'settings_2': {'bit5': False,

                'bit6': False,

                'bit7': False,

                'wind_bft': False,

                'wind_kmph': False,

                'wind_knot': False,

                'wind_mph': False,

                'wind_mps': True},

 'timezone': 0,

 'unknown_01': 0,

 'unknown_18': 96,

 'wind_factor_raw': 18976}

min -> temp_out -> {'val': -40, 'date': '2018-01-01 12:00:00'}

alarm -> hum_out -> {'hi': 70, 'lo': 45}

rel_pressure -> 994.5

abs_pressure -> 990.5

Den lördag 2 november 2019 kl. 11:24:28 UTC+1 skrev Jim Easterbrook:

This means the humidity value is zero. The old stations never reported
values below about 20%, but maybe the new ones are different? If your
station is really reporting zero then I probably need to change the
pywws.conversions module.

Meanwhile you could get round the problem with a user calibration
module, replacing zero values with 'None'.

[Jim Easterbrook]

On 02/11/2019 11:39, Joakim Thulin wrote:
> Below is a dump from testweatherstation -d (I believe it is the same as
> the fixed section of status.ini). Is it possible to get the data you
> requested from this?

No, I'm talking about the data files pywws creates from your station's
logged data. The fixed block data is not relevant to your processing
problems.

You need to go through your data and find where the problems are. If
you're getting zero humidity on a regular basis, rather than as part of
a corrupt record, then pywws will need modifying. If you just have some
corrupt data you need to edit your raw data before it can be processed.
You also need to check the wind direction values.

[Joakim Thulin]

I guess you mean the data in weather/data/raw/2019-11-02.txt ?

[Jim Easterbrook]

On 02/11/2019 12:55, Joakim Thulin wrote:
> I guess you mean the data in weather/data/raw/2019-11-02.txt ?

Yes, that or an earlier file. It depends where the problems are.

[Joakim Thulin]

Got it!

Could you please provide a link on how to understand the comma separated text files?

[Jim Easterbrook]

On 02/11/2019 13:15, Joakim Thulin wrote:
> Got it!
> Could you please provide a link on how to understand the comma separated
> text files?

I admit this isn't well documented. They're CSV files with fields in the
order specified in the pywws.filedata module's "key_list" values.
https://pywws.readthedocs.io/en/latest/api/pywws.filedata.html#pywws.filedata.RawStore

[Joakim Thulin]

Thank you Jim for your support, I collected all raw data from October and the two days of November in a spreadsheet and the low humidity values (5) began appearing with the new weather station yesterday, the same with the odd >15 values on wind_dir. I am deleteing the two raw files from November and start all over here. The november data seem corrupt. Please find attached the spreadsheet.

Den lördag 2 november 2019 kl. 11:24:28 UTC+1 skrev Jim Easterbrook:

This means the humidity value is zero. The old stations never reported
values below about 20%, but maybe the new ones are different? If your
station is really reporting zero then I probably need to change the
pywws.conversions module.

Meanwhile you could get round the problem with a user calibration
module, replacing zero values with 'None'.

[Jim Easterbrook]

On 02/11/2019 14:18, Joakim Thulin wrote:
> Thank you Jim for your support, I collected all raw data from October
> and the two days of November in a spreadsheet and the low humidity
> values (5) began appearing with the new weather station yesterday, the
> same with the odd >15 values on wind_dir. I am deleteing the two raw
> files from November and start all over here. The november data seem
> corrupt. Please find attached the spreadsheet.

That's very odd looking - every other line is nonsense. You can check
what's coming from the weather station's memory with
'pywws-testweatherstation -h 10 -d' to get and decode the last 10
records. If that's similarly strange then your station would appear to
have a different memory layout from the older 3080 class stations. (It
does have different sensor hardware, so who knows what else has changed
in the design.)

[Joakim Thulin]

I cleared the weather station memory, so I haven't got 10 records yet. See attached text file.

The two new raw data entries i got is as follows:

2019-11-02 14:30:00,5,35,25.1,99,5.4,989,0,0,4,1.5,0,1645849.6,0

2019-11-02 14:45:02,38,0,0,4,0.5,128,3.4,25.1,99,16.2,166,3.8,0

Suddenly there is a status value of 166, do you know what that means?

[Joakim Thulin]

It suddenly works!

I wonder what happened.

2019-11-02 15:57:03:pywws.logger:pywws version 19.7.1, build 1661 (05eab44)

2019-11-02 15:57:56:pywws.weatherstation:status {'lost_connection': False, 'rain_overflow': False}

2019-11-02 15:57:57:pywws.process:2019-11-02 14:30:00 rain reset 3779.1 -> 1.5

2019-11-02 15:59:25:pywws.service.ftp:OK

2019-11-02 15:59:25:pywws.service.temperaturnu:server response "ok! (5.3)"

2019-11-02 15:59:26:pywws.service.underground:server response "success"

2019-11-02 15:59:26:pywws.service.underground:2 records sent

[Jim Easterbrook]

On 02/11/2019 15:04, Joakim Thulin wrote:
> I cleared the weather station memory, so I haven't got 10 records yet.
> See attached text file.

This shows perfectly decoded data at addresses 0x0100, 0x0110, 0x0120,
0x0130 and 0x0140, and unset data at 0xfff0 and so on, as you'd expect
from a new station.

Note that your data is at 16 byte intervals, meaning that you have a
1080 class machine, not a 3080 type.

> The two new raw data entries i got is as follows:
> 2019-11-02 14:30:00,5,35,25.1,99,5.4,989,0,0,4,1.5,0,1645849.6,0
> 2019-11-02 14:45:02,38,0,0,4,0.5,128,3.4,25.1,99,16.2,166,3.8,0
>
> Suddenly there is a status value of 166, do you know what that means?

I think pywws is getting data from the wrong address because you've set
the type to '3080' when it should be '1080'.

[Jim Easterbrook]

On 02/11/2019 15:07, Joakim Thulin wrote:
> It suddenly works!
> I wonder what happened.

When live logging with no catchup pywws doesn't need to guess where data
is stored, it uses the pointer that gets updated by the weather station.
When this pointer moves by 16 bytes instead of 20 then pywws knows it's
a 1080 type rather than a 3080 and starts reading data correctly.

[Joakim Thulin]

Pywws is so clever! Now changed to a 1080 type station, everything’s fine and I am happy. Again Jim, thanks for the support. Too bad I can’t use the solar data, but it’s more important the base data is correct

[Chris]

I've been running this type of station for about 6 months now, I can confirm that the solar panel on the top is solely for power and does provide solar data at all, kind of a waste, but hopefully the batteries will last a lot longer.

[Joakim Thulin]

I see, good to know, thanks a lot! The solar panel surely fooled me into believe I got a 3080-type station.