Wrong data reported

[runge....@googlemail.com]

Hi Jim,

hope you are doing fine.

I have a very strange problem and hope you can help me.

It started suddenly without having updated anything.

I get wrong values from time to time, roughly every second day at one point in time.

The wrong values do differ, so the value is changing.

In the log i can see:

2022-08-09 14:37:33:pywws.weatherstation:old data {'illuminance': 1600614.9, 'uv': 0, 'delay': 0, 'hum_in': 46, 'temp_in': 230.5, 'hum_out': 39, 'temp_out': 1052.7, 'abs_pressure': 358.4, 'wind_ave': 13.2, 'wind_gust': 0.1, 'wind_dir': None, 'rain': 1405.5, 'status': {'bit0': True, 'bit1': True, 'bit2': True, 'lost_connection': False, 'rain_overflow': False}}
2022-08-09 14:37:33:pywws.weatherstation:new data {'illuminance': 115859.2, 'uv': 6, 'delay': 0, 'hum_in': 60, 'temp_in': 24.3, 'hum_out': 21, 'temp_out': 30.5, 'abs_pressure': 998.9, 'wind_ave': 2, 'wind_gust': 3.1, 'wind_dir': 4, 'rain': 116.4, 'status': {'lost_connection': False, 'rain_overflow': False}}

Temp_in = 230,5, Temp_out =1052,7 and also the other values are wrong.

These values are recorded and uploaded.

It is very hot the last time here but not so hot :-)

I updated pywws but the error stays also with the latest version.

Here is a bit more log around the error:

2022-08-09 14:36:04:pywws.regulartasks:Templating ws3600.txt
2022-08-09 14:36:08:pywws.service.copy:1 upload
2022-08-09 14:36:08:pywws.service.underground:1 record sent
2022-08-09 14:36:44:pywws.regulartasks:doing task sections ['live']
2022-08-09 14:36:44:pywws.regulartasks:Templating ws3600.txt
2022-08-09 14:36:48:pywws.service.copy:1 upload
2022-08-09 14:36:49:pywws.service.underground:1 record sent
2022-08-09 14:37:04:pywws.weatherstation:live_data new ptr: 000b00
2022-08-09 14:37:04:pywws.process:Generating summary data
2022-08-09 14:37:04:pywws.regulartasks:doing task sections ['logged']
2022-08-09 14:37:04:pywws.regulartasks:Templating ws3600.txt
2022-08-09 14:37:08:pywws.service.copy:1 upload
2022-08-09 14:37:24:pywws.weatherstation:unexpected ptr change 000b00 -> 000a00
2022-08-09 14:37:24:pywws.weatherstation:live_data new ptr: 000a00
2022-08-09 14:37:24:pywws.weatherstation:live_data lost log sync -276.369
2022-08-09 14:37:24:pywws.weatherstation:reset read period 0
2022-08-09 14:37:24:pywws.process:Generating summary data
2022-08-09 14:37:24:pywws.regulartasks:doing task sections ['logged']
2022-08-09 14:37:24:pywws.regulartasks:Templating ws3600.txt
2022-08-09 14:37:28:pywws.service.copy:1 upload
2022-08-09 14:37:32:pywws.weatherstation:unexpected ptr change 000a00 -> 000b00
2022-08-09 14:37:32:pywws.weatherstation:status {'bit0': True, 'bit1': True, 'bit2': True, 'lost_connection': False, 'rain_overflow': False}
2022-08-09 14:37:32:pywws.regulartasks:doing task sections ['live']
2022-08-09 14:37:32:pywws.regulartasks:Templating ws3600.txt
2022-08-09 14:37:32:pywws.weatherstation:live_data new ptr: 000b00
2022-08-09 14:37:32:pywws.weatherstation:missed ptr change time
2022-08-09 14:37:32:pywws.process:Generating summary data
2022-08-09 14:37:32:pywws.process:unexpected data interval 2022-08-09 12:37:30 0:00:08
2022-08-09 14:37:32:pywws.process:2022-08-09 12:37:30 rain jump 116.4 -> 1405.5
2022-08-09 14:37:32:pywws.regulartasks:doing task sections ['logged']
2022-08-09 14:37:32:pywws.regulartasks:Templating ws3600.txt
2022-08-09 14:37:33:pywws.weatherstation:status {'lost_connection': False, 'rain_overflow': False}
2022-08-09 14:37:33:pywws.weatherstation:live_data lost sync -42.7843
2022-08-09 14:37:33:pywws.weatherstation:old data {'illuminance': 1600614.9, 'uv': 0, 'delay': 0, 'hum_in': 46, 'temp_in': 230.5, 'hum_out': 39, 'temp_out': 1052.7, 'abs_pressure': 358.4, 'wind_ave': 13.2, 'wind_gust': 0.1, 'wind_dir': None, 'rain': 1405.5, 'status': {'bit0': True, 'bit1': True, 'bit2': True, 'lost_connection': False, 'rain_overflow': False}}
2022-08-09 14:37:33:pywws.weatherstation:new data {'illuminance': 115859.2, 'uv': 6, 'delay': 0, 'hum_in': 60, 'temp_in': 24.3, 'hum_out': 21, 'temp_out': 30.5, 'abs_pressure': 998.9, 'wind_ave': 2, 'wind_gust': 3.1, 'wind_dir': 4, 'rain': 116.4, 'status': {'lost_connection': False, 'rain_overflow': False}}
2022-08-09 14:37:34:pywws.service.underground:1 record sent
2022-08-09 14:37:36:pywws.service.copy:1 upload
2022-08-09 14:38:16:pywws.weatherstation:setting sensor clock 40.4562
2022-08-09 14:38:16:pywws.regulartasks:doing task sections ['live']
2022-08-09 14:38:16:pywws.regulartasks:Templating ws3600.txt
2022-08-09 14:38:19:pywws.service.underground:1 record sent
2022-08-09 14:38:20:pywws.service.copy:1 upload
2022-08-09 14:39:13:pywws.regulartasks:doing task sections ['live']
2022-08-09 14:39:13:pywws.regulartasks:Templating ws3600.txt
2022-08-09 14:39:16:pywws.service.copy:1 upload
2022-08-09 14:39:16:pywws.service.underground:1 record sent
2022-08-09 14:39:56:pywws.regulartasks:doing task sections ['live']
2022-08-09 14:39:56:pywws.regulartasks:Templating ws3600.txt
2022-08-09 14:40:00:pywws.service.copy:1 upload
2022-08-09 14:40:01:pywws.service.underground:1 record sent
2022-08-09 14:40:32:pywws.weatherstation:station is not logging data
2022-08-09 14:40:36:pywws.weatherstation:station is not logging data
2022-08-09 14:40:44:pywws.regulartasks:doing task sections ['live']
2022-08-09 14:40:44:pywws.regulartasks:Templating ws3600.txt
2022-08-09 14:40:44:pywws.weatherstation:station is not logging data
2022-08-09 14:40:45:pywws.service.underground:1 record sent
2022-08-09 14:40:48:pywws.service.copy:1 upload
2022-08-09 14:41:24:pywws.weatherstation:station is not logging data
2022-08-09 14:41:32:pywws.regulartasks:doing task sections ['live']
2022-08-09 14:41:32:pywws.regulartasks:Templating ws3600.txt
2022-08-09 14:41:32:pywws.process:Generating summary data
2022-08-09 14:41:32:pywws.process:unexpected data interval 2022-08-09 12:37:30 0:00:08
2022-08-09 14:41:32:pywws.process:2022-08-09 12:37:30 rain jump 116.4 -> 1405.5
2022-08-09 14:41:32:pywws.process:2022-08-09 12:41:28 rain reset 1405.5 -> 116.4
2022-08-09 14:41:32:pywws.regulartasks:doing task sections ['logged']
2022-08-09 14:41:32:pywws.regulartasks:Templating ws3600.txt
2022-08-09 14:41:32:pywws.weatherstation:station is not logging data
2022-08-09 14:41:34:pywws.service.underground:1 record sent
2022-08-09 14:41:36:pywws.service.copy:1 upload
2022-08-09 14:42:20:pywws.weatherstation:live_data new ptr: 000b14
2022-08-09 14:42:20:pywws.weatherstation:station is logging data
2022-08-09 14:42:20:pywws.process:Generating summary data
2022-08-09 14:42:20:pywws.process:unexpected data interval 2022-08-09 12:37:30 0:00:08
2022-08-09 14:42:20:pywws.process:2022-08-09 12:37:30 rain jump 116.4 -> 1405.5
2022-08-09 14:42:20:pywws.process:2022-08-09 12:41:28 rain reset 1405.5 -> 116.4
2022-08-09 14:42:20:pywws.process:unexpected data interval 2022-08-09 12:42:18 0:00:50
2022-08-09 14:42:20:pywws.regulartasks:doing task sections ['logged']
2022-08-09 14:42:20:pywws.regulartasks:Templating ws3600.txt
2022-08-09 14:42:21:pywws.regulartasks:doing task sections ['live']
2022-08-09 14:42:21:pywws.regulartasks:Templating ws3600.txt
2022-08-09 14:42:23:pywws.service.underground:1 record sent
2022-08-09 14:42:24:pywws.service.copy:1 upload
2022-08-09 14:43:13:pywws.regulartasks:doing task sections ['live']
2022-08-09 14:43:13:pywws.regulartasks:Templating ws3600.txt

Best regards,

Stefan

[Jim Easterbrook]

On 09/08/2022 17:52, 'runge....@googlemail.com' via pywws wrote:
>
> I have a very strange problem and hope you can help me.
>
> It started suddenly without having updated anything.
> I get wrong values from time to time, roughly every second day at one
> point in time.
> The wrong values do differ, so the value is changing.

I suspect this may be a symptom of aging hardware. It might be worth
trying a user calibration module to filter out (set to None) any really
crazy data.

> 2022-08-09 14:37:32:pywws.weatherstation:unexpected ptr change 000a00 ->
> 000b00

> 2022-08-09 14:37:32:pywws.weatherstation:live_data new ptr: 000b00
> 2022-08-09 14:37:32:pywws.weatherstation:missed ptr change time

> 2022-08-09 14:40:32:pywws.weatherstation:station is not logging data
> 2022-08-09 14:40:36:pywws.weatherstation:station is not logging data

> 2022-08-09 14:42:20:pywws.weatherstation:live_data new ptr: 000b14
> 2022-08-09 14:42:20:pywws.weatherstation:station is logging data
> 2022-08-09 14:42:20:pywws.process:Generating summary data

This looks as if the current data pointer (read from the "fixed block")
has become unreliable. It changed when it shouldn't have, and then data
read from the wrong(?) pointer didn't change.

--
Jim Easterbrook <http://www.jim-easterbrook.me.uk/>

[runge....@googlemail.com]

Hi,

yes, this ptr message is strange. Why is the ptr changing?

I will try filtering out with my calibration module.`

Aging hardware can be a root cause my weather station is used for a long time. 

I am very surprised that it is still working. Especially the outside module.

Do you think resetting the base station is also worth a try?

I think the data is read from the base station and also the pointer change happens there, right?

Thanks and best regards,

Stefan

[Jim Easterbrook]

On 09/08/2022 18:56, 'runge....@googlemail.com' via pywws wrote:
>
> yes, this ptr message is strange. Why is the ptr changing?

I don't know. Maybe the base station's USB hardware is corrupting data.

My old station has an area of memory that is failing. As the pointer
moves through that part of the data I get some silly values.

> Do you think resetting the base station is also worth a try?

Possibly.

> I think the data is read from the base station and also the pointer
> change happens there, right?

Yes. Bad data could be a problem with the outside unit, but all the
pointer stuff is only relevant to the base station.

The first problem in your log was the pointer value decreasing from b00
to a00. This should never happen, so maybe pywws could spot that as an
error and ignore it. Unless the earlier increase to b00 was an error.
It's not easy to reliably detect errors.

[runge....@googlemail.com]

Hi Jim,

thanks,

so I cleared the memory on the base station and I will see if the error reoccurs.

If this does not fix the issue I will try to filter it out with the calibration module.

But let's see if the memory reset perhaps already fix the issue.

Thanks and best regards,

Stefan

[runge....@googlemail.com]

Hi Jim, oh no, it happened again after reset, even with memory just 1/4 filled. 

Perhaps I should get a new station ;-)

Again ptr change, afterwards, pywws even crashes with list index out of range when templating.

2022-08-11 14:40:32:pywws.service.copy:1 upload
2022-08-11 14:40:40:pywws.weatherstation:unexpected ptr change 001e00 -> 001f00
2022-08-11 14:40:40:pywws.weatherstation:status {'bit0': True, 'bit1': True, 'bit2': True, 'lost_connection': False, 'rain_overflow': False}
2022-08-11 14:40:40:pywws.regulartasks:doing task sections ['live']
2022-08-11 14:40:40:pywws.regulartasks:Templating ws3600.txt
2022-08-11 14:40:40:pywws.weatherstation:live_data new ptr: 001f00
2022-08-11 14:40:40:pywws.weatherstation:missed ptr change time
2022-08-11 14:40:40:pywws.process:Generating summary data
2022-08-11 14:40:40:pywws.process:unexpected data interval 2022-08-11 12:40:38 0:00:09
2022-08-11 14:40:40:pywws.livelog:list index out of range
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/dist-packages/pywws/livelog.py", line 87, in live_log
    pywws.process.process_data(context)
  File "/usr/local/lib/python3.9/dist-packages/pywws/process.py", line 756, in process_data
    start = generate_hourly(context.calib_data, context.hourly_data, start)
  File "/usr/local/lib/python3.9/dist-packages/pywws/process.py", line 619, in generate_hourly
    hourly_data.update(hourlygen(calib_data, prev))
  File "/usr/local/lib/python3.9/dist-packages/pywws/filedata.py", line 433, in update
    for k in E:
  File "/usr/local/lib/python3.9/dist-packages/pywws/process.py", line 600, in hourlygen
    acc.add_raw(data)
  File "/usr/local/lib/python3.9/dist-packages/pywws/process.py", line 255, in add_raw
    self.wind_fil.add(data)
  File "/usr/local/lib/python3.9/dist-packages/pywws/process.py", line 218, in add
    self.Ve -= speed * sin_LUT[direction]
IndexError: list index out of range

Is there a possibility in the calibration module to ignore all data if let's say Temp_Out is strange?

As of now, Temp_Out is always absolutely off with way bigger >100 Degree. 

So I would like to detect this and do not want to use any data from this call since I can see that also other values are wrong.

Or is it possible to just ignore all readings after an " unexpected ptr change" happens?

I think this would be a lot better.

How can I do this?

Thanks and best regards,

Stefan

[Jim Easterbrook]

On 11/08/2022 15:10, 'runge....@googlemail.com' via pywws wrote:
> Hi Jim, oh no, it happened again after reset, even with memory just 1/4
> filled.
> Perhaps I should get a new station ;-)
>
> Again ptr change, afterwards, pywws even crashes with list index out of
> range when templating.

Templates often assume a minimum quantity of data. I can't remember if I
included any attempt to detect iteration beyond the range of valid data.

>   File "/usr/local/lib/python3.9/dist-packages/pywws/process.py", line
> 218, in add
>     self.Ve -= speed * sin_LUT[direction]
> IndexError: list index out of range

That's caused by corrupt wind direction data being outside the range 0..15

> Is there a possibility in the calibration module to ignore all data if
> let's say Temp_Out is strange?

Yes, that's the sort of thing it's good for. You'll probably need to set
each value to None, rather than simply clearing the dict.

> As of now, Temp_Out is always absolutely off with way bigger >100 Degree.
> So I would like to detect this and do not want to use any data from this
> call since I can see that also other values are wrong.

I think the wrong pointer means data is being read from the wrong memory
location and so it all becomes nonsense.

> Or is it possible to just ignore all readings after an " unexpected ptr
> change" happens?
> I think this would be a lot better.
> How can I do this?

I agree it would be a sensible thing to do - only accept pointers that
meet certain conditions - but I'll need to modify pywws to do that. I'm
on holiday at the moment but will look into it when I get back in a
couple of weeks.

[runge....@googlemail.com]

Hi Jim,

thanks for your detailed answer.

I wish you a very nice holiday! ;-)

I will try to fix it within the calibration module.

I will be happy if you can check after your holiday the ptr handling.

Best regards,

Stefan

[Jim Easterbrook]

On 11/08/2022 17:08, Jim Easterbrook wrote:
>
> I agree it would be a sensible thing to do - only accept pointers that
> meet certain conditions - but I'll need to modify pywws to do that. I'm
> on holiday at the moment but will look into it when I get back in a
> couple of weeks.

I've added a simple pointer validity check to pywws.weatherstation in
the latest update on GitHub. If you can install this it might help to
reduce the errors.

Jim

[runge....@googlemail.com]

Hi Jim,

sorry for the late reply. I was on holiday.

Thanks for the pointer validity check.

I will try to test it.

Now that the temperatures are lower in my area, I recognized that the wrong data only happens when it is hot ( >  30 degrees).

So I can only see if it is working next Summer :-)

Thanks and best regards,

Stefan

[Jim Easterbrook]

On 27/09/2022 08:53, 'runge....@googlemail.com' via pywws wrote:
>
> Now that the temperatures are lower in my area, I recognized that the
> wrong data only happens when it is hot ( >  30 degrees).
> So I can only see if it is working next Summer :-)

That's an interesting observation. Is the base station in full sunlight?
Overheating could well be the problem.

[runge....@googlemail.com]

Hi Jim,

that's really strange. The base station is not in direct sunlight and it is in a room.

I controlled the indoor temperature measured when the errors occurred. 

Mostly it happened when the indoor temperature was 25 degrees or more.

The outdoor sensors are in direct sunlight and if I check the temperature there I can see that it happens when the outdoor sensor exceeds 30 degrees.

But also there I have weeks with even higher temps when it does not happen.

Very strange, I am not sure if temperature plays a role but it looks like that.

Let's see I will check closely if I can see this happen again.

I have implemented the new version of pywws with your fix.

Is there a way to detect that it happened in the logs?

Do you write a message I can grep in the logs daily so that I can see if it works?

Thanks and best regards,

Stefan

[Jim Easterbrook]

On 28/09/2022 16:16, 'runge....@googlemail.com' via pywws wrote:
>
> Let's see I will check closely if I can see this happen again.
>
> I have implemented the new version of pywws with your fix.
> Is there a way to detect that it happened in the logs?
> Do you write a message I can grep in the logs daily so that I can see if
> it works?

Yes, it should log an error "invalid ptr value" every time it gets an
invalid pointer.

[runge....@googlemail.com]

Hi Jim,

i monitored the fix very closely and did not get any wrong data in the template files.

I use the template files to get the data into my home automation fhem. 

But i recognized today that there was a wrong value reported to wunderground.

At 16:49 in wunderground i have wrong temperature, wrong wind speed and wrong rain reading.

In my home automation the wrong value was not recorded.

Is it possible that the fix only works for templating?

Can this be enhanced to the other channels too?

Here is the pywws log of this time:

2023-09-07 16:45:53:pywws.service.underground:1 record sent
2023-09-07 16:46:38:pywws.regulartasks:doing task sections ['live']
2023-09-07 16:46:38:pywws.regulartasks:Templating ws3600.txt
2023-09-07 16:46:41:pywws.service.copy:1 upload
2023-09-07 16:46:41:pywws.service.underground:1 record sent
2023-09-07 16:47:07:pywws.weatherstation:live_data new ptr: 0013ac
2023-09-07 16:47:07:pywws.process:Generating summary data
2023-09-07 16:47:07:pywws.regulartasks:doing task sections ['logged']
2023-09-07 16:47:07:pywws.regulartasks:Templating ws3600.txt
2023-09-07 16:47:09:pywws.service.copy:1 upload
2023-09-07 16:47:26:pywws.regulartasks:doing task sections ['live']
2023-09-07 16:47:26:pywws.regulartasks:Templating ws3600.txt
2023-09-07 16:47:29:pywws.service.copy:1 upload
2023-09-07 16:47:30:pywws.service.underground:1 record sent
2023-09-07 16:48:14:pywws.regulartasks:doing task sections ['live']
2023-09-07 16:48:14:pywws.regulartasks:Templating ws3600.txt
2023-09-07 16:48:15:pywws.service.underground:1 record sent
2023-09-07 16:48:17:pywws.service.copy:1 upload
2023-09-07 16:49:07:pywws.regulartasks:doing task sections ['live']
2023-09-07 16:49:07:pywws.regulartasks:Templating ws3600.txt
2023-09-07 16:49:09:pywws.service.copy:1 upload
2023-09-07 16:49:11:pywws.service.underground:1 record sent
2023-09-07 16:49:50:pywws.regulartasks:doing task sections ['live']
2023-09-07 16:49:50:pywws.regulartasks:Templating ws3600.txt
2023-09-07 16:49:51:pywws.service.underground:1 record sent
2023-09-07 16:49:53:pywws.service.copy:1 upload
2023-09-07 16:50:38:pywws.regulartasks:doing task sections ['live']
2023-09-07 16:50:38:pywws.regulartasks:Templating ws3600.txt
2023-09-07 16:50:40:pywws.service.underground:1 record sent
2023-09-07 16:50:41:pywws.service.copy:1 upload
2023-09-07 16:51:26:pywws.regulartasks:doing task sections ['live']
2023-09-07 16:51:26:pywws.regulartasks:Templating ws3600.txt
2023-09-07 16:51:28:pywws.service.underground:1 record sent
2023-09-07 16:51:29:pywws.service.copy:1 upload
2023-09-07 16:52:14:pywws.weatherstation:live_data new ptr: 0013c0
2023-09-07 16:52:14:pywws.process:Generating summary data
2023-09-07 16:52:14:pywws.regulartasks:doing task sections ['logged']
2023-09-07 16:52:14:pywws.regulartasks:Templating ws3600.txt
2023-09-07 16:52:15:pywws.regulartasks:doing task sections ['live']
2023-09-07 16:52:15:pywws.regulartasks:Templating ws3600.txt
2023-09-07 16:52:17:pywws.service.copy:1 upload

This line perhaps seems to indicate the problem:

pywws.weatherstation:live_data new ptr: 0013ac

But this line is happening every day and i do not see wrong values every time this line occurs.

Attached are screenshots of wunderground and of my home automation.

I am using pywws version 23.2.0, build 1703.

Thanks,

Stefan

[Jim Easterbrook]

On 07/09/2023 23:03, 'runge....@googlemail.com' via pywws wrote:
>
> i monitored the fix very closely and did not get any wrong data in the
> template files.
> I use the template files to get the data into my home automation fhem.
>
> But i recognized today that there was a wrong value reported to
> wunderground.
> At 16:49 in wunderground i have wrong temperature, wrong wind speed and
> wrong rain reading.
>
> In my home automation the wrong value was not recorded.

The Weather Underground uploader sends "live" data (every 48 seconds)
that is not stored in the data files. Is your home automation also using
"live" data?

> Is it possible that the fix only works for templating?
> Can this be enhanced to the other channels too?

If you remove data spikes in your user calibration module then it should
fix all such problems.

> This line perhaps seems to indicate the problem:
> pywws.weatherstation:live_data new ptr: 0013ac

No, that's a normal pointer increment.

[runge....@googlemail.com]

Hi Jim,

very strange.

No my homeautomation is reading the template file every 120 seconds.

So i do not see every value in my homeautomation.

This month i had 3 wrong datasets in wunderground, it was wind and rain speed.

I do not remove spikes in my data calibration.

I only calibrate temp and hum.

Can you tell my how i can remove spikes in a clever way?

My spikes are 2 - 3 in one month:

Rain with 7582 mm to 16900mm?

Wind i had one spile with 132 km/h to 296 km/h

Temp i had 53,7 C

Thanks and best regards,

Stefan

[Jim Easterbrook]

On 28/09/2023 23:14, 'runge....@googlemail.com' via pywws wrote:
>
> This month i had 3 wrong datasets in wunderground, it was wind and rain
> speed.
>
> I do not remove spikes in my data calibration.
> I only calibrate temp and hum.
>
> Can you tell my how i can remove spikes in a clever way?
> My spikes are 2 - 3 in one month:
> Rain with 7582 mm to 16900mm?

I assume the rain value drops back next reading. In which case just
ignore any jump of more than 20 or 30 mm, depending on how much rain you
could expect in your station's measuring interval.

> Wind i had one spile with 132 km/h to 296 km/h

Wind is harder to deal with as it's very spiky anyway. Ignoring any
value over a high number (150?) might help.

> Temp i had 53,7 C

The "remove_temperature_spikes" example module uses a median filter on
the last 30 minutes and removes the current temperature if it's more
than 2.0 degrees different. You could adjust it if needed.

https://github.com/jim-easterbrook/pywws/blob/master/src/pywws/examples/modules/remove_temperature_spikes.py

[runge....@googlemail.com]

Ah, ok thanks.

The example will help me a lot.

I will try to write some filtering.

Thanks and best regards,

Stefan

[runge....@googlemail.com]

Hi Jim,

ok i implemented the temperature spike example in my calib.py.

It seems to work. 

Can you please help me with the rain adjustment? 

How can i compare to the before value?

I found this in your examples but i am not sure how to use it.

How is this service name defined, also i do not have this context?

Or is there a way like in the temperature example with help of self.raw_data[result['idx']]`?

service_name = os.path.splitext(os.path.basename(__file__))[0]

 last_update = context.status.get_datetime('last update', service_name)
        if last_update:
            last_update = context.calib_data.nearest(last_update)
            self.last_rain = context.calib_data[last_update]['rain']
        else:
            self.last_rain = None

I would like to deal with Wind in a similar way. 

I would like to compare with the before value and if it shoots up over 100 km/h to the before value i filter it out.

Thanks for your help,

Stefan

[runge....@googlemail.com]

Oh, 

no also the temperature is not working for me.

The median = history[(len(history) - 1) / 2]  does not work since the result (len(history) - 1) / 2 is not a integer.

I get:

2023-10-03 00:23:01:pywws.livelog:list indices must be integers or slices, not float

Traceback (most recent call last):

  File "/usr/local/lib/python3.9/dist-packages/pywws/livelog.py", line 81, in live_log
    pywws.process.process_data(context)
  File "/usr/local/lib/python3.9/dist-packages/pywws/process.py", line 754, in process_data
    start = calibrate_data(context.params, context.raw_data, context.calib_data)
  File "/usr/local/lib/python3.9/dist-packages/pywws/process.py", line 547, in calibrate_data
    calib_data.update(calibgen(raw_data[start:]))

  File "/usr/local/lib/python3.9/dist-packages/pywws/filedata.py", line 433, in update
    for k in E:

  File "/usr/local/lib/python3.9/dist-packages/pywws/process.py", line 546, in calibgen
    yield calibrator.calib(data)
  File "/home/pi/Programs/weather/modules/calib.py", line 35, in calib
    median = history[(len(history) - 1) / 2]
TypeError: list indices must be integers or slices, not float

I would be happy if you can point me in the right direcion.

Thanks,

Stefan

[Jim Easterbrook]

On 02/10/2023 23:35, 'runge....@googlemail.com' via pywws wrote:
>
> no also the temperature is not working for me.
> The median = history[(len(history) - 1) / 2]  does not work since the
> result (len(history) - 1) / 2 is not a integer.

The examples were written a long time ago, before Python 3 was common.
Use '//' instead of '/' for integer division.

[Jim Easterbrook]

On 02/10/2023 23:22, 'runge....@googlemail.com' via pywws wrote:
>
> Can you please help me with the rain adjustment?
> How can i compare to the before value?

The temperature processing you already have shows access to nearby data.
For rain you just want the last valid value. Something like this:

if result['rain'] is not None:
before = self.raw_data.before(result['idx'])
while (result['idx'] - before < MINUTEx30
and self.raw_data['before']['rain'] is None):
before = self.raw_data.before(before)
if (self.raw_data['before']['rain'] is not None
and result['rain'] - self.raw_data['before']['rain'] > 500):
result['rain'] = None

This finds the most recent raw data with a value (but no more than 30
minutes old) then removes the current rain if it's more than 500 mm
greater. Adjust the values to suit your climate.

> I would like to deal with Wind in a similar way.
> I would like to compare with the before value and if it shoots up over
> 100 km/h to the before value i filter it out.

This should be very similar to the rain processing, but the previous
value is less useful as wind goes up and down a lot anyway. (The rain
should only ever increase, until it resets to zero.) You might need to
average the last few values and then remove spikes that are greater than
some multiple of the average.

[runge....@googlemail.com]

Hi Jim,

thanks a lot.

// it can be so easy ;-)

Temperature is now working fine.

I am sorry that i do not understand the index operations in python.

I still have a small problem with your rain example.

I get:

Traceback (most recent call last):

  File "/usr/local/lib/python3.9/dist-packages/pywws/livelog.py", line 81, in live_log
    pywws.process.process_data(context)

  File "/usr/local/lib/python3.9/dist-packages/pywws/process.py", line 754, in process_data
    start = calibrate_data(context.params, context.raw_data, context.calib_data)
  File "/usr/local/lib/python3.9/dist-packages/pywws/process.py", line 547, in calibrate_data
    calib_data.update(calibgen(raw_data[start:]))

  File "/usr/local/lib/python3.9/dist-packages/pywws/filedata.py", line 433, in update
    for k in E:

  File "/usr/local/lib/python3.9/dist-packages/pywws/process.py", line 546, in calibgen
    yield calibrator.calib(data)

  File "/home/pi/Programs/weather/modules/calib.py", line 59, in calib

    while (result['idx'] - before < MINUTEx30 and self.raw_data['before']['rain'] is None):

  File "/usr/local/lib/python3.9/dist-packages/pywws/filedata.py", line 215, in __getitem__
    raise TypeError("list indices must be %s" % (datetime))
TypeError: list indices must be <class 'datetime.datetime'>

Can you help me?

Best regards,

Stefan

[Jim Easterbrook]

On 03/10/2023 09:59, 'runge....@googlemail.com' via pywws wrote:
>
> I am sorry that i do not understand the index operations in python.
> I still have a small problem with your rain example.
> I get:

> TypeError: list indices must be <class 'datetime.datetime'>
>
> Can you help me?

I made a silly mistake in my suggestion. 'before' shouldn't be in
quotes. Try this:

if result['rain'] is not None:
before = self.raw_data.before(result['idx'])
while (result['idx'] - before < MINUTEx30
and self.raw_data[before]['rain'] is None):
before = self.raw_data.before(before)
if (self.raw_data[before]['rain'] is not None
and result['rain'] - self.raw_data[before]['rain'] > 500):
result['rain'] = None

[runge....@googlemail.com]

Ah i see.

Thank you so much for your help!

It seems my calib.py is now working.
I added some comments and logging and will monitor now if it filters just fine.

Here is my complete calibration routine calib.py if someone is interested:
from datetime import timedelta
import logging

from pywws.constants import SECOND

logger = logging.getLogger(__name__)

MINUTEx30 = timedelta(minutes=30)

class Calib(object):
    def __init__(self, params, raw_data):
        self.raw_data = raw_data
       
        # Get config offset as params
        self.pressure_offset = float(params.get('config', 'pressure offset'))
        self.hum_in_offset = float(params.get('config', 'hum_in offset'))
        self.hum_out_offset = float(params.get('config', 'hum_out offset'))
        self.hum_out_divider = float(params.get('config', 'hum_out divider'))
       
    def calib(self, raw):
        result = dict(raw)
        # calculate relative pressure
        result['rel_pressure'] = result['abs_pressure'] + self.pressure_offset
       
        # correct hum_in by offset
        if result['hum_in'] is not None:            
            result['hum_in'] = int(result['hum_in'] + self.hum_in_offset + 0.5)
       
        # correct hum_out by offset
        if result['hum_out'] is not None:
            result['hum_out'] = int(result['hum_out'] + self.hum_out_offset - result['hum_out'] / self.hum_out_divider + 0.5)        
       
        # get last 30 mins valid temperatures and remove spikes
        # temp_out
        if result['temp_out'] is not None:            
            history = []
            for data in self.raw_data[result['idx'] - MINUTEx30:
                                      result['idx'] + SECOND]:
                if data['temp_out'] is not None:
                    history.append(data['temp_out'])
            history.sort()
            if len(history) >= 4:
                median = history[(len(history) - 1) // 2]
                if abs(result['temp_out'] - median) > 1.5:
                    logger.warning('temp_out > 1.5 spike detected? %s %s', str(history), str(result['temp_out']))
                if abs(result['temp_out'] - median) > 2.0:
                    logger.warning('temp_out spike > 2.0 from median removed? %s %s', str(history), str(result['temp_out']))
                    result['temp_out'] = None
       
        # temp_in
        if result['temp_in'] is not None:
            history = []
            for data in self.raw_data[result['idx'] - MINUTEx30:
                                      result['idx'] + SECOND]:
                if data['temp_in'] is not None:
                    history.append(data['temp_in'])
            history.sort()
            if len(history) >= 4:
                median = history[(len(history) - 1) // 2]
                if abs(result['temp_in'] - median) > 1.5:
                    logger.warning('temp_in > 1.5 spike detected? %s %s', str(history), str(result['temp_in']))
                if abs(result['temp_in'] - median) > 2.0:
                    logger.warning('temp_in spike > 2.0 from median removed? %s %s', str(history), str(result['temp_out']))
                    result['temp_in'] = None
       
        # remove rain when jumps over 400 mm

        if result['rain'] is not None:
            before = self.raw_data.before(result['idx'])
            while (result['idx'] - before < MINUTEx30 and self.raw_data[before]['rain'] is None):
                before = self.raw_data.before(before)

            if (self.raw_data[before]['rain'] is not None and result['rain'] - self.raw_data[before]['rain'] > 400):
                logger.warning('rain spike > 400 removed? %s %s', str(self.raw_data[before]['rain']), str(result['rain']))
                result['rain'] = None
       
        # remove wind_gust when jumps over 120 km/h
        if result['wind_gust'] is not None:

            before = self.raw_data.before(result['idx'])

            while (result['idx'] - before < MINUTEx30 and self.raw_data[before]['wind_gust'] is None):
                before = self.raw_data.before(before)
            if (self.raw_data[before]['wind_gust'] is not None and result['wind_gust'] - self.raw_data[before]['wind_gust'] > 120):
                logger.warning('wind_gust spike > 120 removed? %s %s', str(self.raw_data[before]['wind_gust']), str(result['wind_gust']))
                result['wind_gust'] = None
       
        # remove wind_ave when jumps over 100 km/h
        if result['wind_ave'] is not None:

            before = self.raw_data.before(result['idx'])

            while (result['idx'] - before < MINUTEx30 and self.raw_data[before]['wind_ave'] is None):
                before = self.raw_data.before(before)
            if (self.raw_data[before]['wind_ave'] is not None and result['wind_ave'] - self.raw_data[before]['wind_ave'] > 100):
                logger.warning('wind_ave spike > 100 removed? %s %s', str(self.raw_data[before]['wind_ave']), str(result['wind_ave']))
                result['wind_ave'] = None
       
        return result

Best regards,
Stefan

[runge....@googlemail.com]

There was a small fix necessary because of None values in Rain.

Here is the new corrected calib coding if someone needs it:

                if abs(result['temp_out'] - median) > 4.0:
                    logger.warning('temp_out spike > 4.0 from median removed? %s %s', str(history), str(result['temp_out']))

                    result['temp_out'] = None
       
        # temp_in
        if result['temp_in'] is not None:
            history = []
            for data in self.raw_data[result['idx'] - MINUTEx30:
                                      result['idx'] + SECOND]:
                if data['temp_in'] is not None:
                    history.append(data['temp_in'])
            history.sort()
            if len(history) >= 4:
                median = history[(len(history) - 1) // 2]

                if abs(result['temp_in'] - median) > 4.0:
                    logger.warning('temp_in spike > 4.0 from median removed? %s %s', str(history), str(result['temp_out']))

                    result['temp_in'] = None
       
        # remove rain when jumps over 400 mm
        if result['rain'] is not None:
            before = self.raw_data.before(result['idx'])
            while (result['idx'] - before < MINUTEx30 and self.raw_data[before]['rain'] is None):
                before = self.raw_data.before(before)
            if (self.raw_data[before]['rain'] is not None and result['rain'] - self.raw_data[before]['rain'] > 400):
                logger.warning('rain spike > 400 removed? %s %s', str(self.raw_data[before]['rain']), str(result['rain']))

                result['rain'] = self.raw_data[before]['rain']

       
        # remove wind_gust when jumps over 120 km/h
        if result['wind_gust'] is not None:
            before = self.raw_data.before(result['idx'])
            while (result['idx'] - before < MINUTEx30 and self.raw_data[before]['wind_gust'] is None):
                before = self.raw_data.before(before)
            if (self.raw_data[before]['wind_gust'] is not None and result['wind_gust'] - self.raw_data[before]['wind_gust'] > 120):
                logger.warning('wind_gust spike > 120 removed? %s %s', str(self.raw_data[before]['wind_gust']), str(result['wind_gust']))
                result['wind_gust'] = None
       
        # remove wind_ave when jumps over 100 km/h
        if result['wind_ave'] is not None:
            before = self.raw_data.before(result['idx'])
            while (result['idx'] - before < MINUTEx30 and self.raw_data[before]['wind_ave'] is None):
                before = self.raw_data.before(before)
            if (self.raw_data[before]['wind_ave'] is not None and result['wind_ave'] - self.raw_data[before]['wind_ave'] > 100):
                logger.warning('wind_ave spike > 100 removed? %s %s', str(self.raw_data[before]['wind_ave']), str(result['wind_ave']))
                result['wind_ave'] = None
       
        return result