randomly stops adding data

[YB322]

G'day everyone

I have been experiencing an issue for the last week, and so far have not been able to determine the cause. I was hoping someone may have some ideas.

I am running the interceptor driver (https://github.com/matthewwall/weewx-interceptor)

Weewx 4.1.1 

SQLite Database

OS Ubuntu 20.04 (Running as a VM on Hyper-V)

Station in a Pantech

At random times the system stops entering data into the database. From what I can see in the log the interceptor driver appears to be receiving the data but it isn't processing. The unit uploads data direct to Weather Underground and Weather Cloud, these services continue to receive data during the outage periods.

I have uploaded the log file here: https://pastebin.com/WgNKTKRT

The last successful database update in this log was Aug 29 08:00:38

If anyone has any ideas it would be very much appreciated.

Thanks

Adam 

[gjr80]

Hi,

I’d say you are suffering from a little temporal displacement:

Aug 29 08:00:38 weewx weewx[2328] DEBUG user.interceptor: raw packet: {'dateTime': 1598684400, 'usUnits': 1, 'rain_total': 0.0, 'temperature_in': 64.6, 'temperature_out': 47.7, 'dewpoint': 41.5, 'windchill': 46.8, 'humidity_in': 40.0, 'humidity_out': 79.0, 'wind_speed': 3.4, 'wind_gust': 5.8, 'wind_dir': 267.0, 'pressure': 30.65, 'barometer': 30.21, 'solar_radiation': 189.98, 'uv': 2.0, 'rain': 0.0}

Aug 29 08:00:38 weewx weewx[2328] DEBUG user.interceptor: mapped packet: {'dateTime': 1598684400, 'usUnits': 1, 'pressure': 30.65, 'barometer': 30.21, 'outHumidity': 79.0, 'inHumidity': 40.0, 'outTemp': 47.7, 'inTemp': 64.6, 'windSpeed': 3.4, 'windGust': 5.8, 'windDir': 267.0, 'radiation': 189.98, 'dewpoint': 41.5, 'windchill': 46.8, 'rain': 0.0, 'UV': 2.0}

Aug 29 08:00:38 weewx weewx[2328] INFO weewx.manager: Added record 2020-08-29 08:00:00 AEST (1598652000) to database 'weewx.sdb'

Aug 29 08:00:38 weewx weewx[2328] INFO weewx.manager: Added record 2020-08-29 08:00:00 AEST (1598652000) to daily summary in 'weewx.sdb'

What station do you have? Does it have a clock and if so is it set correctly? Might help to see a wee_debug report. When posting the wee_debug report check the content for sensitive info such as usernames, passwords and API keys before posting, wee_debug should obfuscate such data but it’s not perfect.

Gary

[YB322]

Thanks Gary

Appreciate you taking a look. I have stared at the log for the last week and not noticed that one.

I have pasted the debug here  https://pastebin.com/vrsDg0G6 

unit is a PanTech WH2900. It does have a time setting, although it does look correct.

Thanks

Adam

[gjr80]

Adam,

Definitely a time issue, what timezone are you in and is the timezone set correctly in you stations console?

Gary

[YB322]

Thanks Gary,

Timezone is Sydney\Australia (AEST - UTC+10). The station doesn't have a timezone setting just date and time, which does look correct.

It is really strange the setup had been working well for years and just started doing it this week. I created a new server with Ubuntu 20.04 as a troubleshooting step, but as you can see the issue came with it.

Really appreciate you taking the time.

Adam

[gjr80]

Thought the 9 hour difference seemed familiar but thought it was the other side of GMT. I'm in Brisbane.

That's interesting about no timezone because the WU format packet coming from your station includes a UTC format date-time string so the station must somehow know what timezone it is in. Actually, just did a google for Pantech 2900 on the desktop this time, is it one of these? If so do you use the WS View app to configure? I have only used the WS View app to configure a GW1000 and in the device settings on WS View there is a timezone.

Gary

[YB322]

Yes, that is the unit I am using. I use the WS View to configure it, the is no option to update the timezone, it may have been an option when I first setup, it was a while ago I don't remember if it was. I will try a factory rest on the station and re-configure and see if there is.

Thanks again for the help.

[YB322]

Just a quick update in case it helps someone.

I was reminded that there was a power outage around the same time this started happening. Not one to believe in coincidence, I restored the database from a backup of the day before the power outage, and filled the missing data using wee_import from WU.

So far it has been 2 days since I have experienced the issue. Possibly there was some kind of database corruption.

Gary - Thanks for your help, appreciate it.

Adam

[YB322]

After a few weeks of success the problem appears to have resurfaced. I have done some extra research, the weather station is randomly throwing records with a date in the future. See below:

A correct log (UTC 19:58 - 05:58 local time):

Sep 15 05:59:08 raspberrypi weewx[15243] DEBUG user.interceptor: POST: PASSKEY=XXXX&stationtype=EasyWeatherV1.5.3&dateutc=2020-09-14+19:58:12&tempinf=64.6&humidityin=57&baromrelin=30.095&baromabsin=30.535&tempf=56.3&humidity=99&winddir=29&windspeedmph=1.1&windgustmph=1.1&rainratein=0.000&eventrainin=0.000&dailyrainin=0.000&weeklyrainin=0.000&monthlyrainin=0.299&totalrainin=82.051&solarradiation=5.14&uv=0&model=WS2900

This is the next log which is incorrect (UTC 05:00 - 15:00 local time) should be 06:00 local:

Sep 15 06:00:11 raspberrypi weewx[15243] DEBUG user.interceptor: raw data: PASSKEY=XXX&stationtype=EasyWeatherV1.5.3&dateutc=2020-09-15+05:00:00&tempinf=64.6&humidityin=57&baromrelin=30.095&baromabsin=30.535&tempf=56.3&humidity=99&winddir=40&windspeedmph=0.7&windgustmph=1.1&rainratein=0.000&eventrainin=0.000&dailyrainin=0.000&weeklyrainin=0.000&monthlyrainin=0.299&totalrainin=82.051&solarradiation=4.77&uv=0&model=WS2900

From here it the time goes back to being correct, but it is ignored as it it before the last record.

It is clearly something with the station itself, but not sure why it is randomly throwing a time in the future, think it is some kind of bug in the latest firmware as I have now seen a couple of other posts with the same issue.

I am not an expert at Python but would it be possible to have a check and drop the packet if date time is more than an hour into the future? 

Thanks

Adam

[vince]

On Monday, September 14, 2020 at 5:03:23 PM UTC-7, YB322 wrote:

It is clearly something with the station itself, but not sure why it is randomly throwing a time in the future, think it is some kind of bug in the latest firmware as I have now seen a couple of other posts with the same issue.

I am not an expert at Python but would it be possible to have a check and drop the packet if date time is more than an hour into the future? 

If you have two things with different date+time, how would you know which one is reporting a timestamp in the future (or in the past) ?

 

[John Kline]

This is done already for catchup at startup time.  See engine.py “Reject any records with a timestamp in the future.”  It uses the current time and allows archive_delay seconds for clock drift.

On Sep 14, 2020, at 6:54 PM, vince <vince...@gmail.com> wrote:



--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/d8f135c6-cbf0-43d6-a76a-25a0d34e2a0do%40googlegroups.com.

[YB322]

Is this check only done at startup? I am seeing this happen during the ecowitt update sent via the station itself. The interceptor driver is receiving an update from the station with a timestamp in the future. Once it is processed no more updates are added to the database until after the time of the incorrect time stamp. After this point the normal operation continues and updates are added to the DB until another incorrect timestamp is received. 

Thanks for taking the time to respond, appreciate it.

Adam

[John Kline]

> Is this check only done at startup?

Yes. It’s a check only at startup during a catchup phase meant to insert archive records that might have been generated by the device since the time of the last archive record inserted in the weewx database.

My reply was a narrow one, responding only to how weewx might determine an archive record is in the future and, therefore, inappropriate.  I’m not endorsing the idea of simply ignoring the record and moving on.  If the underlying problem is that the driver can’t rely on station time, perhaps it should ignore station time and substitute current time for it. Some drivers have this option.

On Sep 15, 2020, at 12:24 AM, YB322 <ma...@adammccarthy.com> wrote:

Is this check only done at startup? I am seeing this happen during the ecowitt update sent via the station itself. The interceptor driver is receiving an update from the station with a timestamp in the future. Once it is processed no more updates are added to the database until after the time of the incorrect time stamp. After this point the normal operation continues and updates are added to the DB until another incorrect timestamp is received. 

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/7a029865-1d35-409a-98cb-b3581bd259d5n%40googlegroups.com.

[Tom Keffer]

There is an underlying assumption in WeeWX that, somewhere, there is a clock that can be depended on. It is simply impossible to write data acquisition software without a definitive time source. In the long run, strategies to get around that are just going to make things worse.

-tk

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/00232CEB-95DA-4114-842C-CD536A58DC89%40johnkline.com.