WeatherCat import Vs Daylight Savings Time

[ni...@kurand.net]

Has anyone successfully imported data from WeatherCat for months where Daylight Savings Time starts or ends? 

When I have tried this, I get the following type of error (example for Oct-2017):

Cannot derive 'interval' for record timestamp: 2017-10-29 01:00:00 BST (1509235200).

**** Raw data is not in ascending date time order.

Looking at the 7 years of data I have logged by WeatherCat, it appears that WeatherCat does not really handle DST transitions very well. Data is generally logging in local time.

For spring DST changes (in March for my UK based timezone), it seems usually to jump forward by two hours at 0100 going to 0300 rather than just one hour as you'd expect. It then tack on an hours worth of data for day 01 at the end of the month. This gives rise to the above error when the end of the month's data is processed. However, for some reason, in my data for 2014, it does only jump forward one hour as expected and there was no extra data logged at the end of the month. 

For autumn DST changes (in October in UK timezone), it seems to jump back an hour at 0200, so it repeats log entries for hour 0100 again. This gives rise to the above error.

It's all a bit of a mess really; I'm not sure if its just my WeatherCat data, or the way I configured it with Adaptive interval set true or something else. 

Does anyone have a fix for this, or do I need to work something out.

[gjr80]

If you are using 'derive' to derive the archive interval from the imported data then the imported records must be in ascending timestamp/date-time order. This is because the derived archive interval is calculated by taking the difference between the timestamp of the current record and the timestamp of the previous record. If the calculated interval is negative it indicates the records being imported are not in timestamp order and the calculated archive interval cannot be trusted. The import will abort.

This should not present a problem at the start of a daylight saving period where clocks jump forward (though the archive interval would be incorrectly calculated for the first record after daylight saving begins). It would certainly cause a problem at the end of a period of daylight saving where clocks jump back in time.

So what to do? WeeWX date-time objects are timezone naive so managing daylight saving is near impossible. I would suggest removing the data to be imported with the duplicate date-times (which set of duplicates I leave up to you) and re-trying the import. You might also want to consider using a fixed interval (if your data does indeed have a fixed interval) rather than deriving it from the data. FWIW, the vantage driver has a similar issue in that each year when daylight saving ends the driver receives archive records with duplicate timestamps, in this case the situation is handled slightly more gracefully in that WeeWX ignores the duplicate timestamped records with a 'PRIMARY KEY must be unique' error. wee_import is somewhat less forgiving.

Gary

[kurand]

Thanks for confirming what I understood from weeimport.py and weathercatimport.py.

I am using "derived' for the import because I originally configured WeatherCat with Adaptive interval set true so I have data logged at varying  rates from 1 minute all the way up to 30+ minute intervals. 

I think in this case, weathercatimport.py needs to use timezone aware date-time objects during its construction of dateTime else data will become misplaced and DST changes will fail as noted. Once this is accomplished, then something needs to get added to deal with the apparent bug in WeatherCat where transitioning from standard time to daylight saving time in spring causes it to skip an extra hour forward sometimes. It looks to me like this can all be isolated in weathercatimport.py's getRawData function. 

I'm curious to know if anyone else has run into this problem, or if it is something specific to my WeatherCat data. Or perhaps WeatherCat import has been used in anger. 

[vhel...@gmail.com]

Hello,

I have succesfully imported Weathercat data from about 7 years. That was about year and half ago. I remember, that I didn't have any problems with daylight saving. And In Finland we have summer time , which is +1 hour. 

I have data logged in 1 minute intervals, and I used X minutes option in import config.

[gjr80]

On Sunday, 28 August 2022 at 01:37:33 UTC+10 vhel...@gmail.com wrote:

Hello,

I have succesfully imported Weathercat data from about 7 years. That was about year and half ago. I remember, that I didn't have any problems with daylight saving. And In Finland we have summer time , which is +1 hour. 

I have data logged in 1 minute intervals, and I used X minutes option in import config.

 

In your case the fixed one minute interval effectively bypasses the ascending date-time order check. I suspect you will find an hours worth of import data will have been discarded each time daylight saving ended and clocks were wound back. This will have been recorded in the log, but unfortunately wee_import is not made aware of the issue (and hence does not report it) as the WeeWX archive management machinery is used for saving imported records to archive it that machinery only logs success or failure and does not otherwise report back. 

Gary

[gjr80]

On Thursday, 25 August 2022 at 23:54:11 UTC+10 kurand wrote:

Thanks for confirming what I understood from weeimport.py and weathercatimport.py.

I am using "derived' for the import because I originally configured WeatherCat with Adaptive interval set true so I have data logged at varying  rates from 1 minute all the way up to 30+ minute intervals. 

I think in this case, weathercatimport.py needs to use timezone aware date-time objects during its construction of dateTime else data will become misplaced and DST changes will fail as noted. Once this is accomplished, then something needs to get added to deal with the apparent bug in WeatherCat where transitioning from standard time to daylight saving time in spring causes it to skip an extra hour forward sometimes. It looks to me like this can all be isolated in weathercatimport.py's getRawData function. 

I'm curious to know if anyone else has run into this problem, or if it is something specific to my WeatherCat data. Or perhaps WeatherCat import has been used in anger. 

 Yes 'aware' datetime objects would be preferred; however, their use raises a number of other issues. Programatically, there is the potential for conflict with the rest of the WeeWX code base which uses naive datetime objects only. Also, we try to avoid external dependencies unless absolutely necessary so hence we need to work within the standard libraries shipped with python. Then there is the issue of daylight saving cutover times often being politically driven, so who knows what impact that may have - we may be solving one issue only to create others.

That being said there are two issues at play here. Firstly, an out of date-time order record caused the entire import to abort. The reasoning for this behaviour was the assumption that a randomly ordered batch of records was provided for import, this is probably too onerous and we could (should be able to) back this off to simply ignore the miscreant record. This should be a simple change. Secondly, the issue of daylight saving. I am happy to have a look at wee_import to see what can be done to support daylight saving using the existing included python libraries. To this end some relevant WeatherCat data files would help, would you be able to provide example WeatherCat files covering both the start and end of daylight saving? If you want you can reply privately to this post and include them. thanks.

Gary

[kurand]

Fortunately, I think use of aware datetime objects in this case can be isolated to weathercatimport.py because the current implementation converts each entry's datatime to a unix epoch timestamp prior inserting it into the row for later processing. For my purposes, I think Python 3.9 and later provides all the necessary datetime and zoneinfo functions to handle the various cases.

As you pointed out, with the current implementation in weathercatimport.py, you can silently(ish) loose the entries during the autumn transition hour from DST to standard time because the later entries with duplicate timestamp in standard time (fold=1) will interact with the entries using the same timestamp in DST (fold=0). However, for WeatherCat data with Adaptive interval set, some of the later standard time entries' timestamps may not co-inside with the earlier DST entries' timestamps, so you could end up with a mix of standard time and DST time entries across that hour. 

Furthermore, with the current implementation, approximately half of a year's worth of entries will be entered into the database using the wrong time (and even date in some edge cases) because processing for the entire year is done using the local time DST setting when the actual import takes place. There will also be an hour long hole (or maybe two) in the data for each year when transitioning from standard time to DST in the spring. Of course, you may not consider these issues to be particularly problematic for historical data...

The import problems seems to be easily divisible into two parts which need separate solutions: 1) transition fro standard time to DST in spring; and 2) transition from DST to standard time in the fall. I have been working on changes to weathercatimport.py to address the transition from DST to standard time (2). It is the easier of the two transitions. I thought I had competed the necessary changes, but when tested it with all of my data (7 years worth), I came across yet another anomaly in my WeatherCat data. I have at least a pair of rows in Oct-2017 sometime after the transition to standard time that have identical timestamps. I presume that WeatherCat has some kind of rounding error, or does not account for common unix timer variations/clock jitter/resolution or something. I'll work on fixing this issue, and then continue with 1). Once complete, I'll mail you the changes (along with sample data) and you can decide what to do with them: either incorporate them into the codebase; modify them to meet Python backward compatibility rules; or dump them in the nearest bin. There clearly aren't too many folks with this kind of historic WeatherCat database to import.

Nick