wee_import from WD logs fails

[Peter Süß]

Hello,
after using my Vantage pro 2 with Weather Display for about 15 years, I think now its time for a change. :)
I am making the first tests for transferring data, before switch the live system.

My problem, I cannot import the log data from wdisplay.

I'm getting this error message:

pi@WEEWX-DEMO:/etc/weewx $ wee_import --import-config=wd-import-test.conf --from=2020-12-01T00:00 --to=2020-12-31T23:59 --verbose

Using WeeWX configuration file /etc/weewx/weewx.conf

Starting wee_import...

Weather Display monthly log files in the '/tmp/wdisplay' directory will be imported

The following options will be used:

     config=/etc/weewx/weewx.conf, import-config=/etc/weewx/wd-import-test.conf

     from=2020-12-01T00:00, to=2020-12-31T23:59

     dry-run=None, calc_missing=True, ignore_invalid_data=True

     monthly logs are in METRIC units

     tranche=250, interval=1

     UV=True, radiation=True ignore extreme temperature and humidity=True

Using database binding 'wx_binding', which is bound to database 'weewx.sdb'

Destination table 'archive' unit system is '0x01' (US).

Missing derived observations will be calculated.

Observations timestamped after 2020-12-01 00:00:00 CET (1606777200) and up to and

including 2020-12-31 23:59:00 CET (1609455540) will be imported.

Starting import ...

Records covering multiple periods have been identified for import.

Period 1 ...

Obtaining raw import data for period 1 ...

Traceback (most recent call last):

  File "/usr/share/weewx/wee_import", line 903, in <module>

    main()

  File "/usr/share/weewx/wee_import", line 833, in main

    source_obj.run()

  File "/usr/share/weewx/weeimport/weeimport.py", line 369, in run

    _raw_data = self.getRawData(period)

  File "/usr/share/weewx/weeimport/wdimport.py", line 598, in getRawData

    _ts = "%s" % int(time.mktime(_tt))

OverflowError: mktime argument out of range

Any ideas? File content is looking ok.

It is happening with all time tests across this 15 years.

Here a month sample data.

Thanks for any ideas...
Peter

[gjr80]

Hi,

Nothing leaps out at me having a quick look at the output and log files you posted. I'll endeavour to have a look at this tomorrow (bit late here). Could I ask you to post the import config file you are using, /etc/weewx/wd-import-test.conf. There should be nothing sensitive in the file.

Gary

[Peter Süß]

Hi Gary,

thanks for your quick answer.
Please check the attached file. All I edited was the path to data and filenames. I limited it to only one type of file, with my last change.

No, problem when it takes time.

Thanks for checking out

Peter

[gjr80]

Peter,

I ran wee_import under WeeWX v4.4.0/python3 using the import config file and log files posted and the import processed without error:

gary@buster31:~$ /home/weewx/bin/wee_import --import-config=/var/tmp/wd-import-test.conf --from=2020-12-01T00:00 --to=2020-12-31T23:59 --verbose
Using WeeWX configuration file /home/weewx/weewx.conf

Starting wee_import...
Weather Display monthly log files in the '/tmp/wdisplay' directory will be imported
The following options will be used:

     config=/home/weewx/weewx.conf, import-config=/var/tmp/wd-import-test.conf

     from=2020-12-01T00:00, to=2020-12-31T23:59
     dry-run=None, calc_missing=True, ignore_invalid_data=True
     monthly logs are in METRIC units
     tranche=250, interval=1
     UV=True, radiation=True ignore extreme temperature and humidity=True
Using database binding 'wx_binding', which is bound to database 'weewx.sdb'
Destination table 'archive' unit system is '0x01' (US).
Missing derived observations will be calculated.

Observations timestamped after 2020-12-01 00:00:00 AEST (1606744800) and up to and
including 2020-12-31 23:59:00 AEST (1609423140) will be imported.

Starting import ...
Records covering multiple periods have been identified for import.

Obtaining raw import data for period 1 ...

Raw import data read successfully for period 1.
Mapping raw import data for period 1 ...
Mapped 44639 records.
Raw import data mapped successfully for period 1.
Saving mapped data to archive for period 1 ...
44639 records identified for import.
Proceeding will save all imported records in the WeeWX archive.
Are you sure you want to proceed (y/n)? y
Unique records processed: 44639; Last timestamp: 2020-12-31 23:59:00 AEST (1609423140)
Mapped data saved to archive successfully for period 1.
Calculating missing derived observations ...
Processing record: 44639; Last record: 2021-01-01 00:00:00 AEST (1609423200)
Recalculating daily summaries...
Records processed: 44000; time: 2020-12-31 13:20:00 AEST (1609384800)
Finished recalculating daily summaries
Finished calculating missing derived observations
Finished import
44639 records were processed and 44639 unique records imported in 249.23 seconds.
Those records with a timestamp already in the archive will not have been
imported. Confirm successful import in the WeeWX log file.

So I am confident there is nothing wrong with the import config file and log files posted. Granted it was a setup.py install but should not really make a difference to wee_import.

You said you had 15 years of WD data, when you did the import were there other older log files in /tmp/wdisplay? The reason I ask is that whilst you can limit the time frame over which wee_import imports data, all that does is apply a filter to the final processing/saving of records. wee_import will still go through and read all log files it finds starting at the oldest. So if you had 15 years of WD data in a directory and tried to import the most recent month all 15 years of log files will be read and processed in a limited manner. The initial processing of the raw import data involves assembling the individual date and time components from each row in the log. That appears to be where your import failed. When I wrote the WD component of wee_import I found a number of data inconsistencies in the WD log files I was using so I guess it is possible you may have similar inconsistencies. Have a look through the oldest WD log file set in /tmp/wdisplay and see if you can spot an obvious error/inconsistency in the date and time data. If you can't spot anything could you post the respective log files here. Alternatively, if you let me know what version of WeeWX you are running I can provide you with an instrumented version of wee_import to log details of the offending record/log.

Gary

[Peter Süß]

Thank you Gary.

This might be the error from my side. I just copied the whole bunch of logs into my import directory.

I will make subdir for every year, then I am going to test again.
Maybe going backward from 2020 to 2005 might be a good idea to find out my problems.

Version of my WeeWX is 4.4.0, running on a fresh Raspian image, installed on Friday.

Let me test tonight and I will report you back.

Peter

[Peter Süß]

Hey Gary,

its looking perfect. I just separated the logs for each year and right now I am testing the 6th year in a row, doing a quick dry-run.

The script is finding some duplicates, but it looks like every complete year from 01/01 to 12/31 should be imported into the database.

One more question from my side: Would you prefer a MySQL database with all this mass of data?

Thank you very much! 

Peter

[gjr80]

Hi Peter,

On Tuesday, 2 February 2021 at 06:03:36 UTC+10 peter.su...@gmail.com wrote:

Hey Gary,

its looking perfect. I just separated the logs for each year and right now I am testing the 6th year in a row, doing a quick dry-run.

The script is finding some duplicates, but it looks like every complete year from 01/01 to 12/31 should be imported into the database.

That is interesting, the screen output you posted originally clearly showed an error decoding a date-time, I am surprised that re-arranging the log files fixed that. The first person that trialled the WD import module was in a similar boat to you; they had many years of WD data and when they first tried to import the data they did it in one run and not only did it take many hours but wee_import eventually ground to a halt and crashed (the exact details escape me). So importing/processing a year at a time may be more manageable and safer.

The duplicates issue is not uncommon, the few suites of WD logs I have worked on all contained the odd duplicate date-time. wee_import will simply ignore the duplicates, the  only issue is when the data for each duplicate is different. Probably not going to make too much difference over 15 years of data at one minute intervals.

One more question from my side: Would you prefer a MySQL database with all this mass of data?

 

I think the consensus is that unless you have good reason to use MySQL/MariaDB stick with SQLite. SQLite is faster and simpler to backup/restore. SQLite will easy handle all of your 15 years of data.

 

Gary

[Peter Süß]

Hi Gary,

today my Raspberry 4 arrived, which will hopefully be my long-running weather computer.
I tried to import all my data on Pi3. It worked back for about 8 years, but then the performance crashed into hell.
The import of one year took about 1 week and was getting worse.

Now I'm starting it all over on a fresh install and report, what will be the result.

Peter

[gjr80]

Hi Peter,

Before you start it off again can you describe how you are doing the import; ie time period per import,  one big import, multiple smaller periods? How is you data organised on disc, one large directory with all file, year by year directories etc.

Gary

[Peter Süß]

Hi Gary,

I'm going to start it all over again this weekend, when having time. I organized all logdata from each year into a separate directory.

When importing it, I just edit the import path in the configfile and then start wee_import with --import-config=myimportconfig.con --verbose, and the let it run....

Here is some 6 years of data, some newer and some older. You can play around with it, if you like.
https://drive.google.com/drive/folders/1zwvlJSlGc0gtypuBaFIYq0V-jCg1e776?usp=sharing

I can just upload the whole bunch of it, if you want.
Thank you for your help!!!

Peter