wee_import fixes - Gary

[Nicolas CUVILLIER]

Gary

I am trying to import my historical data from underground.

If I understood you wrote the utility

I got some problem, and here is how I was able to work around the three issues I encountered:

Note that this are my first lines in Python, I'm more a node's guy, then double check my Python proposals...

in wee_import.py (v4.00.b12), line 1068, current_rain can be None, and the utility fail, and exit, please can you change the test:

if current_rain >= last_rain:

into

if current_rain is not None and current_rain >= last_rain:

in wee_import, line 795, the test fail, I had to remove this test:

WeeWX 4.0.0 or greater is required, found 4.0.0b12. Nothing done, exiting

in wuimport.py, line 297, can you replace the test, which is failing sometime with an exit:

(the expection does not initialise _date, not sure it is what is wanted ?)

if obs == 'epoch':

try:

_date = datetime.date.fromtimestamp(_flat_record['epoch'])

except ValueError:

_flat_record['epoch'] = _flat_record['epoch'] // 1000

by

if _flat_record['epoch'] > 1000000000000:

_flat_record['epoch'] = _flat_record['epoch'] // 1000

# do we need to use a try here ?

# what to put in _date if the function fail ?

_date = datetime.date.fromtimestamp(_flat_record['epoch'])

I am running the import for now, but it will take some time.

I'll send you more comments if I get other problem.

thanks

Nicolas

NB: Is it possible to directly push change proposals in GitHub ? 

[gjr80]

Nicolas,

Thanks for the feedback, I will get onto looking at each issue. Some initial comments below.

Gary

On Friday, 21 February 2020 18:54:29 UTC+10, Nicolas CUVILLIER wrote:

Gary

I am trying to import my historical data from underground.

If I understood you wrote the utility

I got some problem, and here is how I was able to work around the three issues I encountered:

Note that this are my first lines in Python, I'm more a node's guy, then double check my Python proposals...

in wee_import.py (v4.00.b12), line 1068, current_rain can be None, and the utility fail, and exit, please can you change the test:

if current_rain >= last_rain:

into

if current_rain is not None and current_rain >= last_rain:

Probably do need to catch current_rain == None, it really should not occur but if WU was to respond with an invalid rain value under some circumstances wee_import could take this as the value None.

 

in wee_import, line 795, the test fail, I had to remove this test:

WeeWX 4.0.0 or greater is required, found 4.0.0b12. Nothing done, exiting

Was set to 4.0.0 in anticipation of 4.0 release, will set it to an earlier beta.

 

in wuimport.py, line 297, can you replace the test, which is failing sometime with an exit:

(the expection does not initialise _date, not sure it is what is wanted ?)

if obs == 'epoch':

try:

_date = datetime.date.fromtimestamp(_flat_record['epoch'])

except ValueError:

_flat_record['epoch'] = _flat_record['epoch'] // 1000

by

if _flat_record['epoch'] > 1000000000000:

_flat_record['epoch'] = _flat_record['epoch'] // 1000

# do we need to use a try here ?

# what to put in _date if the function fail ?

_date = datetime.date.fromtimestamp(_flat_record['epoch'])

I need to look back at the code again but I believe this was handle the case where the timestamp received was in milliseconds (a la java) rather than seconds since epoch. _date is never used, rather we are looking for the exception that would be thrown if the timestamp was in milliseconds. Looking back at your weewx user thread it appears I need to trap OverflowError as well, odd as I distinctly remember testing that portion of the code. At this stage my preference is for the try..except as it seems more pythonic to me. Would need to better comment the code though. Like I said, need to look at the code.

I am running the import for now, but it will take some time.

I'll send you more comments if I get other problem.

thanks

Nicolas

NB: Is it possible to directly push change proposals in GitHub ? 

Best way to submits changes is via GitHub pull request, though this does require you to fork the WeeWX repo and submit the pull request via your fork.

 

[gjr80]

Below issues should be fixed now. The epoch issue was interesting. What I neglected on my post in weewx-user was that according to WU's new API documentation the epoch field in their station history API response is in seconds since epoch, given the OverflowError encountered it would appear that is not what they are returning. But given WUs recent track record that does not surprise me.

Gary

[Nicolas CUVILLIER]

Thanks Gary

On the test on current_rain, my main goal was more to not have wee_import to hangs...

I will create a new post on interval, because the values returned by WU are quite wrong...

But I need to load all my historical data : June to 2015 to Nov 2017 is done.

My PI is quite busy running days and nights :-)

I will pull service request in future

Nicolas

[gjr80]

The best way to have wee_import calculate interval is really dependent on the underlying WU data and at times the reliability of WU. There are three ways that wee_import calculates interval; firstly by the simple difference in timestamps between successive records, by use of a fixed number defined in the import config file or by use of the archive_interval setting in weewx.conf. If your WU data is patchy or WU is having a bad day (at times I have seen the WU API return a different set of results each time you query the same historical day) deriving interval from successive records may lead to many incorrect interval values. How bad the discrepancy will be and how often it occurs will depend on your data. In such cases if you know the data was posted every x minutes you can use value x in the import config file to set a fixed interval or if x happens to be the same as your archive_interval setting in weewx.conf you can use that value. This is covered in the WU interval section in the Utilities Guide.

Gary