wee_import 4.00.b12 errors

[Nicolas CUVILLIER]

Hello,

I have a running weewx (3.9.2) connected to a VP2, working perfectly.

This weewx is sending data to wunderground (stationID = IRIVIRED4)

This system is running since a month and is part of a local network of weather stations in south Indian Ocean.

I prefer to stay on the same configuration than the other people in the network.

I would like to import my historical data from wunderground (3 years)

As with the new API from WU, I need to use wee_import v4, my idea is to setup a second system,

to get the historical data in mySql, then transfer the missing records to the running system, as the

structure of the archive table did not change.

I setuped a test weewx 4.0.b12 on a PI4, running with the simulator with mySql,

no errors in the log, data is well simulated...

I  put the information for wunderground in weewx.conf (probably not needed for the wee_import)

(station, password, api_key) I used the same parameters as the one used in the working 3.9.2

  (BTW is wee_import use WU parameter from weewx.conf ???)

I put station_id and api_key in the util/import/wu-example.conf
Other parameters are with their default value

Is it needed to have weewx deamon running when you are using wee_import ?

(I tried with and without...., same result)

I ran the command:
wee_import --dry-run --config=/home/weewx/weewx.conf --import-config=/home/test/weewx-4.0.0b12/util/import/wu.conf --date=2019-01-01 --verbose

and I got the error:

Starting wee_import...
Traceback (most recent call last):
  File "/usr/bin/wee_import", line 834, in <module>
    main()
  File "/usr/bin/wee_import", line 783, in main wlog)
  File "/usr/share/weewx/weeimport/weeimport.py", line 320, in sourceFactory log)
  File "/usr/share/weewx/weeimport/wuimport.py", line 82, in __init__ log)
  File "/usr/share/weewx/weeimport/weeimport.py", line 155, in __init__ 'schema': 'schemas.wview.schema'})
  File "/usr/share/weewx/weewx/manager.py", line 1031, in open_manager_with_config default_binding_dict=default_binding_dict)
  File "/usr/share/weewx/weewx/manager.py", line 1002, in get_manager_dict_from_config manager_dict['schema'] = weeutil.weeutil._get_object(schema_name)
  File "/usr/share/weewx/weeutil/weeutil.py", line 1107, in _get_object
    mod = __import__(module)
ImportError: No module named wview_extended

What did I make wrong ?

Thanks

Nicolas - Mauritius

[Thomas Keffer]

I suspect you are using weewx v3.9 code to run a V4.0 configuration file. A new version of WeeWX can run old configuration files, but not the other way around. 

You are running wee_import without qualifying which version you want. The system found your old version, and is attempting to use that. Try again, but this time with the fully qualified path to wee_import. Something like:

/home/weewx/bin/wee_import --dry-run --config=/home/weewx/weewx.conf --import-config=/home/test/weewx-4.0.0b12/util/import/wu.conf --date=2019-01-01 --verbose

-tk

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/34f8818b-a7b6-44d3-9fc1-bdd22318135e%40googlegroups.com.

[Nicolas CUVILLIER]

Thanks Thomas

Probably I made a mistake during the upgrade.

I removed weewx, and re-install directly the v4.0.b12, and it just work :-)

I have 3 years more or less to import. Is it better to do it in one shot, or by smaller part, like every six months ?

Do you have any experience on this ?

Thanks

Nicolas

[Thomas Keffer]

No, I don't. Gary wrote the wee_import utility. 

There's nothing to be lost by just trying it!

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/97428b0f-073d-4320-afc9-0e668ca17d15%40googlegroups.com.

[Nicolas CUVILLIER]

I got an error in wee_import in the bloc

if obs == 'epoch':

try:

_date = datetime.date.fromtimestamp(_flat_record['epoch'])

except ValueError:

_flat_record['epoch'] = _flat_record['epoch'] // 1000

The error was:

Traceback (most recent call last):

  File "/home/weewx/bin/wee_import", line 896, in <module>

    main()

  File "/home/weewx/bin/wee_import", line 826, in main

    source_obj.run()

  File "/home/weewx/bin/weeimport/weeimport.py", line 367, in run

    _raw_data = self.getRawData(period)

  File "/home/weewx/bin/weeimport/wuimport.py", line 299, in getRawData

    _date = datetime.date.fromtimestamp(_flat_record['epoch'])

OverflowError: timestamp out of range for platform time_t

(I am running wee_import --from=2015-06-25 --to=2015-12-31)

I removed the bloc and replace with the line in the expect branch:

_flat_record['epoch'] = _flat_record['epoch'] // 100

I am not sure about exactly the consequences of what I did, the only point is

that I forced the expect because of an error, and remove the blocking error

 

Does it make sense, or will my data be corrupted ?

Is there a better way to have this fixed ?

I can provide in a personal message (email?) my api_key if you want to reproduce

Nicolas

======================

Le jeudi 20 février 2020 19:12:21 UTC+4, Thomas Keffer a écrit :

No, I don't. Gary wrote the wee_import utility. 

There's nothing to be lost by just trying it!

On Thu, Feb 20, 2020 at 6:54 AM Nicolas CUVILLIER <nico...@gmail.com> wrote:

Thanks Thomas

Probably I made a mistake during the upgrade.

I removed weewx, and re-install directly the v4.0.b12, and it just work :-)

I have 3 years more or less to import. Is it better to do it in one shot, or by smaller part, like every six months ?

Do you have any experience on this ?

Thanks

Nicolas

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.

To unsubscribe from this group and stop receiving emails from it, send an email to weewx...@googlegroups.com.

[Thomas Keffer]

Gary will have to answer that one. Gary?

To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/8d0f42b8-854f-42ae-8403-e7c2176d3843%40googlegroups.com.

[Nicolas CUVILLIER]

I spent more time in the src code, and I have other questions to Gary.

Let me switch to the weewx-development group, and close this post.

Nicolas

[gjr80]

Just to wrap this up. Python 3.3 and later raises an OverflowError rather than a ValueError if datetime.date.fromtimestamp() encounters a timestamp that is out of range for values supported by the platform C localtime() or gmtime() functions. Whilst wee_import was developed on python 3.5, this was using a non-armhf 64 bit machine and hence did not strike the same range of values. A quick check on a RPi3 showed the same OverflowError behaviour. Interesting error and easily fixed. Fixed at commit 5355bfb.

Gary