4.6.2. update problem with wee_import

[jannep...@gmail.com]

Hi, I updated from 4.5 to 4.6.2 and my wee_import stop to next message:

I use WS6in1-driver with my Ventus.

.......................

Using WeeWX configuration file /etc/weewx/weewx.conf
Starting wee_import...
A CSV import from source file '/var/tmp/data.csv' has been requested.
Using database binding 'wx_binding', which is bound to database 'weewx.sdb'
Destination table 'archive' unit system is '0x11' (METRICWX).
Missing derived observations will be calculated.
This is a dry run, imported data will not be saved to archive.
Starting dry run import ...
Traceback (most recent call last):
  File "/usr/share/weewx/wee_import", line 899, in <module>
    main()
  File "/usr/share/weewx/wee_import", line 829, in main
    source_obj.run()
  File "/usr/share/weewx/weeimport/weeimport.py", line 372, in run
    _raw_data = self.getRawData(period)
  File "/usr/share/weewx/weeimport/csvimport.py", line 246, in getRawData
    self.map = self.parseMap('CSV', _csv_reader, self.csv_config_dict)
  File "/usr/share/weewx/weeimport/weeimport.py", line 635, in parseMap
    and _val['units'] not in weewx.units.USUnits.values():
AttributeError: 'ListOfDicts' object has no attribute 'values'

..........................................

What this does mean? Any help?

Is there easy way to get back to working configuration (4.5)?

regards

Janne

[gjr80]

Janne,

Could you please let me know whether WeeWX is being run under python 2 or 3? You can tell by looking at the WeeWX log during WeeWX startup (the python version is logged) or you can look at the first line of any of the wee_xxxxxx (eg wee_database) utilities in the /usr/share/weewx directory.

If yo cannot work it out please restart WeeWX and post a log extract showing the full WeeWX startup.

Gary

[jannep...@gmail.com]

hi

python 2. Here is log. When I start weewx, LOOP is working. I can get weather station info to terminal, but maybe not to database. I have a Ventus weather station and I download data every now and then to windows. From there I import the data in csv to weewx.

Janne

Feb 21 14:31:09 janne-Lat weewx[21011] INFO __main__: Initializing weewx version 4.6.2
Feb 21 14:31:09 janne-Lat weewx[21011] INFO __main__: Using Python 2.7.17 (default, Feb 27 2021, 15:10:58) #012[GCC 7.5.0]
Feb 21 14:31:09 janne-Lat weewx[21011] INFO __main__: Platform Linux-5.4.0-99-generic-x86_64-with-LinuxMint-19.3-tricia
Feb 21 14:31:09 janne-Lat weewx[21011] INFO __main__: Locale is 'fi_FI.UTF-8'
Feb 21 14:31:09 janne-Lat weewx[21011] INFO __main__: Using configuration file /etc/weewx/weewx.conf
Feb 21 14:31:09 janne-Lat weewx[21011] INFO __main__: Debug is 0
Feb 21 14:31:09 janne-Lat weewx[21011] INFO weewx.engine: Loading station type WS6in1 (user.ws6in1)
Feb 21 14:31:09 janne-Lat weewx[21011] INFO weewx.engine: StdConvert target unit is 0x11
Feb 21 14:31:09 janne-Lat weewx[21011] INFO weewx.wxservices: StdWXCalculate will use data binding wx_binding
Feb 21 14:31:09 janne-Lat weewx[21011] INFO weewx.engine: Archive will use data binding wx_binding
Feb 21 14:31:09 janne-Lat weewx[21011] INFO weewx.engine: Record generation will be attempted in 'hardware'
Feb 21 14:31:09 janne-Lat weewx[21011] INFO weewx.engine: Using archive interval of 300 seconds (specified in weewx configuration)
Feb 21 14:31:09 janne-Lat weewx[21011] INFO weewx.restx: StationRegistry: Registration not requested.
Feb 21 14:31:09 janne-Lat weewx[21011] INFO weewx.restx: Wunderground: Posting not enabled.
Feb 21 14:31:09 janne-Lat weewx[21011] INFO weewx.restx: PWSweather: Posting not enabled.
Feb 21 14:31:09 janne-Lat weewx[21011] INFO weewx.restx: CWOP: Posting not enabled.
Feb 21 14:31:09 janne-Lat weewx[21011] INFO weewx.restx: WOW: Posting not enabled.
Feb 21 14:31:09 janne-Lat weewx[21011] INFO weewx.restx: AWEKAS: Posting not enabled.
Feb 21 14:31:09 janne-Lat weewx[21011] INFO weewx.engine: 'pyephem' not detected, extended almanac data is not available
Feb 21 14:31:09 janne-Lat weewx[21011] INFO __main__: Starting up weewx version 4.6.2
Feb 21 14:31:09 janne-Lat weewx[21011] INFO weewx.engine: Using binding 'wx_binding' to database 'weewx.sdb'
Feb 21 14:31:09 janne-Lat weewx[21011] INFO weewx.manager: Starting backfill of daily summaries
Feb 21 14:31:09 janne-Lat weewx[21011] INFO weewx.manager: Daily summaries up to date
Feb 21 14:31:09 janne-Lat weewx[21011] INFO weewx.engine: Starting main packet loop.

[jannep...@gmail.com]

Hi, more info.

I fixed with ducktape the first problem by commenting out the line 635 from weeimport.py

            #                        and _val['units'] not in weewx.units.USUnits.values()

But this did not solve my import troubles. Now it looks that csv.conf and weewx.conf does not understand each other. csv.conf is asking for unkonwn units

   Using WeeWX configuration file /etc/weewx/weewx.conf
   Starting wee_import...
   A CSV import from source file '/var/tmp/data.csv' has been requested.
   Using database binding 'wx_binding', which is bound to database 'weewx.sdb'
   Destination table 'archive' unit system is '0x11' (METRICWX).
   Missing derived observations will be calculated.
   This is a dry run, imported data will not be saved to archive.
   Starting dry run import ...

   **** Unknown units 'uv_index' specified for field 'UV' in /var/tmp/csv.conf.
   **** Nothing done, exiting.

If I comment out UV, from my csv.conf wee_import finds next unknown unit... I also have tried to create group_uv to weewx.conf. The unkonwn units are uv_index, percent, degree_compass.

regards

Janne

[gjr80]

Would you also please provide a copy of the import config file you are using and a sample of the data (at least the header line and a few lines of data) you are trying to import.

I would also stop trying to patch the code to make it work; you will likely only continue to encounter other issues, most likely caused by your previous changes. Give me some time to work through the initial problem.

Gary

[jannep...@gmail.com]

Hi Gary

I sent you PM

Janne

[gjr80]

Thank you Janne, let me see what I can do.

Gary

[gjr80]

Janne,

You've encountered a bug that shows when wee_import is run under python 2 using a user defined field map. The fix required was not to wee_import but elsewhere in WeeWX and will appear in the next release (expected to be 4.6.3). In the meantime since you are using WeeWX v4.6.2 you can safely just download the patched file, replace the corresponding file on your system and wee_import should work fine. To download and install the patched file:

1. move aside the file to be patched:

$ mv /usr/share/weewx/weeutil/weeutil.py /usr/share/weewx/weeutil/weeutil_orig.py

2. download the patched file:

$ wget -P /usr/share/weewx/weeutil/ https://raw.githubusercontent.com/weewx/weewx/master/bin/weeutil/weeutil.py

3. stop WeeWX (if it is running) and perform your import

I did a test import using the patched file, your import config file and your CSV data. The raw_datetime_format setting in your import config file needs to be changed to match the format of the date-time field your are using in your source data, it needs to be set as follows:

raw_datetime_format = "%d.%m.%y %H:%M"

With this change your data imported correctly and with out error. Note that you will likely receive the following message during the import:

Warning: Import field 'gustDir' is mapped to WeeWX field 'windGustDir' but the
         import field 'gustDir' could not be found in one or more records.
         WeeWX field 'windGustDir' will be set to 'None' in these records.

This is just a warning that you have a mapped field for which there is no source data. It can be safely ignored.

Gary

[jannep...@gmail.com]

Gary,

Thanks!

I installed the patch, but it did not help. I also tried to uncomment my own fix from weeimport.py, (the fix which you don't recommend to do...) but that returned my first problem back.

My original date.csv has datetime format as in my csv.conf raw_datetime_format expect. Thanks anyway. 

Janne

[gjr80]

Janne,

"It doesn't work" is not very helpful. Please return weeimport.py to it's original 4.6.2 state, ensure you have the patched weeutil.py and then run the import again. Please post an exact copy of the console output including the command issued and responses/error traces.

Gary

[jannep...@gmail.com]

Gary,

yes, my mistake! I used the wrong version of weeimport.py while I was returning the original. I have many... Thanks for advising me. Sorry.

Now everything works and I could import data! Your patch was working great.

Thank you! You are really doing a good work!

best regards

Janne