Importing from cumulus give a python error

[Hyrules Hyrules]

Hi,

I just discovered Weewx and i'm in the process of converting from Cumulus and I'm trying to import

my monthly logs to my weewx database but after uploading the cumulus log files and setting up my cumulus.conf

file I get the following errow when I try a dryrun :

Starting wee_import...

Cumulus monthly log files in the '/var/tmp/cumulus' directory will be imported
The following options will be used:
     config=/etc/weewx/weewx.conf, import-config=/var/tmp/cumulus.conf
     from=None, to=None
     dry-run=True, calc_missing=True, ignore_invalid_data=True
     tranche=250, interval=derive
     UV=True, radiation=True
Using database binding 'wx_binding', which is bound to database 'weewx'
Destination table 'archive' unit system is '0x01' (US).
Missing derived observations will be calculated.
This is a dry run, imported data will not be saved to archive.
Obtaining raw import data for period 1...
Raw import data read successfully for period 1.
Mapping raw import data for period 1...
Traceback (most recent call last):
  File "/usr/bin/wee_import", line 834, in <module>
    main()
  File "/usr/bin/wee_import", line 784, in main
    source_obj.run()
  File "/usr/share/weewx/weeimport/weeimport.py", line 350, in run
    _mapped_data = self.mapRawData(_raw_data, self.archive_unit_sys)
  File "/usr/share/weewx/weeimport/weeimport.py", line 664, in mapRawData
    _temp = float(_row[self.map[_field]['field_name']].strip())
AttributeError: 'NoneType' object has no attribute 'strip'Enter code here...

I`ve been following the docs. Am I missing something ?

My configuration file and sample monthly cumulus log file has been attached to this message. Any help would be appreciated.

My setup :

Ubuntu server 19.04

Davis Vantage Pro 2

Thanks

[gjr80]

Hi,

No you are not missing anything, wee_import is expecting there to be more fields in your Cumulus log file than you currently have. I think I see why and a way ahead, just going to take a day or two to put together an updated Cumulus import module for wee_import.

I trust you are using WeeWX 3.9.2?

Gary

[gjr80]

Hmm, seems my initial diagnosis was somewhat off the mark and the problem is a simple case of having trapped the wrong error. The fix requires a simple change to one line of one of the wee_import .py files. If you are up for it, on your WeeWX machine open the file weeimport.py (weeimport.py will be in /home/weewx/bin/weeimport or /usr/share/weewx/weeimport depending on your WeeWX install type) in a text editor of your choice. Look for the highlighted line (it should be line 665):

                        try:

                            _temp = float(_row[self.map[_field]['field_name']].strip())

                        except TypeError:
                            # perhaps we have a None, so return None for our field
                            _temp = None

and change it to:

                        try:

                            _temp = float(_row[self.map[_field]['field_name']].strip())

                        except (TypeError, AttributeError):
                            # perhaps we have a None, so return None for our field
                            _temp = None

Save the file and then try your dry run import again. If you don't feel you can make the modification let me know and I will provide some download instructions.

One more thing. When I did a test import with the data file you provided I received a warning about multiple interval values. WeeWX will handle multiple different interval values; however, it could cause data integrity issues if the derived interval is wrong. Looking at your data file I see you have a one minute interval though there is a gap of some 7 odd days around 22 September. When you use interval = derive in your import config file this gap will likely result in an interval value equal to the gap for the first record after the gap or the record will be ignored altogether. If all of your data in at one minute intervals you can overcome the effects of this gap/possible incorrect interval value by setting interval = 1 in your import config file (note this will apply a one minute interval to all imported records, so if your historical data later changes to say a five minute interval you would need to split your imports).

Gary

[Hyrules]

Hi Gary,

Thanks for the quick reply. I'm not in a hurry. If you want to work on the cumulus import script of weewx i'll be around waiting if you want to test it with someone. I really want to import my cumulus file. It's possible there are gaps in the files it's not really a problem for me as it is a personal weather station so I don't mind gaps in the data. This will most likely happen at other places in my logs. I don't remember why but I know at some point I changed my intervals to 5 minutes as well so chances are I will need to split the import as you said. As for the version of weewx it's 3.9.2. I don't mind changing code in scripts of weewx either i'm a programmer as well I just don't know python (yet).

Thanks for the help.

Hyrules

--
You received this message because you are subscribed to a topic in the Google Groups "weewx-user" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/weewx-user/0KF7U8svBEA/unsubscribe.
To unsubscribe from this group and all its topics, send an email to weewx-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/a6282230-9da3-4521-b750-12881c12cd1c%40googlegroups.com.

--

Hyrules
-------------------------------------------------------------------
> Computer Analyst / Computer Tech. B.Sc.A , A+
-------------------------------------------------------------------

[gjr80]

OK, since you are more than capable of making the change I would just make the change I detailed and do the import. The fix has ben included in some other changes to that portion of wee_import but these will not appear until 4.0 is released.

Gary

To unsubscribe from this group and all its topics, send an email to weewx...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/a6282230-9da3-4521-b750-12881c12cd1c%40googlegroups.com.

[Hyrules Hyrules]

Yep it seems to be working fine. Tried it with a dryrun and no more errors. Thanks again for the help.

I'm glad I have discovered weewx. Currently toying with the Cheetah generator to reproduce some files I need.