Cumulus Import Fail

[FIA Ranch]

Did not see any recent Cumulus import issues.  Don't know how that is.

This failure looks beyond my scope of knowledge.

Following are the pertinent pieces and parts (I think) other than printing lines out of the reported python files.

Installed = WeeWX Version 4.9.1

This is an actual Cumulus data file line record (wrapped only here)
27/06/13,07:20,71.7,24,33.0,0.0,3.0,338,0.00,0.00,29.714,2.13,85.5,14,0.0,71.7,71.7,0.7,248,0.000,22.308,68.3,225,0.1,339,0.00


Definitions From Cumulus Report (Spaced For Clarity)

Cumulus.conf file entries
source = Cumulus
[Cumulus]
directory = /home/fiaranch/Files/cumulus_data
interval = x        #There are likely missing records
qc = True        #Assume invalid/extreme records
calc_missing = True    #Not sure about this one.
separator = '/'
delimiter = ','
decimal = '.'
ignore_invalid_data = True
tranche = 250
UV_sensor = True
solar_sensor = True
    [[Units]]
    temperature = degree_F
    pressure = mbar
    rain = inch
    speed = mile_per_hour


Dry Run Request
root@weather:~# wee_import --import-config=/home/fiaranch/Files/cumulus.conf --dry-run --verbose

Result
Using WeeWX configuration file /etc/weewx/weewx.conf
Starting wee_import...
Cumulus monthly log files in the '/home/fiaranch/Files/cumulus_data' directory will be imported
The following options will be used:
     config=/etc/weewx/weewx.conf, import-config=/home/fiaranch/Files/cumulus.conf
     from=None, to=None
     dry-run=True, calc_missing=True, ignore_invalid_data=True
     tranche=250, interval=x
     UV=True, radiation=True
Using database binding 'wx_binding', which is bound to database 'weewx.sdb'
Destination table 'archive' unit system is '0x01' (US).
Missing derived observations will be calculated.
This is a dry run, imported data will not be saved to archive.
Starting dry run import ...
Records covering multiple periods have been identified for import.
Period 1 ...
Obtaining raw import data for period 1 ...
Traceback (most recent call last):
  File "/usr/share/weewx/wee_import", line 899, in <module>
    main()
  File "/usr/share/weewx/wee_import", line 829, in main
    source_obj.run()
  File "/usr/share/weewx/weeimport/weeimport.py", line 382, in run
    _raw_data = self.getRawData(period)
  File "/usr/share/weewx/weeimport/cumulusimport.py", line 346, in getRawData
    _line = clean_row.replace(self.decimal, '.')
AttributeError: 'CumulusSource' object has no attribute 'decimal'
Error in sys.excepthook:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/apport_python_hook.py", line 153, in apport_excepthook
    with os.fdopen(os.open(pr_filename,
FileNotFoundError: [Errno 2] No such file or directory: '/var/crash/_usr_share_weewx_wee_import.0.crash'

Original exception was:
Traceback (most recent call last):
  File "/usr/share/weewx/wee_import", line 899, in <module>
    main()
  File "/usr/share/weewx/wee_import", line 829, in main
    source_obj.run()
  File "/usr/share/weewx/weeimport/weeimport.py", line 382, in run
    _raw_data = self.getRawData(period)
  File "/usr/share/weewx/weeimport/cumulusimport.py", line 346, in getRawData
    _line = clean_row.replace(self.decimal, '.')
AttributeError: 'CumulusSource' object has no attribute 'decimal'

Thanks in advance!

[gjr80]

Looks like there were a couple of typos when I added support for decimal points other than a period to the Cumulus and WD import classes. I believe I have fixed it, though am unable to test due to being away from home. You can try the fix by downloading and installing the updated file as follows:

1. move aside your existing /usr/share/weewx/weeimport/cumulusimport.py by renaming it, eg rename to cumulusimport_orig.py

2. download the updated file and save in place of the old cumulusimport.py:

$ wget -P /usr/share/weewx/weeimport https://raw.githubusercontent.com/weewx/weewx/master/bin/weeimport/cumulusimport.py

Try the import again, the 'decimal' attribute issue should be fixed.

Not applicable to you (unless you also wish to import from WD) but the WD import in v4.9.1 will also suffer from the same error. It was fixed in the same commit.

Gary

[FIA Ranch]

Thanks for the reply Gary.  It is always nice to know I am not the only one who's fingers tend to outrun their brain now and then!!  LOL

Getting closer.  That cleaned up the "decimal" issue.

Now it does not like the 'x' interval in the config file.  I chose that because I am sure it is more than likely that there are missing records.

Using WeeWX configuration file /etc/weewx/weewx.conf
Starting wee_import...
Cumulus monthly log files in the '/home/fiaranch/Files/cumulus_data' directory will be imported
The following options will be used:
     config=/etc/weewx/weewx.conf, import-config=/home/fiaranch/Files/cumulus.conf
     from=None, to=None
     dry-run=True, calc_missing=True, ignore_invalid_data=True
     tranche=250, interval=x
     UV=True, radiation=True
Using database binding 'wx_binding', which is bound to database 'weewx.sdb'
Destination table 'archive' unit system is '0x01' (US).
Missing derived observations will be calculated.
This is a dry run, imported data will not be saved to archive.
Starting dry run import ...
Records covering multiple periods have been identified for import.
Period 1 ...
Obtaining raw import data for period 1 ...

Raw import data read successfully for period 1.
Mapping raw import data for period 1 ...
**** Cannot derive 'interval'. Unknown 'interval' setting in /home/fiaranch/Files/cumulus.conf.
**** Nothing done, exiting.

Thanks again!!

David

[gjr80]

If you want to have wee_import set the interval field for all imported records to a fixed number then x needs to be an integer. All imported records will have the interval field set to this number. From the example cumulus import config file:

# x       - Use a fixed interval of x minutes for every record. This
#           setting is best used if the records to be imported are
#           equally based in time but there are some missing records.
#           This setting is recommended for WU imports.
# To import Cumulus records it is recommended that the interval setting
# be set to the value used in Cumulus as the 'data log interval'.

I believe Cumulus also has a setting where the data log interval is set adaptively, if your source data was collected by Cumulus with this adaptive data log interval then wee_import will not import this data correctly. I am still working through some proposed changes by another user to handle the import of such data.

Gary

[FIA Ranch]

That is about the dumbest brain-fart I have had in a long minute!!  Sheesh!!!!

Thanks.  I will rerun it when I get the chance.  I shortened the station interval when I installed WeeWX, but the old interval is obvious from the data files.

I will run it again when I get a few minutes.

Thanks again.

--
You received this message because you are subscribed to a topic in the Google Groups "weewx-user" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/weewx-user/0YG00SoPsIc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to weewx-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/0674be2c-7a22-46b7-a596-a195378723b8n%40googlegroups.com.

[FIA Ranch]

It is running, and running fine.  Up to 2014, so it will be crankin' for awhile.

Thanks again for your assistance.

David

On Sat, Nov 19, 2022, 1:30 PM gjr80 <gjrod...@gmail.com> wrote: