wee_import python error

[Ryan Stasel]

Hi All,

Trying to get data from meteotemplate imported into weewx, and I'm getting the following error:

Traceback (most recent call last):

  File "/usr/share/weewx/wee_import", line 900, in <module>

    main()

  File "/usr/share/weewx/wee_import", line 830, in main

    source_obj.run()

  File "/usr/share/weewx/weeimport/weeimport.py", line 368, in run

    _raw_data = self.getRawData(period)

  File "/usr/share/weewx/weeimport/csvimport.py", line 225, in getRawData

    _csv_reader = csv.DictReader(_clean_data, delimiter=self.delimiter)

  File "/usr/lib/python2.7/csv.py", line 79, in __init__

    self.reader = reader(f, dialect, *args, **kwds)

TypeError: "delimiter" must be an 1-character string

I'm assuming this is due to the following:

pi@raspi-server-misc:~ $ grep ",,," alldata.csv 

"2019-03-20 20:15:00",,,,,,,,,,,,,

"2019-03-20 20:40:00",,,,,,,,,,,,,

"2019-03-20 21:05:00",55.8,55.8,55.8,64.0,43.7,0.9,2.0,313.0,,,29.900,,53.8

"2019-03-22 20:25:00",,,,,,,,,,,,,

"2019-03-26 07:45:00",,,,,,,,,,,,,

"2019-03-26 07:55:00",37.6,37.6,37.6,97.0,36.9,0.0,0.0,223.0,,,30.070,,34.7

"2019-03-30 15:15:00",58.1,58.1,58.1,72.0,49.1,4.9,4.0,5.0,,,30.280,,55.2

"2019-04-02 08:45:00",52.2,52.2,52.2,97.0,51.3,0.0,0.0,145.0,,,29.770,,52.7

Not sure _why_ those are that way, but curious why wee_import wouldn't be able to handle empty entries. Or am I barking up the wrong tree and this is actually something else? 

I thought at first it was the second to last column being blank (think that's solar rad). 

If that's the case (the few lines that are just completely empty), I can just remove those lines. If not, then would love some assistance. The import is about 1.2M lines.

Thanks! 

[Ryan Stasel]

It's neither. Removed both the second to last column (solar rad) that was blank, and the dead lines. 

pi@raspi-server-misc:~ $ grep ",," alldata-mod2.csv | wc

      0       0       0

Still get the error. 

This is on weewx 4.0. 

Not sure what issue is. 

Thanks! 

-Ryan Stasel

[Ryan Stasel]

Commented out the delimiter line in the csv import config, since it says the default is comma, and that's what I'm using. Now I get: 

wee_import --import-config=/home/pi/csv.conf --dry-run

Using WeeWX configuration file /etc/weewx/weewx.conf

Starting wee_import...

A CSV import from source file '/home/pi/alldata-mod2.csv' has been requested.

Using database binding 'wx_binding', which is bound to database 'weewx.sdb'

Destination table 'archive' unit system is '0x01' (US).

Missing derived observations will be calculated.

All WeeWX UV fields will be set to None.

All WeeWX radiation fields will be set to None.

This is a dry run, imported data will not be saved to archive.

Starting dry run import ...

Traceback (most recent call last):

  File "/usr/share/weewx/wee_import", line 900, in <module>

    main()

  File "/usr/share/weewx/wee_import", line 830, in main

    source_obj.run()

  File "/usr/share/weewx/weeimport/weeimport.py", line 401, in run

    _mapped_data = self.mapRawData(_raw_data, self.archive_unit_sys)

  File "/usr/share/weewx/weeimport/weeimport.py", line 918, in mapRawData

    self.map[_field]['units'],

KeyError: 'units'

... which is due to a mistake on my part. Didn't have the unit set for dew point... 

[gjr80]

Just a comment on the original error/delimiter issue. The problem was nothing to do with the source data but rather the import config file; something other than a single character was being read form the import config file and passed to the DictReader. wee_import handles blank fields perfectly well. You are correct that the default delimiter is a comma, consequently removing the problem delimiter setting in your import config file fixed the delimiter error.

Gary

[Ryan Stasel]

That line was default. I added quotes around the comma and it also worked. the line does look like

delimiter = ,

Could it be reading the space prior to the comma as part of the string? 

[gjr80]

Hmm, rather poor form on my part. Whilst the delimiter option does not need to be in quotes per se, a comma certainly does otherwise when configobj parses the config file a single comma is interpreted as a list with two empty string elements. That explains the error. Intrigued to know how that got through testing, will have to dig into my test machine.

Will fix with a commit later this morning.

Gary