CSV Import Error

[rich T]

I'm trying to import a large CSV file into the database and I'm receiving the following error the the dry run.

pi@raspberrypi:/home/weewx/bin $ sudo ./wee_import --import-config=/home/pi/Desktop/clog/csv.conf --dry-run

Starting wee_import...

A CSV import from source file '/home/pi/Desktop/clog/archive_updated1.csv' has been requested.

Using database binding 'wx_binding', which is bound to database 'weewx.sdb'

Destination table 'archive' unit system is '0x01' (US).

Missing derived observations will be calculated.

All WeeWX UV fields will be set to None.

All WeeWX radiation fields will be set to None.

This is a dry run, imported data will not be saved to archive.

Traceback (most recent call last):

  File "./wee_import", line 703, in <module>

    main()

  File "./wee_import", line 653, in main

    source_obj.run()

  File "/home/weewx/bin/weeimport/weeimport.py", line 350, in run

    _mapped_data = self.mapRawData(_raw_data, self.archive_unit_sys)

  File "/home/weewx/bin/weeimport/weeimport.py", line 721, in mapRawData

    self.map[_field]['units'],

KeyError: 'units'

Here is the mapping in the csv.conf:

[[FieldMap]]

        dateTime    = timestamp, unix_epoch

        usUnits     = units

        interval    = interval

        barometer   = barometer, inHg

        pressure    = pressure

        altimeter   = altimeter

        inTemp      = inTemp

        outTemp     = outTemp, degree_F

        inHumidity  = inHumidity

        outHumidity = outHumidity, percent

        windSpeed   = windSpeed, mile_per_hour

        windDir     = windDir, degree_compass

        windGust    = windGust, mile_per_hour

        windGustDir = windGustDir, degree_compass

        rainRate    = rainRate, inch_per_hour

        rain        = rain, inch

        dewpoint    = dewpoint

        windchill   = windchill

        heatindex   = heatindex

        ET          =

        radiation   =

        UV          =

        rxCheckPercent = rxCheckPercent

Here is a small portion of the CSV file:

timestamp,units,interval,barometer,pressure,altimeter,inTemp,outTemp,inHumidity,outHumidity,windSpeed,windDir,windGust,windGustDir,rainRate,rain,dewpoint,windchill,heatindex,ET,radiation,UV,extraTemp1,extraTemp2,extraTemp3,soilTemp1,soilTemp2,soilTemp3,soilTemp4,leafTemp1,leafTemp2,extraHumid1,extraHumid2,soilMoist1,soilMoist2,soilMoist3,soilMoist4,leafWet1,leafWet2,rxCheckPercent,txBatteryStatus,consBatteryVoltage,hail,hailRate,heatingTemp,heatingVoltage,supplyVoltage,referenceVoltage,windBatteryStatus,rainBatteryStatus,outTempBatteryStatus,inTempBatteryStatus

04/09/2018 00:05,1,5,30.101,,,77.2,24.2,20,62,0,,0,,0,0,13.08666521,24.2,24.2,0,,,,,,,,,,,,,,,,,,,,99.9375,,,,,,,,,,,,

04/09/2018 00:10,1,5,30.102,,,77,24,20,63,0,,0,,0,0,13.25922371,24,24,0,,,,,,,,,,,,,,,,,,,,97.375,,,,,,,,,,,,

04/09/2018 00:15,1,5,30.108,,,76.9,23.8,20,63,0,,0,,0,0,13.06932323,23.8,23.8,0,,,,,,,,,,,,,,,,,,,,96.52083333,,,,,,,,,,,,

04/09/2018 00:20,1,5,30.11,,,76.7,23.7,20,63,0,,0,,0,0,12.97437126,23.7,23.7,0,,,,,,,,,,,,,,,,,,,,97.375,,,,,,,,,,,,

04/09/2018 00:25,1,5,30.112,,,76.6,23.5,20,63,0,,0,,0,0,12.78446384,23.5,23.5,0,,,,,,,,,,,,,,,,,,,,77.72916667,,,,,,,,,,,,

04/09/2018 00:30,1,5,30.11,,,76.4,23.5,21,63,0,,0,,0,0,12.78446384,23.5,23.5,0,,,,,,,,,,,,,,,,,,,,99.9375,,,,,,,,,,,,

04/09/2018 00:35,1,5,30.111,,,76.7,23.3,20,64,0,,3,225,0,0,12.95043454,23.3,23.3,0,,,,,,,,,,,,,,,,,,,,99.08333333,,,,,,,,,,,,

04/09/2018 00:40,1,5,30.106,,,76.9,23.3,20,64,0,,0,,0,0,12.95043454,23.3,23.3,0,,,,,,,,,,,,,,,,,,,,99.08333333,,,,,,,,,,,,

04/09/2018 00:45,1,5,30.11,,,77,23.1,20,64,0,,0,,0,0,12.76018685,23.1,23.1,0,,,,,,,,,,,,,,,,,,,,99.08333333,,,,,,,,,,,,

04/09/2018 00:50,1,5,30.114,,,77,23,20,65,0,,0,,0,0,13.01554258,23,23,0,,,,,,,,,,,,,,,,,,,,100,,,,,,,,,,,,

[Andrew Milner]

shouldn't it just be dateTime = timestamp and the format of timestamp be specified in raw_datetime_format ??

[rich T]

I could be wrong, but the raw_datetime_format is the datetime format in the CSV file (i.e. %m/%d%Y). The dateTime  = timestamp, unix_epoch  in the CSV Map basically means convert the timestamp column into  unix_epoch time format. 

[gjr80]

The error in the original post is due to one of the [[FieldMap]] mappings not having WeeWX unit name. You do not need to include mappings for all fields in the CSV source, only those you wish to import, but those you do include must be of the form:

WeeWX field = CSV field name, WeeWX unit name

You are correct in that the raw_datetime_format setting determines how the CSV date-time field is decoded. For the data shown I would expect something like (untested):

raw_datetime_format = "%m/%d/%Y %H:%M"

this assumes the date is in the format month/day/year, just transpose %m and %d if it id day/month/year.

Gary

[rich T]

Gary

I have the raw_datetime_format as "%m/%d/%Y %H:%M", but forgot to include it in previous post.  Now I receive the following error:

pi@raspberrypi:/home/weewx/bin $ sudo ./wee_import --import-config=/home/pi/Desktop/clog/csv.conf --dry-run

Starting wee_import...

A CSV import from source file '/home/pi/Desktop/clog/archive_updated1.csv' has been requested.

Using database binding 'wx_binding', which is bound to database 'weewx.sdb'

Destination table 'archive' unit system is '0x01' (US).

Missing derived observations will be calculated.

All WeeWX UV fields will be set to None.

All WeeWX radiation fields will be set to None.

This is a dry run, imported data will not be saved to archive.

**** Unable to map source data.

**** Field 'units' not found in source data.

**** Nothing done, exiting.

Thanks

Rich

[gjr80]

Rich,

Try deleting the mapping for usUnits from the field map. Long story short, it does not make much sense to be importing/converting a unit system code when units are specified/implied for all the fields. I know this isn't covered in the wee_import guide, will fix that up later.

Gary

[rich T]

Gary

Removed usUnits and interval from the csv.conf and it worked like a champ.

Thanks

Rich

[gjr80]

Just to close the loop on this, CSV imports with a usUnits field mapping are working fine under WeeWX 4.0.0b6. Documentation has been updated and all should be fine once WeeWX v4 is released.

Gary