Problem with wee_import utility (rain field)

[Remy Lavabre]

Good morning,
I am using the wee_import.py utility to integrate AWEKAS data into my weewx database.

command executed: sudo wee_import --config=/etc/weewx/weewx.conf --import-config=/home/pi/Documents/Bresser7in1/csv-Bresser.conf --dry-run --verbose

My CSVFile :

dateTime,outTemp,outHumidity,dewpoint,appTemp,barometer,windchill,windSpeed,windDir,windGust,rain,rainRate,UV,radiation,inTemp,inHumidity,extraTemp1
2023-08-26 16:35:00,23.6,61.0,15.7,18.4,1010.6,23.6,3.2,214,3.9,6.9,0,1.1,101.0,26.1,53.0,24.9,22.0



My wee_import configuration file:

source = CSV

##############################################################################

[CSV]
    file = /home/pi/Documents/Bresser7in1/data5.csv
    delimiter = ','
    decimal = '.'
    interval = conf
    qc = True
    calc_missing = True
    ignore_invalid_data = True
    tranche = 250
    UV_sensor = True
    solar_sensor = True
    raw_datetime_format = %Y-%m-%d %H:%M:%S

    rain = cumulative
    wind_direction = 0,360


    [[FieldMap]]
        dateTime    = dateTime, unix_epoch
        barometer   = barometer, hPa
        inTemp      = inTemp, degree_C
        outTemp     = outTemp, degree_C
        inHumidity  = inHumidity, percent
        outHumidity = outHumidity, percent
        windSpeed   = windSpeed, km_per_hour
        windDir     = windDir, degree_compass
        windGust    = windGust, km_per_hour
        rainRate    = rainRate, mm_per_hour
        rain        = rain, mm
        dewpoint    = dewpoint, degree_C
        windchill   = windchill, degree_C
        radiation   = radiation, watt_per_meter_squared
        UV          = UV, uv_index
        extraTemp1  = extraTemp1, degree_C
        extraTemp2  = extraTemp2, degree_C
        extraTemp3  = extraTemp3, degree_C
        extraTemp4  = extraTemp4, degree_C
        extraTemp6  = extraTemp6, degree_C
        extraHumid1 = extraHumid1, percent
        extraHumid2 = extraHumid2, percent
        extraHumid3 = extraHumid3, percent
        extraHumid4 = extraHumid4, percent



Response from WeeWX :

Using WeeWX configuration file /etc/weewx/weewx.conf
Starting wee_import...
A CSV import from source file '/home/pi/Documents/Bresser7in1/data5.csv' has been requested.
The following options will be used:
     config=/etc/weewx/weewx.conf, import-config=/home/pi/Documents/Bresser7in1/csv-Bresser.conf
     source=/home/pi/Documents/Bresser7in1/data5.csv, from=None, to=None
     dry-run=True, calc_missing=True, ignore_invalid_data=True
     tranche=250, interval=conf, date/time_string_format=%Y-%m-%d %H:%M:%S
     delimiter=',', rain=cumulative, wind_direction=[0.0, 360.0]
     UV=True, radiation=True
Using database binding 'wx_binding', which is bound to database 'weewx.sdb'
Destination table 'archive' unit system is '0x01' (US).
Missing derived observations will be calculated.
This is a dry run, imported data will not be saved to archive.
Starting dry run import ...
Obtaining raw import data for period 1 ...
The following imported field-to-WeeWX field map will be used:
     source field 'dateTime' in units 'unix_epoch' --> WeeWX field 'dateTime'
     source field 'barometer' in units 'hPa' --> WeeWX field 'barometer'
     source field 'inTemp' in units 'degree_C' --> WeeWX field 'inTemp'
     source field 'outTemp' in units 'degree_C' --> WeeWX field 'outTemp'
     source field 'inHumidity' in units 'percent' --> WeeWX field 'inHumidity'
     source field 'outHumidity' in units 'percent' --> WeeWX field 'outHumidity'
     source field 'windSpeed' in units 'km_per_hour' --> WeeWX field 'windSpeed'
     source field 'windDir' in units 'degree_compass' --> WeeWX field 'windDir'
     source field 'windGust' in units 'km_per_hour' --> WeeWX field 'windGust'
     source field 'rainRate' in units 'mm_per_hour' --> WeeWX field 'rainRate'
     source field 'rain' in units 'mm' --> WeeWX field 'rain'
     source field 'dewpoint' in units 'degree_C' --> WeeWX field 'dewpoint'
     source field 'windchill' in units 'degree_C' --> WeeWX field 'windchill'
     source field 'radiation' in units 'watt_per_meter_squared' --> WeeWX field 'radiation'
     source field 'UV' in units 'uv_index' --> WeeWX field 'UV'
     source field 'extraTemp1' in units 'degree_C' --> WeeWX field 'extraTemp1'
     source field 'extraTemp2' in units 'degree_C' --> WeeWX field 'extraTemp2'
     source field 'extraTemp3' in units 'degree_C' --> WeeWX field 'extraTemp3'
     source field 'extraTemp4' in units 'degree_C' --> WeeWX field 'extraTemp4'
     source field 'extraTemp6' in units 'degree_C' --> WeeWX field 'extraTemp6'
     source field 'extraHumid1' in units 'percent' --> WeeWX field 'extraHumid1'
     source field 'extraHumid2' in units 'percent' --> WeeWX field 'extraHumid2'
     source field 'extraHumid3' in units 'percent' --> WeeWX field 'extraHumid3'
     source field 'extraHumid4' in units 'percent' --> WeeWX field 'extraHumid4'
Raw import data read successfully for period 1.
Mapping raw import data for period 1 ...
Mapped 1 records.
Raw import data mapped successfully for period 1.
1 records identified for import.
Unique records processed: 1; Last timestamp: 2023-08-26 16:55:00 CEST (1693061700)
Finished dry run import
1 records were processed and 1 unique records would have been imported.

Everything seems to be correct.
However, when I run the command without the --dry run option, ALL the fields are correctly entered in the weewx database except SYSTEMATIC "rain" and "rainRate" which always remain at 0!
I don't understand why these two fields are never updated in weewx database. Here for example rain is at 6.9 (since 00H00) and in the database rain remains at 0 (option cumulative actived). Same thing for rainRate! And these are the only two fields that pose a problem!
Is it a wee_import.py bug? Does anyone have a solution... Thanks a lot!

[gjr80]

If you are importing a single record and using rain = cumulative the imported rain and rainRate values will always be zero, this is expected behaviour. The reason being that WeeWX records rainfall data on a per archive interval basis; in other words each archive record contains the rainfall that was recorded in that archive period, for example in the last five minutes if using a five minute archive interval. When the import source rainfall data is cumulative or a running total wee_import calculates the rain field value as the difference in this cumulative or running total value between successive records. The If there is only one record then this difference cannot be calculated and hence rain is recorded as zero. Likewise for rainRate; if there is no rain data then there can be no rainRate.

The use of the rain config option in a CSV import config file is covered here in the Utilities Guide.

If you must import a single record at a time you must convert your source rain data to be a per-record value and import using rain = discrete. wee_import was not designed with the regular import of single records in mind, rather it was designed for the bulk import of days/months/years of data. 

Gary

[Remy Lavabre]

Thank you Gary for your very quick and detailed response. Nice of you!
I had read the documentation on the Weewx utilities guide but had not understood it in the sense you indicate...
So I modified my Python program to calculate in the csv file the amount of water that fell over the 5 minute period and modified the configuration file with rain=discrete instead of rain=cumulative.

Thanks again Gary. I'm waiting for the next rain and confirming the resolution of the problem.

Unfortunately no precipitation is expected locally for the next few days... We'll have to be patient! ;-)
Thanks a lot again Gary

[Remy LAVABRE]

Hi Gary,

I followed your advice:

1/ I now have in the rain field of my csv file the amount of rain that fell over the previous 5 minutes.
2/ I changed the config file so that rain=discrete is now the config value

Unfortunately, the integration of the csv file in the weewx database with this new configuration did not solve the problem, the rain and rainRate fields systematically remain at zero while the integration of all the other fields works wonderfully!
Do you think there is something wrong or is it a weewx bug?

My new configuration file :
    rain = discrete
    wind_direction = 0,360

    [[FieldMap]]
        dateTime    = dateTime, unix_epoch
        barometer   = barometer, hPa
        inTemp      = inTemp, degree_C
        outTemp     = outTemp, degree_C
        inHumidity  = inHumidity, percent
        outHumidity = outHumidity, percent
        windSpeed   = windSpeed, km_per_hour
        windDir     = windDir, degree_compass
        windGust    = windGust, km_per_hour
        rainRate    = rainRate, mm_per_hour
        rain        = rain, mm
        dewpoint    = dewpoint, degree_C
        windchill   = windchill, degree_C
        radiation   = radiation, watt_per_meter_squared
        UV          = UV, uv_index
        extraTemp1  = extraTemp1, degree_C

Example of CSV file with rain & rainRate :

dateTime,outTemp,outHumidity,dewpoint,appTemp,barometer,windchill,windSpeed,windDir,windGust,rain,rainRate
2023-08-28 18:30:00,17.6,66.0,11.3,13.8,1009.5,17.6,3.8,0,6.1,0.3,3.6
2023-08-28 18:35:00,16.0,80.0,12.6,13.7,1009.4,16.0,3.8,1,7.1,0.2,6.0


Result of the test :

sudo wee_import --config=/etc/weewx/weewx.conf --import-config=/home/pi/Documents/Bresser7in1/csv-Bresser.conf --dry-run --verbose

Using WeeWX configuration file /etc/weewx/weewx.conf
Starting wee_import...
A CSV import from source file '/home/pi/Documents/Bresser7in1/data5.csv' has been requested.
The following options will be used:
     config=/etc/weewx/weewx.conf, import-config=/home/pi/Documents/Bresser7in1/csv-Bresser.conf
     source=/home/pi/Documents/Bresser7in1/data5.csv, from=None, to=None
     dry-run=True, calc_missing=True, ignore_invalid_data=True
     tranche=250, interval=conf, date/time_string_format=%Y-%m-%d %H:%M:%S

     delimiter=',', rain=discrete, wind_direction=[0.0, 360.0]

Unique records processed: 1; Last timestamp: 2023-08-29 14:30:00 CEST (1693312200)

Finished dry run import
1 records were processed and 1 unique records would have been imported.

Rémy LAVABRE

--
You received this message because you are subscribed to a topic in the Google Groups "weewx-user" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/weewx-user/nWTJnx2UNlU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to weewx-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/0a5739b7-436c-4ccf-b170-576969ae6357n%40googlegroups.com.

[gjr80]

I just setup a WeeWX instance with your import config file in your first post but using rain = discrete and a csv data file using the two records in your last post. I ran wee_import using these files and --dry-run and --verbose and received somewhat different output to what you posted, in my case there were warnings of missing fields in the csv data file:

$ sudo /home/weewx/bin/wee_import --import-config=/var/tmp/csv.conf --verbose
/home/weewx/bin/wee_import:719: DeprecationWarning: The distutils package is deprecated and slated for removal in Python 3.12. Use setuptools or check PEP 632 for potential alternatives
  from distutils.version import StrictVersion
Using WeeWX configuration file /home/weewx/weewx.conf
Starting wee_import...
A CSV import from source file '/var/tmp/remy.csv' has been requested.

The following options will be used:

     config=/home/weewx/weewx.conf, import-config=/var/tmp/csv.conf
     source=/var/tmp/remy.csv, from=None, to=None
     dry-run=None, calc_missing=True, ignore_invalid_data=True

     tranche=250, interval=conf, date/time_string_format=%Y-%m-%d %H:%M:%S

     delimiter=',', rain=discrete, wind_direction=[-360.0, 360.0]

     UV=True, radiation=True
Using database binding 'wx_binding', which is bound to database 'weewx.sdb'
Destination table 'archive' unit system is '0x01' (US).
Missing derived observations will be calculated.

Starting import ...

Obtaining raw import data for period 1 ...
The following imported field-to-WeeWX field map will be used:
     source field 'dateTime' in units 'unix_epoch' --> WeeWX field 'dateTime'
     source field 'barometer' in units 'hPa' --> WeeWX field 'barometer'
     source field 'inTemp' in units 'degree_C' --> WeeWX field 'inTemp'
     source field 'outTemp' in units 'degree_C' --> WeeWX field 'outTemp'
     source field 'inHumidity' in units 'percent' --> WeeWX field 'inHumidity'
     source field 'outHumidity' in units 'percent' --> WeeWX field 'outHumidity'
     source field 'windSpeed' in units 'km_per_hour' --> WeeWX field 'windSpeed'
     source field 'windDir' in units 'degree_compass' --> WeeWX field 'windDir'
     source field 'windGust' in units 'km_per_hour' --> WeeWX field 'windGust'
     source field 'rainRate' in units 'mm_per_hour' --> WeeWX field 'rainRate'
     source field 'rain' in units 'mm' --> WeeWX field 'rain'
     source field 'dewpoint' in units 'degree_C' --> WeeWX field 'dewpoint'
     source field 'windchill' in units 'degree_C' --> WeeWX field 'windchill'
     source field 'radiation' in units 'watt_per_meter_squared' --> WeeWX field 'radiation'
     source field 'UV' in units 'uv_index' --> WeeWX field 'UV'
     source field 'extraTemp1' in units 'degree_C' --> WeeWX field 'extraTemp1'

Raw import data read successfully for period 1.
Mapping raw import data for period 1 ...

Warning: Import field 'inTemp' is mapped to WeeWX field 'inTemp' but the
         import field 'inTemp' could not be found in one or more records.
         WeeWX field 'inTemp' will be set to 'None' in these records.
Warning: Import field 'inHumidity' is mapped to WeeWX field 'inHumidity' but the
         import field 'inHumidity' could not be found in one or more records.
         WeeWX field 'inHumidity' will be set to 'None' in these records.
Warning: Import field 'radiation' is mapped to WeeWX field 'radiation' but the
         import field 'radiation' could not be found in one or more records.
         WeeWX field 'radiation' will be set to 'None' in these records.
Warning: Import field 'UV' is mapped to WeeWX field 'UV' but the
         import field 'UV' could not be found in one or more records.
         WeeWX field 'UV' will be set to 'None' in these records.
Warning: Import field 'extraTemp1' is mapped to WeeWX field 'extraTemp1' but the
         import field 'extraTemp1' could not be found in one or more records.
         WeeWX field 'extraTemp1' will be set to 'None' in these records.
Mapped 2 records.

Raw import data mapped successfully for period 1.

Saving mapped data to archive for period 1 ...
2 records identified for import.
Proceeding will save all imported records in the WeeWX archive.
Are you sure you want to proceed (y/n)? y
Unique records processed: 2; Last timestamp: 2023-08-28 18:35:00 AEST (1693211700)
Mapped data saved to archive successfully ""for period 1.
Calculating missing derived observations ...
Processing record: 2; Last record: 2023-08-29 00:00:00 AEST (1693231200)
Recalculating daily summaries...

Finished recalculating daily summaries
Finished calculating missing derived observations
Finished import
2 records were processed and 2 unique records imported in 2.87 seconds.
Those records with a timestamp already in the archive will not have been
imported. Confirm successful import in the WeeWX log file.

These warnings are inconsequential for your import, but they make me wonder if you have posted the correct output for the command/config/data specified? Importantly, in my test the data was imported as expected:

$ sqlite3 /home/weewx/archive/weewx.sdb
SQLite version 3.37.2 2022-01-06 13:25:41
Enter ".help" for usage hints.
sqlite> SELECT dateTime,datetime(dateTime, 'unixepoch', 'localtime'),outTemp,rain,rainRate FROM archive ORDER BY dateTime DESC LIMIT 6;
1693340400|2023-08-30 06:20:00|32.1130389185491|0.0|0.0
1693340100|2023-08-30 06:15:00|32.3124634933352|0.0|0.0
1693339800|2023-08-30 06:10:00|32.5203063775446|0.0|0.0
1693339500|2023-08-30 06:05:00|32.6532104624026|0.0|0.0
1693211700|2023-08-28 18:35:00|60.8|0.0078740157480315|0.2362204722
1693211400|2023-08-28 18:30:00|63.68|0.0118110236220472|0.14173228332
sqlite>

(note the fields are in the native units (US customary) for my database ie rain and rainRate values are in inches and inches/hour)

Also, the command and response specified in your posts include the use of the --dry-run command line option; if --dry-run is used data will never be saved to database - that is the point of --dry-run. Have you run the same command without --dry-run? If not I suggest you try that and if it does not import correctly post the exact command line used and the exact output received. If you have run wee_import without --dry-run please post the corresponding wee_import output.

One other thing, how are you determining that the data was not imported? The most reliable way is to directly query the database, if you are relying upon WeeWX generated output this can be unreliable/untimely due to caching,  delays in generation etc.

Gary

[Remy LAVABRE]

Thank you very much for your patience Gary,

Indeed I did not provide you with all the data from the CSV file in order to simplify.
However, the fields were in the correct order with the correct wording.

Regarding the --dry-run command, I actually remove it for integration. This is why all my fields are saved in the database with the timestamp (except for rain).

Actually, to make it simple for you, I have a weather station that doesn't have a USB output. The data is sent to AWEKAS. So in python, for 5 minutes I retrieve this data regularly, make an average for certain fields (temperatures, hygrometry etc.) and for others take the maximum (rain rate) during these 5 minutes. For rain, the station gives me the total since 00:00 which I calculate into rain over the 5 minute time slot.
After 5 minutes, I generate a CSV which will be integrated into the weewx database with Wee_import.
Weewx recalculates all the data, generates the html page and the csv file is deleted.
We start again every 5 minutes.

The other day after the changes you told me about there was a very light rain.
I had this rain over two 5-minute time periods with a correct rain rate in the CSV file.
However in the weewx database nothing...

This morning, I did another test manually and put a value of 2mm in the rain field and 4mm/h in the rainRate. I manually tried the wee_import command (exactly the same one that is in the crontab) and I had a correct import this time!
So the problem really comes from me and not from weewx...

Do you think that a rainRate at 0 with a rain value in the rain field could cause a problem during the integration and set both fields to 0?

I will first try to artificially simulate a rain in python in the CSV file and see if this value now integrates correctly or not automatically with the crontab.
Secondly, I will monitor the actual conditions (rain is normally forecast this weekend...) and keep you informed.
Thank you Gary for helping me out and giving me the solution! ;-)

Rémy LAVABRE

You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/80122a9d-9540-41a1-b6b4-77f50fb4448dn%40googlegroups.com.

[gjr80]

Some comments below.

Gary

On Wednesday, 30 August 2023 at 20:02:01 UTC+10 remy.l...@gmail.com wrote:

Actually, to make it simple for you, I have a weather station that doesn't have a USB output. The data is sent to AWEKAS. So in python, for 5 minutes I retrieve this data regularly, make an average for certain fields (temperatures, hygrometry etc.) and for others take the maximum (rain rate) during these 5 minutes. For rain, the station gives me the total since 00:00 which I calculate into rain over the 5 minute time slot.
After 5 minutes, I generate a CSV which will be integrated into the weewx database with Wee_import.
Weewx recalculates all the data, generates the html page and the csv file is deleted.
We start again every 5 minutes.

 

You have a complex, automated system for using wee_import. I would be looking very carefully at what data wee_import is actually being fed, it might not be what you think it is.

 

Do you think that a rainRate at 0 with a rain value in the rain field could cause a problem during the integration and set both fields to 0?

I don't think this is likely. wee_import will import data if the field map is valid and consistent, the field map fields are found in the source data and the field value is (in this case) numeric. 0 is nothing special, just another number. The import processing of one imported field has no impact on any other imported field.

[Remy Lavabre]

Thanks Gary,

My fields, including for rain, incremented automatically yesterday for the first time!
Your solution was the right one and I still don't understand why the previous time the rain was not taken into account in my weewx database....

However, thank you for your explanations which helped me progress and allowed me to resolve my problem!