Missing Raindata after import

[David Schulz]

Hi,

i want to import some old weatherdata via csv file, but after the import, the raindata is missing. i tried the import with "rain = cumulative / discrete".

Here my config:

source = CSV

#######################################

[CSV]

    file = /etc/weewx/import/EXP201901A.CSV

    delimiter = ;

    interval = derive

    qc = True

    calc_missing = True

    ignore_invalid_data = True

    tranche = 1000

    UV_sensor = False

    solar_sensor = False

    raw_datetime_format = %d.%m.%Y %H:%M

    # rain = (discrete | cumulative)

    rain = cumulative

    wind_direction = -360,360

    [[FieldMap]]

        dateTime    = DatumZeit, unix_epoch

        outTemp     = Temp, degree_C

        outHumidity = Feuchte, percent

        barometer   = Luftdruck, hPa

        rain        = Regen, mm

        windSpeed   = Wind, km_per_hour

        windDir     = Richtung, degree_compass

        windGust    = Windboeen, km_per_hour

And here some sample data.

DatumZeit;Temp;Feuchte;Luftdruck;Regen;Wind;Richtung;Windboeen
01.01.2019 00:00;6,9;97;1026,6;0,2;4,7;242;9,7
01.01.2019 00:01;6,9;97;1026,6;0,2;4,7;251;7,9
01.01.2019 00:02;6,9;97;1026,6;0,2;4,7;250;7,9
01.01.2019 00:03;6,9;97;1026,6;0,2;4,7;245;4,7
01.01.2019 00:04;6,9;97;1026,6;0;4,7;250;7,9
01.01.2019 00:05;6,9;97;1026,5;0;4,7;234;7,9
01.01.2019 00:06;6,9;97;1026,5;0;4,7;232;11,2
01.01.2019 00:07;7;97;1026,5;0;4,7;234;4,7
01.01.2019 00:08;7;97;1026,5;0;4,7;241;7,9
01.01.2019 00:09;7;97;1026,5;0;4,7;240;6,5

[David Schulz]

wee_import --verbose --import-config=/etc/weewx/import/csv.conf
/usr/share/weewx/wee_import:719: DeprecationWarning: The distutils package is deprecated and slated for removal in Python 3.12. Use setuptools or check PEP 632 for potential alternatives
  from distutils.version import StrictVersion
Using WeeWX configuration file /etc/weewx/weewx.conf
Starting wee_import...
A CSV import from source file '/etc/weewx/import/EXP201901A.CSV' has been requested.
The following options will be used:
     config=/etc/weewx/weewx.conf, import-config=/etc/weewx/import/csv.conf
     source=/etc/weewx/import/EXP201901A.CSV, from=None, to=None
     dry-run=None, calc_missing=True, ignore_invalid_data=True
     tranche=1000, interval=derive, date/time_string_format=%d.%m.%Y %H:%M
     delimiter=';', rain=cumulative, wind_direction=[-360.0, 360.0]
     UV=False, radiation=False
Using database binding 'wx_binding', which is bound to database 'weewx.sdb'
Destination table 'archive' unit system is '0x01' (US).
Missing derived observations will be calculated.
All WeeWX UV fields will be set to None.
All WeeWX radiation fields will be set to None.
Starting import ...
Obtaining raw import data for period 1 ...
The following imported field-to-WeeWX field map will be used:
     source field 'DatumZeit' in units 'unix_epoch' --> WeeWX field 'dateTime'
     source field 'Temp' in units 'degree_C' --> WeeWX field 'outTemp'
     source field 'Feuchte' in units 'percent' --> WeeWX field 'outHumidity'
     source field 'Luftdruck' in units 'hPa' --> WeeWX field 'barometer'
     source field 'Regen' in units 'mm' --> WeeWX field 'rain'
     source field 'Wind' in units 'km_per_hour' --> WeeWX field 'windSpeed'
     source field 'Richtung' in units 'degree_compass' --> WeeWX field 'windDir'
     source field 'Windboeen' in units 'km_per_hour' --> WeeWX field 'windGust'
Raw import data read successfully for period 1.
Mapping raw import data for period 1 ...
Mapped 1440 records.
Raw import data mapped successfully for period 1.
Saving mapped data to archive for period 1 ...
1440 records identified for import.
Proceeding will save all imported records in the WeeWX archive.
Are you sure you want to proceed (y/n)? y
Unique records processed: 1440; Last timestamp: 2019-01-01 23:59:00 CET (1546383540)
Mapped data saved to archive successfully ""for period 1.
Calculating missing derived observations ...
Processing record: 1440; Last record: 2019-01-02 00:00:00 CET (1546383600)
Recalculating daily summaries...
Records processed: 1000; time: 2019-01-01 16:39:00 CET (1546357140)
Finished recalculating daily summaries
Finished calculating missing derived observations
Finished import
1440 records were processed and 1440 unique records imported in 40.49 seconds.
Those records with a timestamp already in the archive will not have been
imported. Confirm successful import in the WeeWX log file.

[David Schulz]

[gjr80]

You need to be very careful when using the NOAA format reports to verify the import of historical data, this is because only the current month and current year reports are updated each report cycle. Historical reports are not updated so will not reflect and changes to earlier data. The sample data you provided is from January 2019 so I am guessing you did not import into the current month and year on your system. Try deleting all of your NOAA format reports from your WeeWX system, they will all be re-generated on the next report cycle based on the current database data.

Gary 

[David Schulz]

Thanks for the info. But for testing I delete every time the whole DB and the NOAA reports. I have a Weewx test environment specifically for this.

[gjr80]

OK. Let's go straight to the horses head, what happens when you manually query the database archive table for the period concerned. Do each of the rows show the correct rainfall data or is it missing?

Gary

[David Schulz]

There should actually be values in there.

echo "SELECT dateTime, rain FROM archive" | sqlite3 /var/lib/weewx/weewx.sdb
1546297200|
1546297260|
1546297320|0.0
1546297380|0.0

DatumZeit;Temp;Feuchte;Luftdruck;Regen;Wind;Richtung;Windboeen
01.01.2019 00:00;6,9;97;1026,6;0,2;4,7;242;9,7
01.01.2019 00:01;6,9;97;1026,6;0,2;4,7;251;7,9

01.01.2019 00:02;6,9;97;1026,6;0;4,7;250;7,9
01.01.2019 00:03;6,9;97;1026,6;0;4,7;245;4,7

[David Schulz]

I think I should add a period to all values, not a comma.

[gjr80]

No, commas are fine. If the commas were the problem wee_import would be failing hard. Let me finish working through your data.

Gary

[David Schulz]

I think so, because now the graphs are also generated better. But feel free to test.

[David Schulz]

...Dots instead of comma...

[David Schulz]

I changed the config to...

rain = discrete

gjr80 schrieb am Sonntag, 15. Januar 2023 um 00:06:43 UTC+1:

Your import config file is telling wee_import the input rainfall field contains cumulative data, so wee_import uses the difference between the current row rainfall data and the previous row rainfall data to calculate the WeeWX field rain. For the first row there is no previous row so rain is set to None (also note that the 1January 2019 00:00 rain value is considered by WeeWX as containing 2018 data (ie data covering the period from 23:59:00 to 23:59:59 + 1 second on 31 December 2018). The second row is is the first record containing 2019 data. It should calculate the rain field as being 0.2 - 0.2 = 0.0, though from your database extract it appears to have been set to None. This could be a corner case involving the change of year, but as far as I know wee_import should handle change of year without problem. I will need to look further at the wee_import code. The third row will calculate rain as 0 - 0.2 = -0.2 which of course is a nonsense and I believe wee_import would assume there has been a counter overflow and set rain to 0.0. For the fourth row rain is calculated as 0 - 0 = 0.0 which is what you see. 

So other than the minor discrepancy at row two (for rain 0 and None are essentially treated the same in WeeWX output) I think wee_import is working as expected given the import data and import config settings. Analysis of a larger sample of data involving non-zero rainfall data may throw up more information.

Gary

On Sunday, 15 January 2023 at 08:40:05 UTC+10 schulz....@gmail.com wrote:

[gjr80]

I stand corrected. I see my confusion now. Support for non-period decimal points was added to wee_import some time ago. The impetus behind this was a user with Cumulus monthly log files that used a comma as the decimal separator. Similar support was added for WeatherDisplay, WeatherCat and CAV data files but it was only documented in the Cumulus, WeatherDisplay and WeatherCat instructions. Not in the CSV instructions. I will make sure this omission is fixed.

Gary 

[gjr80]

If your import rainfall data is a per row value and not a cumulative total then this is correct.

Gary

[David Schulz]

Jepp. Got it...

decimal = ','

[David Schulz]

thx