Problem with weewx SQL database

[Remy Lavabre]

Good morning,

My WeeWX.sdb database works perfectly. I removed all unnecessary fields with sudo wee_database --drop-columns=XXX,YYY,ZZZ which worked perfectly.

I renamed the field "heatindex1", from the base to "wetbulbTemp" with the command: sudo wee_database --rename-column=heatindex1 --to-name=wetbulbTemp. Flawless.

The problem is when I try to insert CSV data into the database using the wee_import.py utility.
All fields work on import except the one renamed to "wetbulbTemp"...
If I remove the wetbulbTemp field from the import configuration file (*.conf), the import is done without any problem.


With the field renamed "wetbulbTemp", I get the following message:

Using WeeWX configuration file /etc/weewx/weewx.conf
Starting wee_import...
A CSV import from source file '/home/pi/Documents/Bresser7in1/data5.csv' has been requested.
The following options will be used:
      config=/etc/weewx/weewx.conf, import-config=/home/pi/Documents/Bresser7in1/csv-Bresser.conf
      source=/home/pi/Documents/Bresser7in1/data5.csv, from=None, to=None
      dry-run=True, calc_missing=True, ignore_invalid_data=True
      slice=250, interval=conf, date/time_string_format=%Y-%m-%d %H:%M:%S
      delimiter=',', rain=cumulative, wind_direction=[0.0, 360.0]
      UV=True, radiation=True
Using database binding 'wx_binding', which is bound to database 'weewx.sdb'
Destination table 'archive' unit system is '0x01' (US).
Missing derived observations will be calculated.
This is a dry run, imported data will not be saved to archive.
Starting dry run import ...
Obtaining raw import data for period 1 ...
The following imported field-to-WeeWX field map will be used:
      source field 'dateTime' in units 'unix_epoch' --> WeeWX field 'dateTime'
      source field 'barometer' in units 'hPa' --> WeeWX field 'barometer'
      source field 'inTemp' in units 'degree_C' --> WeeWX field 'inTemp'
      source field 'outTemp' in units 'degree_C' --> WeeWX field 'outTemp'
      source field 'inHumidity' in units 'percent' --> WeeWX field 'inHumidity'
      source field 'outHumidity' in units 'percent' --> WeeWX field 'outHumidity'
      source field 'windSpeed' in units 'km_per_hour' --> WeeWX field 'windSpeed'
      source field 'windDir' in units 'degree_compass' --> WeeWX field 'windDir'
      source field 'windGust' in units 'km_per_hour' --> WeeWX field 'windGust'
      source field 'rainRate' in units 'mm_per_hour' --> WeeWX field 'rainRate'
      source field 'rain' in units 'mm' --> WeeWX field 'rain'
      source field 'dewpoint' in units 'degree_C' --> WeeWX field 'dewpoint'
      source field 'windchill' in units 'degree_C' --> WeeWX field 'windchill'
      source field 'radiation' in units 'watt_per_meter_squared' --> WeeWX field 'radiation'
      source field 'UV' in units 'uv_index' --> WeeWX field 'UV'
      source field 'extraTemp1' in units 'degree_C' --> WeeWX field 'extraTemp1'
      source field 'extraTemp2' in units 'degree_C' --> WeeWX field 'extraTemp2'
      source field 'extraTemp3' in units 'degree_C' --> WeeWX field 'extraTemp3'
      source field 'extraTemp4' in units 'degree_C' --> WeeWX field 'extraTemp4'
      source field 'extraHumid1' in units 'percent' --> WeeWX field 'extraHumid1'
      source field 'extraHumid2' in units 'percent' --> WeeWX field 'extraHumid2'
      source field 'extraHumid3' in units 'percent' --> WeeWX field 'extraHumid3'
      source field 'extraHumid4' in units 'percent' --> WeeWX field 'extraHumid4'
      source field 'wetbulbTemp' in units 'degree_C' --> WeeWX field 'wetbulbTemp'
Raw import data read successfully for period 1.
Mapping raw import data for period 1 ...
Traceback (most recent call last):
   File "/usr/share/weewx/wee_import", line 900, in <module>
     hand()
   File "/usr/share/weewx/wee_import", line 830, in main
     source_obj.run()
   File "/usr/share/weewx/weeimport/weeimport.py", line 421, in run
     _mapped_data = self.mapRawData(_raw_data, self.archive_unit_sys)
   File "/usr/share/weewx/weeimport/weeimport.py", line 981, in mapRawData
     weewx.units.obs_group_dict[_field])
   File "/usr/lib/python3.9/collections/__init__.py", line 941, in __getitem__
     return self.__missing__(key) # support subclasses that define __missing__
   File "/usr/lib/python3.9/collections/__init__.py", line 933, in __missing__
     raiseKeyError(key)
KeyError: 'wetbulbTemp'


I couldn't find any help in the WeeWX documentation on this.
What is happening with this renamed field and what is the solution to integrate the CSV data of this field into my modified weewx.sdb database?

Thank you...

[gjr80]

Your problem is that wee_database --add_column/--rename_column makes all necessary changes to the database, but it does not make any change to the WeeWX unit system; in other words WeeWX knows there is a field wetbulbTemp in the database but does not know if it is a temperature, wind speed or rainfall. To fix this you need to add wetbulbTemp to the WeeWX unit system, the usual approach for this is to add a couple of lines of code to user/extensions.py (refer to Assigning a unit group in the Customization Guide), in your case something like the following should work (untested):

import weewx.units
weewx.units.obs_group_dict['wetbulbTemp'] = 'group_temperature'

For the WeeWX daemon this will take effect when WeeWX is restarted, but for the purposes of wee_import it should work straight away.

Gary

[Remy LAVABRE]

Thanks Gary for your response.
I added :

import weewx.units
weewx.units.obs_group_dict['wetbulbTemp'] = 'group_temperature'

in file /usr/share/weewx/weecfg/extension.py

It didn't change anything about the problem when using wee_import.py, I still get the same error.
I specify that I did not create a field ['wetbulbTemp'] in the WeeWX database but simply renamed ['heatindex1'] to ['wetbulbTemp'].

The WeeWX daemon does not enter (in my opinion) into the problem since to use the wee_import.py utility it is strongly advised to stop it... What I do.

I saw in the documentation what you are talking about where it is a question of creating a user/electricity.py service, but unfortunately did not quite understand where this electricity.py file went... :-(

Rémy LAVABRE

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/dbf90df9-0cee-465d-89b1-86b2c77076ebn%40googlegroups.com.

[jterr...@gmail.com]

you should edit /usr/share/weewx/user/extension.py  and not /usr/share/weewx/weecfg/extension.py

[gjr80]

Should have spelt out the full path/file:

/usr/share/weewx/user/extensions.py

Didn’t say the WeeWX daemon has anything to do with your issue, was pointing out that for WeeWX to pickup the new settings (eg for reports, services etc) you need to restart the daemon. This is SOP for changes to .py files. For wee_import you do not need to restart. 

Gary

[Remy LAVABRE]

Hello and thank you for your answer.
By adding the line weewx.units.obs_group_dict['wetbulbTemp'] = 'group_temperature' to the /usr/share/weewx/user/extensions.py file, the wetbulbTemp field is now imported correctly.
Thank you for your help !

However, I have a new question as I haven't been able to understand exactly the weewx documentation on this subject. I clearly saw that the case of adding electricity consumption to the database had been processed but I did not understand the entire explanation given.

To put it simply, I retrieve data in CSV format which I integrate at regular intervals into the weewx database using wee_import.py (my weather station does not have a USB output and the data is retrieved from AWEKAS).
To the weather data, I would have liked to add for the period considered:
- CPU load data (unitless)
- The total and used memory of the Raspberry (kB unit)
- The total and used capacity of the SSD (Gb unit)
- Power consumption (Watt unit)
- Water consumption (unit per liter)

I created the fields I needed in the existing WeeWX database without problem with wee_database

What I couldn't do:
- Create a new base group (for example group_memory with the new Kb and Gb units). What file exactly should be modified or created and where is it located?

Compared to the documentation, I would tend to make a file "/usr/share/weewx/user/memory.py" with in it:

import weewx.units

weewx.units.obs_group_dict['rocketForce'] = 'group_memory'
weewx.units.USUnits['group_memory'] = 'byte'
weewx.units.MetricUnits['group_memory'] = 'byte'
weewx.units.MetricWXUnits['group_memory'] = 'byte'

weewx.units.default_unit_format_dict['byte'] = '%.1f'

weewx.units.default_unit_label_dict['byte'] = ' byte'

Is my approach correct? How to create the units kB and gB?


- For data that is unitless such as charge1, charge5 and charge15 (assuming that the field names of the weewx.sdb database are charge1,charge5,charge15 and that the names of the header fields of the CSV file to import are charge1, charge5 and charge15), will the import be done by putting in the CSV import configuration file for wee_import.py:

source = CSV

[CSV]
....
    [[FieldMap]]
        dateTime    = dateTime, unix_epoch
....
        charge1    = charge1
        charge5    = charge5
        charge15   = charge15

Thank you for your help because I have my data in a CSV file but am a little lost importing it into the weewx database...

Rémy LAVABRE

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/43f0c3c7-27aa-4f9f-b8bf-d9e9b911b779n%40googlegroups.com.

[Karen K]

In /usr/share/weewx/weewx/units.py the wetbulb temperature is defined as outWetbulb instead of wetbulbTemp.

[gjr80]

Some answers/guidance below.

Gary

On Monday, 18 September 2023 at 00:55:03 UTC+10 remy.l...@gmail.com wrote:

However, I have a new question as I haven't been able to understand exactly the weewx documentation on this subject. I clearly saw that the case of adding electricity consumption to the database had been processed but I did not understand the entire explanation given.

To put it simply, I retrieve data in CSV format which I integrate at regular intervals into the weewx database using wee_import.py (my weather station does not have a USB output and the data is retrieved from AWEKAS).
To the weather data, I would have liked to add for the period considered:
- CPU load data (unitless)
- The total and used memory of the Raspberry (kB unit)
- The total and used capacity of the SSD (Gb unit)
- Power consumption (Watt unit)
- Water consumption (unit per liter)

I created the fields I needed in the existing WeeWX database without problem with wee_database

What I couldn't do:
- Create a new base group (for example group_memory with the new Kb and Gb units). What file exactly should be modified or created and where is it located?

Compared to the documentation, I would tend to make a file "/usr/share/weewx/user/memory.py" with in it:

import weewx.units

weewx.units.obs_group_dict['rocketForce'] = 'group_memory'
weewx.units.USUnits['group_memory'] = 'byte'
weewx.units.MetricUnits['group_memory'] = 'byte'
weewx.units.MetricWXUnits['group_memory'] = 'byte'

weewx.units.default_unit_format_dict['byte'] = '%.1f'

weewx.units.default_unit_label_dict['byte'] = ' byte'

Is my approach correct? How to create the units kB and gB?

Not quite. First up, no need for memory.py, just use /usr/share/weewx/user/extensions.py, that is what is is meant for. Next, WeeWX already has a unit named byte which is used with unit group group_data. You can certainly define your own unit group group_memory but why, it just adds the possibility for confusion. The one thing WeeWX does not have is unit names representing kilobyte and gigabyte (and megabyte), these you will need to define or otherwise handle yourself. To add units kilobyte, megabyte and gigabyte you need to define the unit conversion functions for each unit as well as specifying a number format and unit label for each unit. We generally specify unit conversion functions using lambda functions, in the case unit kilobyte we need to specify how to convert from kilobyte to each of the other units that may be used within unit group group_data; ie kilobyte to bit, kilobyte to byte, kilobyte to megabyte and kilobyte to gigabyte. We might do this as follows:

weewx.units.conversionDict['kilobyte'] = {'bit': lambda x: x * 8192.0,
                                          'byte': lambda x: x * 1024.0,
                                          'megabyte': lambda x: x / 1024.0,
                                          'gigabyte': lambda x: x / 1048576.0}

Likewise for each of the other group_data units. To define a unit label and number format for the kilobyte unit you would use something like:

weewx.units.default_unit_label_dict['kilobyte'] = ' kB'

weewx.units.default_unit_format_dict['kilobyte'] = '%.1f'

Again, likewise for the other units.

Overall you would end up with something like:

weewx.units.conversionDict['bit'] = {'byte': lambda x: x / 8.0,
                                     'kilobyte': lambda x: x / 8192.0,
                                     'megabyte': lambda x: x / 8388608.0,
                                     'gigabyte': lambda x: x / 8589934592.0}

weewx.units.conversionDict['byte'] = {'bit': lambda x: x * 8.0,
                                      'kilobyte': lambda x: x / 1024.0,
                                      'megabyte': lambda x: x / 1048576.0,
                                      'gigabyte': lambda x: x / 1073741824.0}

weewx.units.conversionDict['kilobyte'] = {'bit': lambda x: x * 8192.0,
                                          'byte': lambda x: x * 1024.0,
                                          'megabyte': lambda x: x / 1024.0,
                                          'gigabyte': lambda x: x / 1048576.0}

weewx.units.conversionDict['megabyte'] = {'bit': lambda x: x * 8388608.0,
                                          'byte': lambda x: x * 1048576.0,
                                          'kilobyte': lambda x: x * 1024.0,
                                          'gigabyte': lambda x: x / 1024.0}

weewx.units.conversionDict['gigabyte'] = {'bit': lambda x: x * 8589934592.0,
                                          'byte': lambda x: x * 1073741824.0,
                                          'kilobyte': lambda x: x * 1048576.0,
                                          'megabyte': lambda x: x * 1024.0}

weewx.units.default_unit_label_dict['kilobyte'] = ' kB'

weewx.units.default_unit_format_dict['kilobyte'] = '%.1f'

weewx.units.default_unit_label_dict['megabyte'] = ' MB'

weewx.units.default_unit_format_dict['megabyte'] = '%.1f'

weewx.units.default_unit_label_dict['gigabyte'] = ' GB'

weewx.units.default_unit_format_dict['gigabyte'] = '%.1f'

Note the above uses the binary interpretation of a kilobyte; ie 1kB = 1024 bytes, 1MB = 1024kB and 1GB = 1024MB. If you wish to use the Metric interpretation of a kilobyte; ie 1kB = 1000 bytes, 1MB = 1000kB and 1GB = 1000MB some of the above conversion factors will need to be changed accordingly.

- For data that is unitless such as charge1, charge5 and charge15 (assuming that the field names of the weewx.sdb database are charge1,charge5,charge15 and that the names of the header fields of the CSV file to import are charge1, charge5 and charge15), will the import be done by putting in the CSV import configuration file for wee_import.py:

source = CSV

[CSV]
....
    [[FieldMap]]
        dateTime    = dateTime, unix_epoch
....
        charge1    = charge1
        charge5    = charge5
        charge15   = charge15

WeeWX has unit group group_count and unit count to handle unitless fields. All you need to do is assign your fields to group_count in the same way you assigned your wet bulb field to group_temperature.

 

[Remy Lavabre]

Thank you Gary for all these detailed explanations! Thank you also for your patience...
Thanks to you I was finally able to finish integrating the data into the weewx database as I wanted!
Without your precious help, I would NEVER have gotten there!

Just one last quick question, if weewx is updated, do you know if the /usr/share/weewx/user/extensions.py file is modified or if it is kept as is?

Thanks a lot again Gary!

[gjr80]

Anything in the user directory (/home/weewx/bin/user or /usr/share/weewx/user depending on the WeeWX install type) is upgrade safe. Of course, despite this you should still keep a backup of key WeeWX files.

Gary