[TFA Klimalogg Pro] dewpoint and heatindex are not calculated

[Fry3k]

Hello,

I'm new to weewx and very happy that I got it up and running in one day. Now I want to start with the custumization part. As far as I know, derived values are calculated if they are not provided by the hardware (weewx.conf settings: [StdWXCalculate] -> [[Calculations]] -> dewpoint = prefer_hardware, heatindex = prefer_hardware). However, this doesn't happen; the fields in the sqlite database are null. I also tried setting it to software instead of prefer_hardware but with no effect.

I'm using the TFA Klimalogg Pro with the driver by Luc Heijst. The Klimalogg Pro only provides temperature and humidity for the base station and up to 8 sensors (I'm using 4). Is there anything else that must be changed if I want this to work? Has this something to do with the names of the values? The column names in the database are called temp0, temp1, ... / humidity0, humidity1, ... / dewpoint0, dewpoint1, ... How is the setting in weewx.conf settings: [StdWXCalculate] -> [[Calculations]] actually linked to column names in the database, i.e. how does the weewx software know that dewpoint1 is a dewpoint and has to be calculated from temp0 and humidity0?

Greetings,

Fry3k

[gjr80]

Hi,

The Klimalogg driver works best with its own schema, rather than the default weeWX schema. The default weeWX schema includes fields dewpoint and heatindex only (ie not numbered). Even if your driver outputs heatindex0, heatindex1 etc these will not be stored in the database as they are not in the default schema. Did you make the changes to weewx.conf to use the binding kl_binding as per the klimalogg driver readme? If not you will need to so that weewx uses a schema that includes heatindex0, heatindex1 etc.

The Klimalogg driver should automatically calculate heatindex and dewpoint for all your sensors, you can check this by running weeWX directly and looking at the contents of the LOOP packets and ARCHIVE records. You can run weeWX directly as per these instructions http://weewx.com/docs/usersguide.htm#Running_directly

If you cannot get it to work please post a copy of your weewx.conf with any sensitive info removed. Also, stop then start weeWX and post a log extract from when you start weeWX until a few archive records have been saved. Do not edit the log, we need to see the entire startup preamble.

Gary

[Fry3k]

Hi Gary and thanks for your quick reply.

I did everything in the Klimalogg driver readme. The sqlite database shows colomn headers for the Klimalogg specific values (e.g. temp0, temp1, dewpoint0, ...)

By running weewx directly, I get LOOP and REC packets. I assume REC are the archive records.

LOOP packets look like this:

LOOP:   2017-06-01 11:09:32 CEST (1496308172) altimeter: None, appTemp: None, barometer: None, batteryStatus0: 0, batteryStatus1: 0, batteryStatus2: 0, batteryStatus3: 0, batteryStatus4: 0, batteryStatus5: 0, batteryStatus6: 0, batteryStatus7: 0, batteryStatus8: 0, cloudbase: None, dateTime: 1496308172, dewpoint: None, heatindex: None, humidex: None, humidity0: 45, humidity1: 46, humidity2: 50, humidity3: 48, humidity4: 50, humidity5: None, humidity6: None, humidity7: None, humidity8: None, inDewpoint: None, maxSolarRad: None, pressure: None, rainRate: 0, rxCheckPercent: 100, temp0: 78.26, temp1: 82.94, temp2: 75.2, temp3: 78.26, temp4: 76.28, temp5: None, temp6:None, temp7: None, temp8: None, usUnits: 1, windchill: None

REC packets look like this:

REC:    2017-06-01 11:10:00 CEST (1496308200) altimeter: None, appTemp: None, barometer: None, batteryStatus0: 0.0, batteryStatus1: 0.0, batteryStatus2: 0.0, batteryStatus3: 0.0, batteryStatus4: 0.0, batteryStatus5: 0.0, batteryStatus6: 0.0, batteryStatus7: 0.0, batteryStatus8: 0.0, cloudbase: None, dateTime: 1496308200.0, dewpoint: None, ET: None, heatindex: None, humidex: None, humidity0: 45.0, humidity1: 46.0, humidity2: 50.3333333333, humidity3: 48.0, humidity4: 50.0, humidity5: None, humidity6: None, humidity7: None, humidity8:None, inDewpoint: None, interval: 5, maxSolarRad: None, pressure: None, rainRate: 0.0, rxCheckPercent: 100.0, temp0: 78.38, temp1: 82.94, temp2: 75.2, temp3: 78.26, temp4: 76.28, temp5: None, temp6: None, temp7: None, temp8: None, usUnits: 1, windchill: None, windrun: 0.0

By comparing those It seems that the REC don't use dewpoint[0...8] or headindex[0...8] since it shows the now valueless default names dewoint, heatindex and inDewpoint.

weewx.conf and syslog are attached since I seem to have problems to inlucde them into the post with full length (I'm new to Google Groups)

[gjr80]

Hi,

Yes, Google Groups can be a bit finnicky.

I don't actually have a Klimalogg or use the driver so you might have to bear with me while we work through this. It looks like you have the database/schema/binding set up fine. The issue is that heatindex0 etc are not in the loop packets or archive records. Having a closer read of the driver it appears that the default setup is that heatindex and dewpoint are not calculated unless you specify them in the a [[sensor_map]] stanza in weewx.conf (preferred) or modify the driver(not advisable). Try adding something like the following to the [KlimaLogg] section of weewx.conf:

[[sensor_map]]
    temp0 = Temp0
    humidity0 = Humidity0
    heatindex0 = heatindex0
    dewpoint0 = dewpoint0
    temp1 = Temp1
    humidity1 = Humidity1
    heatindex1 = heatindex1
    dewpoint2 = dewpoint1
    temp2 = Temp2
    humidity2 = Humidity2
    heatindex2 = heatindex2
    dewpoint2 = dewpoint2
    temp3 = Temp3
    humidity3 = Humidity3
    heatindex3 = heatindex3
    dewpoint3 = dewpoint3
    temp4 = Temp4
    humidity4 = Humidity4
    heatindex4 = heatindex4
    dewpoint4 = dewpoint4
    temp5 = Temp5
    humidity5 = Humidity5
    temp6 = Temp6
    humidity6 = Humidity6
    temp7 = Temp7
    humidity7 = Humidity7
    temp8 = Temp8
    humidity8 = Humidity8
    rxCheckPercent = SignalQuality
    batteryStatus0 = BatteryStatus0
    batteryStatus1 = BatteryStatus1
    batteryStatus2 = BatteryStatus2
    batteryStatus3 = BatteryStatus3
    batteryStatus4 = BatteryStatus4
    batteryStatus5 = BatteryStatus5
    batteryStatus6 = BatteryStatus6
    batteryStatus7 = BatteryStatus7
    batteryStatus8 = BatteryStatus8

Save weewx.conf then stop weeWX if it is running and run weeWX directly again, does that put heatindex0 and friends in your LOOP and REC data?

Gary

[Fry3k]

Hi Gary,

Thanks again, it worked! The packets now look like this:

LOOP:   2017-06-01 21:35:31 CEST (1496345731) altimeter: None, appTemp: None, barometer: None, batteryStatus0: 0, batteryStatus1: 0, batteryStatus2: 0, batteryStatus3: 0, batteryStatus4: 0, batteryStatus5: 0, batteryStatus6: 0, batteryStatus7: 0, batteryStatus8: 0, cloudbase: None, dateTime: 1496345731, dewpoint: None, dewpoint0: 12.4395622918, dewpoint1: 10.2028620613, dewpoint2: 13.3146548925, dewpoint3: 15.3598579755, dewpoint4: 13.2895672777, heatindex: None, heatindex0: 26.4, heatindex1: 22.4, heatindex2: 26.2, heatindex3: 27.3566888636, heatindex4: 25.8, humidex: None, humidity0: 42, humidity1: 46, humidity2: 45, humidity3: 49, humidity4: 46, humidity5: None, humidity6: None, humidity7: None, humidity8: None, inDewpoint: None, maxSolarRad: None, pressure: None, rainRate: 0, rxCheckPercent: 100, temp0: 26.4, temp1: 22.4, temp2: 26.2, temp3: 27.0, temp4: 25.8, temp5: None, temp6: None, temp7: None, temp8: None, usUnits: 1, windchill: None

REC:    2017-06-01 21:35:00 CEST (1496345700) altimeter: None, appTemp: None, barometer: None, cloudbase: None, dateTime: 1496345700, dewpoint: None, dewpoint0: 12.5292678342, dewpoint1: 10.293701522, dewpoint2: 13.3146548925, dewpoint3: 15.4512548665, dewpoint4: 13.2895672777, ET: None, heatindex: None, heatindex0: 26.5, heatindex1: 22.5, heatindex2: 26.2, heatindex3: 27.4492357563, heatindex4: 25.8, humidex: None, humidity0: 42, humidity1: 46, humidity2: 45, humidity3: 49, humidity4: 46, humidity5: None, humidity6: None, humidity7: None, humidity8: None, inDewpoint: None, interval: 5, maxSolarRad: None, pressure: None, rainRate: 0, temp0: 26.5, temp1: 22.5, temp2: 26.2, temp3: 27.1, temp4: 25.8, temp5: None, temp6: None, temp7: None, temp8: None, usUnits: 1, windchill: None, windrun: 0.0

The SQlite database now contains the all values from the LOOP/REC packets except the ones that weren't in there before (like cloudbase or inDewpoint).

Just a question for understanding: I assume that you found out by reading the drivers source code, that the names of the Klimalogg parameters are what you have written on the right side of the expressions (the left side beeing the column headers in the database). How did the connection between sensor names and database name work before? (since there was no [[sensor_map]] stanza)

One odd thing happened: the values are now saved in celsius instead of fahrenheit in the database but the usUnits field is still set to 1 (see image below)

The new columns containing dewpoint and heatindex use now celsius too. The generated graphs showed celsius before the change (which means that they were converted from the database bevor graph drawing). Now the are just showing the actual value from the database. For me celsius would be just fine but the user guide states that the [StdConvert] -> target_unit should not be changed unless I know exactly what I'm doing (which I am certainly not).

[gjr80]

Hi,

Re the mapping, if you look at the comments in the code around line 1447-1486 you see mention of options sensor_map_id and [[sensor_map]]. If sensor_map_id is 0 (default) or 1 then one either the Klimalogg or weeWX standard sensor maps is used respectively. Since the sensor map also determines what fields are output from the driver (lines 2090-2111) and since heatindex0 and friends are not in either predefined sensor maps, heatindex0 and friends will never be produced (output) if we use the predefined sensor maps. Fortunately, [[sensor_map]] gives us a way around this. We can use the default Klimalogg sensor map as the basis of a custom sensor map, the format of which is the comments mentioned earlier.

The short answer is that KL_SENSOR_MAP (line 1258) is used as the sensor map if no sensor map is specified.

As for the units issue, can't see anything obvious, the Klimalogg driver should ouput metric data and weeWX should take care of any conversions. Need to have a closer look at things on the big screen (rather than this iPad) when I get home later this morning.

Gary

[gjr80]

Ok,

I think I see what is going on. I think you have stumbled onto a deficiency with the Klimalogg driver, I say deficiency rather than bug as I think the issue could be solved in the Klimalogg driver documentation rather than the code. In a nutshell, when you alter the schema used by weeWX you need to make sure that weeWX knows what unit groups each of the new fields belongs to, ie weeWX needs to know that field dewpoint0 is a temperature. This is automatically taken care of by the Klimalogg driver when you tell it to use either the default Klimalogg sensor map (sensor_map_id = 0) or the standard weeWX sensor map (sensor_map_id = 1). However, if you specify a custom sensor map using [[sensor_map]] the Klimalogg driver does not take care of telling weeWX what fields belong in what observation groups. Fortunately this is easy to setup manually in an enduring manner (ie it will survive weeWX upgrades). What I suggest you try is edit bin/user/extensions.py and add the following at the bottom of the file:

# setup units to support custom Klimalogg sensor map
from weewx.units import obs_group_dict
obs_group_dict['temp0'] = 'group_temperature'
obs_group_dict['temp1'] = 'group_temperature'
obs_group_dict['temp2'] = 'group_temperature'
obs_group_dict['temp3'] = 'group_temperature'
obs_group_dict['temp4'] = 'group_temperature'
obs_group_dict['temp5'] = 'group_temperature'
obs_group_dict['temp6'] = 'group_temperature'
obs_group_dict['temp7'] = 'group_temperature'
obs_group_dict['temp8'] = 'group_temperature'
obs_group_dict['humidity0'] = 'group_percent'
obs_group_dict['humidity1'] = 'group_percent'
obs_group_dict['humidity2'] = 'group_percent'
obs_group_dict['humidity3'] = 'group_percent'
obs_group_dict['humidity4'] = 'group_percent'
obs_group_dict['humidity5'] = 'group_percent'
obs_group_dict['humidity6'] = 'group_percent'
obs_group_dict['humidity7'] = 'group_percent'
obs_group_dict['humidity8'] = 'group_percent'
obs_group_dict['dewpoint0'] = 'group_temperature'
obs_group_dict['dewpoint1'] = 'group_temperature'
obs_group_dict['dewpoint2'] = 'group_temperature'
obs_group_dict['dewpoint3'] = 'group_temperature'
obs_group_dict['dewpoint4'] = 'group_temperature'
obs_group_dict['dewpoint5'] = 'group_temperature'
obs_group_dict['dewpoint6'] = 'group_temperature'
obs_group_dict['dewpoint7'] = 'group_temperature'
obs_group_dict['dewpoint8'] = 'group_temperature'
obs_group_dict['heatindex0'] = 'group_temperature'
obs_group_dict['heatindex1'] = 'group_temperature'
obs_group_dict['heatindex2'] = 'group_temperature'
obs_group_dict['heatindex3'] = 'group_temperature'
obs_group_dict['heatindex4'] = 'group_temperature'
obs_group_dict['heatindex5'] = 'group_temperature'
obs_group_dict['heatindex6'] = 'group_temperature'
obs_group_dict['heatindex7'] = 'group_temperature'
obs_group_dict['heatindex8'] = 'group_temperature'
obs_group_dict['rxCheckPercent'] = 'group_percent'
obs_group_dict['batteryStatus0'] = 'group_count'
obs_group_dict['batteryStatus1'] = 'group_count'
obs_group_dict['batteryStatus2'] = 'group_count'
obs_group_dict['batteryStatus3'] = 'group_count'
obs_group_dict['batteryStatus4'] = 'group_count'
obs_group_dict['batteryStatus5'] = 'group_count'
obs_group_dict['batteryStatus6'] = 'group_count'
obs_group_dict['batteryStatus7'] = 'group_count'
obs_group_dict['batteryStatus8'] = 'group_count'

I know you do not use some of those fields but it will not matter. Save extensions.py then stop weeWX if it was running and then start weeWX. You should be able to easily tell if the units issue is fixed if you run weeWX directly and look at the LOOP records. They should display with usUnits = 1 and temperatures should be in degree F. If this is the case your database should be fine.

Let us know how you go. I am not sure what Luc and Matthew had in mind for supporting a custom sensor map (one way ahead would be to add dewpoint0 and friends to the default Klimalogg sensor map, another to document the unit group additions in extensions.py), I will raise the issue with Matthew when he is back in the land of reliable internet.

Gary

[Fry3k]

Hi,

I will have a look at it if i have the time. I have no experience in Python and little in Linux - at the first look it seems that I have to recompile the code, so it may take a little bit to get into it. But I will keep updating on my progress!

[gjr80]

No compiling required, just a bit of copy/paste and a couple of command line commands.
1. Edit extensions.py and paste the code extract above at the end of the file
2. Save extensions.py
3. Stop weeWX
4. Run weeWX directly and observe the LOOP packets

Gary

[Fry3k]

Thank you again, it worked. Didn't know there are Python files that don't need compiling. If I had looked at the extensions.py earlier I would have seen that it looks more like a configuration file. LOOP packets now show usUnits = 1.

Since there are now wrong value/unit type entries in the database, how can I delete it? Just delete the sdb file when weewx is stopped (will it create a new one?). Additionally, what about files like weewx.conf.{random_number}, weewx.conf.dist or weewx.conf.save which have been created over the time using weewx? Are they safe to delete? Since at the beginning there was only the weewx.conf (same issue goes for skin.conf).

[gjr80]

Since there are now wrong value/unit type entries in the database, how can I delete it? Just delete the sdb file when weewx is stopped (will it create a new one?).

Depends what you want to do. If you are happy to lose all of your recorded data to date then you can stop weeWX, delete weewx.sdb then start weeWX. A new database (weewx.sdb) will be automatically created upon startup. If you wish to retain some of your recorded data then you will need to manipulate your database. There are a number of ways you can do this, there are graphical tools for manipulating the database or you can use a bit of SQL directly through the sqlite3 utility. If you want to go down this path:

1. if not already installed then install sqlite3:

$ sudo apt-get install sqlite3

2. stop weeWX
3. make a copy of weewx.sdb
4. start the sqlite3 utility using your weeWX database as the source:

$ sqlite3 /home/weewx/archive/weewx.sdb

5. find the data that you want to get rid of. This could be all data before a given timestamp (ie dateTime value), it could be all records where outTemp < 40 (eg outTemp was incorrectly converted). Whatever condition you come up with that defines the records you want to delete test it with a SELECT query before you actually delete it with a DELETE query, eg:

sqlite> SELECT * FROM archive WHERE dateTime<1496922300;

will display all records with a dateTime before 8 June 2017 11:45:00 GMT. Using something like

sqlite> SELECT datetime(dateTime, 'unixepoch', 'localtime'),* FROM archive WHERE dateTime<1496922300;

will prepend a human readable version of the record timestamp(field dateTime) to the list of record displayed. If you want to query on field outTemp (or some other observational field) something like:

sqlite> SELECT datetime(dateTime, 'unixepoch', 'localtime'),* FROM archive WHERE outTemp<40;

will display all records where outTemp is less than 40. Once you are able to select the records you wish to delete then you can use a DELETE query to delete them. Say you want to delete all records before timestamp 1234567879:

sqlite> DELETE FROM archive WHERE dateTime<123456789;

to delete all records where outTemp<40:

sqlite> DELETE FROM archive WHERE outTemp<40;

6. once you have deleted the desired records exit sqlite with the .quit command. If you need to revert to your back because you deleted the wrong records, just exit sqlite3 using the .quit command, delete the erroneous weewx.sdb and make a copy you backup as weewx.sdb. You can then go back to step 4.
7. rebuild your daily summaries using the wee_database utility:

$ /home/weewx/bin/wee_database --rebuild-daily

8. start weeWX

Manipulating the database though reasonably straightforward can be daunting. You do run the risk of corrupting/losing your data. So if the process puts you off and you have only a small amount of recorded data you may wish to just start with a new database.
 

Additionally, what about files like weewx.conf.{random_number}, weewx.conf.dist or weewx.conf.save which have been created over the time using weewx? Are they safe to delete? Since at the beginning there was only the weewx.conf (same issue goes for skin.conf).

weewx.conf.{random_number} files are backup copies of weewx.conf when some process (such as an extension installer or some other weeWX tool/utility) altered weewx.conf. The seeminly random number is the date-time the file was backed up in YYYMMDDhhmmss format. You can generally safely delete them once whatever it was that made the change (eg a new extension etc) has been shown to work properly. If problems were encountered then having the weewx.conf backup helps you to revert to your previous weeWX config. Personal choice what you do, I prefer to keep them as they take up little space, but then again I make a lot of changes ot my system that sometimes do not work out.

weewx.conf.dist will generally be as a remnant from upgrading weeWX, if the upgrade went smoothly and everything works as it should you can safely delete them. Not so sure about weewx.conf.save but believe it is essentially the same, a remant of an upgrade.

Can't say I have seen the same for skin.conf but then again I don't let weeWX upgrades touch my skins, I work through them manually since they are customised. I suspect they are upgrade remnants as well and provideed your system is operating as expected they can be safely deleted.

Gary