Multiple TH sensors to CSV

[PK]

Due to module version problems (and capacity of the small NSLU"2) I don't use sql storage, but continue with the .csv file.

I have a wmr928nx with 3 extra sensors (1 temp+hum TH2, and 2 temp) and they are reported fine in wflogger.log, and want to log the indoor temp-hum also (TH1).
Examples:

# outdoor sample: temp, hum (are logged)
2013-03-16 13:43:24,971 INFO [station.wmr928nx] Temp_Hum_1  batteryOK: True, Temp.: 2.9 C, Humidity: 39 %, Dew Point: 0 C

# indoor sample: temp1, hum1
2013-03-16 13:43:48,098 INFO [station.wmr928nx] Console batteryOK: True, Temp.: 22 C, Humidity: 22 %, Pressure: 1000, SeaLevelPressure: 1018, Weat

# other sensores samples: temp2, hum2, temp3, temp5
2013-03-16 13:43:38,998 INFO [station.wmr928nx] Temp_Hum_2  batteryOK: True, Temp.: 8.6 C, Humidity: 46 %, Dew Point: 0 C
2013-03-16 13:43:42,662 INFO [station.wmr928nx] Temp_3  batteryOK: True, Temp.: 61.9 C
2013-03-16 13:43:25,004 INFO [station.wmr928nx] Temp_5  batteryOK: True, Temp.: -18.3 C
herStatus: 12, WeatherStatusTxt: Sunny


Having followed the advice in http://code.google.com/p/wfrog/wiki/CustomizationGuide#Support_for_multiple_TH_sensors, I realized that csvstorage.py need to be modified to get the data into wfrog.csv.
I modified the header in wfrog.csv and tried to add the new sensors towards the end in csvstorage.py records:

   columns = [ 'timestamp', 'localtime', 'temp', 'hum', 'wind', 'wind_dir', 'wind_gust', 'wind_gust_dir', 'dew_point', 'rain', 'rain_rate', 'pressure', 'uv_index', 'temp1', 'hum1', 'temp2', 'hum2', 'temp3', 'temp5' ]

    def keys(self, context={}):
        return ['localtime',
                'temp',
                'hum',
                'wind',
                'wind_dir',
                'wind_gust',
                'wind_gust_dir',
                'dew_point',
                'rain',
                'rain_rate',
                'pressure',
                'uv_index',
                'temp',
                'hum',
                'temp',
                'hum',
                'temp',
                'temp',
                'utctime']

This is obviously not enough, since _write.sample stops at appending temp1 in:

       for key in self.columns[2:]:
            sample_row.append(sample[key])

Could someone please hint what more modifications would be needed?
Are the columns or their datatype defined elsewhere also?
Thanks
/Poul

[Jordi Puigsegur Figueras]

Hi,

There are two types of columns: mandatory and optional. You are probably getting an error because the logger doesn't understand temp1. Look at the base.py file used by the database storages:

mandatory_storage_fields = ['TEMP', 'HUM', 'DEW_POINT', 'WIND', 'WIND_DIR', 'WIND_GUST', 

                            'WIND_GUST_DIR', 'RAIN', 'RAIN_RATE', 'PRESSURE']

optional_storage_fields = ['UV_INDEX', 'SOLAR_RAD', 'TEMPINT', 'HUMINT', 'TEMP2', 'HUM2', 

                           'TEMP3', 'HUM3', 'TEMP4', 'HUM4', 'TEMP5', 'HUM5', 

                           'TEMP6', 'HUM6', 'TEMP7', 'HUM7', 'TEMP8', 'HUM8', 

                           'TEMP9', 'HUM9']

These names cannot be changed.  Internal sensors, usually numbered 0, are saved as TEMPINT, HUMINT.  Additional sensors are numbered TEMP2, TEMP3, etc. wfrog considers that the main sensor is sensor number one. Drivers already encapsulate this behaviour. 

It would be good to go a step beyond and make the CSV format able to handle these values by modifying a configuration file. The columns variable used in csv.py should be removed and instead the column list would be read from a configuration file at initialization time. Look how this is done in a database storage by reading the column list of the table meteo and filtering it with self.storage_fields:

def init(self, context=None):

self.db = wfcommon.database.FirebirdDB(self.database, self.user, self.password, self.charset)

table_fields = self._get_table_fields()

# Verify Mandatory fields

assert 'TIMESTAMP_UTC' in table_fields

assert 'TIMESTAMP_LOCAL' in table_fields

for field in self.mandatory_storage_fields:

assert field in table_fields

# Obtain actual storage fields

self.storage_fields = self.mandatory_storage_fields + \

[field for field in self.optional_storage_fields if field in table_fields]

self.logger.info("Table %s detected with fields: %s" % (self.tablename, ', '.join(self.storage_fields)))

The important factor here is that in a CSV order is relevant and might impose backward compatibility problems. Adding or removing a column in a database storage is straightforward while in a CSV it is going to be more difficult. 

Jordi.

2013/3/16 PK <pk2...@gmail.com>

--
You received this message because you are subscribed to the Google Groups "wfrog-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wfrog-dev+...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[PK]

Thanks Jordi
for revealing to me some of the inner wfrog secrets (that is: properties I didn't know of :-)

It set me on a track that succeseeded after ... some trial and error. I do not have the skill or energy now to write a new csv or sqlite storage driver, but went along and modified more as I started: make adjustments so my fixed set of sensors can be appended to the csv storage and html presentation. When 0.9 or 1.0 of wfrog is issued, I may rebuild my wfrog station with a database to be compliant with wfrog development.

So, in accumulator.py default_formulas was expanded.
csvfile.py: columns and keys.
chart.yaml, chart_accumulator.py and table_accumulator.py were expanded.
main.html and main-ftp.html were updated with charts. 
wfrog.csv was expanded (header and each row appended with new variables and ,0.0,... values (grew from 7 to 9 Mbytes for 2 years of data). Row examples:

timestamp,localtime,temp,hum,wind,wind_dir,wind_gust,wind_gust_dir,dew_point,rain,rain_rate,pressure,uv_index,tempint,humint,temp2,hum2,temp3,temp5
1364120867,2013-03-24 11:27:47,1.8,39.0,4.1,13.1,6.7,9,-10.7,0.0,0.0,1031.7,,0,0,0,0,0,0

(for those curious the new charts are: indoor T/H, cellar T/H, furnace (heat stow), freezer)

In the same take, my page got a few enhancements, such as a 6day period also.

Again, thanks for your support.
/Poul

[Jordi Puigsegur Figueras]

Hi,

I am glad that I was able to help. 

Jordi.

2013/3/25 PK <pk2...@gmail.com>