Change of savings sended data from scripts

[KKoPV]

I would suggest to change the logic of saving/logging measuring data from scripts.

 

At the moment, all data are logged into file, if they are successful sended or not, so you can't see quickly the faild data sets.

 

I'd like to change this to save/log only the data, which can't sended to the API.

 

This will have 2 advantages:

- See quickly the "outstanding" data for reload

- Less storage device usage, important for mini computers like Raspi or Cubiboard with SD cards.

 

What do you think about that?

 

Knut

[Patrick Feisthammel]

We need about 11MBytes per Channel and Year for storing all values. This is based on my installation with data about 12 days.

Having (only) 3GB free storage and 12 channels, I can store all values for a range of 23 years.

Compressing the files would give a factor of about 6. So I could store 72 channels for 23 years.

I do not know if limmited storage is realy an issue. All devices using an SD card or similiar replacable storage will not run into an issue. So I thing big enough storage is no problem.

I find it valuable to know that I have all values for feature use. If I make an configuration error (like setting wrong thresholds) I would loose the readings.

Loading the values into PVLng is no issue for non-meetering channels, beause old values are not overwritten. 

For meetering channels with set "adjusting" it is not possible to load missing values, because they would change the offset. The only way is to load the channel compleetly fresh from the data.

On the other side seeing outstanding data for reload would be nice.

Patrick

[KKoPV]

Hi,

 

Am Mittwoch, 15. Januar 2014 09:23:01 UTC+1 schrieb Patrick Feisthammel:

 

I find it valuable to know that I have all values for feature use. If I make an configuration error (like setting wrong thresholds) I would loose the readings.

 

Yes, correct for sensors.

 

For meetering channels with set "adjusting" it is not possible to load missing values, because they would change the offset. The only way is to load the channel compleetly fresh from the data.

 

Ok, therefor we need all the data for a full rebuild.

 

On the other side seeing outstanding data for reload would be nice.

 

So lets adjust the logic:

• All readings will be saved into the defined "data" directory as now (default bin/data).
• Only not saved / outstanding data will also stored below "data" directory  (default bin/data/missing/) with the same sub structure (GUID, date etc.)

There you can easy find out, if something went wrong.

 

Knut