Irregularly sampled data (change of value)

[Michael Bironneau]

Hello,

Has anyone had any experience using OpenTSDB to store/query time series where the data is sampled on change of value? For example, if you have, say, a thermometer, instead of reading the temperature every minute, you get a reading every time the temperature changes by more than some amount, leading to a timeseries with irregular intervals and where you assume step interpolation instead of linear interpolation. For example:

12:00:00 21.0

12:00:30 21.1

12:10:00 21.2

In that example, the value of the 10m downsampled series by average from 12:00:00 to 12:10:00 should not be 21.05 but (0.5*21 + 9.5*21.1)*(1/10) =  21.095. 

We've been looking at adding step interpolation support and change-of-value aggregators (this is as far as I've got with the idea), but it's a bit hackish. It's also missing a more lenient 'lookback' rule that aligns the last datapoint before the query's start time to the start time in case step interpolation method is specified. This would work with some application logic that would insert the latest value of each timeseries every hour (at least) so that each row would always contain a point for every series and prevent long backscans. 

Thanks

Michael