questions about rate calculation

[Huafeng Lyu]

Hi, 

I was using the 'rate' feature of opentsdb to calculate the networking i/o rate for hosts, and had some problems.

Our metric 'host.networking.incoming.bytes' is a counter with the maximum value of 2^32-1. The metric reports a datapoint every minute. The rate calculation works well when we calculate with each raw datapoint, i.e., every minute.

The problems are:

1. When downsampling is used. When we want to calculate the rate hourly or even daily, we need to use downsampling before rate conversion. We were using opentsdb 2.2 so there's no 'last' function so we had to use 'max' instead. This caused inaccuracy. I noticed that opentsdb 2.3 introduced 'last' which I guess should resolve this problem, but is there a solution for opentsdb 2.2?

2. Further, if the downsampling period is very long, and the counter has reached the maximum value and rolls to 0 for several times, neither 'last' nor 'max' works well for this case. What's the best practice for this scenario?

Also, what do other time series databases handle the rate calculation? How do they handle the problems list above?

Thanks.

--huafeng

[ManOLamancha]

On Monday, February 27, 2017 at 12:26:51 AM UTC-8, Huafeng Lyu wrote:

Hi, 

I was using the 'rate' feature of opentsdb to calculate the networking i/o rate for hosts, and had some problems.

Our metric 'host.networking.incoming.bytes' is a counter with the maximum value of 2^32-1. The metric reports a datapoint every minute. The rate calculation works well when we calculate with each raw datapoint, i.e., every minute.

The problems are:

1. When downsampling is used. When we want to calculate the rate hourly or even daily, we need to use downsampling before rate conversion. We were using opentsdb 2.2 so there's no 'last' function so we had to use 'max' instead. This caused inaccuracy. I noticed that opentsdb 2.3 introduced 'last' which I guess should resolve this problem, but is there a solution for opentsdb 2.2?

Hi, we don't have the ability to change the downsampling/rate order yet. There is a patch that may make it into 2.4 that can help but the real fix will be in 3.0 with query operation ordering. 

2. Further, if the downsampling period is very long, and the counter has reached the maximum value and rolls to 0 for several times, neither 'last' nor 'max' works well for this case. What's the best practice for this scenario?

Unfortunately the best solution right now for this is to emit rates on the source side. It's a good case to handle though and we can handle it in the downsampler with the rate options that are in 2.2. We just have to add it. Mind firing off an issue and we'll see if we can't get it in 3.0? Thanks!