Missing Data Points and Resets

979 views
Skip to first unread message

ili...@luminatewireless.com

unread,
Mar 21, 2017, 10:10:19 PM3/21/17
to Prometheus Users
Hi there,

I see that Prometheus has some support for handling counter resets and missing data points in the raw metric time series data. Would it be possible to get an exact list of which functions adjust for these kinds of issues, and how? 

Also, could someone provide a pointer to the code which handles this?

Thanks,
Ilina

Brian Brazil

unread,
Mar 22, 2017, 5:10:47 AM3/22/17
to Ilina Mitra, Prometheus Users
On 22 March 2017 at 02:10, <ili...@luminatewireless.com> wrote:
Hi there,

I see that Prometheus has some support for handling counter resets and missing data points in the raw metric time series data. Would it be possible to get an exact list of which functions adjust for these kinds of issues, and how? 

rate, increase, and somewhat irate.
 
Also, could someone provide a pointer to the code which handles this?


I'm also giving a talk on Wednesday on this topic at CloudNativeCon. 

--

Ilina Mitra

unread,
Mar 30, 2017, 8:36:16 PM3/30/17
to Brian Brazil, Prometheus Users
Ok, so I see that rate and increase handle counter resets. What about missing data points? Which functions handle those, and how?

Ilina Mitra

unread,
Mar 30, 2017, 8:40:27 PM3/30/17
to Brian Brazil, Prometheus Users
Ah sorry I see that rate, delta, and increase have extrapolation for missing data points. Are these the only functions that handle missing data points?

On Thu, Mar 30, 2017 at 5:35 PM, Ilina Mitra <ili...@luminatewireless.com> wrote:
Ok, so I see that rate and increase handle counter resets. What about missing data points? Which functions handle those, and how?

Brian Brazil

unread,
Mar 31, 2017, 3:24:06 AM3/31/17
to Ilina Mitra, Prometheus Users
On 31 March 2017 at 01:40, Ilina Mitra <ili...@luminatewireless.com> wrote:
Ah sorry I see that rate, delta, and increase have extrapolation for missing data points. Are these the only functions that handle missing data points?

They're the only functions with explicit thought put into it. Whether the other functions handle missing data depends on what you're doing, and generally you should try to be resilient.

Brian
 

On Thu, Mar 30, 2017 at 5:35 PM, Ilina Mitra <ili...@luminatewireless.com> wrote:
Ok, so I see that rate and increase handle counter resets. What about missing data points? Which functions handle those, and how?

On Wed, Mar 22, 2017 at 2:10 AM, Brian Brazil <brian.brazil@robustperception.io> wrote:
On 22 March 2017 at 02:10, <ili...@luminatewireless.com> wrote:
Hi there,

I see that Prometheus has some support for handling counter resets and missing data points in the raw metric time series data. Would it be possible to get an exact list of which functions adjust for these kinds of issues, and how? 

rate, increase, and somewhat irate.
 
Also, could someone provide a pointer to the code which handles this?


I'm also giving a talk on Wednesday on this topic at CloudNativeCon. 

--





--

Ilina Mitra

unread,
Mar 31, 2017, 2:37:31 PM3/31/17
to Brian Brazil, Prometheus Users
Well, in our use case, it's very possible some of the nodes we are scraping might "go down", in which case we will have missing data points from those nodes. The nodes individually are not resilient, though we have failover mechanisms that enable resilience of the system of nodes as a whole.

Is it possible to configure the period of time for which data is extrapolated? From reading through the code of extrapolatedRate, it doesn't seem to be configurable to me, but I want to make sure I'm not missing anything.

Thanks,
Ilina

Brian Brazil

unread,
Mar 31, 2017, 2:59:54 PM3/31/17
to Ilina Mitra, Prometheus Users
On 31 March 2017 at 19:37, Ilina Mitra <ili...@luminatewireless.com> wrote:
Well, in our use case, it's very possible some of the nodes we are scraping might "go down", in which case we will have missing data points from those nodes. The nodes individually are not resilient, though we have failover mechanisms that enable resilience of the system of nodes as a whole.

Is it possible to configure the period of time for which data is extrapolated? From reading through the code of extrapolatedRate, it doesn't seem to be configurable to me, but I want to make sure I'm not missing anything.

It'll do the right thing as long as rate() has enough range to get points around the missed scrape.

Brian
 

Thanks,
Ilina

On Fri, Mar 31, 2017 at 12:24 AM, Brian Brazil <brian.brazil@robustperception.io> wrote:
On 31 March 2017 at 01:40, Ilina Mitra <ili...@luminatewireless.com> wrote:
Ah sorry I see that rate, delta, and increase have extrapolation for missing data points. Are these the only functions that handle missing data points?

They're the only functions with explicit thought put into it. Whether the other functions handle missing data depends on what you're doing, and generally you should try to be resilient.

Brian
 

On Thu, Mar 30, 2017 at 5:35 PM, Ilina Mitra <ili...@luminatewireless.com> wrote:
Ok, so I see that rate and increase handle counter resets. What about missing data points? Which functions handle those, and how?

On Wed, Mar 22, 2017 at 2:10 AM, Brian Brazil <brian.brazil@robustperception.io> wrote:
On 22 March 2017 at 02:10, <ili...@luminatewireless.com> wrote:
Hi there,

I see that Prometheus has some support for handling counter resets and missing data points in the raw metric time series data. Would it be possible to get an exact list of which functions adjust for these kinds of issues, and how? 

rate, increase, and somewhat irate.
 
Also, could someone provide a pointer to the code which handles this?


I'm also giving a talk on Wednesday on this topic at CloudNativeCon. 

--





--




--
Reply all
Reply to author
Forward
0 new messages