metrics expiry/age/ttl setting

1,277 views
Skip to first unread message

Maciek Strzelecki

unread,
Dec 5, 2017, 5:27:23 AM12/5/17
to Prometheus Users
I have a custom exporter builtusing the golang prom client. 

I want to ensure that metrics that were scraped more than X minutes ago will be removed and not exporterd by the exporter. Which setting controls that behaviour?

const (
    // DefMaxAge is the default duration for which observations stay
    // relevant.
    DefMaxAge time.Duration = 10 * time.Minute
    // DefAgeBuckets is the default number of buckets used to calculate the
    // age of observations.
    DefAgeBuckets = 5
    // DefBufCap is the standard buffer size for collecting Summary observations.
    DefBufCap = 500
)

Are these the droids i am looking for?

Brian Brazil

unread,
Dec 5, 2017, 6:05:34 AM12/5/17
to Maciek Strzelecki, Prometheus Users
No, those are for Summary. Why do you want old metrics removed?


--

Maciek Strzelecki

unread,
Dec 5, 2017, 8:17:06 AM12/5/17
to Prometheus Users
I am affraid im exposing some old "ghost" metrics that are never ever updated anymore. (im checking landingpages, i have metrics like

fpc_status{page="www.google.com"} 200
fpc_loads_within_limit{page="www.google.com"} 1
fpc_load_time{page="www.google.com"} 0,2


but in half an hour i might get an updated list of pages , and www.google.com is no longer among them. Then the metrics are "dead" they have a value when they were relevant, but not anymore. Such metrics will never ever get another update.

Is the default behaviour of golang prom client taking care of purging these dead metrics after a while? Is there a variable to set that expiry?

I know that in principle i should construct the metrics in such a way that they are always relevant, well-behaving and non-disappearing... but im lacking a good idea how to organize it.

Brian Brazil

unread,
Dec 5, 2017, 8:20:22 AM12/5/17
to Maciek Strzelecki, Prometheus Users
On 5 December 2017 at 13:17, Maciek Strzelecki <strzelec...@gmail.com> wrote:
I am affraid im exposing some old "ghost" metrics that are never ever updated anymore. (im checking landingpages, i have metrics like

fpc_status{page="www.google.com"} 200
fpc_loads_within_limit{page="www.google.com"} 1
fpc_load_time{page="www.google.com"} 0,2


but in half an hour i might get an updated list of pages , and www.google.com is no longer among them. Then the metrics are "dead" they have a value when they were relevant, but not anymore. Such metrics will never ever get another update.

Is the default behaviour of golang prom client taking care of purging these dead metrics after a while? Is there a variable to set that expiry?

I know that in principle i should construct the metrics in such a way that they are always relevant, well-behaving and non-disappearing... but im lacking a good idea how to organize it.

It sounds like what you want to do is have a blackbox/snmp style exporter, where you pass the URL along with the scrape and return the information for just that URL for that scrape.

Brian
 



 


On Tuesday, December 5, 2017 at 12:05:34 PM UTC+1, Brian Brazil wrote:
On 5 December 2017 at 10:27, Maciek Strzelecki <strzelec...@gmail.com> wrote:
I have a custom exporter builtusing the golang prom client. 

I want to ensure that metrics that were scraped more than X minutes ago will be removed and not exporterd by the exporter. Which setting controls that behaviour?

const (
    // DefMaxAge is the default duration for which observations stay
    // relevant.
    DefMaxAge time.Duration = 10 * time.Minute
    // DefAgeBuckets is the default number of buckets used to calculate the
    // age of observations.
    DefAgeBuckets = 5
    // DefBufCap is the standard buffer size for collecting Summary observations.
    DefBufCap = 500
)

Are these the droids i am looking for? 

No, those are for Summary. Why do you want old metrics removed?


--

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.
 To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/d9d8a075-3c93-4d3e-b4d0-68b80ab16b78%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.



--

Maciek Strzelecki

unread,
Dec 5, 2017, 8:50:38 AM12/5/17
to Prometheus Users
The reason i turned away from blackbox exporter was the configuration scheme. It was forcing me to specify targets within prometheus.yaml config file. *If* prometheus had a conf.d-like dir where snippets for additional jobs could be dropped, that would be perfect. In theory i could leverage the saltstack reactor system to poll whether the external page-list config file was changed, then rerun the prometheus state  thus rendering the joint, updated configuration.... But doing a dedicated configurable exporter proved to be more reliable approach.

:-)

Back to my original question - is there a way to set that kind of expiry?
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To post to this group, send email to promethe...@googlegroups.com.



--

Brian Brazil

unread,
Dec 5, 2017, 8:53:31 AM12/5/17
to Maciek Strzelecki, Prometheus Users
On 5 December 2017 at 13:50, Maciek Strzelecki <strzelec...@gmail.com> wrote:
The reason i turned away from blackbox exporter was the configuration scheme. It was forcing me to specify targets within prometheus.yaml config file. *If* prometheus had a conf.d-like dir where snippets for additional jobs could be dropped, that would be perfect.

Have you looked at file_sd_configs? The last part can be a glob.
 
In theory i could leverage the saltstack reactor system to poll whether the external page-list config file was changed, then rerun the prometheus state  thus rendering the joint, updated configuration....

That's also a standard approach.
 
But doing a dedicated configurable exporter proved to be more reliable approach.

:-)

Back to my original question - is there a way to set that kind of expiry?

There isn't. What you want is the targets to live in Prometheus.
 
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.
 To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/2c86ac5a-4357-4e29-80f6-741fc0433afb%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.



--

Maciek Strzelecki

unread,
Dec 5, 2017, 9:04:57 AM12/5/17
to Prometheus Users
Thanks Brian!

sd_configs was covering for moving targets, yes, but another part of blackbox exporter proved to be a pain, the part specifying the module to probe with. Unfortunately we had some "custom stuff" that would result in duplicating the job n-times with different modules to use and the sd_configs for given module... And we backed out of it, into our own exporter. (even if the config was working, it still lacked the metrics we had in our previous icinga based checks :))

Thx a ton for referencing https://github.com/mattbostock/webdriver_exporter !



--
Reply all
Reply to author
Forward
0 new messages