Drop metric after recording rule calculated the value

[ono...@gmail.com]

I have a metric which is generated by our 3k hosts for per-country usage. Meaning that each host may receive request from that country and it will be:

n hosts * n countries

As a result, we calculate per datacenter aggregated value with recording rule, and this is the value we are interested in. DC value is taken from hostname (there is a room for improvement here of course, but this is not the question). 

Do we have a way to drop metrics after 5min aggregated value calculated in recording rule?

[Björn Rabenstein]

On 26 March 2017 at 14:37, <ono...@gmail.com> wrote:
> Do we have a way to drop metrics after 5min aggregated value calculated in
> recording rule?

Not yet, at least not directly.

You could set up a 2nd Prometheus server that federates the aggregated
metrics from your 1st server, and then have a short retention time on
your 1st server and a long retention time on the 2nd.

Variable retention time per metric might or might not be a feature in
the future...

--
Björn Rabenstein, Engineer
http://soundcloud.com/brabenstein

SoundCloud Ltd. | Rheinsberger Str. 76/77, 10115 Berlin, Germany
Managing Director: Alexander Ljung | Incorporated in England & Wales
with Company No. 6343600 | Local Branch Office | AG Charlottenburg |
HRB 110657B

[Yaroslav Molochko]

Are you interested in this feature? Any pitfalls you see implementing it (I believe you already thought about it https://github.com/prometheus/prometheus/issues/1381 )? I’m interested in getting the feature, and might invest some time implementing it in a near feature, maybe you could point me out into the right direction to don’t waste your time reviewing something you would not merge at all. 

[Brian Brazil]

On 26 March 2017 at 13:37, <ono...@gmail.com> wrote:

I have a metric which is generated by our 3k hosts for per-country usage. Meaning that each host may receive request from that country and it will be:

n hosts * n countries

There are ~200 countries, so that's a bit above the recommended cardinality of 10 for a metric. You may wish to consider doing this analysis via logs rather than metrics.

Brian

 

As a result, we calculate per datacenter aggregated value with recording rule, and this is the value we are interested in. DC value is taken from hostname (there is a room for improvement here of course, but this is not the question). 

Do we have a way to drop metrics after 5min aggregated value calculated in recording rule?

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/819e51a0-d2db-4fc3-8e59-2aa877cc68cd%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Brian Brazil

www.robustperception.io

[Yaroslav Molochko]

Yep, that is how it was done previously, but we want to make it realtime (30sec delay max) and make alerts based on it. We deal with blocking of our solution at some countries, and prometheus should give us realtime information where we stand for that country. We have the same data in hadoop as well, but it is available 30 minutes after the block.  

[Björn Rabenstein]

On 26 March 2017 at 14:54, Yaroslav Molochko <ono...@gmail.com> wrote:
> Are you interested in this feature? Any pitfalls you see implementing it (I
> believe you already thought about it
> https://github.com/prometheus/prometheus/issues/1381 )? I’m interested in
> getting the feature, and might invest some time implementing it in a near
> feature, maybe you could point me out into the right direction to don’t
> waste your time reviewing something you would not merge at all.

Yeah, so this is one of the "can of worms"-type feature requests.
Definitely follow up on that issue #1381 before starting to do
anything. (I think it is also on the list of possible GSoC projects,
but it might also be discouraged there because of its many links to
semantic subtleties and its technical impact on many parts of the
Prometheus stack.)

In different news: Prometheus v2 will come with a completely rewritten
storage layer, so any feature that is not low hanging fruit should
probably be implemented in that new storage layer.