Rate of Sums vs Sum of Rates

[ili...@luminatewireless.com]

Hi there,

I was reading about the functions offered by Prometheus and came across the following:

"Note that when combining irate() with an aggregation operator (e.g. sum()) or a function aggregating over time (any function ending in _over_time), always take a irate() first, then aggregate. Otherwise irate() cannot detect counter resets when your target restarts."

In a system that has hierarchical federation, how would aggregation work? For example, let's say I have server A and server B, and server C collects data from A and B. If I want to evaluate a rule that requires aggregating data from servers A and B, would Prometheus compute the sum of rates instead of the rate of sums, so as to defend against counter resets and missing data?

Thanks,
Ilina

[Julius Volz]

Federation creates no problem here. There's two possibilities:

1) You federate raw counter data from A and B into C. In this case, you'd have to apply the rate yourself in C before summing:

  sum by(foo) (irate(my_federated_counter_total[1m]))

2) A and B already compute rates via recording rules and C then federates over those rates. Then you'd just need to sum.

In both cases, you end up doing the rate first and the sum second. The only way you can get the two the wrong way around is if you record the sum of raw counters via recording rules first, and then take a rate over the result of that recording rule. Otherwise, Prometheus prevents you to take the range ([1m]) of something that is not a pure time series selector, but some arbitrary expression.

 

Thanks,
Ilina

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/0ee2621a-6a28-4a4b-ae69-148edc10d0a3%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Ilina Mitra]

Ok thanks for the clarification. So in general, the philosophy is to sum the rates instead of calculating the rate of the sums, right?

[Julius Volz]

Yep. And Prometheus makes it kinda difficult to do the wrong thing in the normal case.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CAMsVFO29682GnqxLe%2Bw6VE%3DsZV0OhrMh9SKABxhZX2BdNqAOfA%40mail.gmail.com.

[Ilina Mitra]

Cool, thanks for the quick response! :)