Federation for cloudwatch_exporter - best practice?

[Albert Serrallé]

Hello,

I'm trying to figure out how to federate my cloudwatch_exporter metrics.

The default metrics are 10m old, I understand that this is needed because cloudwatch consolidate metrics over time. My prometheus is scraping the exporter and saving the values with the original timestamp. Federation queries (as they are instant queries) cannot retrieve those metrics.

So, I think that my only options are:

• set_timestamp -> false (fake illusion of real time metric, which are really 10m old)
• delay_seconds -> 1s-5m (metric might not be consolidated in cloudwatch)

There's anything else I can try? I don't like any of those two methods for the exposed reasons. I've been looking at the changes in stale logic for 2.0, but I don't know if this applies to me or how to do it:

If you're federating instance-level metrics, switch to only aggregated metrics

So, what's the recommended way to overcome this?

Many thanks.

[Albert Serrallé]

For the records, it looks like there's a third option:

• honor_timestamps -> false (fake ilusion of real time metric)

[Brian Brazil]

You can't with federation, it's merely a more obvious case where instance-level metrics don't work. See https://www.robustperception.io/federation-what-is-it-good-for

In this case if you need the metrics in a different Prometheus, get that Prometheus to scrape the cloudwatch exporter.

Brian

 

Many thanks.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/030245a2-9e9e-4493-9a39-eebf53753e47%40googlegroups.com.

--

Brian Brazil

www.robustperception.io

[Albert Serrallé]

That's how is usually done in my company (and I suspect that's very prevalent but wrong in the end, if I understand that link correctly). In my company, at team level we have a lot of metrics scraped into a single prometheus. Then, at company level we just "replicate" everything with the corresponding team label.

The official recommendation would be to create aggregation rules in the "team" prometheus, and federate that from the "company" prometheus, right?

On Wednesday, 1 April 2020 21:42:20 UTC+2, Brian Brazil wrote:

On Wed, 1 Apr 2020 at 20:11, Albert Serrallé <albert...@gmail.com> wrote:

Hello,

I'm trying to figure out how to federate my cloudwatch_exporter metrics.

The default metrics are 10m old, I understand that this is needed because cloudwatch consolidate metrics over time. My prometheus is scraping the exporter and saving the values with the original timestamp. Federation queries (as they are instant queries) cannot retrieve those metrics.

So, I think that my only options are:

• set_timestamp -> false (fake illusion of real time metric, which are really 10m old)
• delay_seconds -> 1s-5m (metric might not be consolidated in cloudwatch)

There's anything else I can try? I don't like any of those two methods for the exposed reasons. I've been looking at the changes in stale logic for 2.0, but I don't know if this applies to me or how to do it:

If you're federating instance-level metrics, switch to only aggregated metrics

So, what's the recommended way to overcome this?

You can't with federation, it's merely a more obvious case where instance-level metrics don't work. See https://www.robustperception.io/federation-what-is-it-good-for

In this case if you need the metrics in a different Prometheus, get that Prometheus to scrape the cloudwatch exporter.

Brian

 

Many thanks.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to promethe...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/030245a2-9e9e-4493-9a39-eebf53753e47%40googlegroups.com.

--

Brian Brazil

www.robustperception.io

[Brian Brazil]

On Wed, 1 Apr 2020 at 21:00, Albert Serrallé <albert...@gmail.com> wrote:

That's how is usually done in my company (and I suspect that's very prevalent but wrong in the end, if I understand that link correctly). In my company, at team level we have a lot of metrics scraped into a single prometheus. Then, at company level we just "replicate" everything with the corresponding team label.

Yes, that would not be the recommended way to do things. It's more complex, less reliable, and less performant.

The official recommendation would be to create aggregation rules in the "team" prometheus, and federate that from the "company" prometheus, right?

Only if there's an aggregation that makes sense, which is probably not the case for CloudWatch - and you'd still mess up the timestamps.

Brian

 

On Wednesday, 1 April 2020 21:42:20 UTC+2, Brian Brazil wrote:

On Wed, 1 Apr 2020 at 20:11, Albert Serrallé <albert...@gmail.com> wrote:

Hello,

I'm trying to figure out how to federate my cloudwatch_exporter metrics.

The default metrics are 10m old, I understand that this is needed because cloudwatch consolidate metrics over time. My prometheus is scraping the exporter and saving the values with the original timestamp. Federation queries (as they are instant queries) cannot retrieve those metrics.

So, I think that my only options are:

• set_timestamp -> false (fake illusion of real time metric, which are really 10m old)
• delay_seconds -> 1s-5m (metric might not be consolidated in cloudwatch)

There's anything else I can try? I don't like any of those two methods for the exposed reasons. I've been looking at the changes in stale logic for 2.0, but I don't know if this applies to me or how to do it:

If you're federating instance-level metrics, switch to only aggregated metrics

So, what's the recommended way to overcome this?

You can't with federation, it's merely a more obvious case where instance-level metrics don't work. See https://www.robustperception.io/federation-what-is-it-good-for

In this case if you need the metrics in a different Prometheus, get that Prometheus to scrape the cloudwatch exporter.

Brian

 

Many thanks.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to promethe...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/030245a2-9e9e-4493-9a39-eebf53753e47%40googlegroups.com.

--

Brian Brazil

www.robustperception.io

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/f03663b6-1dba-4930-a016-077a2328745c%40googlegroups.com.

--

Brian Brazil

www.robustperception.io