cloudwatch_exporter - Minimum calls per unit of time and resource

30 views
Skip to first unread message

Albert Serrallé Ríos

unread,
Mar 5, 2020, 2:10:36 PM3/5/20
to Prometheus Users

Hello,


I've raised the scrape_interval to 30m and I've noticed that there's a minimum amount of calls that, no matter the value for scrape_interval, will be executed.

Is there any theoretical minimum of GetMetricStatistics and ListMetrics calls? Could that be configured or is it "hardcoded"?


Context: I'm trying to reduce the Cloudwatch costs as much as possible in dev/staging environments.


Thanks!

Brian Brazil

unread,
Mar 5, 2020, 2:21:44 PM3/5/20
to Albert Serrallé Ríos, Prometheus Users
On Thu, 5 Mar 2020 at 19:10, Albert Serrallé Ríos <aserra...@gmail.com> wrote:

Hello,


I've raised the scrape_interval to 30m and I've noticed that there's a minimum amount of calls that, no matter the value for scrape_interval, will be executed.

Is there any theoretical minimum of GetMetricStatistics and ListMetrics calls? Could that be configured or is it "hardcoded"?


You can expect at least one ListMetrics per configured metric, and then one GetMetricStatistics per relevant metric it finds. These should only ever be happening due to a scrape.

Brian
 


Context: I'm trying to reduce the Cloudwatch costs as much as possible in dev/staging environments.


Thanks!

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/bb4258f4-56c4-45cd-8d4c-3cec48de0bd0%40googlegroups.com.


--

Albert Serrallé Ríos

unread,
Mar 5, 2020, 2:36:36 PM3/5/20
to Prometheus Users
I have 11 metrics configured, and my metrics show 100/105 ListMetrics calls and 380/399 GetMetricStatistics calls every 5 minutes.

Certainly, the scraping interval affects the number of calls to GetMetricStatistics (I've tested 5 minute vs 30 minute scrape, and the number of calls is is reduced by 1/3 aprox) but in a way that is not obvious to me.

Also, the number of ListMetrics calls seems unecessarily high...

On Thursday, March 5, 2020 at 8:21:44 PM UTC+1, Brian Brazil wrote:
On Thu, 5 Mar 2020 at 19:10, Albert Serrallé Ríos <aserra...@gmail.com> wrote:

Hello,


I've raised the scrape_interval to 30m and I've noticed that there's a minimum amount of calls that, no matter the value for scrape_interval, will be executed.

Is there any theoretical minimum of GetMetricStatistics and ListMetrics calls? Could that be configured or is it "hardcoded"?


You can expect at least one ListMetrics per configured metric, and then one GetMetricStatistics per relevant metric it finds. These should only ever be happening due to a scrape.

Brian
 


Context: I'm trying to reduce the Cloudwatch costs as much as possible in dev/staging environments.


Thanks!

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to promethe...@googlegroups.com.

Brian Brazil

unread,
Mar 5, 2020, 7:52:02 PM3/5/20
to Albert Serrallé Ríos, Prometheus Users
On Thu, 5 Mar 2020 at 19:37, Albert Serrallé Ríos <aserra...@gmail.com> wrote:
I have 11 metrics configured, and my metrics show 100/105 ListMetrics calls and 380/399 GetMetricStatistics calls every 5 minutes.

Certainly, the scraping interval affects the number of calls to GetMetricStatistics (I've tested 5 minute vs 30 minute scrape, and the number of calls is is reduced by 1/3 aprox) but in a way that is not obvious to me.

Also, the number of ListMetrics calls seems unecessarily high...

Can you share your config?

I'd also suspect that there's more than one Prometheus hitting the exporter.

Brian
 

On Thursday, March 5, 2020 at 8:21:44 PM UTC+1, Brian Brazil wrote:
On Thu, 5 Mar 2020 at 19:10, Albert Serrallé Ríos <aserra...@gmail.com> wrote:

Hello,


I've raised the scrape_interval to 30m and I've noticed that there's a minimum amount of calls that, no matter the value for scrape_interval, will be executed.

Is there any theoretical minimum of GetMetricStatistics and ListMetrics calls? Could that be configured or is it "hardcoded"?


You can expect at least one ListMetrics per configured metric, and then one GetMetricStatistics per relevant metric it finds. These should only ever be happening due to a scrape.

Brian
 


Context: I'm trying to reduce the Cloudwatch costs as much as possible in dev/staging environments.


Thanks!

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to promethe...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/bb4258f4-56c4-45cd-8d4c-3cec48de0bd0%40googlegroups.com.


--

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/691ad00d-ea13-4018-919d-053fbc2b4d90%40googlegroups.com.


--

Albert Serrallé Ríos

unread,
Mar 6, 2020, 3:29:58 AM3/6/20
to Prometheus Users
To make things easier, I've just removed everything except for one metric:

region: us-east-1metrics:
- aws_namespace: AWS/CloudFront
  aws_metric_name: Requests
  aws_statistics: [Sum]
  aws_dimensions: [DistributionId, Region]
  aws_dimensions_select:
    Region: [Global]

Then prometheus:

global:
  scrape_interval:     30m
  scrape_timeout:      10s

  external_labels:
    monitor: 'prometheus-local'
    account: 'es-newconstruction-pre'

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:8080']

  - job_name: 'cloudwatch-exporter'
    honor_labels: false
    metrics_path: '/metrics'
    static_configs:
      - targets:
        - 'localhost:8001'

What I can see now is:
  • 20 ListMetrics per 5 minutes
  • 60 GetMetricStatistics per 5 minutes
I have 3 Cloudfront distributions in that environment.

Thanks for helping out on this, Brian.

Brian Brazil

unread,
Mar 6, 2020, 3:44:01 AM3/6/20
to Albert Serrallé Ríos, Prometheus Users
I would expect 1 ListMetrics then, and 3 GetMetricStatistics. If you manually hit /metrics how much do the request counters go up by?

Brian
 
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/bd5ef7f3-d5eb-4128-8017-0b27f3a41098%40googlegroups.com.


--

Albert Serrallé Ríos

unread,
Mar 6, 2020, 3:59:56 AM3/6/20
to Prometheus Users
Ok, found it... As you said, someone else is hitting the /metrics endpoint, the health probes!

(ノ_<。)


Gonna use that feature now...

Thanks for the help!
Reply all
Reply to author
Forward
Message has been deleted
0 new messages