Prometheus dropping metrics from CloudWatch-Exporter

[m.przy...@biotcloud.com]

Hello,

I've a problem with getting data from cloudwatch-exporter for prometheus, collector is pulling metric properly, but in Prom logs all the time i see metrics are dropped with the following info:

level=warn ts=2018-09-05T13:28:58.66692415Z caller=scrape.go:948 component="scrape manager" scrape_pool=aws-cloudwatch-exporter target=http://aws-billing-prometheus-cloudwatch-exporter.monitoring:80/metrics msg="Error on ingesting samples that are too old or are too far into the future" num_dropped=12

As far as i read its related to time CloudWatch metrics are generated (or something like that ;-) ) - main goal is to get metrics for AWS/Billing namespace, i believe it's related to wrong setting in range_seconds on cloudwatch-exporter config or scrape intervals on Prom side. Here's my config for both - maybe someone here had similar problem and can advise me how to proceed to get it working"

Cloudwatch-Exporter:

config: |-
  region: us-east-1
  metrics:
  - aws_namespace: "AWS/Billing"
    aws_dimensions: [Currency,ServiceName]
    aws_dimensions_select:
      Currency: [USD]
    aws_metric_name: EstimatedCharges
    aws_statistics: [Average]
    range_seconds: 86400

Prom scrape config:

  - job_name: aws-cloudwatch-exporter
    scrape_interval: 60s
    scrape_timeout: 30s
    metrics_path: "/metrics"
    static_configs:
      - targets:
        - ['100.109.36.161:9106']

[Brian Brazil]

On 5 September 2018 at 14:40, <m.przy...@biotcloud.com> wrote:

Hello,

I've a problem with getting data from cloudwatch-exporter for prometheus, collector is pulling metric properly, but in Prom logs all the time i see metrics are dropped with the following info:

level=warn ts=2018-09-05T13:28:58.66692415Z caller=scrape.go:948 component="scrape manager" scrape_pool=aws-cloudwatch-exporter target=http://aws-billing-prometheus-cloudwatch-exporter.monitoring:80/metrics msg="Error on ingesting samples that are too old or are too far into the future" num_dropped=12

As far as i read its related to time CloudWatch metrics are generated (or something like that ;-) ) - main goal is to get metrics for AWS/Billing namespace, i believe it's related to wrong setting in range_seconds on cloudwatch-exporter config or scrape intervals on Prom side. Here's my config for both - maybe someone here had similar problem and can advise me how to proceed to get it working"

Billing is only generated daily or so, so the timestamps can be quite old. I'd suggest setting set_timestamp to false for any metrics like that which don't get an update at least hourly.

Brian

 

Cloudwatch-Exporter:

config: |-
  region: us-east-1
  metrics:
  - aws_namespace: "AWS/Billing"
    aws_dimensions: [Currency,ServiceName]
    aws_dimensions_select:
      Currency: [USD]
    aws_metric_name: EstimatedCharges
    aws_statistics: [Average]
    range_seconds: 86400

Prom scrape config:

  - job_name: aws-cloudwatch-exporter
    scrape_interval: 60s
    scrape_timeout: 30s
    metrics_path: "/metrics"
    static_configs:
      - targets:
        - ['100.109.36.161:9106']

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/f0fdba93-b3a1-484f-8fe7-11fa35edd200%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Brian Brazil

www.robustperception.io

[m.przy...@biotcloud.com]

@Brian Brazil - thank you, somehow i've didn't see this setting - after applying it to false i've collected metrics. 

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To post to this group, send email to promethe...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/f0fdba93-b3a1-484f-8fe7-11fa35edd200%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Brian Brazil

www.robustperception.io