AWS Cloudwatch Exporter metrics missing in Promethues

[sanjay kurra]

Hi Monitoring Gurus,

I'm using Cloudwatch exporter to scape metrics from Cloudwatch and I see metrics being scrapped from Cloudwatch through Cloudwatch Exporter. 

# HELP aws_s3_bucket_size_bytes_average CloudWatch metric AWS/S3 BucketSizeBytes Dimensions: [BucketName, StorageType] Statistic: Average Unit: Bytes

# TYPE aws_s3_bucket_size_bytes_average gauge

aws_s3_bucket_size_bytes_average{job="aws_s3",instance="",storage_type="StandardStorage",bucket_name="XXXXXXXXXXXXXX",} 4317749.0 160099116

0000

aws_s3_bucket_size_bytes_average{job="aws_s3",instance="",storage_type="StandardStorage",bucket_name="XXXXXXXXXXXX",} 1.2971384E7 1600991160000

# HELP aws_s3_number_of_objects_average CloudWatch metric AWS/S3 NumberOfObjects Dimensions: [BucketName, StorageType] Statistic: Average Unit: Count

# TYPE aws_s3_number_of_objects_average gauge

aws_s3_number_of_objects_average{job="aws_s3",instance="",storage_type="AllStorageTypes",bucket_name="XXXXXXXXXX",} 3.0 1600991160000

aws_s3_number_of_objects_average{job="aws_s3",instance="",storage_type="AllStorageTypes",bucket_name="XXXXXXXXXX",} 3596.0 1600991160000

However when I try to view through Prometheus using below configurations, I could see Ec2 and Lambda metrics but not S3.

  - job_name: 'AWS Monitoring'

    metrics_path: /metrics

    static_configs:

      - targets:

         - AWS

    relabel_configs:

      - source_labels: [_address_]

        target_label: instance

      - target_label: _address_

        replacement: xx.xx.xx.xx:8080'

Do I need to add any additional config to view S3 metrics ?

[Sanju k]

Even tried with below -

- job_name: 'S3 Monitoring'

    metrics_path: /s3

    static_configs:

      - targets: ['xx.xx.xx.xx:8080']

[Brian Candler]

Can you scrape the exporter with 'curl'? What's the exact curl command line you use?

[Sanju k]

Thanks for your reply. 

Yes I was able to scrape the metrics using curl -v http://xx.xx.xx.xx:8080/s3 

Little back ground on this - I'm using cloudwatch exporter to pull metrics from Cloudwatch and send data to Prometheus through Prometheus Proxy and Agent (https://github.com/pambrose/prometheus-proxy)

Cloud Watch -> Cloud Watch Exporter - > Prometheus Proxy/Agent -> Prometheus

So I see S3 metrics when I curl Prometheus Proxy. I can Ec2 metrics in Prometheus and not S3 when I use same config in Prometheus.

- job_name: 'S3 Monitoring'

    metrics_path: /s3

    static_configs:

      - targets: ['xx.xx.xx.xx:8080']

# HELP aws_s3_bucket_size_bytes_average CloudWatch metric AWS/S3 BucketSizeBytes Dimensions: [BucketName, StorageType] Statistic: Average Unit: Bytes

# TYPE aws_s3_bucket_size_bytes_average gauge

aws_s3_bucket_size_bytes_average{job="aws_s3",instance="",storage_type="StandardStorage",bucket_name="XXXXXXXXXXXXXX",} 4317749.0 160099116

0000

aws_s3_bucket_size_bytes_average{job="aws_s3",instance="",storage_type="StandardStorage",bucket_name="XXXXXXXXXXXX",} 1.2971384E7 1600991160000

# HELP aws_s3_number_of_objects_average CloudWatch metric AWS/S3 NumberOfObjects Dimensions: [BucketName, StorageType] Statistic: Average Unit: Count

# TYPE aws_s3_number_of_objects_average gauge

aws_s3_number_of_objects_average{job="aws_s3",instance="",storage_type="AllStorageTypes",bucket_name="XXXXXXXXXX",} 3.0 1600991160000

aws_s3_number_of_objects_average{job="aws_s3",instance="",storage_type="AllStorageTypes",bucket_name="XXXXXXXXXX",} 3596.0 1600991160000

[Brian Candler]

On Saturday, 26 September 2020 20:04:57 UTC+1, Sanju k wrote:

Yes I was able to scrape the metrics using curl -v http://xx.xx.xx.xx:8080/s3 

Then that should give you the same as:

- job_name: 'S3 Monitoring'

    metrics_path: /s3

    static_configs:

      - targets: ['xx.xx.xx.xx:8080']

I suggest you run tcpdump to capture the http exchange from curl and the http exchange from prometheus, and compare them.