AWS CloudWatch Exporter Multiply Regions

[Dmitriy Alekseev]

Hello, I created 2 scrapers in docker-compose:

cloudwatch-eu-west-1:
  image: prom/cloudwatch-exporter
  container_name: cloudwatch-eu-west-1
  volumes:
    - './config:/config'
  command:
    - '/config/cloudwatch-eu-west-1.yml'
  networks:
    - monitor-net
  depends_on:
    - prometheus
  labels:
    org.label-schema.group: 'monitoring'
  environment:
    - AWS_REGION=eu-west-1

cloudwatch-ap-northeast-1:
  image: prom/cloudwatch-exporter
  container_name: cloudwatch-ap-northeast-1
  volumes:
    - './config:/config'
  command:
    - '/config/cloudwatch-ap-northeast-1.yml'
  networks:
    - monitor-net
  depends_on:
    - prometheus
  labels:
    org.label-schema.group: 'monitoring'
  environment:
    - AWS_REGION=ap-northeast-1

EC2 with this exporters has IAM role to all required cloudwatch/tags permissions, not limited by regions or anything else.

EC2 located at eu-west-1 region and successfully scrape data from eu-west-1 cloudwatch. Scrape duration takes 6s.

But I have issue with ap-northeast-1: scrape duration takes 68s and don't return any data. I double checked it by directly requesting data from scraper: docker-compose exec sqsexporter-eu-west-1 wget -O- http://cloudwatch-ap-northeast-1:9106/metrics

Also I already use jmal98/sqs-exporter for both regions and it works fine. I noticed the scrape duration, and it makes me worried: eu-west-1 takes 1s when ap-northeast-1 takes 13s which is greatly longer.

Does anyone has same issues? I need to use this exporters strictly in exact AWS region where they deployed at? This not sounds like this mandatory, but reality says exactly this.

[Stuart Clark]

In general I would expect API calls to other regions to take longer. You
might find a small improvement by locating the exporters within the
different regions, but it might not be worth the extra cost/complexity.
How frequently are you wanting to scrape those exporters? As CloudWatch
isn't real-time I don't gerneally scrape them at a high frequency.

--
Stuart Clark

[Stuart Clark]

On 2021-09-06 12:58, Dmitriy Alekseev wrote:
> My issue now that prom/cloudwatch-exporter can't get any data from
> ap-northeast-1 at all, when it works for eu-west-1 and I don't know
> where to dig.About frequency: well what the recommended interval? I
> tied put 5m. But I don't understand what details level will I get then
> from Prometheus. I want use CloudWatch exporter to get my logs custom
> metrics counters.
>

You should be able to enable debug logging within the Cloudwatch
Exporter right down to the API calls being made. You can then see what
is being sent and the replies received, which might show some errors.

Regarding scrape intervals the maximum that can be used is generally
2-2.5 minutes, so you should reduce the interval from 5 minutes (due to
staleness handling).

Have you configured the exporter with the details of what metrics you
are wanting to include?

--
Stuart Clark

[Dmitriy Alekseev]

Yeah, I provided a different config files for each of docker containers to use.

Thanks for debug note, --debug - looks like what I need. Will check and come back here.

Thanks

Best Regards,
Dmitriy Alekseev
DevOps Engineer

[Dmitriy Alekseev]

Can't enable debug logging on docker container. Created issue at https://github.com/prometheus/cloudwatch_exporter/issues/358

Best Regards,
Dmitriy Alekseev
DevOps Engineer

On 06/09/2021 15:20, Stuart Clark wrote:

[Dmitriy Alekseev]

Now I'm totally lost, when I run java exporter out of docker on same EC2 it works fine. It takes same amount of time: ~69s, but it returns metrics data.

Best Regards,
Dmitriy Alekseev
DevOps Engineer

On 06/09/2021 15:20, Stuart Clark wrote:

[Dmitriy Alekseev]

Added to prometheus targets, and it again not contains any data and stopped to provide metrics data over curl directly.

Don't think that without debug this can be solved :(

Best Regards,
Dmitriy Alekseev
DevOps Engineer