Unable to load AWS credentials from any provider in the chain

[julia...@covus.de]

Hey @ all,

I just started working on a monitoring project in my company. I need to include AWS-Cloudwatch.

So Prometheus is up and running. I now want to include AWS-Cloudwatch metrics.

But as soon as I launched the docker-container this error message appears:

Feb 13, 2018 10:14:28 AM io.prometheus.cloudwatch.CloudWatchCollector collect
WARNING: CloudWatch scrape failed
com.amazonaws.SdkClientException: Unable to load AWS credentials from any provider in the chain
	at com.amazonaws.auth.AWSCredentialsProviderChain.getCredentials(AWSCredentialsProviderChain.java:131)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.getCredentialsFromContext(AmazonHttpClient.java:1115)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.runBeforeRequestHandlers(AmazonHttpClient.java:764)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:728)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:721)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:704)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:672)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:654)
	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:518)
	at com.amazonaws.services.cloudwatch.AmazonCloudWatchClient.doInvoke(AmazonCloudWatchClient.java:965)
	at com.amazonaws.services.cloudwatch.AmazonCloudWatchClient.invoke(AmazonCloudWatchClient.java:941)
	at com.amazonaws.services.cloudwatch.AmazonCloudWatchClient.getMetricStatistics(AmazonCloudWatchClient.java:630)
	at io.prometheus.cloudwatch.CloudWatchCollector.scrape(CloudWatchCollector.java:365)
	at io.prometheus.cloudwatch.CloudWatchCollector.collect(CloudWatchCollector.java:443)
	at io.prometheus.client.CollectorRegistry.collectorNames(CollectorRegistry.java:99)
	at io.prometheus.client.CollectorRegistry.register(CollectorRegistry.java:50)
	at io.prometheus.client.Collector.register(Collector.java:128)
	at io.prometheus.client.Collector.register(Collector.java:121)
	at io.prometheus.cloudwatch.WebServer.main(WebServer.java:25)

2018-02-13 10:14:28.729:INFO:oejs.Server:jetty-8.y.z-SNAPSHOT
2018-02-13 10:14:28.795:INFO:oejs.AbstractConnector:Started SelectChann...@0.0.0.0:9106

My Acces key ID and Secret access key are set in .bash_profile and in .aws/credentials
I double checked that.

I looked quite a while on the internet for a solution, but nothing worked for me :/
Some suggestions?

BTW my config file looks like this, just for testing:

---
region: us-east-1
metrics:
- aws_namespace: AWS/ApplicationELB
  aws_metric_name: UnHealthyHostCount

- aws_namespace: AWS/ApplicationELB
  aws_metric_name: HealthyHostCount

and also my IAM settings for Cloudwatch:

and finally my brew config:

brew config
HOMEBREW_VERSION: 1.5.4
ORIGIN: https://github.com/Homebrew/brew
HEAD: 021fe22e3c982b08346422d1e7897e929a709380
Last commit: 3 days ago
Core tap ORIGIN: https://github.com/Homebrew/homebrew-core
Core tap HEAD: 964bafecf9f6c94780d7727d44ba94329907138a
Core tap last commit: 2 days ago
HOMEBREW_PREFIX: /usr/local
HOMEBREW_AWS_ACCESS_KEY_ID: set
HOMEBREW_AWS_SECRET_ACCESS_KEY: set
CPU: quad-core 64-bit skylake
Homebrew Ruby: 2.3.3 => /usr/local/Homebrew/Library/Homebrew/vendor/portable-ruby/2.3.3/bin/ruby
Clang: 9.0 build 900
Git: 2.14.3 => /Applications/Xcode.app/Contents/Developer/usr/bin/git
Curl: 7.54.0 => /usr/bin/curl
Perl: /usr/bin/perl
Python: /usr/local/opt/python/libexec/bin/python => /usr/local/Cellar/python/2.7.14_3/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python
Ruby: /usr/bin/ruby => /System/Library/Frameworks/Ruby.framework/Versions/2.3/usr/bin/ruby
Java: 9.0.4, 1.8.0_152
macOS: 10.13.2-x86_64
Xcode: 9.2
CLT: N/A
X11: 2.7.11 => /opt/X11

Best,
Avendretter

[am...@lumindigital.com]

did you ever find a solution?

[ashishkuma...@gmail.com]

Facing same issue. Trying it on ubuntu machine. AWS is configured on the host and I can fetch CW metrics from that host using AWS CLI but when I run the container, I 'm getting the same error. 

My guess, the aws sdk inside the container is not getting credentials from the host. 

Can anyone help here?

[ashishkuma...@gmail.com]

As I thought the issue is with passing aws crendentials to the container. Just pass these two KV pair to your container. 

-e AWS_ACCESS_KEY_ID=xxxxxxxxxxxxxxxxx -e AWS_SECRET_ACCESS_KEY=xxxxxxxxxxxxxxxxx