Cloudwatch Exporter DNS issue for aws endpoints

[adrianmiron]

Hi,

The application cannot resolve the aws private DNS endpoint, according to the error at least (log spam below). However inside the container i can resolve it just fine from bash :

$k exec -it cloudwatch-exporter-6854db6b5-njlhp -- bash

root@cloudwatch-exporter-6854db6b5-njlhp:/# nslookup monitoring.us-east-1.amazonaws.com
Server: 10.100.0.10
Address: 10.100.0.10#53

Non-authoritative answer:
Name: monitoring.us-east-1.amazonaws.com
Address: 192.168.104.173
Name: monitoring.us-east-1.amazonaws.com
Address: 192.168.201.22
Name: monitoring.us-east-1.amazonaws.com
Address: 192.168.19.87

root@cloudwatch-exporter-6854db6b5-njlhp:/#

$k logs cloudwatch-exporter-6854db6b5-njlhp
2020-09-24 09:32:12.401:INFO::main: Logging initialized @2591ms to org.eclipse.jetty.util.log.StdErrLog
2020-09-24 09:32:12.594:INFO:oejs.Server:main: jetty-9.4.z-SNAPSHOT; built: 2019-02-15T16:53:49.381Z; git: eb70b240169fcf1abbd86af36482d1c49826fa0b; jvm 11.0.8+10
2020-09-24 09:32:12.698:INFO:oejsh.ContextHandler:main: Started o.e.j.s.ServletContextHandler@6f603e89{/,null,AVAILABLE}
2020-09-24 09:32:12.791:INFO:oejs.AbstractConnector:main: Started ServerConnector@7813cb11{HTTP/1.1,[http/1.1]}{0.0.0.0:9106}
2020-09-24 09:32:12.792:INFO:oejs.Server:main: Started @2982ms
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.amazonaws.util.XpathUtils (file:/cloudwatch_exporter.jar) to method com.sun.org.apache.xpath.internal.XPathContext.getDTMManager()
WARNING: Please consider reporting this to the maintainers of com.amazonaws.util.XpathUtils
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Sep 24, 2020 9:35:46 AM io.prometheus.cloudwatch.CloudWatchCollector collect
WARNING: CloudWatch scrape failed
com.amazonaws.SdkClientException: Unable to execute HTTP request: monitoring.eu-east-1.amazonaws.com
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleRetryableException(AmazonHttpClient.java:1207)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1153)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:802)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:770)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:744)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:704)
at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:686)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:550)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:530)
at com.amazonaws.services.cloudwatch.AmazonCloudWatchClient.doInvoke(AmazonCloudWatchClient.java:2594)
at com.amazonaws.services.cloudwatch.AmazonCloudWatchClient.invoke(AmazonCloudWatchClient.java:2561)
at com.amazonaws.services.cloudwatch.AmazonCloudWatchClient.invoke(AmazonCloudWatchClient.java:2550)
at com.amazonaws.services.cloudwatch.AmazonCloudWatchClient.executeListMetrics(AmazonCloudWatchClient.java:1706)
at com.amazonaws.services.cloudwatch.AmazonCloudWatchClient.listMetrics(AmazonCloudWatchClient.java:1678)
at io.prometheus.cloudwatch.CloudWatchCollector.listDimensions(CloudWatchCollector.java:391)
at io.prometheus.cloudwatch.CloudWatchCollector.getDimensions(CloudWatchCollector.java:339)
......
at java.base/java.lang.Thread.run(Unknown Source)
Caused by: java.net.UnknownHostException: monitoring.eu-east-1.amazonaws.com
at java.base/java.net.InetAddress$CachedAddresses.get(Unknown Source)
at java.base/java.net.InetAddress.getAllByName0(Unknown Source)
at java.base/java.net.InetAddress.getAllByName(Unknown Source)
at java.base/java.net.InetAddress.getAllByName(Unknown Source)
at com.amazonaws.SystemDefaultDnsResolver.resolve(SystemDefaultDnsResolver.java:27)
at com.amazonaws.http.DelegatingDnsResolver.resolve(DelegatingDnsResolver.java:38)
at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:112)
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:374)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.base/java.lang.reflect.Method.invoke(Unknown Source)
at com.amazonaws.http.conn.ClientConnectionManagerFactory$Handler.invoke(ClientConnectionManagerFactory.java:76)
at com.amazonaws.http.conn.$Proxy8.connect(Unknown Source)