Connect to CloudWatch through proxy

[jp.ho...@gmail.com]

We are running Jenkins in AWS on an EC2 instance with the Monitoring plugin installed. We are attempting to setup the pushing of metrics to Cloudwatch so we can have alarms on certain metrics. We have followed the steps for setting up Cloudwatch which has the following:

• JAVA_OPTS to include -Djavamelody.cloudwatch-namespace=MyCompany/Jenkins
• Installed AWS CloudWatch Library plugin

After restarting Jenkins, we are not getting any metrics getting pushed to Cloudwatch. Unfortunately our EC2 in AWS instances are not Internet connected and need to go through a proxy which is setup on a separate instance in AWS. I have had a look at the Cloudwatch.java class and it doesn't seem to have any configuation to cater for the use of a proxy.

The Jenkins log is showing the following errors...

Aug 25, 2017 3:03:48 AM net.bull.javamelody.internal.common.JavaLogger warn

WARNING: exception while collecting data: java.io.IOException: Error connecting to AWS CloudWatch

java.io.IOException: Error connecting to AWS CloudWatch

at net.bull.javamelody.internal.model.CloudWatch.send(CloudWatch.java:166)

at net.bull.javamelody.internal.model.Collector.collect(Collector.java:357)

at net.bull.javamelody.internal.model.Collector.collectWithoutErrors(Collector.java:319)

at net.bull.javamelody.internal.model.Collector.collectLocalContextWithoutErrors(Collector.java:308)

at net.bull.javamelody.FilterContext$CollectTimerTask.run(FilterContext.java:79)

at java.util.TimerThread.mainLoop(Timer.java:555)

at java.util.TimerThread.run(Timer.java:505)

Caused by: com.amazonaws.SdkClientException: Unable to execute HTTP request: Connect to monitoring.ap-southeast-2.amazonaws.com:443 [monitoring.ap-southeast-2.amazonaws.com/54.240.193.95] failed: connect timed out

at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleRetryableException(AmazonHttpClient.java:1069)

at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1035)

at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:742)

at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:716)

at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699)

at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667)

at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649)

at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513)

at com.amazonaws.services.cloudwatch.AmazonCloudWatchClient.doInvoke(AmazonCloudWatchClient.java:1073)

at com.amazonaws.services.cloudwatch.AmazonCloudWatchClient.invoke(AmazonCloudWatchClient.java:1049)

at com.amazonaws.services.cloudwatch.AmazonCloudWatchClient.executePutMetricData(AmazonCloudWatchClient.java:957)

at com.amazonaws.services.cloudwatch.AmazonCloudWatchClient.putMetricData(AmazonCloudWatchClient.java:934)

at net.bull.javamelody.internal.model.CloudWatch.send(CloudWatch.java:163)

... 6 more

Caused by: org.apache.http.conn.ConnectTimeoutException: Connect to monitoring.ap-southeast-2.amazonaws.com:443 [monitoring.ap-southeast-2.amazonaws.com/54.240.193.95] failed: connect timed out

at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:150)

at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:353)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:498)

at com.amazonaws.http.conn.ClientConnectionManagerFactory$Handler.invoke(ClientConnectionManagerFactory.java:76)

at com.amazonaws.http.conn.$Proxy25.connect(Unknown Source)

at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:380)

at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)

at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)

at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)

at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)

at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)

at com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72)

at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1190)

at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1030)

... 17 more

Caused by: java.net.SocketTimeoutException: connect timed out

at java.net.PlainSocketImpl.socketConnect(Native Method)

at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)

at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)

at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)

at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)

at java.net.Socket.connect(Socket.java:589)

at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:337)

at com.amazonaws.http.conn.ssl.SdkTLSSocketFactory.connectSocket(SdkTLSSocketFactory.java:132)

at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:141)

... 33 more

Aug 25, 2017 3:03:49 AM net.bull.javamelody.internal.common.JavaLogger warn

WARNING: exception while collecting data: java.lang.NullPointerException

java.lang.NullPointerException

at net.bull.javamelody.internal.model.CloudWatch.getInstance(CloudWatch.java:125)

at net.bull.javamelody.internal.model.MetricsPublisher.getMetricsPublishers(MetricsPublisher.java:44)

at net.bull.javamelody.internal.model.Collector.collect(Collector.java:336)

at net.bull.javamelody.internal.model.Collector.collectWithoutErrors(Collector.java:319)

at net.bull.javamelody.NodesCollector.collectWithoutErrorsNow(NodesCollector.java:173)

at net.bull.javamelody.NodesCollector.collectWithoutErrors(NodesCollector.java:147)

at net.bull.javamelody.NodesCollector$2.run(NodesCollector.java:115)

at java.util.TimerThread.mainLoop(Timer.java:555)

at java.util.TimerThread.run(Timer.java:505)

[evernat]

Hi,

Are you allowed to set a http proxy for the (Jenkins) JVM ?

If yes, you can add system properties in JAVA_OPTS to set the http proxy.
For example:
-Dhttp.proxyHost=10.0.0.100 -Dhttp.proxyPort=80 -Dhttps.proxyHost=10.0.0.100 -Dhttps.proxyPort=443 -Dhttp.nonProxyHosts="localhost|127.*|*.foo.com‌​"

bye,
Emeric

[jp.ho...@gmail.com]

Thanks Emeric,

I setup the proxy in the JAVA_OPTS for just the -Dhttps.proxyHost -Dhttps.proxyPort system properties and it is now successfully pushing metrics to Cloudwatch.

Cheers

[evernat]

Hi,

Can you send a screenshot of the AWS CloudWatch dashboard ?
It would be great to have an example of real use.

Thanks,
Emeric