Hi,
When initially installing Rundeck, I configured the S3 Log Storage plugin [1] and all was working well. The other day I configure SSL in Rundeck for both the webapp and so I can make LDAPS queries for authentication. I folllowed this documentation [2]. My ssl.properties file references the custom keystore and truststore as detailed in the documentation. Both SSL for the webapp and LDAPS work properly.
However, this seems to have broken the S3 Log storage plugin functionality. /var/log/rundeck/service.log shows this error [3]. My basic understanding is that the java client on my Rundeck server does not have the CA certificates associated with AWS S3 in the custom truststore (defined in ssl.properties) and that since the default java truststore (/etc/pki/ca-trust/extracted/java/cacerts) should have these, I should import them into my custom truststore. Problem is I have no idea which specific CA/alias I would have to export from the default truststore.
I then found a few posts suggesting to pull the certificates directly from AWS and import them into my custom truststore. So I tried the following:
echo -n | openssl s_client -connect s3.amazonaws.com:443
I copied the contents between -----BEGIN CERTIFICATE----- and -----END CERTIFICATE----- and placed in a file, then executed:
keytool -importcert -trustcacerts -file aws-s3.cert -alias aws-s3 -keystore /etc/rundeck/ssl/truststore
Restarted Rundeck, but, the problem still persists. I'm not familiar with java and most of my research returns results speaking in generalities without specific examples.
Any assistance is much appreciated.
[1]
https://github.com/rundeck-plugins/rundeck-s3-log-plugin[2]
http://rundeck.org/docs/administration/configuring-ssl.html[3] ...
Aug 05, 2015 9:16:40 AM org.rundeck.plugins.S3LogFileStoragePlugin store
SEVERE: Unable to execute HTTP request: peer not authenticated
com.amazonaws.AmazonClientException: Unable to execute HTTP request: peer not authenticated
at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:363)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:199)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:2994)
at com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1174)
at org.rundeck.plugins.S3LogFileStoragePlugin.store(S3LogFileStoragePlugin.java:226)
at com.dtolabs.rundeck.core.logging.ExecutionFileStorage$store.call(Unknown Source)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:45)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:128)
at rundeck.services.LogFileStorageService$_storeLogFile_closure9.doCall(LogFileStorageService.groovy:704)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:324)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1207)
at groovy.lang.ExpandoMetaClass.invokeMethod(ExpandoMetaClass.java:1110)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1016)
at groovy.lang.Closure.call(Closure.java:423)
at groovy.lang.Closure.call(Closure.java:439)
at org.codehaus.groovy.runtime.IOGroovyMethods.withStream(IOGroovyMethods.java:1186)
at org.codehaus.groovy.runtime.ResourceGroovyMethods.withInputStream(ResourceGroovyMethods.java:1642)
at org.codehaus.groovy.runtime.dgm$870.invoke(Unknown Source)
at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoMetaMethodSiteNoUnwrapNoCoerce.invoke(PojoMetaMethodSite.java:271)
at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:53)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:45)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116)
at rundeck.services.LogFileStorageService.storeLogFile(LogFileStorageService.groovy:703)
at rundeck.services.LogFileStorageService.this$2$storeLogFile(LogFileStorageService.groovy)
at rundeck.services.LogFileStorageService$this$2$storeLogFile$5.callCurrent(Unknown Source)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:49)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:133)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:153)
at rundeck.services.LogFileStorageService.runStorageRequest(LogFileStorageService.groovy:106)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:324)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1207)
at groovy.lang.ExpandoMetaClass.invokeMethod(ExpandoMetaClass.java:1110)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1016)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1152)
at groovy.lang.ExpandoMetaClass.invokeMethod(ExpandoMetaClass.java:1110)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1016)
at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.callCurrent(PogoMetaClassSite.java:66)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:49)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:133)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:141)
at rundeck.services.LogFileStorageService$_afterPropertiesSet_closure1.doCall(LogFileStorageService.groovy:73)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSiteNoUnwrapNoCoerce.invoke(PogoMetaMethodSite.java:207)
at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.call(PogoMetaMethodSite.java:68)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:45)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116)
at rundeck.services.TaskRunner.run(TaskRunner.groovy:28)
at org.springframework.core.task.SimpleAsyncTaskExecutor$ConcurrencyThrottlingRunnable.run(SimpleAsyncTaskExecutor.java:251)
at java.lang.Thread.run(Thread.java:745)
Caused by: javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated
at sun.security.ssl.SSLSessionImpl.getPeerCertificates(SSLSessionImpl.java:431)
at org.apache.http.conn.ssl.AbstractVerifier.verify(AbstractVerifier.java:128)
at org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:390)
at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:148)
at org.apache.http.impl.conn.AbstractPoolEntry.open(AbstractPoolEntry.java:149)
at org.apache.http.impl.conn.AbstractPooledConnAdapter.open(AbstractPooledConnAdapter.java:121)
at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:562)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:415)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:820)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:754)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:732)
at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:315)
... 64 more
ERROR LogFileStorageService: Storage request [ID#106] error: Unable to execute HTTP request: peer not authenticated
ERROR LogFileStorageService: Storage request [ID#106] FAILED 1 attempts, giving up