Cannot get Rundeck to startup with HTTPS enabled on CentOS 7

[Brian Clark]

Hello,

I am really excited to try out Rundeck, but hit a roadblock when I couldn't get Rundeck to startup after configuring it to use HTTPS. The service.log shows a lot of:  java.security.UnrecoverableKeyException: Cannot recover key

Anyone able to get Rundeck working successfully, via a new RPM install on CentOS 7.x?

Here's the main error that I keep seeing in the logs:

2016-01-22 17:11:41.089:WARN:oejuc.AbstractLifeCycle:FAILED SslContextFactory@351c421c(/etc/rundeck/ssl/keystore,/etc/rundeck/ssl/truststore): java.security.UnrecoverableKeyException: Cannot recover key

java.security.UnrecoverableKeyException: Cannot recover key

        at sun.security.provider.KeyProtector.recover(KeyProtector.java:328)

        at sun.security.provider.JavaKeyStore.engineGetKey(JavaKeyStore.java:138)

        at sun.security.provider.JavaKeyStore$JKS.engineGetKey(JavaKeyStore.java:55)

        at java.security.KeyStore.getKey(KeyStore.java:804)

        at sun.security.ssl.SunX509KeyManagerImpl.<init>(SunX509KeyManagerImpl.java:131)

        at sun.security.ssl.KeyManagerFactoryImpl$SunX509.engineInit(KeyManagerFactoryImpl.java:68)

        at javax.net.ssl.KeyManagerFactory.init(KeyManagerFactory.java:259)

        at org.eclipse.jetty.util.ssl.SslContextFactory.getKeyManagers(SslContextFactory.java:1077)

        at org.eclipse.jetty.util.ssl.SslContextFactory.doStart(SslContextFactory.java:288)

        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:59)

        at org.eclipse.jetty.server.ssl.SslSelectChannelConnector.doStart(SslSelectChannelConnector.java:607)

        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:59)

        at org.eclipse.jetty.server.Server.doStart(Server.java:272)

        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:59)

        at com.dtolabs.rundeck.RunServer.run(RunServer.java:118)

        at com.dtolabs.rundeck.RunServer.main(RunServer.java:78)

I had an IRC conversation but we couldn't figure out the issue (see below).

Thanks,
Brian

[16:57] <brian-stats> hello, having an issue getting ssl working for my rundeck server

[16:57] <brian-stats> brand new install

[16:57] <brian-stats> getting a lot of this in the logs:  java.security.UnrecoverableKeyException: Cannot recover key

[16:58] <brian-stats> followed the instructions here exactly:  http://rundeck.org/docs/administration/configuring-ssl.html

[16:58] <brian-stats> running on CentOS 7.2

[17:05] <Alex-SF> hi brian-stats

[17:05] <Alex-SF> brian-stats: maybe incorrect passphrase? http://stackoverflow.com/questions/15967650/caused-by-java-security-unrecoverablekeyexception-cannot-recover-key

[17:05] <Alex-SF> password

[17:06] <brian-stats> I don't think that's it

[17:06] <brian-stats> I recreated the keystore to ensure that I typed the password correctly

[17:07] <Alex-SF> paths pointing to desired location for trust/keystore?

[17:07] == blalor has changed nick to blalor_afk

[17:07] <brian-stats> and used the same password for both keypass and keystore

[17:07] == conn1 [~co...@79.97.1.211] has quit [Ping timeout: 240 seconds]

[17:08] <brian-stats> yes. my ssl.properties file points to the correct keystore and truststore files

[17:08] <brian-stats> Is there a requirement on the permissions for these two files?

[17:09] <Alex-SF> brian-stats: anything else in the log file? maybe something related

[17:09] <Alex-SF> brian-stats: just read perms

[17:09] <brian-stats> I have the files as owned by root but permissions are 644

[17:09] <Alex-SF> the rundeck process needs to read them

[17:09] <brian-stats> so they are world-readable

[17:11] <Alex-SF> brian-stats: is this RPM or deb install?

[17:11] <brian-stats> RPM

[17:11] <Alex-SF> can you check /etc/rundeck/profile to see if rundeck.ssl.config is passed as java flag?

[17:12] <brian-stats> yes, give me a sec

[17:13] <brian-stats> where do I check for that?

[17:14] <Alex-SF> brian-stats: grep RDECK_JVM

[17:15] <Alex-SF> make sure that line isn't commented

[17:15] <brian-stats> I do have this line uncommented in my profile file:  export RDECK_JVM="$RDECK_JVM -Drundeck.ssl.config=/etc/rundeck/ssl/ssl.properties -Dserver.https.port=${RDECK_HTTPS_PORT}"

[17:15] <Alex-SF> ok

[17:16] <Alex-SF> is /etc/rundeck/ssl/ssl.properties the right path? it might be /etc/rundeck/ssl.properties

[17:16] <brian-stats> on my system, it is /etc/rundeck/ssl/ssl.properties

[17:17] <Alex-SF> ok just want to be sure they are as intended

[17:17] <Alex-SF> can you show the log file?

[17:17] <brian-stats> sure. which one?

[17:17] <Alex-SF> service.log

[17:18] <brian-stats> 2016-01-22 17:11:17.029:INFO:oejs.Server:jetty-7.6.0.v20120127 2016-01-22 17:11:19.403:INFO:oejw.StandardDescriptorProcessor:NO JSP Support for /, did not find org.apache.jasper.servlet.JspServlet 2016-01-22 17:11:20.528:INFO:/:Initializing Spring root WebApplicationContext INFO  BootStrap: Starting Rundeck 2.6.2-1... INFO  BootStrap: using rdeck.base config property: /var/lib/rundeck INFO  BootStrap: loaded configuration: /etc/rundeck/framework.properties INFO  Boo

[17:19] <brian-stats> doesn't look like that copied in correctly

[17:19] <Alex-SF> gist is better

[17:20] <brian-stats> take a look here:  http://pastebin.com/5ccqyUHf

[17:23] <Alex-SF> hmm, not much more than what you already said

[17:25] <brian-stats> the last line before the first error is related to grails

[17:26] <Alex-SF> which line#?

[17:26] <brian-stats> 10

[17:27] <brian-stats> I checked, and my /etc/rundeck/rundeck-config.properties entry for grails.serverURL looks ok

[17:27] <Alex-SF> brian-stats: even if that was wrong it should not throw that error

[17:27] <brian-stats> it is the full URL, https://servername:4443

[17:28] <Alex-SF> right

[17:28] <Alex-SF> only thing google searches seem to indicate is password or file corruption.

[17:29] <brian-stats> yes, that's what I found too

[17:29] == okok [~o...@96.56.63.50] has quit [Quit: peace]

[17:30] == jyaworski [~jyaworski@fsf/member/jyaworski] has quit [Ping timeout: 264 seconds]

[17:33] <brian-stats> here's something interesting

[17:33] <brian-stats> https://gist.github.com/anonymous/3ee429dff611ef6c40e9

[17:34] <brian-stats> When I try and start up rundeck, it seems to still have some references to the normal HTTP port of 4440

[17:35] <brian-stats> even though I have it configured everywhere I can to use SSL and Port 4443

[17:35] <Alex-SF> right

[17:35] <Alex-SF> check /etc/rundeck/framework.properties

[17:35] <Alex-SF> look at framework.server.port

[17:36] <brian-stats> this is correctly set to 4443

[17:37] <brian-stats> and the framework.server.url has the full url of https://servername:4443

[17:37] <Alex-SF> grep 4440 /etc/rundeck/*

[17:38] <brian-stats> and the framework.server.url has the full url of https://servername:4443

[17:38] <brian-stats> ]$ sudo grep 4440 /etc/rundeck/* /etc/rundeck/profile:RDECK_HTTP_PORT=4440 grep: /etc/rundeck/ssl: Is a directory

[17:38] <Alex-SF> i think that var isnt even used

---

[17:42] <brian-stats> I have to take off, and log out of htis IRC

[17:42] <brian-stats> I can create a new topic on the mailing list

[17:42] <brian-stats> do you think that would help?

[17:42] <brian-stats> not sure if you also follow the mailing list

[17:42] <Alex-SF> ya

[Brian Clark]

I found the issue. The problem was that I had two '$' characters in my password. That was somehow interfering with how they were saved by keytool. When I created a new keystore with a password that did not contain any $ characters, it worked fine.

Thanks Alex-SF for trying to help me via IRC. If you have any control over the Runscope SSL configuration documentation, I suggest putting in a warning against trying to use the $ character in either the key or keystore passwords.

Thanks!

Brian

[Rob]

This post helped me!!!    Thank you!

[Art Hill]

I realize that your issue was solved, but I found this page because I had a very similar error.  I was using a random password generator for the keypass and storepass.  Turns out that there is a limit to how long those may be.  I don't know what that limit is, but I can tell you that a 12 character password works.  (And no, I did not have a "$" in either, I experimented with only upper, lower, and numbers with no special.)  

My error:

2018-09-27 14:16:35.157:WARN:oejuc.AbstractLifeCycle:FAILED SslContextFactory@130ad58d(/etc/keystore/ecckeystore.jks,/etc/rundeck/ssl/truststore): java.io.IOException: Keystore was tampered with, or password was incorrect

I hope this helps someone.

On Friday, January 22, 2016 at 5:50:15 PM UTC-6, Brian Clark wrote:

[Michael Ellis]

Brian,

I am trying to get Rundeck to work with https.  For now I followed the directions, created a keystore and copied to truststore in /etc/rundeck/ssl . I used the adminadmin password for now so did not edit the ssl.properties file.  Changed the framework and rundeck-config files to use port 4443.  I made sure firewalld is off.  Can not get the login page to come up.  Works fine with http.  using latest version.  How did you get htis to work?

[rac...@rundeck.com]

Hi Michael,

I have a procedure that it works for me:

1. Create keystore:

keytool -keystore /etc/rundeck/ssl/keystore -alias rundeck -genkey -keyalg RSA -keypass password -storepass password

2. Copy keystore as truststore (as you did it).

3.- Edit /etc/rundeck/ssl/ssl.properties file:

keystore=/etc/rundeck/ssl/keystore
keystore.password=password
key.password=password
truststore=/etc/rundeck/ssl/truststore
truststore.password=password

4.-Edit /etc/rundeck/framework.properties file:

framework.server.port = 4443
framework.server.url = https://localhost:4443

5. Edit /etc/rundeck/rundeck-config.properties file:

grails.serverURL=https://localhost:4443

6. Edit or create /etc/sysconfig/rundeckd file (on CentOS/RHEL, in Debian/Ubuntu the path is /etc/defaults/rundeckd):

export RUNDECK_WITH_SSL=true

Hope it helps!