"Unable to start embedded Jetty server" and "Cannot recover key" when trying to start rundeck with SSL

[Tulio Campos]

Version: rundeck-3.3.3

OS: Red Hat Enterprise Linux release 8.2 (Ootpa)
Installation: RPM/DEB

I did these steps (these password are examples and only made to differentiate one from another):
keytool -keystore /etc/rundeck/ssl/keystore -alias rundeck -genkey -keyalg RSA -keypass password123 -storepass password789

then

cp /etc/rundeck/ssl/keystore /etc/rundeck/ssl/truststore

edited /etc/rundeck/ssl/ssl.properties to look like this

keystore=/etc/rundeck/ssl/keystorework
keystore.password=password789
key.password= password123
truststore=/etc/rundeck/ssl/truststorework
truststore.password=password789 

then edited /etc/rundeck/framework.properties to look like this

....

framework.server.name = hostname.domain
framework.server.hostname = hostname.domain
framework.server.port = 4443
framework.server.url = https://hostname.domain:4443

...

then edited /etc/rundeck/rundeck-config.properties to look like this

...

grails.serverURL=https://hostname.domain:4443

...

then created /etc/sysconfig/rundeckd with:

export RUNDECK_WITH_SSL=true
export RDECK_HTTPS_PORT=4443

I also tried with

RUNDECK_WITH_SSL=true
RDECK_HTTPS_PORT=4443

and then tried to run it with:

java -Drundeck.jaaslogin=true -Djava.security.auth.login.config=/etc/rundeck/jaas-loginmodule.conf -Dloginmodule.name=RDpropertyfilelogin -Drdeck.config=/etc/rundeck -Drundeck.server.configDir=/etc/rundeck -Dserver.datastore.path=/var/lib/rundeck/data/rundeck -Drundeck.server.serverDir=/var/lib/rundeck -Drdeck.projects=/var/lib/rundeck/projects -Dlog4j.configurationFile=/etc/rundeck/log4j2.properties -Dlogging.config=file:/etc/rundeck/log4j2.properties -Drdeck.runlogs=/var/lib/rundeck/logs -Drundeck.server.logDir=/var/lib/rundeck/logs -Drundeck.config.location=/etc/rundeck/rundeck-config.properties -Drundeck.ssl.config=/etc/rundeck/ssl/ssl.properties -Djava.io.tmpdir=/tmp/rundeck -Drundeck.server.workDir=/tmp/rundeck -Dserver.https.port=4443 -Drdeck.base=/var/lib/rundeck -Xmx1024m -Xms256m -XX:MaxMetaspaceSize=256m -server -jar /var/lib/rundeck/bootstrap/rundeck-3.3.3-20200910.war --skipinstall

And got this erros
[2020-11-10T14:13:23,569] ERROR boot.SpringApplication - Application run failed
org.springframework.boot.web.server.WebServerException: Unable to start embedded Jetty server
        at org.springframework.boot.web.embedded.jetty.JettyWebServer.start(JettyWebServer.java:166) ~[spring-boot-2.1.13.RELEASE.jar!/:2.1.13.RELEASE]
        at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.startWebServer(ServletWebServerApplicationContext.java:297) ~[spring-boot-2.1.13.RELEASE.jar!/:2.1.13.RELEASE]
        at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.finishRefresh(ServletWebServerApplicationContext.java:163) ~[spring-boot-2.1.13.RELEASE.jar!/:2.1.13.RELEASE]
        at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:552) ~[spring-context-5.1.14.RELEASE.jar!/:5.1.14.RELEASE]
        at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:141) ~[spring-boot-2.1.13.RELEASE.jar!/:2.1.13.RELEASE]
        at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:744) [spring-boot-2.1.13.RELEASE.jar!/:2.1.13.RELEASE]
        at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:391) [spring-boot-2.1.13.RELEASE.jar!/:2.1.13.RELEASE]
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:312) [spring-boot-2.1.13.RELEASE.jar!/:2.1.13.RELEASE]
        at grails.boot.GrailsApp.run(GrailsApp.groovy:96) [grails-core-4.0.3.jar!/:4.0.3]
        at grails.boot.GrailsApp.run(GrailsApp.groovy:456) [grails-core-4.0.3.jar!/:4.0.3]
        at grails.boot.GrailsApp.run(GrailsApp.groovy:443) [grails-core-4.0.3.jar!/:4.0.3]
        at grails.boot.GrailsApp$run.call(Unknown Source) [grails-core-4.0.3.jar!/:4.0.3]
        at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47) [groovy-2.5.6.jar!/:2.5.6]
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:115) [groovy-2.5.6.jar!/:2.5.6]
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:135) [groovy-2.5.6.jar!/:2.5.6]
        at rundeckapp.Application.main(Application.groovy:30) [classes!/:?]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_265]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_265]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_265]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_265]
        at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:48) [rundeck-3.3.3-20200910.war:?]
        at org.springframework.boot.loader.Launcher.launch(Launcher.java:87) [rundeck-3.3.3-20200910.war:?]
        at org.springframework.boot.loader.Launcher.launch(Launcher.java:51) [rundeck-3.3.3-20200910.war:?]
        at org.springframework.boot.loader.WarLauncher.main(WarLauncher.java:58) [rundeck-3.3.3-20200910.war:?]
Caused by: java.security.UnrecoverableKeyException: Cannot recover key
        at sun.security.provider.KeyProtector.recover(KeyProtector.java:315) ~[?:1.8.0_265]
        at sun.security.provider.JavaKeyStore.engineGetKey(JavaKeyStore.java:143) ~[?:1.8.0_265]
        at sun.security.provider.JavaKeyStore$JKS.engineGetKey(JavaKeyStore.java:57) ~[?:1.8.0_265]
        at sun.security.provider.KeyStoreDelegator.engineGetKey(KeyStoreDelegator.java:96) ~[?:1.8.0_265]
        at sun.security.provider.JavaKeyStore$DualFormatJKS.engineGetKey(JavaKeyStore.java:71) ~[?:1.8.0_265]
        at java.security.KeyStore.getKey(KeyStore.java:1023) ~[?:1.8.0_265]
        at sun.security.ssl.SunX509KeyManagerImpl.<init>(SunX509KeyManagerImpl.java:133) ~[?:1.8.0_265]
        at sun.security.ssl.KeyManagerFactoryImpl$SunX509.engineInit(KeyManagerFactoryImpl.java:70) ~[?:1.8.0_265]
        at javax.net.ssl.KeyManagerFactory.init(KeyManagerFactory.java:256) ~[?:1.8.0_265]
        at org.eclipse.jetty.util.ssl.SslContextFactory.getKeyManagers(SslContextFactory.java:1234) ~[jetty-util-9.4.26.v20200117.jar!/:9.4.26.v20200117]
        at org.eclipse.jetty.util.ssl.SslContextFactory$Server.getKeyManagers(SslContextFactory.java:2260) ~[jetty-util-9.4.26.v20200117.jar!/:9.4.26.v20200117]
        at org.eclipse.jetty.util.ssl.SslContextFactory.load(SslContextFactory.java:374) ~[jetty-util-9.4.26.v20200117.jar!/:9.4.26.v20200117]
        at org.eclipse.jetty.util.ssl.SslContextFactory.doStart(SslContextFactory.java:245) ~[jetty-util-9.4.26.v20200117.jar!/:9.4.26.v20200117]
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72) ~[jetty-util-9.4.26.v20200117.jar!/:9.4.26.v20200117]
        at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169) ~[jetty-util-9.4.26.v20200117.jar!/:9.4.26.v20200117]
        at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117) ~[jetty-util-9.4.26.v20200117.jar!/:9.4.26.v20200117]
        at org.eclipse.jetty.server.SslConnectionFactory.doStart(SslConnectionFactory.java:92) ~[jetty-server-9.4.26.v20200117.jar!/:9.4.26.v20200117]
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72) ~[jetty-util-9.4.26.v20200117.jar!/:9.4.26.v20200117]
        at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169) ~[jetty-util-9.4.26.v20200117.jar!/:9.4.26.v20200117]
        at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117) ~[jetty-util-9.4.26.v20200117.jar!/:9.4.26.v20200117]
        at org.eclipse.jetty.server.AbstractConnector.doStart(AbstractConnector.java:320) ~[jetty-server-9.4.26.v20200117.jar!/:9.4.26.v20200117]
        at org.eclipse.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:81) ~[jetty-server-9.4.26.v20200117.jar!/:9.4.26.v20200117]
        at org.eclipse.jetty.server.ServerConnector.doStart(ServerConnector.java:231) ~[jetty-server-9.4.26.v20200117.jar!/:9.4.26.v20200117]
        at org.springframework.boot.web.embedded.jetty.SslServerCustomizer$SslValidatingServerConnector.doStart(SslServerCustomizer.java:248) ~[spring-boot-2.1.13.RELEASE.jar!/:2.1.13.RELEASE]
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72) ~[jetty-util-9.4.26.v20200117.jar!/:9.4.26.v20200117]
        at org.springframework.boot.web.embedded.jetty.JettyWebServer.start(JettyWebServer.java:147) ~[spring-boot-2.1.13.RELEASE.jar!/:2.1.13.RELEASE]
        ... 23 more

I feel like I messed something while trying to use

java -jar rundeck.war --encryptpwd Jetty
For example, when I tried to "encrypt" the password it asked for a user and I used rundeck.

Thank you

[rac...@rundeck.com]

Hi Tulio,

Could you try these steps?

Regards!

[Tulio Campos]

I did from step 5 to 14 and got:

root@hostname:/etc/rundeck/ssl $ curl -k -I https://hostname.domain:4443/user/login
curl: (7) Failed to connect to hostname.domain port 4443: Connection refused

Also I think this is relevant:

root@hostname:/etc/rundeck/ssl $ keytool -importkeystore -destkeystore /etc/rundeck/ssl/keystore -srckeystore rundeck.p12 -srcstoretype pkcs12
Importing keystore rundeck.p12 to /etc/rundeck/ssl/keystore...
Enter destination keystore password:
Enter source keystore password:
Entry for alias 1 successfully imported.
Import command completed:  1 entries successfully imported, 0 entries failed or cancelled

Warning:
The JKS keystore uses a proprietary format. It is recommended to migrate to PKCS12 which is an industry standard format using "keytool -importkeystore -srckeystore /etc/rundeck/ssl/keystore -destkeystore /etc/rundeck/ssl/keystore -deststoretype pkcs12".

[rac...@rundeck.com]

Hi Tulio,

Check if SELinux is enabled.  If so,  try to disable it to discard.

Greetings.

[Tulio Campos]

I can't disable SELinux. It's against company policy and an extra layer of security.

What I just noticed is that I managed to make it "work" manually running:
java -Drundeck.jaaslogin=true -Djava.security.auth.login.config=/etc/rundeck/jaas-loginmodule.conf -Dloginmodule.name=RDpropertyfilelogin -Drdeck.config=/etc/rundeck -Drundeck.server.configDir=/etc/rundeck -Dserver.datastore.path=/var/lib/rundeck/data/rundeck -Drundeck.server.serverDir=/var/lib/rundeck -Drdeck.projects=/var/lib/rundeck/projects -Dlog4j.configurationFile=/etc/rundeck/log4j2.properties -Dlogging.config=file:/etc/rundeck/log4j2.properties -Drdeck.runlogs=/var/lib/rundeck/logs -Drundeck.server.logDir=/var/lib/rundeck/logs -Drundeck.config.location=/etc/rundeck/rundeck-config.properties -Djava.io.tmpdir=/tmp/rundeck -Drundeck.server.workDir=/tmp/rundeck -Dserver.http.port=4440 -Drdeck.base=/var/lib/rundeck -Xmx1024m -Xms256m -XX:MaxMetaspaceSize=256m -server -Drundeck.ssl.config=/etc/rundeck/ssl/ssl.properties -Dserver.https.port=4443 -jar /var/lib/rundeck/bootstrap/rundeck-3.3.3-20200910.war --skipinstall

But with

systemctl start rundeckd

it doesn't work, even though I copied that command line from the running process that started.

[rac...@rundeck.com]

Hi Tulio,

Remember that those parameters could be defined at rundeckd file (on /etc/sysconfig path in your case) to override the “default” params, take a look at this. This avoids some problems at the moment of upgrade your instance.

Hope it helps!

​

[Tulio Campos]

What I noticed is that the service runs on "rundeck" user while my testing is on root. Seems like permissions related but I made sure everything was chown rundeck and chmod was at least readable by this user.
And the files I would have to worry are the new ones which are the keystore, truststore, the keys (which I don't think I need them since it's on the store), ssl.properties, and the sysconfig file and that's all.
Am I missing any file that is used on SSL configuration that needs a different permission to the rundeck user?

[rac...@rundeck.com]

Hi Tulio,

What do you see on the service.log file at the moment of Rundeck systemd launch? It’s different of your first post? Perhaps the /etc/init.d/rundeckd or /etc/rundeck/profile files are edited?. Maybe a good approach is to follow these steps (defined from scratch) from a test/virtual environment to compare the file differences between both environments.

Regards!

​

[Tulio Campos]

I finally found out after checking the service.log which is REALLY BIG in terms of logs.
For some reason some random files in my rundeck had their owner changed to root, including some logs while other weren't changed. After a long time checking all these errors I managed to start it with SSL on the right user.

Thanks a lot for the help. It was of great use in my troubleshooting.

[rac...@rundeck.com]

Good news then Tulio, cheers!