CAS 7 Embedded Tomcat with PEM certs

[Jeremiah Garmatter]

Has the APR functionality been removed or replaced on CAS 7's embedded Tomcat server?

I wanted to use PEM encoded SSL certificates instead of the Java keystore. When I use the APR configurations in cas.properties I receive errors that the properties failed to bind and they are no longer recognized by CAS 7.0.4.1. Here are the configurations I tried:

cas.server.tomcat.apr.enabled=true

cas.server.tomcat.apr.ssl-certificate-file=/opt/workspace/cas7/my_certs/cert.pem

cas.server.tomcat.apr.ssl-certificate-key-file=/opt/workspace/cas7/my_certs/key.pem

This page doesn't mention any configurations for APR: https://apereo.github.io/cas/7.0.x/installation/Servlet-Container-Embedded-Tomcat-APR.html

If this feature has been removed, is there another method to use PEM encoded SSL certs on the embedded Tomcat server?

[Jeremiah Garmatter]

I tracked down a few PEM SSL options here: https://apereo.github.io/cas/7.0.x/installation/Configuring-Servlet-Container-Embedded.html

Specifically, "server.ssl.certificate-private-key" and "server.ssl.certificate". I put my self-signed cert and key paths into the respective server.ssl setting but when I run the cas.war file I receive java keystore errors.

The last errors relate to bad padding with a "PKCS12KeyStore". Looks like the embedded Tomcat server wants a pkcs12 keystore only. Is there a way to disable the keystore check so the embedded Tomcat will use the cert and key I specified?

The errors:

Exception in thread "main" java.lang.reflect.InvocationTargetException
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:118)
at java.base/java.lang.reflect.Method.invoke(Method.java:580)
at org.springframework.boot.loader.launch.Launcher.launch(Launcher.java:91)
at org.springframework.boot.loader.launch.Launcher.launch(Launcher.java:53)
at org.springframework.boot.loader.launch.WarLauncher.main(WarLauncher.java:57)
Caused by: org.springframework.context.ApplicationContextException: Failed to start bean 'webServerStartStop'
at org.springframework.context.support.DefaultLifecycleProcessor.doStart(DefaultLifecycleProcessor.java:287)
at org.springframework.context.support.DefaultLifecycleProcessor$LifecycleGroup.start(DefaultLifecycleProcessor.java:467)
at java.base/java.lang.Iterable.forEach(Iterable.java:75)
at org.springframework.context.support.DefaultLifecycleProcessor.startBeans(DefaultLifecycleProcessor.java:256)
at org.springframework.context.support.DefaultLifecycleProcessor.onRefresh(DefaultLifecycleProcessor.java:201)
at org.springframework.context.support.AbstractApplicationContext.finishRefresh(AbstractApplicationContext.java:979)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:628)
at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:146)
at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:762)
at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:464)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:334)
at org.springframework.boot.builder.SpringApplicationBuilder.run(SpringApplicationBuilder.java:149)
at org.apereo.cas.web.CasWebApplication.main(CasWebApplication.java:57)
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
... 4 more
Caused by: org.springframework.boot.web.server.WebServerException: Unable to start embedded Tomcat server
at org.springframework.boot.web.embedded.tomcat.TomcatWebServer.start(TomcatWebServer.java:249)
at org.springframework.boot.web.servlet.context.WebServerStartStopLifecycle.start(WebServerStartStopLifecycle.java:44)
at org.springframework.context.support.DefaultLifecycleProcessor.doStart(DefaultLifecycleProcessor.java:284)
... 17 more
Caused by: java.lang.IllegalArgumentException: standardService.connector.startFailed
at org.apache.catalina.core.StandardService.addConnector(StandardService.java:222)
at org.springframework.boot.web.embedded.tomcat.TomcatWebServer.addPreviouslyRemovedConnectors(TomcatWebServer.java:306)
at org.springframework.boot.web.embedded.tomcat.TomcatWebServer.start(TomcatWebServer.java:234)
... 19 more
Caused by: org.apache.catalina.LifecycleException: Protocol handler start failed
at org.apache.catalina.connector.Connector.startInternal(Connector.java:1047)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:171)
at org.apache.catalina.core.StandardService.addConnector(StandardService.java:219)
... 21 more
Caused by: java.lang.IllegalArgumentException: Get Key failed: Given final block not properly padded. Such issues can arise if a bad key is used during decryption.
at org.apache.tomcat.util.net.AbstractJsseEndpoint.createSSLContext(AbstractJsseEndpoint.java:114)
at org.apache.tomcat.util.net.AbstractJsseEndpoint.initialiseSsl(AbstractJsseEndpoint.java:70)
at org.apache.tomcat.util.net.NioEndpoint.bind(NioEndpoint.java:199)
at org.apache.tomcat.util.net.AbstractEndpoint.bindWithCleanup(AbstractEndpoint.java:1286)
at org.apache.tomcat.util.net.AbstractEndpoint.start(AbstractEndpoint.java:1372)
at org.apache.coyote.AbstractProtocol.start(AbstractProtocol.java:635)
at org.apache.catalina.connector.Connector.startInternal(Connector.java:1044)
... 23 more
Caused by: java.security.UnrecoverableKeyException: Get Key failed: Given final block not properly padded. Such issues can arise if a bad key is used during decryption.
at java.base/sun.security.pkcs12.PKCS12KeyStore.engineGetKey(PKCS12KeyStore.java:454)
at java.base/sun.security.util.KeyStoreDelegator.engineGetKey(KeyStoreDelegator.java:93)
at java.base/java.security.KeyStore.getKey(KeyStore.java:1075)
at org.apache.tomcat.util.net.SSLUtilBase.getKeyManagers(SSLUtilBase.java:393)
at org.apache.tomcat.util.net.SSLUtilBase.createSSLContext(SSLUtilBase.java:268)
at org.apache.tomcat.util.net.AbstractJsseEndpoint.createSSLContext(AbstractJsseEndpoint.java:112)
... 29 more
Caused by: javax.crypto.BadPaddingException: Given final block not properly padded. Such issues can arise if a bad key is used during decryption.
at java.base/com.sun.crypto.provider.CipherCore.unpad(CipherCore.java:861)
at java.base/com.sun.crypto.provider.CipherCore.fillOutputBuffer(CipherCore.java:941)
at java.base/com.sun.crypto.provider.CipherCore.doFinal(CipherCore.java:734)
at java.base/com.sun.crypto.provider.PBES2Core.engineDoFinal(PBES2Core.java:203)
at java.base/javax.crypto.Cipher.doFinal(Cipher.java:2244)
at java.base/sun.security.pkcs12.PKCS12KeyStore.lambda$engineGetKey$0(PKCS12KeyStore.java:370)
at java.base/sun.security.pkcs12.PKCS12KeyStore$RetryWithZero.run(PKCS12KeyStore.java:257)
at java.base/sun.security.pkcs12.PKCS12KeyStore.engineGetKey(PKCS12KeyStore.java:361)
... 34 more

[Jeremiah Garmatter]

I could use some help,

After my pem issues (see prior emails in chain) I tried my hand at the keystore.

I used the gradle task to build a generic keystore with ./gradlew createKeystore. This worked fine to get me through the rest of my build process.

Now I am ready to deploy to a test cluster but I can not get my new keystore to work.

I created a keystore, generated new csr, and got the csr approved and retrieved my new certificates, and imported them (see keytool commands below).

No errors during that process.

Then I move the keystore to /etc/cas/ and update /etc/cas/properties with:

server.ssl.key-store=/etc/cas/cas.jks
server.ssl.key-store-password=REDACTED

Which are the same settings used with the default "thekeystore" and password "changeit".

However, every time I restart my CAS instance the embedded tomcat server throws the same errors as my pem attempts earlier in this email chain. I included a summary of the trace below.

I'm very confused by this issue. I would love to simplify my deployment by keeping CAS and Tomcat together.

Am I missing something very simple? Has anyone else had luck with their own keystore on the embedded tomcat instance? Is it because I have SANs in my cert or is something messed up with the key?

Create new keystore and key:

keytool -genkey -alias sso -keystore cas.jks -storepass 'REDACTED' -keyalg rsa -keysize 2048 -dname "CN=My host,O=my org,L=my localisty,S=Ohio,C=US,OU=Information Technology,EMAILADDRESS=my work email" -ext san=dns:host1,dns:host2,etc...

Then the CSR:

keytool -certreq -alias sso -keystore cas.jks -storepass 'REDACTED' -file my.csr

Got it approved.

Split the chain into individual cert files.

Then imported the individual certificates with these commands:

keytool -import -alias root -keystore cas.jks -storepass 'REDACTED' -file root.pem

keytool -import -alias usertrust -keystore cas.jks -storepass 'REDACTED' -file usertrust.pem

keytool -import -alias sso -keystore cas.jks -storepass 'REDACTED' -file host.pem

Error summary:

WARN [org.springframework.boot.web.servlet.context.AnnotationConfigServletWebServerApplicationContext] - <Exception encountered during context initialization - cancelling refresh attempt: org.springframework.context.ApplicationContextException: Failed to start bean 'webServerStartStop'>

Exception in thread "main" java.lang.reflect.InvocationTargetException

Caused by: org.springframework.context.ApplicationContextException: Failed to start bean 'webServerStartStop'

Caused by: org.springframework.boot.web.server.WebServerException: Unable to start embedded Tomcat server

Caused by: java.lang.IllegalArgumentException: standardService.connector.startFailed

Caused by: org.apache.catalina.LifecycleException: Protocol handler start failed

Caused by: java.lang.IllegalArgumentException: Get Key failed: Given final block not properly padded. Such issues can arise if a bad key is used during decryption.

Caused by: java.security.UnrecoverableKeyException: Get Key failed: Given final block not properly padded. Such issues can arise if a bad key is used during decryption.

Caused by: javax.crypto.BadPaddingException: Given final block not properly padded. Such issues can arise if a bad key is used during decryption.

[Jeremiah Garmatter]

In case this helps anyone in the future. I was able to track down the issue. The error stacktrace  is misleading as it mentions some sort of key padding issue on PKCS12 keys. The real issue is due to key store passwords and default values for the related configurations.

My keystore does not use the default "changeit" password. I set this option in cas.properties but I still received the padding error.

server.ssl.key-store-password=my password

Once I added this configuration, my key and cert were accepted. I do not have an additional password on the key file so I do not understand why this had to be set to the same password as the above config.

server.ssl.key-password=my password

Curious about how this might impact my original plan I tried again with the PEM-encoded certificate. My key file does not have a password on it so I set:

server.ssl.key-password=

In other words, I set the configuration to an empty string. Sure enough, CAS accepted the PEM certificate and private key.

I would've thought that an empty string is the default option as that is how it is depicted on the following CAS documentation: https://apereo.github.io/cas/7.0.x/installation/Configuring-Servlet-Container-Embedded.html

I can only imagine CAS passes some other value, possibly "changeit" by default into the server.ssl.key-password. Hopefully this helps others stuck on the same problem.