Opencast 15 with internal HTTPS fails to start web server

[s.la...@googlemail.com]

Hi there,

I upgraded our opencast test cluster from v13 to v15.
Now when I start opencast it throws the following error on me:

2024-04-11T15:49:00,027 | ERROR | (Activator:623) - Unable to start Pax Web server: Get Key failed: Given final block not properly padded. Such issues can arise if a bad key is used during decryption.
java.security.UnrecoverableKeyException: Get Key failed: Given final block not properly padded. Such issues can arise if a bad key is used during decryption.
at sun.security.pkcs12.PKCS12KeyStore.engineGetKey(PKCS12KeyStore.java:422) ~[?:?]
at sun.security.util.KeyStoreDelegator.engineGetKey(KeyStoreDelegator.java:90) ~[?:?]
at java.security.KeyStore.getKey(KeyStore.java:1057) ~[?:?]
at sun.security.ssl.SunX509KeyManagerImpl.<init>(SunX509KeyManagerImpl.java:145) ~[?:?]
at sun.security.ssl.KeyManagerFactoryImpl$SunX509.engineInit(KeyManagerFactoryImpl.java:76) ~[?:?]
at javax.net.ssl.KeyManagerFactory.init(KeyManagerFactory.java:271) ~[?:?]
at org.eclipse.jetty.util.ssl.SslContextFactory.getKeyManagers(SslContextFactory.java:1249) ~[?:?]
at org.eclipse.jetty.util.ssl.SslContextFactory$Server.getKeyManagers(SslContextFactory.java:2364) ~[?:?]
at org.eclipse.jetty.util.ssl.SslContextFactory.load(SslContextFactory.java:373) ~[?:?]
at org.eclipse.jetty.util.ssl.SslContextFactory.doStart(SslContextFactory.java:244) ~[?:?]
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73) ~[?:?]
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169) ~[?:?]
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117) ~[?:?]
at org.eclipse.jetty.server.SslConnectionFactory.doStart(SslConnectionFactory.java:97) ~[?:?]
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73) ~[?:?]
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169) ~[?:?]
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117) ~[?:?]
at org.eclipse.jetty.server.AbstractConnector.doStart(AbstractConnector.java:323) ~[?:?]
at org.eclipse.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:81) ~[?:?]
at org.eclipse.jetty.server.ServerConnector.doStart(ServerConnector.java:234) ~[?:?]
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73) ~[?:?]
at org.eclipse.jetty.server.Server.doStart(Server.java:401) ~[?:?]
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73) ~[?:?]
at org.ops4j.pax.web.service.jetty.internal.JettyServerWrapper.start(JettyServerWrapper.java:626) ~[?:?]
at org.ops4j.pax.web.service.jetty.internal.JettyServerController.start(JettyServerController.java:109) ~[?:?]
at org.ops4j.pax.web.service.internal.Activator.performConfiguration(Activator.java:555) ~[?:?]
at org.ops4j.pax.web.service.internal.Activator.updateController(Activator.java:445) ~[?:?]
at org.ops4j.pax.web.service.internal.Activator.lambda$updateServerControllerFactory$1(Activator.java:347) ~[?:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) [?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:829) [?:?]
Caused by: javax.crypto.BadPaddingException: Given final block not properly padded. Such issues can arise if a bad key is used during decryption.
at com.sun.crypto.provider.CipherCore.unpad(CipherCore.java:981) ~[?:?]
at com.sun.crypto.provider.CipherCore.fillOutputBuffer(CipherCore.java:1062) ~[?:?]
at com.sun.crypto.provider.CipherCore.doFinal(CipherCore.java:853) ~[?:?]
at com.sun.crypto.provider.PBES2Core.engineDoFinal(PBES2Core.java:323) ~[?:?]
at javax.crypto.Cipher.doFinal(Cipher.java:2202) ~[?:?]
at sun.security.pkcs12.PKCS12KeyStore.lambda$engineGetKey$0(PKCS12KeyStore.java:363) ~[?:?]
at sun.security.pkcs12.PKCS12KeyStore$RetryWithZero.run(PKCS12KeyStore.java:252) ~[?:?]
at sun.security.pkcs12.PKCS12KeyStore.engineGetKey(PKCS12KeyStore.java:357) ~[?:?]
... 33 more

We are using opencast with internal HTTPS/SSL without nginx as reverse proxy (our institutional loadbalancer is doing the reverse proxying for us).
I also tried v14 which throws the same error. Downgrading to v13 works, so opencast starts correctly.

So it seems to be a problem with SSL since v14? I also recreated my SSL key and certificate (using the documented way) to check if there was a problem. I'm using self signed SSL certificates for all opencast nodes.

Does anyone have a clue what is wrong? Do I miss something our could it be a bug?

[Greg Logan]

My initial, glib, response is using the internal https systems rather than nginx ;)

I think there's a disconnect here in your configuration. If your institutional load balancer is terminating the ssl connection you should configure Opencast as if you have a local nginx doing it - Opencast itself does not handle the ssl bits at all in either case.

G

To unsubscribe from this group and stop receiving emails from it, send an email to users+un...@opencast.org.

[s.la...@googlemail.com]

I found the error in my configuration. There were slight changes to org.ops4j.pax.web.cfg which aren't mentioned in https://docs.opencast.org/r/14.x/admin/#upgrade/ nor https://docs.opencast.org/r/15.x/admin/#upgrade/ (but they are changed in the HTTPS documentation ;-)). It was this change: https://github.com/opencast/opencast/commit/8f062357ece1c1bf54c4594aa301101e220411d4

Now it works with internal https.

[Katrin Ihler]

Related PR that explains the reason for this change: https://github.com/opencast/opencast/pull/5124.

This is mentioned in the release notes for example for 14.2, but it's a bit hidden. If you think this should be documented in the upgrade guide, would you be willing to create a small pull request?

Katrin

-- 
ELAN e.V.
Karlstr. 23
D-26123 Oldeburg

elan-ev.de