OC 8 Update - Servers offline

75 views
Skip to first unread message

hans.t...@gmail.com

unread,
Feb 6, 2021, 12:49:09 PM2/6/21
to Opencast Users
Hi,

I did an update on our OC 8 three Server setup. After that, all the servers (Admin node, Presentation an Worker) are marked as offline.

Looking in the logs I find those entries which maybe related to the issue:


####################################################

2021-02-06T18:43:53,972 | WARN  | (JavaLog:302) -
org.eclipse.persistence.exceptions.DatabaseException:
Internal Exception: com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException: Duplicate entry 'mh_default_org--1-medien.example.at ' for key 'PRIMARY'
Error Code: 1062
Call: INSERT INTO oc_organization_node (organization, port, name) VALUES (?, ?, ?)
        bind => [3 parameters bound]
######################################################


And also:

####################################################
2021-02-06T18:08:03,481 | WARN  | (ServiceRegistryJpaImpl$JobProducerHeartbeat:3389) - Added org.opencastproject.caption@https://medien-admin.example.at to the watch list
2021-02-06T18:08:03,499 | WARN  | (ServiceRegistryJpaImpl$JobProducerHeartbeat:3378) - Unable to reach org.opencastproject.ingest@https://medien-admin.example.at : {}
org.opencastproject.security.api.TrustedHttpClientException: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
        at org.opencastproject.kernel.security.TrustedHttpClientImpl.execute(TrustedHttpClientImpl.java:394) ~[?:?]
        at org.opencastproject.kernel.security.TrustedHttpClientImpl.execute(TrustedHttpClientImpl.java:346) ~[?:?]
        at org.opencastproject.serviceregistry.impl.ServiceRegistryJpaImpl$JobProducerHeartbeat.run(ServiceRegistryJpaImpl.java:3350) [131:opencast-serviceregistry:8.10.0]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:?]
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:?]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:?]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:?]
        at java.lang.Thread.run(Thread.java:748) [?:?]
Caused by: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
        at sun.security.ssl.Alert.createSSLException(Alert.java:131) ~[?:?]
        at sun.security.ssl.TransportContext.fatal(TransportContext.java:324) ~[?:?]
        at sun.security.ssl.TransportContext.fatal(TransportContext.java:267) ~[?:?]
        at sun.security.ssl.TransportContext.fatal(TransportContext.java:262) ~[?:?]
        at sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(CertificateMessage.java:654) ~[?:?]
        at sun.security.ssl.CertificateMessage$T12CertificateConsumer.onCertificate(CertificateMessage.java:473) ~[?:?]
###########################################################



##################################################
2021-02-06T18:11:04,367 | WARN  | (ServiceRegistryJpaImpl$JobProducerHeartbeat:3378) - Unable to reach org.opencastproject.distribution.aws.s3@https://medien.example.at : {}
org.opencastproject.security.api.TrustedHttpClientException: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
        at org.opencastproject.kernel.security.TrustedHttpClientImpl.execute(TrustedHttpClientImpl.java:394) ~[?:?]
        at org.opencastproject.kernel.security.TrustedHttpClientImpl.execute(TrustedHttpClientImpl.java:346) ~[?:?]
        at org.opencastproject.serviceregistry.impl.ServiceRegistryJpaImpl$JobProducerHeartbeat.run(ServiceRegistryJpaImpl.java:3350) [131:opencast-serviceregistry:8.10.0]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:?]
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:?]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:?]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:?]
        at java.lang.Thread.run(Thread.java:748) [?:?]
Caused by: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
        at sun.security.ssl.Alert.createSSLException(Alert.java:131) ~[?:?]
        at sun.security.ssl.TransportContext.fatal(TransportContext.java:324) ~[?:?]
        at sun.security.ssl.TransportContext.fatal(TransportContext.java:267) ~[?:?]
        at sun.security.ssl.TransportContext.fatal(TransportContext.java:262) ~[?:?]
        at sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(CertificateMessage.java:654) ~[?:?]
        at sun.security.ssl.CertificateMessage$T12CertificateConsumer.onCertificate(CertificateMessage.java:473) ~[?:?]
        at sun.security.ssl.CertificateMessage$T12CertificateConsumer.consume(CertificateMessage.java:369) ~[?:?]
        at sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:377) ~[?:?]
        at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:444) ~[?:?]
        at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:422) ~[?:?]
        at sun.security.ssl.TransportContext.dispatch(TransportContext.java:182) ~[?:?]
        at sun.security.ssl.SSLTransport.decode(SSLTransport.java:149) ~[?:?]
        at sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1143) ~[?:?]
        at sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1054) ~[?:?]
        at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:394) ~[?:?]
        at org.apache.http.conn.ssl.SSLSocketFactory.createLayeredSocket(SSLSocketFactory.java:570) ~[?:?]
        at org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:554) ~[?:?]
        at org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:415) ~[?:?]
        at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:180) ~[?:?]
        at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:326) ~[?:?]
        at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:605) ~[?:?]
        at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:440) ~[?:?]
        at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:835) ~[?:?]


Can somebody see whats going wrong?


Kind regards,
Hans

Rainer Rillke

unread,
Feb 6, 2021, 12:51:44 PM2/6/21
to Opencast Users, hans.t...@gmail.com
Hi Hans,

my bet for the last 2 error messages is that you are either using self-signed certificates or the certificate chain is incomplete or the root certificate is not trusted by Java. Please have a look at https://docs.opencast.org/develop/admin/#configuration/https/self-signed-certificates/

Kind regards
Rillke

hans.t...@gmail.com

unread,
Feb 6, 2021, 1:57:25 PM2/6/21
to Opencast Users, rainer...@gmail.com, hans.t...@gmail.com
Thank you, fixed in in the meantime myself. 
You are right, it had to do with the certificate chain. I had to include my ca-chain .pem in the Nginx configuration.
I think this is more strict now when chacking the server certificate.

Kind regards,
Hans

Greg Logan

unread,
Feb 6, 2021, 9:32:26 PM2/6/21
to Opencast Users
Correct, this fix was the result of a security issue where validation wasn't being done at all.

In terms of the duplicate key issue, that sounds to me like a misconfiguration in the organization config files. What changes have you made to those files?

G

--
To unsubscribe from this group and stop receiving emails from it, send an email to users+un...@opencast.org.

hans.t...@gmail.com

unread,
Feb 7, 2021, 1:49:02 AM2/7/21
to Opencast Users, Greg Logan
Hi Greg,

changed nothing recently and definetely nothing during the update. I fixed it somehow now by deleting everything in the oc_organization_node table. After a restart everything seems ok now.

Kind regards,
Hans

Reply all
Reply to author
Forward
0 new messages