go server / agent upgraded to 21.2.0 and agent lost contact

[Prakash K]

Hello everyone,

We have recently upgraded both server and agent to 21.2.0 and agent is still lost contact. Appreciate your thoughts and help. 

Go-server -> up and running.
Go-agent -> go-agent service is up and running but the agent is still in lost contact at server side.

we restarted both server and agent and no luck.

I also have another question reg go-agent-1, 2 3.. are these still applicable to new version 21.2.0?
When I run "service go-agent-1" we are still getting java version , home path error. Not sure if agent-1, 2, 3 are still valid in 21.2.0.

In the /var/log/go-agent/go-agent-launcher.log, we see the below error. Your help will be highly appreciated as we have several PROD instances running using 18.x versions and we need to upgrade them to 21.2.0 asap. so we are trying an upgrade in DEV and facing with these issues. Thank you.

Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target at java.base/sun.security.provider.certpath.SunCertPathBuilder.build(Unknown Source) at java.base/sun.security.provider.certpath.SunCertPathBuilder.engineBuild(Unknown Source) at java.base/java.security.cert.CertPathBuilder.build(Unknown Source) ... 51 common frames omitted 2021-05-26 13:58:56,480 ERROR [WrapperJarAppMain] ServerBinaryDownloader:88 - Couldn't update admin/agent-launcher.jar. Sleeping for 1m. Error: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target at java.base/sun.security.ssl.Alert.createSSLException(Unknown Source) at java.base/sun.security.ssl.TransportContext.fatal(Unknown Source) at java.base/sun.security.ssl.TransportContext.fatal(Unknown Source) at java.base/sun.security.ssl.TransportContext.fatal(Unknown Source) at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(Unknown Source) at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.onCertificate(Unknown Source) at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.consume(Unknown Source) at java.base/sun.security.ssl.SSLHandshake.consume(Unknown Source) at java.base/sun.security.ssl.HandshakeContext.dispatch(Unknown Source) at java.base/sun.security.ssl.HandshakeContext.dispatch(Unknown Source) at java.base/sun.security.ssl.TransportContext.dispatch(Unknown Source) at java.base/sun.security.ssl.SSLTransport.decode(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.decode(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:436) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:384) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108) at com.thoughtworks.go.agent.launcher.ServerBinaryDownloader.fetchUpdateCheckHeaders(ServerBinaryDownloader.java:104) at com.thoughtworks.go.agent.launcher.ServerBinaryDownloader.downloadIfNecessary(ServerBinaryDownloader.java:80) at com.thoughtworks.go.agent.launcher.AgentLauncherImpl.doLaunch(AgentLauncherImpl.java:88) at com.thoughtworks.go.agent.launcher.AgentLauncherImpl.lambda$launch$0(AgentLauncherImpl.java:68) at com.thoughtworks.go.logging.LogConfigurator.runWithLogger(LogConfigurator.java:62) at com.thoughtworks.go.agent.launcher.AgentLauncherImpl.launch(AgentLauncherImpl.java:68) at com.thoughtworks.go.agent.bootstrapper.AgentBootstrapper.go(AgentBootstrapper.java:76) at com.thoughtworks.go.agent.bootstrapper.AgentBootstrapper.lambda$main$0(AgentBootstrapper.java:57) at com.thoughtworks.go.logging.LogConfigurator.runWithLogger(LogConfigurator.java:53) at com.thoughtworks.go.agent.bootstrapper.AgentBootstrapper.main(AgentBootstrapper.java:57) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.base/java.lang.reflect.Method.invoke(Unknown Source) at com.thoughtworks.gocd.Boot.run(Boot.java:90) at com.thoughtworks.gocd.Boot.main(Boot.java:56) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.base/java.lang.reflect.Method.invoke(Unknown Source) at org.tanukisoftware.wrapper.WrapperJarApp.run(WrapperJarApp.java:451) at java.base/java.lang.Thread.run(Unknown Source) Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target at java.base/sun.security.validator.PKIXValidator.doBuild(Unknown Source) at java.base/sun.security.validator.PKIXValidator.engineValidate(Unknown Source) at java.base/sun.security.validator.Validator.validate(Unknown Source) at java.base/sun.security.ssl.X509TrustManagerImpl.checkTrusted(Unknown Source) at java.base/sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(Unknown Source) ... 46 common frames omitted Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target at java.base/sun.security.provider.certpath.SunCertPathBuilder.build(Unknown Source) at java.base/sun.security.provider.certpath.SunCertPathBuilder.engineBuild(Unknown Source) at java.base/java.security.cert.CertPathBuilder.build(Unknown Source) ... 51 common frames omitted

[Ashwanth Kumar]

> PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

This error usually means, SSL Certificate is not trusted on the client side (whoever is making a request). In this case it's the agent. Couple of patterns I have seen earlier:

1. Did you happen to downgrade the JVM version (to something old or is it already running something old?) and you're using a LetsEncrypt cert by any chance? I would recommend using the latest JVM that's supported by GoCD. 
2. Are you using any self-signed SSL certificate on the server (behind a reverse proxy or such)? If yes, you might want to import that into the agent's JVM truststore.

Thanks,

--
You received this message because you are subscribed to the Google Groups "go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email to go-cd+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/go-cd/592e3f9d-8634-4ccd-8f11-99ff49d7cb21n%40googlegroups.com.

--

Ashwanth Kumar / ashwanthkumar.in

[Aravind SV]

I agree with Ashwanth. It might also be a matter of trying to upgrade from a very old version to the latest, without considering changes in the versions in between.

For instance, 20.2.0 made some SSL/TLS changes which will need to be considered if upgrading from an old version. My suggestion would be to set up a test server and test agent and try the upgrade there, reading all the release notes in between for any major changes. The biggest changes usually will be Java version changes, one-time DB upgrade (around 20.5.0, I think) and SSL/TLS changes.

To view this discussion on the web visit https://groups.google.com/d/msgid/go-cd/CAD9m7CzQvsdtA1yyeGKMUnRUO0nbxhe5i9YPA23tUUwj6fzwTQ%40mail.gmail.com.

[Prakash K]

Thank you Ashwanth and Aravind. 

We have upgraded both server and agent to 21.2.0.. both are independent servers. Agents services are running active, not a problem. As part of the upgrade, we have upgraded java to jdk-13.0.2. Am I missing anything else along with this java upgrade?

To view this discussion on the web visit https://groups.google.com/d/msgid/go-cd/m2o8cxetca.fsf%40arvindsv.com.

[Prakash K]

Anoth quick note, we have already done the major upgrade to 20.x which includes db upgrade and TLS changes which all went well.  

[Marques Lee]

GoCD ships with JDK 15, unless you’re using the universal zip version. I’d try running on JDK 15 and see if you see the same behavior. All of our builds have been running JDK15 so it’s definitely the most (only, even?) exercised. 15 has our full confidence, so as a baseline try that to see if your situation changes.

To view this discussion on the web visit https://groups.google.com/d/msgid/go-cd/CAKD75xstizd__eoTxRm8Sx0zUCRySrhr5uXBu12QMMGfJ1YZVg%40mail.gmail.com.

[Prakash K]

All good champs.. It was a silly mistake.. JDK directory did not have perms set for go user.. Thanks for all your time and efforts, Cheers. 

We now have a server using JDK 11 and an agent using 13. Trying to match both to use the same version, higher version as possible

To view this discussion on the web visit https://groups.google.com/d/msgid/go-cd/CAPKX9jYLPj6D-xVDvqUoBckdxYoCogWeyMEtZc2pwX-1d6t9rg%40mail.gmail.com.