Issue with go-agent on Rocky 8

[Rasmus M]

Hi there,

I have a bit of an issue with the GoCD agent on a Rocky 8 x86 installation.

Whenever I launch the systemd go-agent, the following exception is produced in the /var/log/go-agent/go-agent.log file:

2022-04-06 19:32:44,065 ERROR [scheduler-1] AgentController:92 - [Agent Loop] Error occurred during loop:  
java.lang.reflect.InaccessibleObjectException: Unable to make field private java.lang.String java.util.regex.Pattern.pattern accessible: module java.base does not "opens java.util
.regex" to unnamed module @4e04a765
       at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(Unknown Source)
       at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(Unknown Source)
       at java.base/java.lang.reflect.Field.checkCanSetAccessible(Unknown Source)
       at java.base/java.lang.reflect.Field.setAccessible(Unknown Source)
       at com.google.gson.internal.reflect.UnsafeReflectionAccessor.makeAccessible(UnsafeReflectionAccessor.java:44)
       at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory.getBoundFields(ReflectiveTypeAdapterFactory.java:159)
       at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory.create(ReflectiveTypeAdapterFactory.java:102)
       at com.google.gson.Gson.getAdapter(Gson.java:458)
       at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory.createBoundField(ReflectiveTypeAdapterFactory.java:117)
       at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory.getBoundFields(ReflectiveTypeAdapterFactory.java:166)
       at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory.create(ReflectiveTypeAdapterFactory.java:102)
       at com.google.gson.Gson.getAdapter(Gson.java:458)
       at com.google.gson.internal.bind.CollectionTypeAdapterFactory.create(CollectionTypeAdapterFactory.java:53)
       at com.google.gson.Gson.getAdapter(Gson.java:458)
       at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory.createBoundField(ReflectiveTypeAdapterFactory.java:117)
       at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory.getBoundFields(ReflectiveTypeAdapterFactory.java:166)
       at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory.create(ReflectiveTypeAdapterFactory.java:102)
       at com.google.gson.Gson.getDelegateAdapter(Gson.java:541)
       at com.thoughtworks.go.remote.adapter.RuntimeTypeAdapterFactory.create(RuntimeTypeAdapterFactory.java:238)
       at com.google.gson.Gson.getAdapter(Gson.java:458)
       at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory.createBoundField(ReflectiveTypeAdapterFactory.java:117)
       at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory.getBoundFields(ReflectiveTypeAdapterFactory.java:166)
       at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory.create(ReflectiveTypeAdapterFactory.java:102)
       at com.google.gson.Gson.getAdapter(Gson.java:458)
       at com.google.gson.internal.bind.CollectionTypeAdapterFactory.create(CollectionTypeAdapterFactory.java:53)
       at com.google.gson.Gson.getAdapter(Gson.java:458)
       at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory.createBoundField(ReflectiveTypeAdapterFactory.java:117)
       at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory.getBoundFields(ReflectiveTypeAdapterFactory.java:166)
       at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory.create(ReflectiveTypeAdapterFactory.java:102)
       at com.google.gson.Gson.getAdapter(Gson.java:458)
       at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory.createBoundField(ReflectiveTypeAdapterFactory.java:117)
       at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory.getBoundFields(ReflectiveTypeAdapterFactory.java:166)
       at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory.create(ReflectiveTypeAdapterFactory.java:102)
       at com.google.gson.Gson.getAdapter(Gson.java:458)
       at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory.createBoundField(ReflectiveTypeAdapterFactory.java:117)
       at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory.getBoundFields(ReflectiveTypeAdapterFactory.java:166)
       at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory.create(ReflectiveTypeAdapterFactory.java:102)
       at com.google.gson.Gson.getDelegateAdapter(Gson.java:541)
       at com.thoughtworks.go.remote.adapter.RuntimeTypeAdapterFactory.create(RuntimeTypeAdapterFactory.java:238)
       at com.google.gson.Gson.getAdapter(Gson.java:458)
       at com.google.gson.Gson.fromJson(Gson.java:931)
       at com.google.gson.Gson.fromJson(Gson.java:897)
       at com.google.gson.Gson.fromJson(Gson.java:846)
       at com.google.gson.Gson.fromJson(Gson.java:817)
       at com.thoughtworks.go.agent.RemotingClient.getWork(RemotingClient.java:79)
       at com.thoughtworks.go.agent.AgentHTTPClientController.retrieveWork(AgentHTTPClientController.java:153)
       at com.thoughtworks.go.agent.AgentHTTPClientController.work(AgentHTTPClientController.java:130)
       at com.thoughtworks.go.agent.AgentController.loop(AgentController.java:86)
       at jdk.internal.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
       at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
       at java.base/java.lang.reflect.Method.invoke(Unknown Source)
       at org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:65)
       at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
       at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
       at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source)
       at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
       at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
       at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
       at java.base/java.lang.Thread.run(Unknown Source)

The Rocky 8 VM is currently using Java 17:

[root@mbv01 ~]# java --version
openjdk 17.0.2 2022-01-18 LTS
OpenJDK Runtime Environment 21.9 (build 17.0.2+8-LTS)
OpenJDK 64-Bit Server VM 21.9 (build 17.0.2+8-LTS, mixed mode, sharing)

But I have also tried with java-8-openjdk and java-11-openjdk without it having any effect on the produced Exception.

Do you have any tips on how to resolve this?

Best Regards

Rasmus

[Rasmus M]

Just as a followup,

The current go-agent I have installed is version 22.1.0-13913:

[root@mbv01 ~]# yum list --installed | grep gocd
go-agent.noarch                             22.1.0-13913                                   @gocd

While the server is running version v21.3.0.

gocd_manager.1.djkifkhv5351@mbs01    | jvm 1    | [Wed Apr 06 16:47:28 CEST 2022]            GoCD Version: 21.3.0-13067
gocd_manager.1.djkifkhv5351@mbs01    | jvm 1    | [Wed Apr 06 16:47:28 CEST 2022]            Java Version: 15.0.2

Could the version difference between the client and the server be the cause of this?

Best Regards

Rasmus

[Chad Wilson]

Hi Rasmus

Thanks for the details.

It's a little difficult to explain, but the agent code is actually "bootstrapped" from the server when it starts. Basically your "agent version" is actually just the version of a "bootstrapper" that talks to the server, downloads the agent code that matches the server's version, and then launches the "actual" agent using the same JVM as the bootstrapper. That creates some complications with Java compatibility over time which you are going through here due to breaking changes made in Java 16/17. In any case, the issue is that your server version 21.3.0 has agent code that will be dynamically bootstrapped to your agents that will not work on Java 16-17 without tweaks to the agent config.

Technically only agent bootstrapper + server 22.1.0 support running Java 16+: https://www.gocd.org/releases/#22-1-0

Having said that, it's not possible to get the exception as above with Java 8 and Java 11 (this exception is specific to Java 17) so when you were trying to switch JVMs it's likely there is an issue and it's actually still using the bundled Java 17. The agent bootstrapper does log the JVM at startup IIRC so you should be able to see the version it thinks it is using.

For that specific server version your agent host would need to have Java 13-15 available (as documented)

In your specific case you have at least these options:

1. ensure your agent bootstrapper 22.1.0 is running with Java 13-15 without changing server version. Java 11 won't work with your chosen server version, even if you do correct it to use a different Java version.
   
2. upgrade server to 21.4.0 (still with Java 15 on server) or 22.1.0 (with any of Java 11-17 on server) without changing your agent bootstrapper version. The agent code bundled with 21.4.0+ should work fine under Java 17 with your agent bootstrapper.
3. without changing any versions you mention above, use the instructions at https://github.com/gocd/gocd/issues/10265 to modify your AGENT_STARTUP_ARGS to work with Java 17. Note the instructions under "If you experience other issues and downgrade your server to <= 21.3.0 while leaving agents running under a 22.1.0+ bootstrapper and Java 17 you are likely to get errors like these in your agents logs." which is your specific case.
4. upgrade server to 22.1.0 and run both server and agent under the default bundled JVM of Java 17. This is probably the recommended path due to security issues that exist on 21.3.0 and 21.4.0, as well as moving back to LTS Java versions. There are no major functional changes between these versions and 22.1.0.
   

-Chad

--
You received this message because you are subscribed to the Google Groups "go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email to go-cd+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/go-cd/70bcb389-92e0-4fd8-9840-75ef0aa95dden%40googlegroups.com.

[Rasmus M]

Hi Chad,

Thank you so much for your detailed answer!

Before I checked in on your answer though, I had figured that option number 4 seemed like the right way to go:))

But, nonetheless it was very nice to read your detailed description about the issue and the potential ways to resolve it. 10/10

It all appears to work perfectly now!

Kind Regards

Rasmus