After Upgrade to 19.8.0, config/token missing from primary install folder (Ubuntu)

[Jeff Vincent]

I've recently updated from 19.4.0 (I think) to 19.8.0 (on Ubuntu 16.04).  On all of my agents I have four (4) agent processes running.

For some reason, the /var/lib/go-agent/config/token file gets wiped out during the upgrade and the agent won't connect to the server any more with the error (below) in the log.

I can clean out the guid.txt and it will regenerate a guid and new token, but then I have to go update the GUID in all the environments that the agent belonged to.  

How can I force the go server to resend the token for the same guid?  Or is there another way to fix this?

2019-09-18 14:21:37,126 ERROR [scheduler-1] AgentHTTPClientController:103 - Error occurred when agent tried to ping server:

org.springframework.remoting.RemoteAccessException: Could not access HTTP invoker remote service at [https://goserver/go/remoting/remoteBuildRepository]; nested exception is java.lang.RuntimeException: Couldn't load token from filesystem

        at org.springframework.remoting.httpinvoker.HttpInvokerClientInterceptor.convertHttpInvokerAccessException(HttpInvokerClientInterceptor.java:226)

        at org.springframework.remoting.httpinvoker.HttpInvokerClientInterceptor.invoke(HttpInvokerClientInterceptor.java:153)

        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)

        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213)

        at com.sun.proxy.$Proxy10.ping(Unknown Source)

        at com.thoughtworks.go.agent.AgentHTTPClientController.ping(AgentHTTPClientController.java:98)

        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)

        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

        at java.base/java.lang.reflect.Method.invoke(Unknown Source)

        at org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:65)

        at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)

        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)

        at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source)

        at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)

        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)

        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

        at java.base/java.lang.Thread.run(Unknown Source)

Caused by: java.lang.RuntimeException: Couldn't load token from filesystem

        at com.thoughtworks.go.util.ExceptionUtils.bomb(ExceptionUtils.java:41)

        at com.thoughtworks.go.config.IdFileService.load(IdFileService.java:47)

        at com.thoughtworks.go.config.TokenService.load(TokenService.java:22)

        at com.thoughtworks.go.config.DefaultAgentRegistry.token(DefaultAgentRegistry.java:39)

        at com.thoughtworks.go.agent.GoHttpClientHttpInvokerRequestExecutor.doExecuteRequest(GoHttpClientHttpInvokerRequestExecutor.java:70)

        at org.springframework.remoting.httpinvoker.AbstractHttpInvokerRequestExecutor.executeRequest(AbstractHttpInvokerRequestExecutor.java:137)

        at org.springframework.remoting.httpinvoker.HttpInvokerClientInterceptor.executeRequest(HttpInvokerClientInterceptor.java:202)

        at org.springframework.remoting.httpinvoker.HttpInvokerClientInterceptor.executeRequest(HttpInvokerClientInterceptor.java:184)

        at org.springframework.remoting.httpinvoker.HttpInvokerClientInterceptor.invoke(HttpInvokerClientInterceptor.java:150)

        ... 16 common frames omitted

Caused by: java.io.FileNotFoundException: File 'config/token' does not exist

        at org.apache.commons.io.FileUtils.openInputStream(FileUtils.java:297)

        at org.apache.commons.io.FileUtils.readFileToString(FileUtils.java:1805)

        at com.thoughtworks.go.config.IdFileService.load(IdFileService.java:45)

        ... 23 common frames omitted

[Akshay Dewan]

Hi Jeff,

I'm working on trying to reproduce this. In the meantime, would you mind creating an issue for this at https://github.com/gocd/gocd/issues ?

Thanks

--
You received this message because you are subscribed to the Google Groups "go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email to go-cd+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/go-cd/fcfcff41-903e-4ed3-9672-8a1f08544445%40googlegroups.com.

--

Akshay Dewan

Phone: +91 99201 57907

ThoughtWorks

[Aditya Sood]

Hi Jeff,

Here is what I tried.

I got a Vagrant box of Ubuntu 14.04 and installed the GoCD server v19.4.0 using the Debian package. I then upgraded my GoCD server to GoCD v19.8.0. With GoCD the agents auto-upgrade once the server is upgraded. My agent upgraded fine. 

I then separately upgraded an agent using apt-get and that worked fine to. I could always see the /var/lib/go-agent/config/token file. 

Could you confirm the version you upgraded from was 19.4.0 or was it an earlier version of GoCD?

[Jeff]

I apologize, but I think I jumped to a conclusion regarding the cause and am unable to reproduce it.  I have VMWare snapshots for only one of my go-agent VMs.  When reverted that VM back to each of the 3 previous older states, two at least are for sure running 19.4.0 (go-agent), the token file seems to be missing on them as well. 

What I don't know is when or how the token file was removed.  I wasn't paying close enough attention apparently to the state of the system before I attempted the upgrade .

I'll have to keep an eye on it next time, and do better at preserving the pre-upgrade state.  Sorry for the hassle.

-Jeff

--
You received this message because you are subscribed to the Google Groups "go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email to go-cd+un...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/go-cd/72ce4449-37b6-442b-bf5f-bf3a8c174f2e%40googlegroups.com.