Error when updating from 20.8.0 to 20.9.0

[Nikolay Kirov]

Hello there,

Current setup is: GoCD running on K8s cluster using gocd/gocd-server:v20.8.0 server image. Database is postgresql using pgpool on single replica. I have tried to look into the database trying to get results where state column is empty, but I didn't get such results on any of the tables. 

I'm trying to update our GoCD build server from 20.8.0 to 20.9.0 and I'm getting the following error:

java.lang.RuntimeException: org.mybatis.spring.MyBatisSystemException: nested exception is org.apache.ibatis.executor.result.ResultMapException: Error attempting to get column 'state' from result set. Cause: java.lang.IllegalArgumentException: No enum constant com.thoughtworks.go.domain.JobStat
e.Passed
at com.thoughtworks.go.server.initializers.ApplicationInitializer.onApplicationEvent(ApplicationInitializer.java:158)
at com.thoughtworks.go.server.initializers.ApplicationInitializer.onApplicationEvent(ApplicationInitializer.java:49)
at org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:172)
at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:165)
at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:139)
at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:393)
at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:347)
at org.springframework.context.support.AbstractApplicationContext.finishRefresh(AbstractApplicationContext.java:883)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:545)
at org.springframework.web.context.ContextLoader.configureAndRefreshWebApplicationContext(ContextLoader.java:443)
at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:325)
at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:107)
at org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:933)
at org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:553)
at org.eclipse.jetty.server.handler.ContextHandler.startContext(ContextHandler.java:892)
at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:356)
at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1445)
at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1409)
at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:825)
at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:275)
at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:524)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
at org.eclipse.jetty.deploy.bindings.StandardStarter.processBinding(StandardStarter.java:46)
at org.eclipse.jetty.deploy.AppLifeCycle.runBindings(AppLifeCycle.java:188)
at org.eclipse.jetty.deploy.DeploymentManager.requestAppGoal(DeploymentManager.java:513)
at org.eclipse.jetty.deploy.DeploymentManager.addApp(DeploymentManager.java:154)
at com.thoughtworks.go.server.Jetty9Server.startHandlers(Jetty9Server.java:179)
at com.thoughtworks.go.server.Jetty9Server.start(Jetty9Server.java:129)
at com.thoughtworks.go.server.GoServer.startServer(GoServer.java:62)
at com.thoughtworks.go.server.GoServer.go(GoServer.java:54)
at com.thoughtworks.go.server.util.GoLauncher.main(GoLauncher.java:42)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.base/java.lang.reflect.Method.invoke(Unknown Source)
at com.thoughtworks.gocd.Boot.run(Boot.java:90)
at com.thoughtworks.gocd.Boot.main(Boot.java:56)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.base/java.lang.reflect.Method.invoke(Unknown Source)
at org.tanukisoftware.wrapper.WrapperJarApp.run(WrapperJarApp.java:451)
at java.base/java.lang.Thread.run(Unknown Source)

[Aravind SV]

Hello Nikolay, ,

That’s strange. Can you open an issue on GitHub?

I think you should look for a row where the state column is “Passed”. Looks like there’s some data that is wrong (not sure how). The valid values of JobState are here.

Someone who looks at the issue there might be able to help more.

Regards,
Aravind

[Nikolay Kirov]

Amazing, this helped me identify the malicious pipeline which had state "Passed" in builds table. I really appreciate it.

This was most likely our error, since a while back when we were using pgpool with two replica there were tons of issues with the database, many builds were getting stuck in "Unknown" state and in the database we had to manually set state and result columns to "Completed" and to fill in manually the completedbytransactionid column and we ended up using only a single replica to avoid this.

However something else is amiss as well, I'm unable to open history of this pipeline I get error "There was an unknown error performing the operation. Possible reason ()". Can you give me some hints which is the table and row that is being accessed for the build history?

Thanks in advance.

[Aravind SV]

Ooh. Once you start changing the DB manually … all bets are off. Regarding the pipeline history page, it’ll likely be joining a few tables. Here are some pointers that might help:

1. See this query and this one. These might not be the right ones, but will give you a sense of the kind of things being done. You’ll need to follow the “<include>” tags to find the queries which will be merged into this one.
   
2. Outdated DB schema diagram: Might give you a sense of the connections. It’s outdated though. So, don’t rely on it too much.
   

Cheers,
Aravind