Go Server didn't start due to mismatch in calculated and existing natural order of pipeline instance

[Akshay Mankar]

Hi,

 

Our Go server failed to start with this error: Calculated natural ordering 29.0 is not the same as the existing naturalOrder 30.0, for pipeline runSSTTests_DEV07, with id 16451.

We found this row in the pipelines table in cruise.h2.db:

 

ID	NAME	BUILDCAUSETYPE	BUILDCAUSEBY	LABEL	BUILDCAUSEMESSAGE	COUNTER	LOCKED	NATURALORDER	COMMENT
16451	runSSTTests_DEV07	ManualForcedBuildCause	Null	29	Forced by timer	29	FALSE	30.0	null

 

And changed it to this:

 

ID	NAME	BUILDCAUSETYPE	BUILDCAUSEBY	LABEL	BUILDCAUSEMESSAGE	COUNTER	LOCKED	NATURALORDER	COMMENT
16451	runSSTTests_DEV07	ManualForcedBuildCause	Null	29	Forced by timer	29	FALSE	29.0	null

 

And the GoServer started!

I have two questions:

1. What could have caused this?

2. Would this fix cause any trouble in future?

 

We are using Go version 16.3.0(3183-60d6fd31f00527da00f2affdd8f7f7732e7be103) running on Windows Server 2012R2

Here is the full stack trace from the logs:

 

2016-09-12 14:23:19,463 ERROR [main] GoServer:76 - ERROR: Failed to start Go server.

java.lang.RuntimeException: java.lang.RuntimeException: Calculated natural ordering 29.0 is not the same as the existing naturalOrder 30.0, for pipeline runSSTTests_DEV07, with id 16451

                at com.thoughtworks.go.server.initializers.ApplicationInitializer.onApplicationEvent(ApplicationInitializer.java:142)

                at com.thoughtworks.go.server.initializers.ApplicationInitializer.onApplicationEvent(ApplicationInitializer.java:48)

                at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:97)

                at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:327)

                at org.springframework.context.support.AbstractApplicationContext.finishRefresh(AbstractApplicationContext.java:941)

                at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:475)

                at org.springframework.web.context.ContextLoader.configureAndRefreshWebApplicationContext(ContextLoader.java:383)

                at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:283)

                at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:111)

                at org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:800)

                at org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:444)

                at org.eclipse.jetty.server.handler.ContextHandler.startContext(ContextHandler.java:791)

                at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:294)

                at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1349)

                at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1342)

                at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:741)

                at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:505)

                at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)

                at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)

                at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114)

                at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)

                at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)

                at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)

                at org.eclipse.jetty.server.Server.start(Server.java:387)

                at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114)

                at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)

                at org.eclipse.jetty.server.Server.doStart(Server.java:354)

                at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)

                at com.thoughtworks.go.server.Jetty9Server.start(Jetty9Server.java:90)

                at com.thoughtworks.go.server.GoServer.startServer(GoServer.java:71)

                at com.thoughtworks.go.server.GoServer.go(GoServer.java:63)

                at com.thoughtworks.go.server.util.GoLauncher.main(GoLauncher.java:31)

                at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

                at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

                at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

                at java.lang.reflect.Method.invoke(Method.java:606)

                at com.simontuffs.onejar.Boot.run(Boot.java:306)

                at com.simontuffs.onejar.Boot.main(Boot.java:159)

Caused by: java.lang.RuntimeException: Calculated natural ordering 29.0 is not the same as the existing naturalOrder 30.0, for pipeline runSSTTests_DEV07, with id 16451

                at com.thoughtworks.go.util.ExceptionUtils.bomb(ExceptionUtils.java:28)

                at com.thoughtworks.go.domain.PipelineTimelineEntry.updateNaturalOrder(PipelineTimelineEntry.java:180)

                at com.thoughtworks.go.server.domain.PipelineTimeline.add(PipelineTimeline.java:100)

                at com.thoughtworks.go.server.persistence.PipelineRepository$1.populateFrom(PipelineRepository.java:176)

                at com.thoughtworks.go.server.persistence.PipelineRepository$1.doInHibernate(PipelineRepository.java:90)

                at org.springframework.orm.hibernate3.HibernateTemplate.doExecute(HibernateTemplate.java:406)

                at org.springframework.orm.hibernate3.HibernateTemplate.execute(HibernateTemplate.java:339)

                at com.thoughtworks.go.server.persistence.PipelineRepository.updatePipelineTimeline(PipelineRepository.java:75)

                at com.thoughtworks.go.server.domain.PipelineTimeline.updateTimelineOnInit(PipelineTimeline.java:186)

                at com.thoughtworks.go.server.initializers.ApplicationInitializer.onApplicationEvent(ApplicationInitializer.java:126)

                ... 37 more

--

Regards,
Akshay Mankar | @akshaymankar

[Varsha Varadarajan]

Hi Akshay,

Natural order is the order of pipelines based on the order in which modifications were made to its material. We load and store this information in memory on server startup for features like Fan-In and Fan-out, comparing pipelines etc. 

As for your questions,

1. What could have caused this?

We had analyzed a similar issue earlier and concluded that a possible cause is that someone modified the database to delete a pipeline row from the pipelines table. However, if you can attach the logs from the time you first saw this exception, it will be useful.

2. Would this fix cause any trouble in future?

As far as we know, it is not likely to cause an issue. However, the next time you encounter this issue, its best to set the natural order to 0. The GoCD server, upon startup will calculate the correct natural order.  Please keep track of the values before setting them to 0.

Thank You,

Varsha