Migrating a GoCD instance between instances, startup hanging on Hibernate task

[Michael McCarthy]

Hi all,

I was wondering if anyone had seen anything similar to this before, and help greatly appreciated.

The background is we're doing a lift and shift of a GoCD instance with a Postgres DB from one location to another. The version of GoCD on either end is 17.11.0 (old I know) and we've followed the instructions on taking the DB backup.

The issue is that GoCD seems to hang on startup when this DB import is done (but not with a clean db). The connection itself seems OK as we see these logs:

2021-03-15 17:13:04,136 INFO  [main] GoServer:94 - Including addons: addons/go-postgresql-17.10.0-5380-2282ba5.jar

2021-03-15 17:13:09,788 INFO  [main] PostgresqlDatabase:93 - [DB] Using connection configuration jdbc:postgresql://localhost:5432/cruisedb1 [User: cruisedb1] [Password Encrypted: false]

2021-03-15 17:13:10,499 INFO  [main] PostgresqlDatabase:71 - [DB] Changelog exists. Skipping creation of a new table.

However, the go-server log's last lines of logging are:

2021-03-15 17:13:31,065 INFO  [main] BuildAssignmentService:253 - [Configuration Changed] Removing jobs for pipelines that no longer exist in configuration.

2021-03-15 17:13:31,167 INFO  [main] PipelineRepository:86 - Start updating pipeline timeline

This seems to be a Hibernate task - see https://github.com/gocd/gocd/blob/17.11.0/server/src/com/thoughtworks/go/server/persistence/PipelineRepository.java#L86. It just sits there though. The Java process is running, but there's no logging, and nothing I can see from top suggests there is any activity going on.

I notice a lot of the backup and restore docs speak about backup and restoring the db but it isn't clear whether there is any data that would stop you loading that dump up in a different location or not - something specific to the 'old home' of the db data that you'd need to amend before you could use that as the database for a newer system.

I'm bumping up logback logging in a number of classes but there's nothing I can get to log out post that line above.

Again any pointers would be massively appreciated.

Many thanks

Michael

[Aravind SV]

Yes, 17.11 is quite old and it’s hard to say what the problem could be. A couple of things I’d look at:

1. Run: kill -3 go-server-process-id and it will show you stack traces, which might tell you what the main thread is doing.
   
2. The pipeline timeline update query is quite time consuming, but everything else being the same, it should be as fast as the older setup. Maybe some DB indexes are missing?
   
3. You could see if the same behaviour is seen if you move to a new DB schema on the old DB server itself to remove any suspicion on the new DB server’s performance.
   

I notice a lot of the backup and restore docs speak about backup and restoring the db but it isn’t clear whether there is any data that would stop you loading that dump up in a different location or not - something specific to the ’old home’ of the db data that you’d need to amend before you could use that as the database for a newer system.

I don’t think there is anything like that.

Cheers,
Aravind

[Michael McCarthy]

Many thanks Aravind,

I've started to go through some of those ideas, appreciated. It is not an indexes thing but good shout there.

I noticed the free memory take a nose dive very very low when GoCD is starting up, and when that message is displayed it dwindles down and down and then eventually gets to around 100mb free before rising up again as Java disappears from 'top'. I've checked the memory profile and its quite different from the original machine - almost 2gb less RAM and no swap at all. I'll get those changed and try again tomorrow.

I expected to see some Hibernate logging but can't see any so I'll try and get the logback-include tweaked to help there and report back.

Many thanks

Michael

[Michael McCarthy]

Thanks all, this is solved. 

The hunch on the memory ended up being correct. With an extra 2gb of RAM added, the task was able to continue and the UI started. Presumably somewhere there was an OOM but I couldn't find it, I'll dig deeper and post back if I see where it was.

Thanks again

Michael

[Aravind SV]

Good to know. Thank you for following up. I’m sure it’ll help someone, at some point. :)

Cheers,
Aravind