Fresh install of docker GOCD Server(v24.3.0) stuck on Upgrading database

[Lawrence]

Anyone else stuck on upgrading database on a fresh install of the docker container? I'm able to run v24.2.0 container no problem, but v24.3.0 didn't complete after an hour.

This could be a me problem, but not sure what I could have done since I just downloaded the image and started it.

[Chad Wilson]

Maybe you can share the specific lines it gets stuck on, and something of your environment (OS, container runtime version etc)? Doing anything unusual with volume mounts or entry points or a custom container image?

If you are upgrading an existing install perhaps there is a lock that's stuck on your database?

The containers are all automatically tested that they start up correctly within latest Docker on Linux (albeit in a docker-in-docker type of environment) and within Kubernetes versions and they also work OK for me on MacOS within Colima/Docker.

On Tue, 24 Sept 2024, 04:49 Lawrence, <lawrenc...@gmail.com> wrote:

Anyone else stuck on upgrading database on a fresh install of the docker container? I'm able to run v24.2.0 container no problem, but v24.3.0 didn't complete after an hour.

This could be a me problem, but not sure what I could have done since I just downloaded the image and started it.

--
You received this message because you are subscribed to the Google Groups "go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email to go-cd+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/go-cd/caf591ea-eeb8-4745-a875-779860a2bf8an%40googlegroups.com.

[Ash]

I have the same issue

[Lawrence]

It's happening on a fresh install of the docker container, so no data. Just downloaded the image and started it without mapping any drives

[Lawrence]

Forgot to add details.

Tried it on Windows 11 with Docker desktop v4.34.3 and Synology DSM 7.2.1-69057 update 5 using container manager.

They both have a log of:

stdout wrapper  | <-- Wrapper Stopped
stdout wrapper  | TERM trapped.  Shutting down.
stdout "jvm 1    | 2024-10-18 23:01:09,736 INFO  [WrapperJarAppMain] DatabaseMigrator:53 - ************************************************************************"
stdout jvm 1    | ************************************************************************
stdout "jvm 1    | 2024-10-18 23:01:09,735 INFO  [WrapperJarAppMain] DatabaseMigrator:53 - WARNING: Shutting down your server at this point will lead to a database corruption. Please wait until the database upgrade completes."
stdout jvm 1    | WARNING: Shutting down your server at this point will lead to a database corruption. Please wait until the database upgrade completes.
stdout "jvm 1    | 2024-10-18 23:01:09,734 INFO  [WrapperJarAppMain] DatabaseMigrator:53 - ************************************************************************"
stdout jvm 1    | ************************************************************************
stdout "jvm 1    | 2024-10-18 23:01:09,733 INFO  [WrapperJarAppMain] DatabaseMigrator:44 - Upgrading database, this might take a while depending on the size of the database."
stdout "jvm 1    | 2024-10-18 23:01:09,186 WARN  [WrapperJarAppMain] ConnectionManager:117 - The file config/db.properties specified by `go.db.config` does not exist."
stdout "jvm 1    | 2024-10-18 23:01:03,332 WARN  [WrapperJarAppMain] AbstractHandler:88 - No Server set for ResourceHandler@447a0fea{STOPPED}"
stdout "jvm 1    | 2024-10-18 23:01:03,212 WARN  [WrapperJarAppMain] Server:366 - ErrorPageMapper not supported for Server level Error Handling"
stdout "jvm 1    | 2024-10-18 23:01:03,060 INFO  [WrapperJarAppMain] JettyServer:194 - Configuring Jetty using /go-working-dir/config/jetty.xml"
stdout jvm 1    | Using classpath resource `jar:onejar:lib/server-launcher-24.3.0-19261-main.jar!/config/logback.xml'.
stdout jvm 1    | Could not find file `config/logback.xml'. Attempting to load from classpath.
stdout jvm 1    | [Fri Oct 18 23:01:00 GMT 2024]        Operating System: Linux(4.4.302+)
stdout jvm 1    | [Fri Oct 18 23:01:00 GMT 2024]            Java Version: 21.0.4
stdout jvm 1    | [Fri Oct 18 23:01:00 GMT 2024]            GoCD Version: 24.3.0-19261
stdout jvm 1    | [Fri Oct 18 23:01:00 GMT 2024]   Application arguments: []
stdout jvm 1    | [Fri Oct 18 23:01:00 GMT 2024]   Working directory    : /go-working-dir
stdout jvm 1    | [Fri Oct 18 23:01:00 GMT 2024] Starting process:
stdout jvm 1    | WrapperManager: Initializing...
stdout wrapper  | Launching a JVM...
stdout wrapper  |
stdout wrapper  |   Licensed to ThoughtWorks for GoCD Server
stdout wrapper  |     https://wrapper.tanukisoftware.com
stdout "wrapper  |   Copyright (C) 1999-2024 Tanuki Software, Ltd. All Rights Reserved.
stdout  ]0;go-server wrapper  | Java Service Wrapper Standard Edition 64-bit 3.5.59
stdout wrapper  | --> Wrapper Started as Console
stdout Running go-server...
stdout $ exec /usr/local/sbin/tini -g -- /go-working-dir/bin/go-server console
stdout $ sed -i -e s...@wrapper.logfile=.*@wrapper.logfile=/go-working-dir/logs/go-server-wrapper.log@g -e s...@wrapper.java.command=.*@wrapper.java.command=/gocd-jre/bin/java@g -e s...@wrapper.working.dir=.*@wrapper.working.dir=/go-working-dir@g /go-server/wrapper-config/wrapper.conf
stdout /docker-entrypoint.sh: Running custom scripts in /docker-entrypoint.d/ ...

[Chad Wilson]

At a time when it is stuck, can you exec into the container and send kill -3 to the java process for the server? Then check the stdout and share the big thread dump (in normal chronological order please, not reversed 🙏)

I need to see a thread dump to have an idea why it is stuck, as we don't have any steps to replicate this inside or outside a container.

-Chad

To view this discussion on the web visit https://groups.google.com/d/msgid/go-cd/28b822e9-02a0-4138-9d4d-a96054e7c217n%40googlegroups.com.

[Ashish Nayyar]

I am not running in docker 

You received this message because you are subscribed to a topic in the Google Groups "go-cd" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/go-cd/9NawU8Q0QC8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to go-cd+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/go-cd/CAA1RwH-b_a7UbgAWcG4A9yr0cM3ZKHRUdGoeiymkAxznSg1X0w%40mail.gmail.com.

[Chad Wilson]

I understand that, but you replied to a thread where the original poster is specifically running on docker.

I already replied to your other thread requesting exactly the same. Need a thread dump minimally to why any process is stuck if the logs dont tell us. Doesn't matter if it's in a container or not. Please read the reply to your original message.

To view this discussion on the web visit https://groups.google.com/d/msgid/go-cd/CAK3wOE6RCPj5r53j%3DQw4S3ybEjK9Gkk%2B4GWAz%2BaUPYd51ut8QQ%40mail.gmail.com.

[Chad Wilson]

Hi Lawrence

It looks like another user might has quite likely got to the bottom of this problem: https://github.com/gocd/gocd/issues/13214

TL;DR - a library upgrade included in 24.3.0 (but not 24.2.0) changed the random string generation Liquibase database upgrades/migrations rely upon to a highly secure mechanism. In some environments without sufficient entropy things would just block waiting for more entropy to generate random numbers/strings.

While I cannot replicate this with MacOS/Colima containers, and I have not seen an issue with our build/test Linux VMs/containers on AWS it's possible other host or virtual environments would have issues, especially those without much entropy and running Linux kernels < 5.6 on "real" hardware.

If someone wants to try and validate it is fixed, perhaps see if you have the same issue with an experimental build like gocdexperimental/gocd-server:v24.4.0-19635 which has upgraded libraries that address the underlying problem.

-Chad

[Lawrence Spear]

Thanks for the update. I'll check out the next version to see if I still have the issue. Sorry I didn't continue the debugging process here. An employee left suddenly and I inherited their projects, which ate up any free time for debugging.

You received this message because you are subscribed to a topic in the Google Groups "go-cd" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/go-cd/9NawU8Q0QC8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to go-cd+un...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/go-cd/CAA1RwH-MfTs_-SUVXN_yXfaWNfSmWXoOHYvP7XdC8LfUrhC2mQ%40mail.gmail.com.

[Chad Wilson]

Ahh, sorry to hear that - no problem.

In any case, 24.4.0 is released now, and should address this issue.

To view this discussion visit https://groups.google.com/d/msgid/go-cd/CALT3rQDw2XWZTb%3DfU%3DN9KnmL4MwjF6jK4haS6dis9bY1a_F_Og%40mail.gmail.com.