GOCD restart issue

[Vijayakumaran A.]

Team,we have just restarted the gocd container now gocd not started.

Having below in docker logs please help 

[Sriram Narayanan]

Looks like something is wrong with the container system. /go-working-dir/ is a runtime symlink directory created here https://github.com/gocd/docker-gocd-server/blob/master/docker-entrypoint.sh#L41

If a directory is not getting created, then there is an underlying issue unrelated to GoCD and entirely related to the underlying container system. Please see if a container restart helps you continue. You should also check the container system's logs. To share something that we saw at our project two weeks ago, one of the EKS clusters was still running at version 1.23 and we would often see the error "Out of storage space" (sometimes once a day). When I'd exec into the container, I found that we were unable to make directories but were able to create files. We upgrades to EKS 1.24 and that error has not recurred since then.

I also see that you are using GoCD 23.4 and the H2DB. If this is a GoCD setup that you are using for your project, then I urge you to use the postgres database instead. See https://github.com/gocd/gocd-database-migrator

 

--
You received this message because you are subscribed to the Google Groups "go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email to go-cd+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/go-cd/5df023fd-f859-4a6b-99e7-92e3063653e7n%40googlegroups.com.

[Chad Wilson]

Please share how you are starting the container and what mounts you are using, especially to /godata.

Your container may have permissions issues writing to /godata or issues mounting the filesystem in writeable mode. Is it some kind of NFS volume? Is it mounted by another container/host already and thus is being mounted in read-only mode? is this a Helm/Kubernetes install, or just a regular container on a server?

You can exec into the container and check /godata and /go-working-dir are writable by the `go` user.

-Chad

To view this discussion on the web visit https://groups.google.com/d/msgid/go-cd/CANiY96bSuG28hLLP_A%2Bij%3DKt0xO%3Digxb3zawjKXcj_bjnAWpLg%40mail.gmail.com.

[Vijayakumaran A.]

We have started container using docker start servicename.

Here is the composer file of gocd server part 

server:
    container_name: gocd-server
    image: gocd/gocd-server:v23.4.0
    restart: always
    ports:
      - "8153:8153"
    volumes:
      - ./gocd/data:/godata
      - ./gocd/data/home:/home/go
      - ./gocd/scripts/server:/docker-entrypoint.d
      - ./gocd/scripts/shared:/shared
      - ./gocd/passwd:/godata/config/passwd
      - ./gocd/docker-elastic-agents-3.2.3-415.jar:/go-working-dir/plugins/external
    networks:
      - gocd

[Vijayakumaran A.]

its is just regular container in linux server and used local disk for volume

[Chad Wilson]

Did someone recently change something in your compose volume mounts?

It's probably not a good idea to mount things directly into /go-working-dir like the line "- ./gocd/docker-elastic-agents-3.2.3-415.jar:/go-working-dir/plugins/external"

The container entrypoint expects to own and manage these locations with symlinks, as Ram mentioned earlier. This is why the /go-working-dir path is not documented - it is internal and you shouldn't interfere with it or couple your config to undocumented internals.

This mount being there at container start is probably going to prevent the entrypoint from linking things to /godata correctly and you will probably end up with a `plugins` directory which is not writable by GoCD and errors like you had below.

Since you already have a /godata volume - put the plugin jars directly into ./gocd/data/plugins/external (on your host machine), and remove the individual jar mount. Basically, please follow the documentation: https://hub.docker.com/r/gocd/gocd-server

If removing that plugin mount still doesn't work, check the other mounts and the permissions of those directories as seen by the container when it starts by execing into it and exploring.

-Chad

To view this discussion on the web visit https://groups.google.com/d/msgid/go-cd/d1136435-78d7-42b6-9b1c-5c2d2a49ded2n%40googlegroups.com.