Help Needed with GoCD Docker Setup and cruise-config.xml Configuration

31 views
Skip to first unread message

Satya Elipe

unread,
Apr 10, 2024, 5:34:55 PMApr 10
to go...@googlegroups.com


Dear GoCD Support Team,


I hope this message finds you well. I am currently encountering an issue with setting up GoCD on Docker and specifically with configuring the cruise-config.xml file to be recognized correctly in my setup. Despite following the official documentation and trying various configurations, I've hit a stumbling block that I hope you can help me with.


Issue Summary:

I have a GoCD server running in a Docker container, and I'm attempting to ensure that the cruise-config.xml file is correctly picked up from a specified location. My goal is to mount this configuration file from an external volume into the GoCD server container so that the server uses this configuration instead of the default one.

Configuration Details:

  • GoCD Server Image Version: gocd/gocd-server:v22.3.0
  • Docker Version: Docker version 25.0.3, build 4debf41
  • Host Operating System: Amazon Linux 2023

Docker Run Command:

docker run -d \

  --name gocd-server \

  -p 8153:8153 \

  -p 8154:8154 \

  -v /mnt/gocd_efs:/var/lib/go-server \

  -v /mnt/gocd_efs/etc_go/cruise-config.xml:/go-working-dir/config/cruise-config.xml \

  -v /mnt/gocd_efs/etc_go:/etc/go \

  custom-gocd-server



Issue Encountered:

When including the -v /mnt/gocd_efs/etc_go/cruise-config.xml:/go-working-dir/config/cruise-config.xml volume mount, the GoCD server fails to start correctly, with logs indicating an inability to create or copy necessary files within /go-working-dir/config. 


Without this mount, the server starts but does not load the desired configuration, defaulting instead to the initial configuration without any of our pipeline configurations.


Log snippet:
```
jvm 1    | 2024-04-10 20:55:17,429 ERROR [Thread-79] GoFileConfigDataSource:436 - Unable to load config file: /go-working-dir/config/cruise-config.xml The value of 'serverId' uniquely identifies a Go server instance. This field cannot be modified.

```



Questions:

  • Is there a recommended approach to ensure cruise-config.xml is correctly recognized and used by the GoCD server when running in Docker?
  • Could this issue be related to how volumes are mounted or permissions within the container?


Any assistance or insights you could provide on this matter would be greatly appreciated. I am happy to provide any further information or logs as needed.

Thank you for your time and support.



Best regards,

Satya

Chad Wilson

unread,
Apr 10, 2024, 9:48:57 PMApr 10
to go...@googlegroups.com
Hi Satya

When running a standard GoCD docker server image, you should mount a writable volume to the entire /godata directory which includes the config as noted at https://hub.docker.com/r/gocd/gocd-server, but not directly to an individual config file.

If you do not, even if you get the config working properly, you'll lose other things such as artifacts at startup which you generally do not want.

You are getting this error likely because the server ID inside the go config does not match what is inside the configuration history, which is normally at /godata/db/config.git (The GoCD Docker entrypoint script creates the symlinks into /go-working-dir within the docker-entrypoint.sh), so these things need to be mounted together and consistent.

I don't think your mounts to /var/lib/go-server or /etc/go will be doing anything, as these folders are not used for off-the-shelf docker server images, since the server is not installed as an rpm/deb package when creating a container. You can/should likely remove these.

In a general sense, all you should need to do is mount a location in EFS to /godata, similar to what the Helm chart does: https://github.com/gocd/helm-chart/blob/c734ad2263b1d2885229d00267c428e88f868504/gocd/templates/gocd-server-deployment.yaml#L122-L124 That will put all logs, artifacts, databases and config on your volume. Some folks decide to also mount /home/go if they want to use external storage for things like Git SSH keys or other shell defaults affecting the go user, but that's optional.

-Chad

--
You received this message because you are subscribed to the Google Groups "go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email to go-cd+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/go-cd/CADKEDRrF7KRFP9jNsg631STyokXS1Njqutdshd0ZKdwHMRTy6g%40mail.gmail.com.

Satya Elipe

unread,
Apr 11, 2024, 6:11:57 AMApr 11
to go...@googlegroups.com
Thank you Chad. 

I see my next issue with that, wherein cruise-config.xml is still not seen by the container and uses default or generates its own. 

Here is the data on the EFS mount:
```
[ec2-user@ip-10-154-168-49 ~]$ ls -al /mnt/gocd_efs
total 108252
drwxr-xr-x. 15 ec2-user root      6144 Apr 11 09:38 .
drwxr-xr-x.  3 root     root        22 Apr 10 15:57 ..
drwxr-xr-x.  4 ec2-user root      6144 Dec 16  2020 artifacts
-rwxr-xr-x.  1 root     root        32 Apr 11 09:38 cipher.aes
-rwxr-xr-x.  1 root     root        32 Apr 11 09:38 cipher.aes.16540.2020-12-16@14:08:08~
drwxr-xr-x.  2 ec2-user root      6144 Apr 11 08:10 config
-rwxr-xr-x.  1 root     root     14950 Apr 11 09:38 cruise-config.xml
-rwxr-xr-x.  1 root     root      7382 Apr 11 09:38 cruise-config.xml.1125.2021-01-19@11:20:26~
-rwxr-xr-x.  1 root     root     10460 Apr 11 09:38 cruise-config.xml.invalid.2021-09-27-16-03-30
-rwxr-xr-x.  1 root     root     47442 Apr 11 09:38 cruise-config.xsd
-rwxr-xr-x.  1 ec2-user root 110355285 Apr 10 09:28 cruise.war
drwxr-xr-x.  3 ec2-user root      6144 Dec 16  2020 data
drwxr-xr-x.  5 ec2-user root      6144 Jul  7  2022 db
drwxr-xr-x.  3 ec2-user root      6144 Apr 10 18:17 etc_go
drwxr-xr-x. 12 ec2-user root      6144 Apr 10 09:29 felix-cache
-rwxr-xr-x.  1 root     root       799 Apr 11 09:38 go_update_server.pub
drwxr-xr-x.  3 root     root      6144 Apr 11 09:38 godata
-rwxr-xr-x.  1 root     root      4120 Apr 11 09:38 jetty.xml
drwxr-xr-x.  2 ec2-user root      6144 Apr 11 08:11 logs
drwxr-xr-x.  3 ec2-user root      6144 Dec 16  2020 pipelines
-rwxr-xr-x.  1 ec2-user root       131 Sep 27  2021 plugin-slack.notifier.log
drwxr-xr-x.  4 ec2-user root      6144 Apr  4 14:59 plugins
drwxr-xr-x. 11 ec2-user root      6144 Apr 10 09:29 plugins_work
drwxr-xr-x.  2 ec2-user root      6144 Apr 10 09:28 run
-rwxr-xr-x.  1 root     root       144 Apr 11 09:38 users.passwd
drwxr-xr-x.  3 ec2-user root      6144 Apr 10 09:28 work```

And, /mnt/gocd_efs/cruise-config.xml:
```
[ec2-user@ip-10-154-168-49 ~]$ cat /mnt/gocd_efs/cruise-config.xml
<?xml version="1.0" encoding="utf-8"?>
<cruise xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="cruise-config.xsd" schemaVersion="139">
        <server jobTimeout="10" agentAutoRegisterKey="abcb673ad8de1f6df1afa41eff9ba70e" webhookSecret="e7e978e8-b4b1-498f-bd15-ce4c8f6f4240" tokenGenerationKey="6a945808-d166-40f2-87ab-a1533fc1edba">
    <siteUrls>
      <siteUrl>https://gocd-brxt.corp.mendeley.com/</siteUrl>
      <secureSiteUrl>https://gocd-brxt.corp.mendeley.com/</secureSiteUrl>
    </siteUrls>
    <security>
      <authConfigs>
        <authConfig id="file-auth-config" pluginId="cd.go.authentication.passwordfile">
          <property>
            <key>PasswordFilePath</key>
            <value>/etc/go/users.passwd</value>
          </property>
        </authConfig>
      </authConfigs>
      <admins>
        <user>api_user</user>
      </admins>
    </security>
    <mailhost hostname="relay.corp.mendeley.com" port="25" from="no-r...@gocd-atlas.test.corp.mendeley.com" admin="atlas...@elsevier.com" />
    <artifacts>
      <artifactsDir>artifacts</artifactsDir>
    </artifacts>
  </server>
  <elastic>
    <agentProfiles>
      <agentProfile id="brxt_gocd.shared-agent-staging-elastic" clusterProfileId="eks_brxt_staging_cluster">```


And, mounting into the container:
```
  docker run -d \
  --name gocd-server \
  -p 8153:8153 \
  -p 8154:8154 \
  -v /mnt/gocd_efs:/godata \
  custom-gocd-server```

And the data in the container:
```
bash-5.1$ ls -al /go-working-dir
total 107804
drwxrwxr-x    1 go       root         16384 Apr 11 09:42 .
drwxr-xr-x    1 root     root            90 Apr 11 09:41 ..
lrwxrwxrwx    1 go       root            17 Apr 11 09:41 artifacts -> /godata/artifacts
lrwxrwxrwx    1 go       root            14 Apr 11 09:41 bin -> /go-server/bin
lrwxrwxrwx    1 go       root            14 Apr 11 09:41 config -> /godata/config
-rw-r--r--    1 go       root     110355285 Apr 11 09:41 cruise.war
lrwxrwxrwx    1 go       root            10 Apr 11 09:41 db -> /godata/db
drwxr-xr-x   11 go       root           141 Apr 11 09:42 felix-cache
lrwxrwxrwx    1 go       root            14 Apr 11 09:41 lib -> /go-server/lib
lrwxrwxrwx    1 go       root            12 Apr 11 09:41 logs -> /godata/logs
lrwxrwxrwx    1 go       root            15 Apr 11 09:41 plugins -> /godata/plugins
drwxr-xr-x   11 go       root         16384 Apr 11 09:42 plugins_work
lrwxrwxrwx    1 go       root            14 Apr 11 09:41 run -> /go-server/run
drwxr-xr-x    3 go       root            52 Apr 11 09:41 work
lrwxrwxrwx    1 go       root            18 Apr 11 09:41 wrapper -> /go-server/wrapper
lrwxrwxrwx    1 go       root            25 Apr 11 09:41 wrapper-config -> /go-server/wrapper-config```

And "/godata/config/cruise-config.xml" in the is not same as the one from the efs mount:
```
bash-5.1$ cat /godata/config/cruise-config.xml
<?xml version="1.0" encoding="utf-8"?>
<cruise xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="cruise-config.xsd" schemaVersion="139">
  <server agentAutoRegisterKey="44eb6cdc-d8d9-4604-abfa-f061bbcbbee1" webhookSecret="506660f5-21e9-40c4-aa67-78c5cb37cde1" serverId="f256fbc9-d378-4649-a978-c4ddaca865b7" tokenGenerationKey="a7dc8993-3209-4459-ac18-85c1e89a5a60">
    <artifacts>
      <artifactsDir>artifacts</artifactsDir>
    </artifacts>
  </server>
</cruise>```

Wonder where is it going wrong and the container not seeing it, do we have to mount them differently ?

Here is what container logs say:
```
/docker-entrypoint.sh: Creating directories and symlinks to hold GoCD configuration, data, and logs
$ ln -sv /godata/artifacts /go-working-dir/artifacts
'/go-working-dir/artifacts' -> '/godata/artifacts'
$ mkdir -v -p /godata/config
created directory: '/godata/config'
$ ln -sv /godata/config /go-working-dir/config
'/go-working-dir/config' -> '/godata/config'
$ ln -sv /godata/db /go-working-dir/db
'/go-working-dir/db' -> '/godata/db'
$ mkdir -v -p /godata/logs
created directory: '/godata/logs'
$ ln -sv /godata/logs /go-working-dir/logs
'/go-working-dir/logs' -> '/godata/logs'
$ ln -sv /godata/plugins /go-working-dir/plugins
'/go-working-dir/plugins' -> '/godata/plugins'
/docker-entrypoint.sh: Creating directories and symlinks to hold GoCD wrapper binaries
$ ln -sv /go-server/bin /go-working-dir/bin
'/go-working-dir/bin' -> '/go-server/bin'
$ ln -sv /go-server/lib /go-working-dir/lib
'/go-working-dir/lib' -> '/go-server/lib'
$ ln -sv /go-server/run /go-working-dir/run
'/go-working-dir/run' -> '/go-server/run'
$ ln -sv /go-server/wrapper /go-working-dir/wrapper
'/go-working-dir/wrapper' -> '/go-server/wrapper'
$ ln -sv /go-server/wrapper-config /go-working-dir/wrapper-config
'/go-working-dir/wrapper-config' -> '/go-server/wrapper-config'
$ cp -rfv /go-server/config/logback-include.xml /go-working-dir/config/logback-include.xml
'/go-server/config/logback-include.xml' -> '/go-working-dir/config/logback-include.xml'
$ install-gocd-plugins
$ git-clone-config
$ cd /godata/config
/docker-entrypoint.sh: Running custom scripts in /docker-entrypoint.d/ ...
$ sed -i -e s...@wrapper.logfile=.*@wrapper.logfile=/go-working-dir/logs/go-server-wrapper.log@g -e s...@wrapper.java.command=.*@wrapper.java.command=/gocd-jre/bin/java@g -e s...@wrapper.working.dir=.*@wrapper.working.dir=/go-working-dir@g /go-server/wrapper-config/wrapper.conf
$ exec /usr/local/sbin/tini -g -- /go-working-dir/bin/go-server console
Running go-server...
wrapper  | Failed to set JVM input handle to non blocking mode: Bad file descriptor (9)
wrapper  | Failed to set JVM input handle to close on JVM exit: Bad file descriptor (9)
wrapper  | JVM exited while loading the application.
wrapper  | Unable to set JVM's stdin: Bad file descriptor
wrapper  | --> Wrapper Started as Console
wrapper  | Java Service Wrapper Standard Edition 64-bit 3.5.50
wrapper  |   Copyright (C) 1999-2022 Tanuki Software, Ltd. All Rights Reserved.
wrapper  |     http://wrapper.tanukisoftware.com
wrapper  |   Licensed to ThoughtWorks for GoCD Server
wrapper  |
wrapper  | Launching a JVM...
jvm 1    | wrapper  | --> Wrapper Started as Console
jvm 1    | wrapper  | Java Service Wrapper Standard Edition 64-bit 3.5.50
jvm 1    | wrapper  |   Copyright (C) 1999-2022 Tanuki Software, Ltd. All Rights Reserved.
jvm 1    | wrapper  |     http://wrapper.tanukisoftware.com
jvm 1    | wrapper  |   Licensed to ThoughtWorks for GoCD Server
jvm 1    | wrapper  |
jvm 1    | wrapper  | Launching a JVM...
wrapper  | Launching a JVM...
jvm 2    | WrapperManager: Initializing...
jvm 2    | [Thu Apr 11 08:10:31 GMT 2024] Starting process:
jvm 2    | [Thu Apr 11 08:10:31 GMT 2024]   Working directory    : /go-working-dir
jvm 2    | [Thu Apr 11 08:10:31 GMT 2024]   Application arguments: []
jvm 2    | [Thu Apr 11 08:10:31 GMT 2024]            GoCD Version: 22.3.0-15301
jvm 2    | [Thu Apr 11 08:10:31 GMT 2024]            Java Version: 17.0.5
jvm 2    | [Thu Apr 11 08:10:31 GMT 2024]        Operating System: Linux(6.1.82-99.168.amzn2023.x86_64)
jvm 2    | Could not find file `config/logback.xml'. Attempting to load from classpath.
jvm 2    | Using classpath resource `jar:onejar:lib/server-launcher-22.3.0-15301-main.jar!/config/logback.xml'.
jvm 2    | 2024-04-11 08:10:33,278 INFO  [WrapperJarAppMain] Jetty9Server:193 - Configuring Jetty using /go-working-dir/config/jetty.xml
jvm 2    | 2024-04-11 08:10:33,356 WARN  [WrapperJarAppMain] Server:357 - ErrorPageMapper not supported for Server level Error Handling
jvm 2    | 2024-04-11 08:10:33,436 WARN  [WrapperJarAppMain] AbstractHandler:96 - No Server set for ResourceHandler@67d4b2e7{STOPPED}
jvm 2    | 2024-04-11 08:10:39,298 WARN  [WrapperJarAppMain] ConnectionManager:117 - The file config/db.properties specified by `go.db.config` does not exist.
jvm 2    | 2024-04-11 08:10:40,046 INFO  [WrapperJarAppMain] DatabaseMigrator:40 - Upgrading database, this might take a while depending on the size of the database.
jvm 2    | ************************************************************************
jvm 2    | 2024-04-11 08:10:40,047 INFO  [WrapperJarAppMain] DatabaseMigrator:49 - ************************************************************************
jvm 2    | WARNING: Shutting down your server at this point will lead to a database corruption. Please wait until the database upgrade completes.
jvm 2    | 2024-04-11 08:10:40,048 INFO  [WrapperJarAppMain] DatabaseMigrator:49 - WARNING: Shutting down your server at this point will lead to a database corruption. Please wait until the database upgrade completes.
jvm 2    | ************************************************************************
jvm 2    | 2024-04-11 08:10:40,053 INFO  [WrapperJarAppMain] DatabaseMigrator:49 - ************************************************************************
jvm 2    | Apr 11, 2024 8:10:40 AM liquibase.database
jvm 2    | INFO: Set default schema name to PUBLIC
jvm 2    | Apr 11, 2024 8:10:40 AM liquibase.lockservice
jvm 2    | INFO: Successfully acquired change log lock
jvm 2    | Apr 11, 2024 8:10:40 AM liquibase.servicelocator
jvm 2    | INFO: Cannot load service: liquibase.parser.ChangeLogParser: liquibase.parser.core.json.JsonChangeLogParser Unable to get public no-arg constructor
jvm 2    | Apr 11, 2024 8:10:40 AM liquibase.servicelocator
jvm 2    | INFO: Cannot load service: liquibase.parser.ChangeLogParser: liquibase.parser.core.yaml.YamlChangeLogParser Unable to get public no-arg constructor
jvm 2    | Apr 11, 2024 8:10:40 AM liquibase.servicelocator
jvm 2    | INFO: Cannot load service: liquibase.change.Change: liquibase.change.core.LoadDataChange Unable to get public no-arg constructor
jvm 2    | Apr 11, 2024 8:10:40 AM liquibase.servicelocator
[ec2-user@ip-10-154-168-49 ~]$
[ec2-user@ip-10-154-168-49 ~]$
[ec2-user@ip-10-154-168-49 ~]$
[ec2-user@ip-10-154-168-49 ~]$
[ec2-user@ip-10-154-168-49 ~]$ docker exec gocd-server cat /go-working-dir/logs/go-server.log
2024-04-11 08:10:33,278 INFO  [WrapperJarAppMain] Jetty9Server:193 - Configuring Jetty using /go-working-dir/config/jetty.xml
2024-04-11 08:10:33,356 WARN  [WrapperJarAppMain] Server:357 - ErrorPageMapper not supported for Server level Error Handling
2024-04-11 08:10:33,436 WARN  [WrapperJarAppMain] AbstractHandler:96 - No Server set for ResourceHandler@67d4b2e7{STOPPED}
2024-04-11 08:10:39,298 WARN  [WrapperJarAppMain] ConnectionManager:117 - The file config/db.properties specified by `go.db.config` does not exist.
2024-04-11 08:10:40,046 INFO  [WrapperJarAppMain] DatabaseMigrator:40 - Upgrading database, this might take a while depending on the size of the database.
2024-04-11 08:10:40,047 INFO  [WrapperJarAppMain] DatabaseMigrator:49 - ************************************************************************
2024-04-11 08:10:40,048 INFO  [WrapperJarAppMain] DatabaseMigrator:49 - WARNING: Shutting down your server at this point will lead to a database corruption. Please wait until the database upgrade completes.
2024-04-11 08:10:40,053 INFO  [WrapperJarAppMain] DatabaseMigrator:49 - ************************************************************************
2024-04-11 08:10:42,164 INFO  [WrapperJarAppMain] DatabaseMigrator:57 - Database upgrade completed successfully.
2024-04-11 08:10:42,170 INFO  [WrapperJarAppMain] DataMigrationRunner:34 - Running data migrations...
2024-04-11 08:10:42,371 INFO  [WrapperJarAppMain] DataMigrationRunner:49 - Data migration took 24 ms
2024-04-11 08:10:42,383 INFO  [WrapperJarAppMain] DataMigrationRunner:49 - Data migration took 1 ms
2024-04-11 08:10:42,388 INFO  [WrapperJarAppMain] DataMigrationRunner:39 - Data migrations completed.
2024-04-11 08:10:43,745 WARN  [WrapperJarAppMain] ConfigurationFactory:136 - No configuration found. Configuring ehcache from ehcache-failsafe.xml  found in the classpath: jar:file:/go-working-dir/work/jetty-0_0_0_0-8153-cruise_war-_go-any-/webapp/WEB-INF/lib/ehcache-2.10.9.2.jar!/ehcache-failsafe.xml
2024-04-11 08:10:43,865 WARN  [WrapperJarAppMain] EhCacheProvider:93 - Could not find configuration [com.thoughtworks.go.domain.AccessToken]; using defaults.
2024-04-11 08:10:43,935 WARN  [WrapperJarAppMain] EhCacheProvider:93 - Could not find configuration [com.thoughtworks.go.domain.NotificationFilter]; using defaults.
2024-04-11 08:10:44,071 WARN  [WrapperJarAppMain] EhCacheProvider:93 - Could not find configuration [com.thoughtworks.go.domain.User]; using defaults.
2024-04-11 08:10:44,124 WARN  [WrapperJarAppMain] EhCacheProvider:93 - Could not find configuration [com.thoughtworks.go.domain.Plugin]; using defaults.
2024-04-11 08:10:44,159 WARN  [WrapperJarAppMain] EhCacheProvider:93 - Could not find configuration [com.thoughtworks.go.domain.EnvironmentVariable]; using defaults.
2024-04-11 08:10:44,218 WARN  [WrapperJarAppMain] EhCacheProvider:93 - Could not find configuration [com.thoughtworks.go.domain.User.notificationFilters]; using defaults.
2024-04-11 08:10:44,316 WARN  [WrapperJarAppMain] EhCacheProvider:93 - Could not find configuration [org.hibernate.cache.UpdateTimestampsCache]; using defaults.
2024-04-11 08:10:44,328 WARN  [WrapperJarAppMain] EhCacheProvider:93 - Could not find configuration [org.hibernate.cache.StandardQueryCache]; using defaults.
2024-04-11 08:10:47,228 WARN  [WrapperJarAppMain] BrokerService:2116 - Temporary Store limit is 51200 mb (current store usage is 0 mb). The data directory: /go-working-dir only has 5097 mb of usable space. - resetting to maximum available disk space: 5097 mb
2024-04-11 08:10:47,478 INFO  [WrapperJarAppMain] ConnectionManager:98 - Done loading query extensions, found com.thoughtworks.go.server.database.h2.H2QueryExtensions@7b3c0f16
2024-04-11 08:10:48,190 INFO  [WrapperJarAppMain] AESCipherProvider:61 - AES cipher not found. Creating a new cipher file
2024-04-11 08:10:55,360 INFO  [WrapperJarAppMain] GoConfigMigration:94 - Upgrading config file from version 139 to version 139
2024-04-11 08:10:55,419 INFO  [WrapperJarAppMain] GoConfigMigration:102 - Finished upgrading config file
2024-04-11 08:10:55,424 INFO  [WrapperJarAppMain] GoConfigMigrator:106 - [Config Save] Starting Config Save post upgrade using FullConfigSaveNormalFlow
2024-04-11 08:10:55,518 INFO  [Thread-79] DefaultPluginJarChangeListener:67 - Plugin load starting: /go-working-dir/plugins/bundled/gocd-ldap-authentication-plugin.jar
2024-04-11 08:10:56,057 INFO  [WrapperJarAppMain] CachedGoConfig:223 - About to notify config listeners
2024-04-11 08:10:56,059 INFO  [WrapperJarAppMain] CachedGoConfig:231 - Finished notifying all listeners
2024-04-11 08:10:56,083 WARN  [Thread-79] PluginSettingsMetadataLoader:63 - Failed to fetch plugin settings metadata for plugin cd.go.authentication.ldap. Maybe the plugin does not implement plugin settings and view?
2024-04-11 08:10:56,088 WARN  [Thread-79] PluginSettingsMetadataLoader:64 - Plugin: cd.go.authentication.ldap - Metadata load info: [{extension='authorization', configuration='null', view='null', error='The plugin sent a response that could not be understood by Go. Plugin returned with code '500' and the following response: 'This is an invalid request type :go.plugin-settings.get-configuration''}]
2024-04-11 08:10:56,089 WARN  [Thread-79] PluginSettingsMetadataLoader:65 - Not all plugins are required to implement the request above. This error may be safe to ignore.
2024-04-11 08:10:56,090 INFO  [Thread-79] DefaultPluginJarChangeListener:74 - Plugin load finished: /go-working-dir/plugins/bundled/gocd-ldap-authentication-plugin.jar
2024-04-11 08:10:56,133 INFO  [Thread-79] DefaultPluginJarChangeListener:67 - Plugin load starting: /go-working-dir/plugins/bundled/gocd-file-based-secrets-plugin.jar
2024-04-11 08:10:56,314 ERROR [Thread-79] DefaultPluginManager:136 - This is an invalid request type :go.plugin-settings.get-configuration
2024-04-11 08:10:56,314 WARN  [Thread-79] PluginSettingsMetadataLoader:63 - Failed to fetch plugin settings metadata for plugin cd.go.secrets.file-based-plugin. Maybe the plugin does not implement plugin settings and view?
2024-04-11 08:10:56,314 WARN  [Thread-79] PluginSettingsMetadataLoader:64 - Plugin: cd.go.secrets.file-based-plugin - Metadata load info: [{extension='secrets', configuration='null', view='null', error='com.thoughtworks.go.plugin.api.exceptions.UnhandledRequestTypeException: This is an invalid request type :go.plugin-settings.get-configuration'}]
2024-04-11 08:10:56,314 WARN  [Thread-79] PluginSettingsMetadataLoader:65 - Not all plugins are required to implement the request above. This error may be safe to ignore.
2024-04-11 08:10:56,441 INFO  [WrapperJarAppMain] BuildAssignmentService:250 - [Configuration Changed] Removing jobs for pipelines that no longer exist in configuration.
2024-04-11 08:10:56,442 INFO  [Thread-79] DefaultPluginJarChangeListener:74 - Plugin load finished: /go-working-dir/plugins/bundled/gocd-file-based-secrets-plugin.jar
2024-04-11 08:10:56,478 INFO  [Thread-79] DefaultPluginJarChangeListener:67 - Plugin load starting: /go-working-dir/plugins/bundled/gocd-yum-repository-poller-plugin.jar
2024-04-11 08:10:56,479 INFO  [WrapperJarAppMain] InvalidateAuthenticationOnSecurityConfigChangeFilter:78 - [Configuration Changed] Security Configuration is changed. Updating the last changed time.
2024-04-11 08:10:56,498 INFO  [WrapperJarAppMain] PipelineRepository:78 - Start updating pipeline timeline
2024-04-11 08:10:56,879 WARN  [Thread-79] PluginSettingsMetadataLoader:63 - Failed to fetch plugin settings metadata for plugin yum. Maybe the plugin does not implement plugin settings and view?
2024-04-11 08:10:56,879 WARN  [Thread-79] PluginSettingsMetadataLoader:64 - Plugin: yum - Metadata load info: [{extension='package-repository', configuration='null', view='null', error='The plugin sent a response that could not be understood by Go. Plugin returned with code '400' and the following response: 'Invalid request name go.plugin-settings.get-configuration''}]
2024-04-11 08:10:56,879 WARN  [Thread-79] PluginSettingsMetadataLoader:65 - Not all plugins are required to implement the request above. This error may be safe to ignore.
2024-04-11 08:10:56,998 INFO  [WrapperJarAppMain] PipelineRepository:84 - Pipeline timeline updated
2024-04-11 08:10:56,999 INFO  [WrapperJarAppMain] PipelineSqlMapDao:132 - Loading active pipelines into memory.
2024-04-11 08:10:56,999 INFO  [WrapperJarAppMain] PipelineSqlMapDao:383 - Retriving Active Pipelines from Database...
2024-04-11 08:10:57,012 ERROR [Thread-79] PluginInfoBuilder:54 - Failed to fetch Plugin Settings metadata for plugin yum. Maybe the plugin does not implement plugin settings and view?
2024-04-11 08:10:57,018 INFO  [Thread-79] DefaultPluginJarChangeListener:74 - Plugin load finished: /go-working-dir/plugins/bundled/gocd-yum-repository-poller-plugin.jar
2024-04-11 08:10:57,057 INFO  [Thread-79] DefaultPluginJarChangeListener:67 - Plugin load starting: /go-working-dir/plugins/bundled/gocd-json-config-plugin.jar
2024-04-11 08:10:57,303 INFO  [Thread-82] PipelineSqlMapDao:419 - Loading pipeline history to cache...Started
2024-04-11 08:10:57,305 INFO  [Thread-82] PipelineSqlMapDao:423 - Loading pipeline history to cache...Done
2024-04-11 08:10:57,310 WARN  [WrapperJarAppMain] PipelineSqlMapDao:450 - No pipelines found in Config, Skipping material revision caching.
2024-04-11 08:10:57,310 INFO  [Thread-81] PipelineSqlMapDao:413 - Loading Active Pipelines to cache...Started
2024-04-11 08:10:57,311 INFO  [Thread-81] PipelineSqlMapDao:416 - Loading Active Pipelines to cache...Done
2024-04-11 08:10:57,311 INFO  [WrapperJarAppMain] PipelineSqlMapDao:134 - Done loading active pipelines into memory.
2024-04-11 08:10:58,904 INFO  [Thread-79] ConfigRepositoryInitializer:108 - [Config Repository Initializer] Start initializing the config repositories for plugin 'json.config.plugin'
2024-04-11 08:10:58,909 INFO  [Thread-79] ConfigRepositoryInitializer:112 - [Config Repository Initializer] Done initializing the config repositories for plugin 'json.config.plugin'
2024-04-11 08:10:58,956 INFO  [Thread-79] DefaultPluginJarChangeListener:74 - Plugin load finished: /go-working-dir/plugins/bundled/gocd-json-config-plugin.jar
2024-04-11 08:10:59,014 INFO  [Thread-79] DefaultPluginJarChangeListener:67 - Plugin load starting: /go-working-dir/plugins/bundled/gocd-filebased-authentication-plugin.jar
2024-04-11 08:10:59,226 INFO  [WrapperJarAppMain] ConsoleActivityMonitor:78 - Found '0' building jobs. Added them with '2024-04-11T08:10:57.311Z' as the last heard time
2024-04-11 08:10:59,245 INFO  [WrapperJarAppMain] ConsoleActivityMonitor:79 - Found '0' scheduled jobs. Added them with '2024-04-11T08:10:57.311Z' as the last heard time
2024-04-11 08:10:59,261 INFO  [WrapperJarAppMain] RailsAssetsService:68 - Found rails assets manifest file named .sprockets-manifest-33237067f8bbd7d7d3ed31a1f9d514b9.json
2024-04-11 08:10:59,286 INFO  [WrapperJarAppMain] RailsAssetsService:72 - Successfully read rails assets manifest file located at /go-working-dir/work/jetty-0_0_0_0-8153-cruise_war-_go-any-/webapp/WEB-INF/rails/public/assets/.sprockets-manifest-33237067f8bbd7d7d3ed31a1f9d514b9.json
2024-04-11 08:10:59,316 WARN  [Thread-79] PluginSettingsMetadataLoader:63 - Failed to fetch plugin settings metadata for plugin cd.go.authentication.passwordfile. Maybe the plugin does not implement plugin settings and view?
2024-04-11 08:10:59,317 WARN  [Thread-79] PluginSettingsMetadataLoader:64 - Plugin: cd.go.authentication.passwordfile - Metadata load info: [{extension='authorization', configuration='null', view='null', error='The plugin sent a response that could not be understood by Go. Plugin returned with code '500' and the following response: 'Request go.plugin-settings.get-configuration is not supported by plugin.''}]
2024-04-11 08:10:59,317 WARN  [Thread-79] PluginSettingsMetadataLoader:65 - Not all plugins are required to implement the request above. This error may be safe to ignore.
2024-04-11 08:10:59,318 INFO  [Thread-79] DefaultPluginJarChangeListener:74 - Plugin load finished: /go-working-dir/plugins/bundled/gocd-filebased-authentication-plugin.jar
2024-04-11 08:10:59,359 INFO  [Thread-79] DefaultPluginJarChangeListener:67 - Plugin load starting: /go-working-dir/plugins/bundled/gocd-yaml-config-plugin.jar
2024-04-11 08:11:01,493 INFO  [Thread-79] ConfigRepositoryInitializer:108 - [Config Repository Initializer] Start initializing the config repositories for plugin 'yaml.config.plugin'
2024-04-11 08:11:01,499 INFO  [Thread-79] ConfigRepositoryInitializer:112 - [Config Repository Initializer] Done initializing the config repositories for plugin 'yaml.config.plugin'
2024-04-11 08:11:01,538 INFO  [Thread-79] DefaultPluginJarChangeListener:74 - Plugin load finished: /go-working-dir/plugins/bundled/gocd-yaml-config-plugin.jar
2024-04-11 08:11:01,579 INFO  [Thread-79] DefaultPluginJarChangeListener:67 - Plugin load starting: /go-working-dir/plugins/external/gocd-slack-notifier-1.4.0.jar
2024-04-11 08:11:01,710 ERROR [Thread-79] PluginLoader:121 - Failed to load plugin: plugins_work/gocd-slack-notifier-1.4.0.jar. Plugin is invalid. Reasons [Class [GoNotificationPlugin] is annotated with @Extension but cannot be constructed. Reason: java.lang.RuntimeException: Unable to find go_notify.conf. Please make sure you've set it up right.., No extensions found in this plugin. Please check for @Extension annotations]
2024-04-11 08:11:01,723 INFO  [Thread-79] DefaultPluginJarChangeListener:74 - Plugin load finished: /go-working-dir/plugins/external/gocd-slack-notifier-1.4.0.jar
2024-04-11 08:11:01,763 INFO  [Thread-79] DefaultPluginJarChangeListener:67 - Plugin load starting: /go-working-dir/plugins/external/gocd-ec2-elastic-agent-plugin-2.2.2.jar
2024-04-11 08:11:02,693 INFO  [Thread-79] GoConfigDao:108 - Config update request by anonymous is in queue - com.thoughtworks.go.config.update.ReplaceElasticAgentInformationCommand@424dd8d0
2024-04-11 08:11:02,705 INFO  [Thread-79] GoConfigDao:111 - Config update request com.thoughtworks.go.config.update.ReplaceElasticAgentInformationCommand@424dd8d0 by anonymous is being processed
2024-04-11 08:11:02,832 INFO  [Thread-79] MagicalGoConfigXmlWriter:85 - [Serializing Config] Generating config partial.
2024-04-11 08:11:02,908 INFO  [Thread-79] GoFileConfigDataSource:587 - [Configuration Changed] Saving updated configuration.
2024-04-11 08:11:02,936 INFO  [Thread-79] CachedGoConfig:223 - About to notify config listeners
2024-04-11 08:11:02,941 INFO  [Thread-79] BuildAssignmentService:250 - [Configuration Changed] Removing jobs for pipelines that no longer exist in configuration.
2024-04-11 08:11:02,944 INFO  [Thread-79] CachedGoConfig:231 - Finished notifying all listeners
2024-04-11 08:11:02,945 INFO  [Thread-79] GoConfigDao:126 - Config update request by anonymous is completed
2024-04-11 08:11:02,985 WARN  [Thread-79] PluginSettingsMetadataLoader:63 - Failed to fetch plugin settings metadata for plugin com.continuumsecurity.elasticagent.ec2. Maybe the plugin does not implement plugin settings and view?
2024-04-11 08:11:02,986 WARN  [Thread-79] PluginSettingsMetadataLoader:64 - Plugin: com.continuumsecurity.elasticagent.ec2 - Metadata load info: [{extension='elastic-agent', configuration='null', view='null', error='java.lang.NullPointerException: Cannot invoke "com.continuumsecurity.elasticagent.ec2.Request.ordinal()" because the return value of "com.continuumsecurity.elasticagent.ec2.Request.fromString(String)" is null'}]
2024-04-11 08:11:02,986 WARN  [Thread-79] PluginSettingsMetadataLoader:65 - Not all plugins are required to implement the request above. This error may be safe to ignore.
2024-04-11 08:11:03,022 INFO  [Thread-79] DefaultPluginJarChangeListener:74 - Plugin load finished: /go-working-dir/plugins/external/gocd-ec2-elastic-agent-plugin-2.2.2.jar
2024-04-11 08:11:03,057 INFO  [Thread-79] DefaultPluginJarChangeListener:67 - Plugin load starting: /go-working-dir/plugins/external/gocd-groovy-dsl-config-plugin-2.0.0-241.jar
2024-04-11 08:11:03,972 INFO  [Thread-79] ConfigRepositoryInitializer:108 - [Config Repository Initializer] Start initializing the config repositories for plugin 'cd.go.contrib.plugins.configrepo.groovy'
2024-04-11 08:11:03,977 INFO  [Thread-79] ConfigRepositoryInitializer:112 - [Config Repository Initializer] Done initializing the config repositories for plugin 'cd.go.contrib.plugins.configrepo.groovy'
2024-04-11 08:11:04,109 ERROR [Thread-79] PluginInfoBuilder:54 - Failed to fetch Plugin Settings metadata for plugin cd.go.contrib.plugins.configrepo.groovy. Maybe the plugin does not implement plugin settings and view?
2024-04-11 08:11:04,119 INFO  [Thread-79] DefaultPluginJarChangeListener:74 - Plugin load finished: /go-working-dir/plugins/external/gocd-groovy-dsl-config-plugin-2.0.0-241.jar
2024-04-11 08:11:21,924 WARN  [WrapperJarAppMain] DeprecationWarning:43 - Using @Deprecated Class com.thoughtworks.go.server.web.DefaultHeadersFilter
2024-04-11 08:14:11,360 WARN  [qtp1067095815-35] FeatureToggleRepository:69 - Toggles file, /go-working-dir/config/go.feature.toggles does not exist. Saying there are no toggles.
2024-04-11 08:15:57,695 INFO  [qtp1067095815-35] ServerVersionInfoManager:71 - [Go Update Check] Starting update check at: Thu Apr 11 08:15:57 GMT 2024
2024-04-11 08:15:58,220 INFO  [qtp1067095815-27] ServerVersionInfoManager:86 - [Go Update Check] Update check done at: Thu Apr 11 08:15:58 GMT 2024, latest available version: 23.5.0-18179```

Many thanks 
Satya

Satya Elipe

unread,
Apr 11, 2024, 6:53:38 AMApr 11
to go...@googlegroups.com
Just to add:

I see the right content in the file in the /godata/:

```

bash-5.1$ ls -al /godata/cruise-config.xml

-rwxr-xr-x    1 root     root         14950 Apr 11 09:38 /godata/cruise-config.xml

bash-5.1$ cat /godata/cruise-config.xml

  <elastic>```



But then looks like the file getting changed after the sym links creation:

```

bash-5.1$ ls -al /go-working-dir/

total 107804

drwxrwxr-x    1 go       root         16384 Apr 11 09:42 .

drwxr-xr-x    1 root     root            90 Apr 11 09:41 ..

lrwxrwxrwx    1 go       root            17 Apr 11 09:41 artifacts -> /godata/artifacts

lrwxrwxrwx    1 go       root            14 Apr 11 09:41 bin -> /go-server/bin

lrwxrwxrwx    1 go       root            14 Apr 11 09:41 config -> /godata/config

-rw-r--r--    1 go       root     110355285 Apr 11 09:41 cruise.war

lrwxrwxrwx    1 go       root            10 Apr 11 09:41 db -> /godata/db

drwxr-xr-x   11 go       root           141 Apr 11 09:42 felix-cache

lrwxrwxrwx    1 go       root            14 Apr 11 09:41 lib -> /go-server/lib

lrwxrwxrwx    1 go       root            12 Apr 11 09:41 logs -> /godata/logs

lrwxrwxrwx    1 go       root            15 Apr 11 09:41 plugins -> /godata/plugins

drwxr-xr-x   11 go       root         16384 Apr 11 09:42 plugins_work

lrwxrwxrwx    1 go       root            14 Apr 11 09:41 run -> /go-server/run

drwxr-xr-x    3 go       root            52 Apr 11 09:41 work

lrwxrwxrwx    1 go       root            18 Apr 11 09:41 wrapper -> /go-server/wrapper

lrwxrwxrwx    1 go       root            25 Apr 11 09:41 wrapper-config -> /go-server/wrapper-config

bash-5.1$ cat /godata/config/cruise-config.xml

<?xml version="1.0" encoding="utf-8"?>

<cruise xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="cruise-config.xsd" schemaVersion="139">

  <server agentAutoRegisterKey="44eb6cdc-d8d9-4604-abfa-f061bbcbbee1" webhookSecret="506660f5-21e9-40c4-aa67-78c5cb37cde1" serverId="f256fbc9-d378-4649-a978-c4ddaca865b7" tokenGenerationKey="a7dc8993-3209-4459-ac18-85c1e89a5a60">

    <artifacts>

      <artifactsDir>artifacts</artifactsDir>

    </artifacts>

  </server>

</cruise>```



Is that been happening by the entrypoint script as such ? 


For information, here is my dockerfile:

```

# Use the official GoCD server image from gocd.org as the base
FROM gocd/gocd-server:v22.3.0

# Environment variables can be set if needed
ENV GO_SERVER_SYSTEM_PROPERTIES=""

# Switch to root user to install additional packages
USER root

# Install additional packages required for your setup
RUN apk update && apk add --no-cache \
    unzip \
    git \
    openssh-client \
    jq \
    ca-certificates \
    curl \
    zip \
 && rm -rf /var/cache/apk/* /var/lib/apt/lists/*

# No custom entrypoint script is used, rely on the base image's entrypoint

# Switch back to the GoCD user for safety
USER go

# Volumes where GoCD expects data to persist, you will mount your EFS to these locations
VOLUME ["/var/lib/go-server", "/etc/go"]

# Expose ports
EXPOSE 8153 8154

# The ENTRYPOINT from the base image is used, ensuring GoCD starts up as expected```



Many thanks

Satya

Chad Wilson

unread,
Apr 11, 2024, 11:42:04 AMApr 11
to go...@googlegroups.com
Just some housekeeping - there is no "GoCD Support Team". That would be a commercial thing you should expect to pay for :-)

It's quite likely your mount path and/or file permissions are wrong - or you at least need to fix these first and then see what else might be a problem.

If you are mounting with -v /mnt/gocd_efs:/godata from the host perspective, your cruise-config.xml needs to be at /mnt/gocd_efs/config/cruise-config.xml, and your config DB needs to be at /mnt/gocd_efs/db/config.git etc etc.

/godata/config/cruise-config.xml would be the only place GoCD will look for config INSIDE the container, so the content of /godata/cruise-config.xml is irrelevant - the root of /godata is not looked at, as you can see from the symlinks created in /go-working-dir

In any case, your volume listing is a mess, as you also seem to have a "godata" dir inside the same folder that you are trying to mounting as godata, and you have other files in the root which should not be in the root of a "godata" folder (cipher.aes, cruise-config.xml etc), so it looks like you have mounted the same thing to multiple different target dirs in a container and not cleaned up between runs. I strongly advise you mount a clean/empty folder from within your EFS volume and ensure you only have the expected data/folders in the expected places. It's extremely difficult to debug or understand what might be going wrong with such a mess, and may lead to unpredictable behaviour

If you start with a fresh volume (or empty directory within the volume) with nothing in it and let GoCD create its own structure of files inside /godata, you can compare the permissions and file locations and then make sure you copy your existing config/artifacts/databases carefully into the correct places, with the correct permissions.

Note when copying files around on the host (rather than inside the container) that you may need to correct all relevant file ownership on your volume inside /godata to be owned and writeable by the "go" user (UID=1000, probably the same as ec2-user on the host) - NOT root, as the container runs with a non-root user by default. This is something container orchestration such as Kubernetes can manage for you, but if you are doing manual containers and importing content from elsewhere to prepare volume you need to be comfortable with managing things like this yourself, or you will have unpredictable behaviour.

-Chad

Satya Elipe

unread,
Apr 12, 2024, 10:52:58 AMApr 12
to go...@googlegroups.com
Hi Chad,

Thank you for your assistance so far.


I’ve encountered a new issue where agents are launched correctly, and I can verify their status using "check agent status," which provides all the necessary details of the agents. However, these agents are not being assigned to their respective jobs.


I noticed that the Elastic agent plugin updates the /var/lib/go-agent/config/autoregister.properties file with a new "agent.auto.register.key" for every new job, resulting in constant key changes. My server’s "cruise-config.xml" retains a different key. Is this behavior expected? Perhaps this could be the issue why agents are not getting assigned jobs. 



Here is the glimpse of the autoregister.properties file:

agent.auto.register.key=b57ea434-9000-4d14-893a-47a422563680

agent.auto.register.hostname=EA_i-07a877b3997f767a9

agent.auto.register.elasticAgent.agentId=i-07a877b3997f767a9

agent.auto.register.elasticAgent.pluginId=com.continuumsecurity.elasticagent.ec2

agent.auto.register.environments=staging,sandbox```


I've done extensive research and attempted various solutions, but I'm still at a loss. Any insights or suggestions you could provide would be greatly appreciated.


Many thanks,

Satya


Chad Wilson

unread,
Apr 12, 2024, 11:14:28 AMApr 12
to go...@googlegroups.com
On Fri, Apr 12, 2024 at 10:52 PM Satya Elipe <satya...@gmail.com> wrote:

I noticed that the Elastic agent plugin updates the /var/lib/go-agent/config/autoregister.properties file with a new "agent.auto.register.key" for every new job, resulting in constant key changes.


Elastic agent auto-registration keys are generated ephemerally for each agent request, so you can expect it to be different for each elastic agent instance - it won't match what's in cruise-config.xml (that is only for static/persistent agents). So the changing key is not a problem on its own.

However I cannot help you specifically with the EC2 elastic agent plugin as this is a third party plugin which I have never used - not affiliated with GoCD core. You'll need to check the actual logs from an agent startup to see whether an individual elastic agent is registering correctly with the server and what error it is receiving (rather than guessing why the agent might not be appearing on the server and picking up jobs).

-Chad

 
Reply all
Reply to author
Forward
0 new messages