issues loading cruise-config.xml on gocd version 23.4

[Sri Sai Sruthi JAGANNADDAM]

Hi team,

We are facing an an issue upgrading to GOCD version 23.4 from 21.2, 

On further investigation we found there is an issue with 3 plugins as observed in logs, when we remove the 3 bundled jars the server loads perfectly fine

 gocd-file-based-secrets-plugin.jar        gocd-json-config-plugin.jar          gocd-yaml-config-plugin.jar

But we are not sure what would be the impact if we remove these 3 jars as they are bundled with gocd. Do we know how can we resolve the issue with this plugins.

Below are the error logs:

023-11-10 17:24:13,705 INFO  [goPluginLocationMonitor] DefaultPluginJarChangeListener:67 - Plugin load starting: /var/lib/go-server/plugins/bundled/gocd-file-b
ased-secrets-plugin.jar
2023-11-10 17:24:13,729 INFO  [WrapperJarAppMain] GoConfigMigration:91 - Upgrading config file from version 139 to version 139
2023-11-10 17:24:13,730 ERROR [goPluginLocationMonitor] DefaultPluginManager:136 - This is an invalid request type :go.plugin-settings.get-configuration
2023-11-10 17:24:13,731 WARN  [goPluginLocationMonitor] PluginSettingsMetadataLoader:63 - Failed to fetch plugin settings metadata for plugin cd.go.secrets.file
-based-plugin. Maybe the plugin does not implement plugin settings and view?
2023-11-10 17:24:13,731 WARN  [goPluginLocationMonitor] PluginSettingsMetadataLoader:64 - Plugin: cd.go.secrets.file-based-plugin - Metadata load info: [{extens
ion='secrets', configuration='null', view='null', error='com.thoughtworks.go.plugin.api.exceptions.UnhandledRequestTypeException: This is an invalid request typ
e :go.plugin-settings.get-configuration'}]
2023-11-10 17:24:13,732 WARN  [goPluginLocationMonitor] PluginSettingsMetadataLoader:65 - Not all plugins are required to implement the request above. This erro
r may be safe to ignore.
2023-11-10 17:24:13,771 INFO  [goPluginLocationMonitor] DefaultPluginJarChangeListener:74 - Plugin load finished: /var/lib/go-server/plugins/bundled/gocd-file-b
ased-secrets-plugin.jar
2023-11-10 17:24:13,779 INFO  [goPluginLocationMonitor] DefaultPluginJarChangeListener:67 - Plugin load starting: /var/lib/go-server/plugins/bundled/gocd-json-c
onfig-plugin.jar
2023-11-10 17:24:14,073 INFO  [goPluginLocationMonitor] ConfigRepositoryInitializer:108 - [Config Repository Initializer] Start initializing the config reposito
ries for plugin 'json.config.plugin'
2023-11-10 17:24:14,113 INFO  [goPluginLocationMonitor] GoFileConfigDataSource:264 - Config file changed at 2023-11-10T16:57:20.392Z
2023-11-10 17:24:14,114 INFO  [goPluginLocationMonitor] GoFileConfigDataSource:265 - Reloading config file: /etc/go/cruise-config.xml
2023-11-10 17:24:14,319 INFO  [WrapperJarAppMain] GoConfigMigration:99 - Finished upgrading config file
2023-11-10 17:24:14,319 INFO  [WrapperJarAppMain] GoConfigMigrator:107 - [Config Save] Starting Config Save post upgrade using FullConfigSaveNormalFlow
2023-11-10 17:24:21,478 ERROR [goPluginLocationMonitor] GoFileConfigDataSource:466 - Unable to load config file: /etc/go/cruise-config.xml Parameter 'TEST_JOB_R
ESOURCEJOB_RESOURCE' is not defined. All pipelines using this parameter directly or via a template must define it., Resource name '#{JOB_RESOURCE}' is not valid
. Valid names much match '^[-\w\s|.]*$'
com.thoughtworks.go.config.exceptions.GoConfigInvalidException: Parameter 'TEST_JOB_RESOURCEJOB_RESOURCE' is not defined. All pipelines using this parameter dir
ectly or via a template must define it., Resource name '#{JOB_RESOURCE}' is not valid. Valid names much match '^[-\w\s|.]*$'
        at com.thoughtworks.go.config.MagicalGoConfigXmlLoader.validateCruiseConfig(MagicalGoConfigXmlLoader.java:133)
        at com.thoughtworks.go.config.MagicalGoConfigXmlLoader.preprocessAndValidate(MagicalGoConfigXmlLoader.java:122)
        at com.thoughtworks.go.config.FullConfigSaveFlow.preprocessAndValidate(FullConfigSaveFlow.java:97)
        at com.thoughtworks.go.config.FullConfigSaveNormalFlow.execute(FullConfigSaveNormalFlow.java:59)
        at com.thoughtworks.go.config.GoFileConfigDataSource.forceLoad(GoFileConfigDataSource.java:427)
        at com.thoughtworks.go.config.GoFileConfigDataSource.load(GoFileConfigDataSource.java:268)

[Sriram Narayanan]

On Sat, Nov 11, 2023 at 1:47 AM Sri Sai Sruthi JAGANNADDAM <sjagan...@mdsol.com> wrote:

Hi team,

We are facing an an issue upgrading to GOCD version 23.4 from 21.2, 

On further investigation we found there is an issue with 3 plugins as observed in logs, when we remove the 3 bundled jars the server loads perfectly fine

 gocd-file-based-secrets-plugin.jar        gocd-json-config-plugin.jar          gocd-yaml-config-plugin.jar

But we are not sure what would be the impact if we remove these 3 jars as they are bundled with gocd. Do we know how can we resolve the issue with this plugins.

It is safe to ignore these errors. The messages are because these plugins are not implementing specific UI configuration features (and do not need to - as per the follow-on messages).

We should ideally have better error messages for these.

In case you are using these three plugins, then please do test whether the functionality works for you (there should not be an issue).

-- Sriram

 

---

The information in this email and any attachments are intended solely for the recipient(s) to whom it is addressed, and may be confidential and/or privileged. Any unauthorized distribution or copying of this transmittal or its attachments is prohibited. If you are not a named recipient or have received this email in error: (i) you should not read, disclose, or copy it, (ii) please notify the sender of your receipt by reply email and delete this email and all attachments.

--
You received this message because you are subscribed to the Google Groups "go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email to go-cd+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/go-cd/da994589-e544-4445-a268-a12285133a14n%40googlegroups.com.

[Sri Sai Sruthi JAGANNADDAM]

This error is invalidating cruise-config.xml

[Jason Smyth]

Hi there,

The initial error/warning messages talk about gocd-file-based-secrets-plugin.jar in particular and, based on my reading, I believe Sriram is correct that they can be safely ignored.

I recommend focussing on this part of the log:

2023-11-10 17:24:21,478 ERROR [goPluginLocationMonitor] GoFileConfigDataSource:466 - Unable to load config file: /etc/go/cruise-config.xml Parameter 'TEST_JOB_R
ESOURCEJOB_RESOURCE' is not defined. All pipelines using this parameter directly or via a template must define it., Resource name '#{JOB_RESOURCE}' is not valid
. Valid names much match '^[-\w\s|.]*$'
com.thoughtworks.go.config.exceptions.GoConfigInvalidException: Parameter 'TEST_JOB_RESOURCEJOB_RESOURCE' is not defined. All pipelines using this parameter dir
ectly or via a template must define it., Resource name '#{JOB_RESOURCE}' is not valid. Valid names much match '^[-\w\s|.]*$'
        at com.thoughtworks.go.config.MagicalGoConfigXmlLoader.validateCruiseConfig(MagicalGoConfigXmlLoader.java:133)
        at com.thoughtworks.go.config.MagicalGoConfigXmlLoader.preprocessAndValidate(MagicalGoConfigXmlLoader.java:122)
        at com.thoughtworks.go.config.FullConfigSaveFlow.preprocessAndValidate(FullConfigSaveFlow.java:97)
        at com.thoughtworks.go.config.FullConfigSaveNormalFlow.execute(FullConfigSaveNormalFlow.java:59)
        at com.thoughtworks.go.config.GoFileConfigDataSource.forceLoad(GoFileConfigDataSource.java:427)
        at com.thoughtworks.go.config.GoFileConfigDataSource.load(GoFileConfigDataSource.java:268)

This explicitly states that the configuration file is invalid, and provides 2 distinct, but possibly related, reasons:

1. Parameter 'TEST_JOB_RESOURCEJOB_RESOURCE' is not defined.
   

1. • In my experience, this error is thrown under the following conditions:
     • There is a template that includes references to parameters.
     • There is a pipeline that is based on that template that does not define one (or more) of the parameters that the template requires.
   • I suspect the reason this exception is being thrown post-upgrade is explained by the second part of the error.

1. Resource name '#{JOB_RESOURCE}' is not valid. Valid names much match '^[-\w\s|.]*$'
   

1. • I suspect this error means that somewhere in the configuration is a resource that is defined via a parameter reference.
   • I also suspect that there has been a change in GoCD somewhere between version 21.2 and 23.4 that invalidates this use-case.
   • If both of the above are true, then:
     • The change may have been intentional. If so, it is likely documented in the release notes for the version where it was introduced.
     • The change may be an unavoidable side-effect of some other intentional change. If so, whether or not it is documented in the release notes probably depends on whether the side-effect was noticed during testing.
     • The change may be an unintentional regression. If so, it is like not documented in the release notes and we should probably have a GitHub issue created for it so that it can be tracked and, hopefully, fixed.

> But we are not sure what would be the impact if we remove these 3 jars as they are bundled with gocd.

Based on their names, it seems reasonable to assume that the impact is as follows:

• gocd-file-based-secrets-plugin.jar: Removing this will disable GoCD's file-based secrets plugin. If you are using file-based secrets management in your configuration, it will stop working.
  
• gocd-json-config-plugin.jar: Removing this will disable GoCD's ability to import pipeline definitions defined as JSON files in a source code repository. If you are using the Configuration Repository feature and one or more of your repos depend on plugin ID json.config.plugin, they will stop working.
• gocd-yaml-config-plugin.jar: Removing this will disable GoCD's ability to import pipeline definitions defined as YAML files in a source code repository. If you are using the Configuration Repository feature and one or more of your repos depend on plugin ID yaml.config.plugin, they will stop working.

The fact that you say the server loads fine if you remove these 3 plugins suggests to me that you are using at least one of them and the data that is invalidating the cruise-config.xml file is coming from that plugin's output. Given the error message at the bottom of the log trace, though, I do not believe the issue is with any of these plugins. Rather, the issue seems to stem from a change in GoCD itself.

Hope this helps,

Jason Smyth