Offline config validation and normalisation

[Vasco Figueira]

Hi,

We are planning to run a large deployment and have all config managed by an external git repo. Users can spin up a docker image of the Go server and mess around but will have to commit their changes to a config repository which will be the authoritative source of truth.

The server polls (and pulls from) the repo into /etc/go and Go server automatically picks that up and commits to its internal git repo that we're not touching (as advised).

There are two things Go does that we would like to have as part of this workflow of ours: validation and normalisation. Concretely,

1. we would like to validate the file before committing it to our config repo;
   
2. we would like to normalise the file as Go does (remove comments, reorder elements, etc) so that it doesn't differ when on the server's filesystem.

The code for doing that is available (all hail open-source), it can be done by just hooking things in the right places, but I wonder if

1. There's any easy/"proven" way of achieving these goals
2. Whether there's any plan to factor out these "utility" concerns into a separate artefact that we could more easily deal with

Thanks.

Yours,

Vasco

[srinivas upadhya]

You can you MagicalGoConfigXmlLoader and MagicalGoConfigXmlWriter i guess.

Clone & Build code:

$ git clone https://github.com/gocd/gocd.git gocd-master

$ cd gocd-master

$ export JAVA_HOME=`/usr/libexec/java_home -v 1.7` (set JAVA_HOME if not set)

$ ./bn clean cruise:prepare SKIP_WAR=Y

create maven project & depend on config-server

        <dependency>

            <groupId>com.thoughtworks.go</groupId>

            <artifactId>config-server</artifactId>

            <version>1.0</version>

        </dependency>

For reference on how to use:

this should be the stacktrace (important parts) for config file write:

~/server/src/com/thoughtworks/go/server/service/GoConfigService.java - updateConfigFromUI

~/server/src/com/thoughtworks/go/config/GoConfigFileDao.java - updateConfig

~/server/src/com/thoughtworks/go/config/GoConfigDataSource.java - configAsXml

~/config/config-server/src/com/thoughtworks/go/config/MagicalGoConfigXmlWriter.java - write

this should be the stacktrace (important parts) for config file read:

~/server/src/com/thoughtworks/go/config/GoConfigDataSource.java - load

~/config/config-server/src/com/thoughtworks/go/config/MagicalGoConfigXmlLoader.java - loadConfigHolder

If you re-use config-server code for internal config file read & write then it should guarantee things you are looking for (validation + formatting).

If you add some code there then do send in a PR :)

[Vasco Figueira]

Thanks Srinivas,

Something like this? It appears to work.

https://github.com/vlfig/test-gocd-validation/blob/master/src/main/java/vlfig/gocd/validation/ConfigValidator.java

Thank you.

[srinivas upadhya]

Looks good to me. I would handle exceptions Go throws if file is invalid. Also, do not forget to update dependency version in pom.xml everytime Go Server is upgraded.

Thanks for keeping it open-source. Let us know if you need more help :)

[Vasco Figueira]

Hi Srinivas,

I'm using it as a commit/push hook so all I need is a return code. Believe me, I wish I had time to contribute to Go. :-)

One other thing:

Go expands templates when writing the config file, doesn't it?

If I pipe two validations in sequence (reading from f1, writing to f2, reading from f2, writing to f3) using a config (f1) that has a pipeline that uses a template it fails on the second validation because a stage from the template is now part of that pipeline (and it cannot both define stages and use a template). Verifying the actual intermediate config I see that the stages are now inline, in the pipeline definition.

f1:

        <pipeline name="myorg-cd-deploy-docker-image" template="deploy-myorg">

            <params>

                <param name="UPSTREAM_PIPELINE_NAME">coreplatform-build-helpers</param>

            </params>

            <materials>

                <pipeline pipelineName="coreplatform-build-helpers" stageName="test"/>

            </materials>

        </pipeline>

f2:

   <pipeline name="myorg-cd-deploy-docker-image" template="deploy-myorg">

      <params>

        <param name="UPSTREAM_PIPELINE_NAME"> coreplatform-build-helpers</param>

      </params>

      <materials>

        <pipeline pipelineName="coreplatform-build-helpers" stageName="test" />

      </materials>

      <stage name="deploy"> <!-- this comes from the template definition -->

        <jobs>

          <job name="invoke_ansible_deployer">

            <environmentvariables>

      ...

Is this by design? Circumventable? A Good Thing? Thoughts?

Again, thank you.

Vasco

[srinivas upadhya]

Oooops! Totally missed that. Here you are trying to write GoConfigHolder.config to file. Try writing GoConfigHolder.configForEdit to output file.

[Vasco Figueira]

Fixed. Cheers

[srinivas upadhya]

Great! May be you can write a blog post about it here (code). Other Go users might find it helpful.