Storing job configurations with source code in VCS

[Patrick van der Velde]

Hi All

We are trying to improve our build server setup and are trying to solve the problem that users can change the configuration of their builds through the UI, but those changes are not stored in source control, so it is hard to determine what has changed and who made those changes. Additionally (and maybe even more importantly) it is hard to rebuild an old version of the software because that may require a different job configuration. I know there are several plugins to handle backing up configuration changes to VCS / etc. but those backup to their own repositories (if they work) and we want the configuration for a build job to be stored with the source code for that build job, not in a separate repository. Because if we store the definition for the build job and the source code in the same location we could easily recreate a build environment as it was for that specific commit.

One option would be to use the Job-DSL plugin to create the jobs from a configuration file that is in VCS. The only catch with that would be that editing the configuration would require users to update the configuration text file which I suspect would increase the chance of errors and also makes it harder for the users to test configuration changes. Ideally we would like to keep the UI as the way to edit configurations but then just have that write to the configuration file.

I suspect this problem cannot be solved with our current version of jenkins (latest LTS 1.642.2) but I'm wondering if there are any plugins / combination of plugins / other apps that could make this possible?

thanks

Petrik

[Victor Martinez]

Hi,
You can easily add some tests to verify your jobDSL jobs locally before committing/pushing them.

Look at the below entry https://groups.google.com/forum/m/#!topic/job-dsl-plugin/xBOxlCMc6Qg

What I also do it is running a local jenkins docker container and apply those changes as the final validation

I hope it helps

[Patrick van der Velde]

Hi Victor

Well that's a good start. Being able to verify that the configuration works is good. However I would really prefer having some kind of UI to create the configurations to make it easier for users to edit their configurations. I'm guessing there are no plug-ins that allow me to achieve this.

Thanks

[Tom Moore]

On Tuesday, March 22, 2016 at 3:25:17 PM UTC-4, Patrick van der Velde wrote:

Hi Victor

Well that's a good start. Being able to verify that the configuration works is good. However I would really prefer having some kind of UI to create the configurations to make it easier for users to edit their configurations. I'm guessing there are no plug-ins that allow me to achieve this.

Thanks

We do a variant of this where our makefiles contain a build timestamp which is needed by our application and we want it captured so if we ever rebuild, we can exactly reproduce the binary (sha1 match on binary)

1) Setup your build job as normal, except do NOT use the delete workspace before building option.  If you do, this strategy wont work.
2) Run the build once to initialize your workspace to the scm you are connecting to.
3) Update your build job to have a pre-scm step that compares the job's config.xml with a copy that exists in your workspace repository:
   If the config.xml does not exist in the workspace, copy it local and commit it to your repository.
   If the config,xml exists in the workspace, and is different, then copy it local and commit the changes to the repository.
   If the config.xml exists and is the same as the workspace, do nothing.

After adding the pre-scm step, your config.xml will be evaluated and saved to the repository if modified.  Since this happens before the SCM step, then  the revision that actually gets built will contain the correct config.xml for the build.

[Craig Rodrigues]

Hi,

Take a look at the SCM Sync configuration plugin https://wiki.jenkins-ci.org/display/JENKINS/SCM+Sync+configuration+plugin

It allows users to modify their job configuration, but keeps track of the changes in Git or Subversion.

--

Craig

[Patrick van der Velde]

Hi

I've tried the SCM configuration plugin and while it allows us to track what changes the users have made it doesn't allow us to keep the build job configuration for each project with the project code (or at least linked to the project code somehow.

Also when we tested the SCM configuration plugin we found it to be very flaky in operation. It's hard to configure, it stores information we don't need and half the time it doesn't actually store the configuration changes.

Petrik

[Martin d'Anjou]

You could look at the Simple Build For Pipeline Plugin, combined with some TBD way of automatically creating job configurations (perhaps something like the BitBucket Branch Source Plugin). The Simple Build For Pipeline has a base class that can be expanded to support a custom mini-DSL code job declaration, and implement a Pipeline DSL flow inside a plugin of your own writing. With that plugin, users are presented with a high level declarative view of their flow, so they can turn knobs on their flow. This declarative view could be stored in their source code repo (git) as a "Jenkinsfile". The BitBucket Branch Source Plugin can scan BitBucket repositories and create jobs for projects that contain a special file literally named the "Jenkinsfile" (optionally one job per branch, even one job per pull-request if you wish). This Jenkinsfile could contain the call to the extension of Simple Build. The extension of Simple Build has to be a plugin that is uploaded to Jenkins. It provides many advantages: it can be unit tested as a Jenkins plugin on the command line (mvn test), and the flow becomes available as global variable in Jenkins (thus making it available to any job configuration in Jenkins). Conceptually, you could have multiple custom Simple Build extensions (each one as a different plugin of your own, or many in the same plugin of your own): one per type of job you need to run.

Conceptually, that is the closest I have come to being able to test the job configurations before I deploy them, the closest I have come to give users the ability tweak their build parameters in their own source code without making everyone an administrator. But I am still lacking the ability to create multiple jobs per project (i.e. multiple Jenkins jobs for each user (git) repository). Unfortunately, there can only be one Jenkingfile per branch, I wish I could have many since I don't want my pipeline all in the same job config.

Conceptually, if the Simple Build plugin extension is kept backwards compatible (which should be doable with good enough unit tests on the custom plugin), rerunning older builds, even years later, should be possible.

It is still a work in progress for me.

Hope this helps,
Martin

[Patrick van der Velde]

That sounds like a promising approach. I will go investigate that