best approach to automate/manage jenkins jobs?

[phil swenson]

Hi, we have a large number of jenkins jobs and would like to automate and version control their configuration.

From what I can tell, most people just manually config their jobs via the UI.  Unless there are only a few very simple jobs, this leads to an unmanageable mess.

I think the jobs should be coded in a DRY fashion, version controlled, and deployed via a scripted system.

Here are the approaches I am aware of:

1) scripting via command line (jenkins cli)

2) scripting via the rest web services

3) chef cookbook http://community.opscode.com/cookbooks/jenkins

What are most people doing?

Any recommendations?

thanks

phil

[Mark Waite]

I called the command line to extract the job definition, then stored that definition in source control.  I could replay it at any time.

That did not update job definitions when they changed.  It did not detect changes (other than through the version control system diff mechanism).  It was not attempting to code Jenkins job definitions in a DRY fashion.

It was simple to maintain and helped me reconstruct environments more readily on those rare times when I needed to reconstruct them.

Mark Waite

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--

Thanks!

Mark Waite

[Andrew Kujtan]

I use an ant task that generates my jobs based off a template job it pulls down. For me, the CLI tends to be really flaky though, when re-generating ~1500 jobs 10-20 will usually fail.

 

-Andrew

[John Vacz]

https://wiki.jenkins-ci.org/display/JENKINS/SCM+Sync+configuration+plugin

[Benjamin Lau]

This sounds similar to the technique described in this post:
http://www.blackpepper.co.uk/generating-new-jenkins-jobs-from-templates-and-parameterised-builds/

I've used this technique where I need to have lots of different build
which are mostly similar and need to be able to run in parallel.

Ben

[Les Mikesell]

Our jobs usually amount to polling an SCM location and running a build
or ant script found there and archiving the results from some standard
locations, so there's not much complexity in the jenkins job itself.
So, we just do nightly backups of the jenkins master with a program
called backuppc which has efficient on-line storage and a nice
interface to access any individual files or directories, and new jobs
are created in the GUI as copies of existing similar jobs and editing
the SCM target.

--
Les Mikesell
lesmi...@gmail.com

[James Nord (jnord)]

Hi Phil,

 

There is also the Job DSL plugin[1] and Cloudbees Templates[2].

The DSL plugin (not used it) can store the DSL in source control – you can add the Cloudbees templates (done it) to an SCM like git and commit & push them after you make any changes.

 

/James

 

--

[James Nord (jnord)]

Finger slipped – this time with some links.

 

[1] https://wiki.jenkins-ci.org/display/JENKINS/Job+DSL+Plugin

[2] http://www.cloudbees.com/jenkins-enterprise-by-cloudbees-features-templates-plugin.cb

 

From: James Nord (jnord)
Sent: 21 February 2014 16:20
To: jenkinsci-users
Subject: RE: best approach to automate/manage jenkins jobs?

 

Hi Phil,

 

There is also the Job DSL plugin[1] and Cloudbees Templates[2].

The DSL plugin (not used it) can store the DSL in source control – you can add the Cloudbees templates (done it) to an SCM like git and commit & push them after you make any changes.

 

/James

 

From: jenkins...@googlegroups.com [mailto:jenkins...@googlegroups.com] On Behalf Of phil swenson
Sent: 21 February 2014 15:15
To: jenkinsci-users
Subject: best approach to automate/manage jenkins jobs?

 

Hi, we have a large number of jenkins jobs and would like to automate and version control their configuration.

--

[phil swenson]

thanks for the replies!

No chef stories?  interesting...

[Richard Bywater]

To add another (that I didn't see listed but apologies if it was), we use Jenkins Job Builder - http://ci.openstack.org/jenkins-job-builder/

Richard.

--

[William Soula]

We use Jenkins Job Builder for job configuration.  This allows anyone to make changes to the job config, let the config live with the source code, and still have it under SCM.  We have the yaml in a folder in the source called jenkins, so I then have a job monitor that folder and run Jenkins Job Builder against that folder to reconfigure the job if the config has changed.

http://ci.openstack.org/jenkins-job-builder/

Will

[William Soula]

Dang I tried to make sure I had read all the replies before I sent this same thing.

Will

[phil swenson]

thanks for the tip on jenkins job builder!  looks great

[Christian Willman]

I don't like any of the open source templating plugins. I think they're rather inflexible and inelegant. I like the Cloudbees templating plugin but it's not necessarily cheap and you need to be familiar (or become familiar) with jelly.

We're in the same boat -- 99% of jobs share 99% of the same configuration. We create different code branches for different release dates, and sometimes we do parallel development, meaning we handle a lot of eerily-similar jobs. Additionally, we have many, many developers and we do not want them to directly touch the Jenkins job configuration UI. Reason being is partly because a global support team is responsible when a developer unknowingly hoses his/her job but still needs to cut a release by midnight.

I solved this by externalizing the templating system. At first we wrote a Rails app which simply accepts some input fields (job name, SCM URL, etc), clones and populates a template config.xml, and POSTs that over to Jenkins. 

But now we have too many Jenkins master instances and many active branches of very similar code, so manually creating jobs via a web UI is too inefficient. So I wrote a standalone application which contains a series of template job config.xml types and a means to parse Maven POMs as input to those templates. For instance, the job name is automatically determined via the Maven GAV. 

I've also baked-in support for arbitrary plugins; basically, anything that can contribute extra XML passages to a main template can be added to the tool in about five minutes. Everything supports variable interpolation, too. So a tech lead sets up Project X, June release with the following POM property:

<downstreamJobs>project-y-$version,project-z-$version</downstreamJobs>

When the job is created, $version resolves to "june-release" and then the downstreamJobs section is translated into a valid XML snippet and spliced into the template config.xml. The great thing is, in August, a fresh-out-of-college dev can create a brand new Project X, October release without any knowledge of the internals.

The final great thing is that this standalone application maps a SVN URL to an appropriate Jenkins master instance (in our case, the mappings are by business unit, which is fairly logical and standard). The app lives in an app server with a queue in front of it, and our SVN installation contains a post-commit hook which adds a SVN URL to that queue when the commit contains a special message ("+makeJob", inspired by Crucible smart commits). 

Now novice developers with very little knowledge about the CI system can create new jobs without ever entering in any technical details, and they'll receive an email containing a link to the new job as soon as it's created. Our goal is to allow developers to go from local commits -> artifact repository without ever worrying about the build system in between. It was a lot cheaper for us to engineer a good job templating system once (or twice, really) than to eternally field support requests.

This might be overkill for you; but then again, if you're tasked with managing thousands of jobs across a developer base with varying degrees of Jenkins competency, it might just be up your alley. If anything, realize that externalizing your templating system is a valid option.

PS -- I'd love to open source our templating system if anyone finds it interesting. My day job is not keen on releasing software so I would need some strong words of encouragement.

[Les Mikesell]

On Thu, Mar 6, 2014 at 3:46 AM, Christian Willman <cew...@gmail.com> wrote:
>
> But now we have too many Jenkins master instances and many active branches
> of very similar code, so manually creating jobs via a web UI is too
> inefficient.

Just curious - what is it that drives the decision to use different
jenkins masters? Would you overload a single master, or are they
split by location, phase of development, work group, or what?

--
Les Mikesell
lesmi...@gmail.com

[Christian Willman]

Great and valid question. Indeed the instances are split by work group. In the "old days" before I assumed the role of build master, developers could only modify jobs on their respective Jenkins instances. But since we've locked down and automated Jenkins, we allow every authenticated user to delete jobs, enable/disable jobs, run builds, whatever. Everything but creating/editing jobs is fair game, because never do we rely on Jenkins as a source of truth for job configuration.

It means we don't have to worry about the problems which will occur when one scales Jenkins vertically, by adding more and more slaves to a single master (a simple example is slower instance startup times due to the increased need to deserialize job configs to memory). Here is a link to Kohsuke's presentation on elastic build environments which briefly mentions horizontal vs vertical scaling. A lot of the Cloudbees marketing material also mentions this well.

Cheers, Christian.

[Gregg Lowrimore]

I'd love to learn more about this approach Christian.

I'm currently neck deep in the Job DSL area and stuck trying to incorporate some custom builder templates I've created, but can't seem to configure them in properly. Your approach might be a better solution for us.