How to structure Jenkins with a build that has 100's of projects?

[Matt Klein]

Hi All,

I'm interested incorporating Jenkins into our build process.  Our current build is comprised of 100's of libraries and executables for many different target platforms.  I'm struggling with how to structure this in Jenkins.

• Should every library/executable have a corresponding job entry in Jenkins?
• Can I use job dependencies to order the build process?
• How would I kick off a build to do everything if it's broken up into many jobs?

Thanks,

Matt

[Slide]

See comments inline

On Tue, Oct 28, 2014 at 2:41 PM, Matt Klein <mkle...@gmail.com> wrote:

Hi All,

I'm interested incorporating Jenkins into our build process.  Our current build is comprised of 100's of libraries and executables for many different target platforms.  I'm struggling with how to structure this in Jenkins.

• Should every library/executable have a corresponding job entry in Jenkins?

I would say yes

 

• Can I use job dependencies to order the build process?

Yes, you could, but I would recommend using build flow, it allows you greater control 

• How would I kick off a build to do everything if it's broken up into many jobs?

Build flow would be easy for this.

 

Thanks,

Matt

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Website: http://earl-of-code.com

[Matt Klein]

Thanks for the tips. I did a search and it appears build flow is a plugin. I will check that out.

Sounds like you are not 100% sure on the many project approach. Can you elaborate a bit on the trades you considered for that approach vs. wrapping many projects into a few jobs?

Thanks again,
Matt

[Patrick van Dissel]

Do yourself a favor and look into the Job DSL plugin
(https://github.com/jenkinsci/job-dsl-plugin).
Managing Jenkins by hand is a hell.. so use the JobDSL and script it al
together. And of course put the scripts in version control.

Most important JobDSL doc pages:
-
https://github.com/jenkinsci/job-dsl-plugin/blob/master/docs/Job-DSL-Commands.md
-
https://github.com/jenkinsci/job-dsl-plugin/blob/master/docs/Job-reference.md
-
https://github.com/jenkinsci/job-dsl-plugin/blob/master/docs/View-Reference.md

Watch the following presentation to get an intro into the usages of the
JobDSL plugin:
- https://www.youtube.com/watch?v=Gyccyj6lA8k

If you use Gradle for your JobDSL scripts repo, you can use my Gradle
plugin to compile the DSL scripts to XML locally without having a
Jenkins running. Handy for quick offline testing. But note, it does NO
validation that it generated correct Jenkins XMLs, it only makes sure it
is valid Groovy and that you used the JobDSL in a way that compiles. It
uses the Jenkins JobDSL plugin code to compile the JobDSL, but the
plugin itself also does no checks on it being valid Jenkins XML.
My Gradle plugin can be found at:
- https://github.com/pvdissel/gradle-jenkins-jobdsl

If you want ideas on how to improve release flows, look into Continuous
Delivery. Start by looking at:
- http://continuousdelivery.com
This is THE bible on Continuous Delivery. Enough stuff to read/watch
available on there.

Or if you want to start with the real basics, start with reading the
following book:
- http://martinfowler.com/books/duvall.html

About the jobs structure..
It really depends how you're going to use Jenkins.
Start simpel, and only add what is nessesary.
The amount of jobs does not really matter. For your idea, with my team
we currently have a Jenkins setup of 1 master with 3 slaves running a
bit more then 1000 jobs. With multiple jobs for single
apps/libraries/.., eg. CI, Deploy, DEV, TEST, ACC, PRO, etc. Almost all
managed with JobDSL scripts.

/Patrick

[Maciej Jaros]

Patrick van Dissel (2014-10-29 02:35):

[...]
About the jobs structure..
It really depends how you're going to use Jenkins.
Start simpel, and only add what is nessesary.
The amount of jobs does not really matter. For your idea, with my team
we currently have a Jenkins setup of 1 master with 3 slaves running a
bit more then 1000 jobs. With multiple jobs for single
apps/libraries/.., eg. CI, Deploy, DEV, TEST, ACC, PRO, etc. Almost all
managed with JobDSL scripts.

The JobDSL sounds very interesting! But how do you configure authorization in a complex environment? I mean I'd like to put DSL in SVN, but do not want to put passwords there. So two issues really:

1. SSH authorization (executing and sending files).
2. SVN authorization.

Regards,
Nux.

[Patrick van Dissel]

The JobDSL (currently) does not support having you global Jenkins config
files managed by the DSL. So, we have the global Jenkins configs in a
separate closed repository which we checked out manually once and is now
kept insync with a Jenkins job :). But solutions like
Puppet/Chef/Ansible could also help you there, we're skipping those
solution and are currently moving to use Docker (https://www.docker.com)
instead.

For the authorization, we use a separate Jenkins job which executes a
Groovy script. That groovy script looks at our landscape service;
- for each application
-- finds the team of that application
-- adds authentication section to the jobs
We run this job as downstream dependency of the JobDSL "Seed" job.

Next week I will give a talk on the JobDSL at the dutch JFall conference
(http://www.nljug.org/jfall/2014/) in The Netherlands. It will be mostly
code, showing an extensive CI and CD setup with the JobDSL. And also
showing steps how to start with the JobDSL. I will record the session
and share the slides, video and demo code afterwards.
I'll add the links here as soon as I have them shared :)

/Patrick

On 10/29/2014 12:47 PM, Maciej Jaros wrote:
> Patrick van Dissel (2014-10-29 02:35):
>> [...]
>> About the jobs structure..
>> It really depends how you're going to use Jenkins.
>> Start simpel, and only add what is nessesary.
>> The amount of jobs does not really matter. For your idea, with my team
>> we currently have a Jenkins setup of 1 master with 3 slaves running a
>> bit more then 1000 jobs. With multiple jobs for single
>> apps/libraries/.., eg. CI, Deploy, DEV, TEST, ACC, PRO, etc. Almost all
>> managed with JobDSL scripts.
>
> The JobDSL sounds very interesting! But how do you configure
> authorization in a complex environment? I mean I'd like to put DSL in
> SVN, but do not want to put passwords there. So two issues really:
>

> 1. SSH authorization (executing and sending files).
> 2. SVN authorization.
>
> Regards,
> Nux.

>
> --
> You received this message because you are subscribed to the Google
> Groups "Jenkins Users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to jenkinsci-use...@googlegroups.com

> <mailto:jenkinsci-use...@googlegroups.com>.

[Patrick van Dissel]

Hi,

As promised, I just published the video of my conference talk about:
"Cultivating the Jenkins job jungle with Groovy"

You can find the video and all info at:
- http://youtu.be/8jQm2wqX4xc

I hope you find it interesting :)

Regards,
Patrick