Best way to re-configure Jobs on-the-fly

[Kenneth Baltrinic]

I have been tasked to look at how we can set up a self service CI system for a large development shop, much along the lines of how travis CI works.  There are big differences between what we need and what Travis CI provides but the basic idea of having a yaml file within the repository itself specify what the build server should do is what we are aiming for.  

Jenkins-Job-Builder from OpenStack seems to provide what we need from the standpoint of being able to take yaml files and create Jenkins jobs from them.  The part that I am seeking advice on is how to make this work from within a job, specifically we are looking for a workflow something like this:

• Given a job that is configured with an SCM Poll to poll a given git repository.
  
• When that poll triggers.
  
• Get the latest source
• Evaluate the build.yml file to create the rest of the jenkins job to executed 
• Execute the rest of the build job.

We have a working prototype right now that manages this with two separate jobs.  One to watch the SMC and rebuild the downstream job based on the build.yml and a separate downstream job that kicks off when the first completes successfully.  (Given the first might fail if the yaml files is invalid, etc.)  The problem with this approach is that we have hundreds of builds already, doubling the number of builds as this approach will do, is not considered acceptable.

Any advice or thoughts on what might be the best approach to doing this will be greatly appreciated.

--Ken

[James Nord]

Hi

What you are attempting sounds like the litterate job type - have you looked at this plugin?

/James

--
Sent from my Android phone with K-9 Mail. Please excuse my brevity.

[Stephen Connolly]

Your effort might be best placed helping us beef up the (stub) travis.yaml parser to one that is more useable.

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/49B39C84-10A8-4079-94B3-44C346EB45B6%40teilo.net.
For more options, visit https://groups.google.com/d/optout.

--
Sent from my phone

[Patrick van Dissel]

> We have a working prototype right now that manages this with two
separate jobs. One to watch the SMC and rebuild the downstream job
based on the build.yml and a separate downstream job that kicks off when
the first completes successfully. (Given the first might fail if the
yaml files is invalid, etc.) The problem with this approach is that we
have hundreds of builds already, doubling the number of builds as this
approach will do, is not considered acceptable.

This is exactly what we have also, but instead of a custom yaml file we
use the Jenkins Job DSL plugin:
- https://github.com/jenkinsci/job-dsl-plugin

About the amount of jobs, we think it's not a problem :) You can "hide"
those additional jobs in different views if wanted. It's just one
additional job per project, and each project already has atleast 6 jobs
anyway.

We have a seperate service running, which injects the initial "Seed" job
(currently trigger by a state change of a story in Jira). This seed job
runs the jobDsl file of that project to generate the rest of the jobs of
that project.
With this we've have complete story based development (aka feature
branches). So for each story we create the whole pipeline of that
application. Other services take care of setting up the story-based
infrastructure.

For us, the JobDSL is a perfect fit. It provides the flexibility to the
teams to configure their own application pipelines which go beyond the
basic CI setups that TravisCI and lookalikes provide.

/Patrick

On 12/02/2014 03:27 PM, Stephen Connolly wrote:
> Your effort might be best placed helping us beef up the (stub)
> travis.yaml parser to one that is more useable.
>
> On Tuesday, December 2, 2014, James Nord <te...@teilo.net

> <mailto:te...@teilo.net>> wrote:
>
> Hi
>
> What you are attempting sounds like the litterate job type - have
> you looked at this plugin?
>
> /James
>
>
>
> On 2 December 2014 13:31:19 GMT+00:00, Kenneth Baltrinic
> <ken...@baltrinic.com

> <javascript:_e(%7B%7D,'cvml','ken...@baltrinic.com');>> wrote:
>
> I have been tasked to look at how we can set up a self service
> CI system for a large development shop, much along the lines of
> how travis CI works. There are big differences between what we
> need and what Travis CI provides but the basic idea of having a
> yaml file within the repository itself specify what the build
> server should do is what we are aiming for.
>
> Jenkins-Job-Builder from OpenStack seems to provide what we need
> from the standpoint of being able to take yaml files and create
> Jenkins jobs from them. The part that I am seeking advice on is
> how to make this work from within a job, specifically we are
> looking for a workflow something like this:
>

> * Given a job that is configured with an SCM Poll to poll a
> given git repository.
> * When that poll triggers.
> o Get the latest source
> o Evaluate the build.yml file to create the rest of the
> jenkins job to executed
> o Execute the rest of the build job.

>
> We have a working prototype right now that manages this with two
> separate jobs. One to watch the SMC and rebuild the downstream
> job based on the build.yml and a separate downstream job that
> kicks off when the first completes successfully. (Given the
> first might fail if the yaml files is invalid, etc.) The
> problem with this approach is that we have hundreds of builds
> already, doubling the number of builds as this approach will do,
> is not considered acceptable.
>
> Any advice or thoughts on what might be the best approach to
> doing this will be greatly appreciated.
>
> --Ken
>
>
> --
> Sent from my Android phone with K-9 Mail. Please excuse my brevity.
>
> --
> You received this message because you are subscribed to the Google
> Groups "Jenkins Users" group.
> To unsubscribe from this group and stop receiving emails from it,
> send an email to jenkinsci-use...@googlegroups.com

> <javascript:_e(%7B%7D,'cvml','jenkinsci-users%2Bunsu...@googlegroups.com');>.

> To view this discussion on the web visit
> https://groups.google.com/d/msgid/jenkinsci-users/49B39C84-10A8-4079-94B3-44C346EB45B6%40teilo.net

> <https://groups.google.com/d/msgid/jenkinsci-users/49B39C84-10A8-4079-94B3-44C346EB45B6%40teilo.net?utm_medium=email&utm_source=footer>.

> For more options, visit https://groups.google.com/d/optout.
>
>
>
> --
> Sent from my phone
>

> --
> You received this message because you are subscribed to the Google
> Groups "Jenkins Users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to jenkinsci-use...@googlegroups.com

> <mailto:jenkinsci-use...@googlegroups.com>.

> To view this discussion on the web visit

> https://groups.google.com/d/msgid/jenkinsci-users/CA%2BnPnMwUjERstjqpbUjjEQdZJLHEPPP1oFJb2a%3D_Fq%2BO8Fy4XQ%40mail.gmail.com
> <https://groups.google.com/d/msgid/jenkinsci-users/CA%2BnPnMwUjERstjqpbUjjEQdZJLHEPPP1oFJb2a%3D_Fq%2BO8Fy4XQ%40mail.gmail.com?utm_medium=email&utm_source=footer>.

[Kenneth Baltrinic]

Gentlemen,

Thank you all for your responses.  Here are my thoughts on the combination of replies.  

The literate build plugin is closest in concept to what we are looking for but it seems too limited in capabilities for what we need.  If it could handle a full DSL such as the JobDSL, BuildFlow and Workflow plugins provide, it would exactly we need. Its focus however seems to be as a replacement for simple freestyle builds which basically run shell scripts.  We have fairly complex builds that use a number of Jenkins plugins to integrate with various build tools, etc.  We would not want to lose that functionality.  Also it does not appear, based on the state of the code and todo list, to be production ready.  

As for Patricks approach of hiding the extra jobs, I am not sure I can make that fly here but its still on our list of possible solutions if we must.  Our concern is the sheer number of jobs, each getting its own folder structure on our slave nodes and incurring other overhead.  Does anyone have any practical advice about how many jobs one can reasonably pile onto a single Jenkins installation (one master, about a dozen slaves in our current case).  Are there any practical limits, best practices around this?

As for the travis.yml parser, I couldn't find anything on google about a travis.yml parser for Jenkins.   However while conceptually what travis does and what we want are the same, (build definition incorporated into the source repo) the underlying build capabilities we need to support are very different from what travis supports.  Assuming it is attempting to provide a travis-like feature set, it would be very much in the same position as the literate build.

Any further thoughts?

--Ken

>     <javascript:_e(%7B%7D,'cvml','jenkinsci-users%2Bunsubscribe@googlegroups.com');>.

>     To view this discussion on the web visit
>     https://groups.google.com/d/msgid/jenkinsci-users/49B39C84-10A8-4079-94B3-44C346EB45B6%40teilo.net
>     <https://groups.google.com/d/msgid/jenkinsci-users/49B39C84-10A8-4079-94B3-44C346EB45B6%40teilo.net?utm_medium=email&utm_source=footer>.
>     For more options, visit https://groups.google.com/d/optout.
>
>
>
> --
> Sent from my phone
>
> --
> You received this message because you are subscribed to the Google
> Groups "Jenkins Users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to jenkinsci-use...@googlegroups.com

> <mailto:jenkinsci-users+unsub...@googlegroups.com>.

[Daniel Beck]

Maybe check out the YAML Project Plugin? It should be available on the experimental update center.

> > <javascript:_e(%7B%7D,'cvml','jenkinsci-users%2Bunsu...@googlegroups.com');>.

> > To view this discussion on the web visit
> > https://groups.google.com/d/msgid/jenkinsci-users/49B39C84-10A8-4079-94B3-44C346EB45B6%40teilo.net
> > <https://groups.google.com/d/msgid/jenkinsci-users/49B39C84-10A8-4079-94B3-44C346EB45B6%40teilo.net?utm_medium=email&utm_source=footer>.
> > For more options, visit https://groups.google.com/d/optout.
> >
> >
> >
> > --
> > Sent from my phone
> >
> > --
> > You received this message because you are subscribed to the Google
> > Groups "Jenkins Users" group.
> > To unsubscribe from this group and stop receiving emails from it, send
> > an email to jenkinsci-use...@googlegroups.com

> > <mailto:jenkinsci-use...@googlegroups.com>.

> > To view this discussion on the web visit
> > https://groups.google.com/d/msgid/jenkinsci-users/CA%2BnPnMwUjERstjqpbUjjEQdZJLHEPPP1oFJb2a%3D_Fq%2BO8Fy4XQ%40mail.gmail.com
> > <https://groups.google.com/d/msgid/jenkinsci-users/CA%2BnPnMwUjERstjqpbUjjEQdZJLHEPPP1oFJb2a%3D_Fq%2BO8Fy4XQ%40mail.gmail.com?utm_medium=email&utm_source=footer>.
> > For more options, visit https://groups.google.com/d/optout.
>

> --
> You received this message because you are subscribed to the Google Groups "Jenkins Users" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/e8dffb79-0fb8-4a3c-af54-c2ed5ccf94ef%40googlegroups.com.

[Stephen Connolly]

On Tuesday, December 2, 2014, Kenneth Baltrinic <ken...@baltrinic.com> wrote:

Gentlemen,

Thank you all for your responses.  Here are my thoughts on the combination of replies.  

The literate build plugin is closest in concept to what we are looking for but it seems too limited in capabilities for what we need.  If it could handle a full DSL such as the JobDSL, BuildFlow and Workflow plugins provide, 

Well I know that workflow will be getting some of the branch-api support, so watch that space

 

it would exactly we need. Its focus however seems to be as a replacement for simple freestyle builds which basically run shell scripts.  We have fairly complex builds that use a number of Jenkins plugins to integrate with various build tools, etc.  

Ha! You have not looked closely enough... You just need to do the right transformation from the literate description

 

We would not want to lose that functionality.  

Keep in mind that all Publishers are supported out of the box also

 

Also it does not appear, based on the state of the code and todo list, to be production ready.  

If you don't need pull request support, it's nearly there. 

Pull request support would open up the possibility of drive-by hacking. I do not view the plugin as 1.0 ready until people can have pull request support without the risk of drive-by hacking 

 

As for Patricks approach of hiding the extra jobs, I am not sure I can make that fly here but its still on our list of possible solutions if we must.  Our concern is the sheer number of jobs, each getting its own folder structure on our slave nodes and incurring other overhead.  Does anyone have any practical advice about how many jobs one can reasonably pile onto a single Jenkins installation (one master, about a dozen slaves in our current case).  Are there any practical limits, best practices around this?

As for the travis.yml parser, I couldn't find anything on google about a travis.yml parser for Jenkins.  

It's in literate-api

Amadeus have a more full implementation but I cannot recall if they published it yet

 

 However while conceptually what travis does and what we want are the same, (build definition incorporated into the source repo) the underlying build capabilities we need to support are very different from what travis supports.  Assuming it is attempting to provide a travis-like feature set, it would be very much in the same position as the literate build.

Any further thoughts?

You can provide your own literate format and add your own parser that builds the job the way you want and presto you are standing on the shoulder of literate ;-) 

To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/e8dffb79-0fb8-4a3c-af54-c2ed5ccf94ef%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Kenneth Baltrinic]

I want to thank everyone for their input.  At this time we have decided to pursue the workflow plug-in and its "load" step as the best way forward.  Given that we are dealing with trying to configure complex build flows, the workflow plugins other features should prove useful well beyond the scope of this current proof of concept.