a lot of jobs

[Marcin Biegan]

Hi,

I wanted to ask how do you handle configuring multiple jobs. We have
about 10 maven projects to build in at least 2 versions (trunk/
branch), which makes 20 jobs, some more for additional tests,
releases, maybe deployment, verification, etc. Manually adding
permissions and adjusting configuration of jobs is cumbersome. Is
there a working plugin which would allow to set some 'parent' job
which would define default configuration? (Template Project Plugin is
not enough, because it 'works' only at job creation). I know also
about Configuration Slicing Plugin, but this will not work in my case
as it requires admin permissions.

Thanks
Marcin Biegan

[Marek Gimza]

Marcin,

Do you know about the "MultiJob" plugin?

https://wiki.jenkins-ci.org/display/JENKINS/Multijob+Plugin

Regards,
Mgimza

[Marcin Biegan]

It does not suite my needs, it just allows to organize multiple jobs
which are supposed to all be triggered by the same event, am I
correct?

My jobs are pretty similar to each other - ex. build jobs have the
same configuration except svn url, downstream jobs list and maybe some
details (like email notification triggers), all other properties are
equal. But they are triggered at different times as they relate to
different projects. What I'm looking for is more like configuration
inheritance, where I'd need to change job configuration property once,
in only one place (in the 'base' job) instead of per each job (plus
possibility to override at job level).
Maybe there is possibility to use configuration slicing plugin without
admin permissions? That would partially work

Thanks
Marcin

On Nov 15, 5:48 pm, Marek Gimza <marekgi...@gmail.com> wrote:
> Marcin,
>
> Do you know about the "MultiJob" plugin?
>
> https://wiki.jenkins-ci.org/display/JENKINS/Multijob+Plugin
>
> Regards,
> Mgimza
>

[krishna chaitanya kurnala]

Hi

You can consider Configuration Slicing Plugin.

thanks

Krishna

Krishna Chaitanya

[Sami Tikka]

I type some Groovy commands into the Script Console you can find under the Manage Jenkins link in the sidebar.

-- Sami

[J Arrizza]

We have 420+ jobs. They are currently in Hudson but I am trying to convert to Jenkins, see below.

To manage them I wrote a nice set of ruby scripts where I declared the jobs and load them into a database (i.e. a ruby hash).

Then I use that DB to check if a job is new and needs to be created, or if it's old and needs to be deleted. The job creation is automated, but I do the job deletion manually - just in case.

It also compares the job settings against the DB contents and if they don't match, it will show me the differences and then, if I have the save_flag set to true, it will also update the Hudson job to match. This is a great feature. If someone manually changes the job, I know about it and I can revert their changes or change my scripts to match their updates.

I used ruby since I could easily create base classes, derived classes etc to accommodate the fact that most of the jobs are nearly identical. There are only a few base classes and all the other ones derive from them. 

Going back to your post, you can use the same technique. In your case you'd have say 10, or most likely fewer, base classes and 2 derived classes, one each for the trunk/branch jobs. The jobs would be created from the derived classes and then be stored in the DB (again just a simple hash). 

Once the DB is setup, a simple loop through it would configure all your jobs in one fell swoop. For example, mine sets up email addresses, sets job priorities, appointed slave nodes, job triggers, SCM triggers, Timer triggers, Locks, Xvnc, SCM repositories and a few others.

I put these scripts into our source control and now I have an audit trail of all the changes made to the jobs configurations. If something bad really happens, I could even revert to a prior version of the job configurations (it may require some changes of course).

And finally, the scripts provide an abstraction layer away from Hudson. I could convert to Jenkins by changing the job creation/update/delete portion of the scripts and voila, I have the exact same jobs on a Jenkins server.

But all is not peaches and cream. 

As I said above I'm using Hudson. I used the ruby gem hudson-remote-api to shield me from some of the low level details. Unfortunately that gem does not:

1) handle plug-ins

2) handle Jenkins

The first means I  had to "manually" change the XML nodes for each plugin. There is some basic structural similarities in the XML across all plugins but there are enough exceptions that it is a headache.

The second means I would have to re-engineer the same hacks against the Jenkins XML nodes. There is a gem jenkins-remote-api but it is much less developed than hudson-remote-api.

The "solution" to both is to have all Hudson and Jenkins plug-in developers provide a uniform scripting front-end to their configuration XML. A highly unlikely scenario but you never know...

Not sure how much that helps you, but perhaps it can give you some potential avenues to try.

John

[William Soula]

What you describe sounds very similar to the jenkins-job-builder project from openstack:

https://github.com/openstack-ci/jenkins-job-builder

Will

---

From: jenkins...@googlegroups.com [jenkins...@googlegroups.com] on behalf of J Arrizza [cppg...@gmail.com]
Sent: Tuesday, November 20, 2012 8:39 PM
To: jenkins...@googlegroups.com
Subject: Re: a lot of jobs

[J Arrizza]

It does look similar. However, the parameters it handles ( e.g blockBuildWhenDownstreamBuilding) are part of the default Jenkins behavior. At some point, they will also have to add external plugin support. This is where the bigger problem is waiting for them. If they can find a very simple, very elegant way to handle a diversity of Jenkins plugins. 

One way is they could use a plugin for each Jenkins plugin. And if they want to support Hudson and thereby the conversion to Jenkins, they would have to provide two plugins, one each for Hudson and Jenkins.

The YAML database is a nice touch. It might work well for them, if they keep it minimal and simple. In their doc they imply it should be checked in to source control. the idea being that all aspects of the jobs are held in YAML. I suspect that approach will eventually cause problems for them.

My preference is to build the db every time I run the scripts. This allows a bunch of fairly complex behavior to be captured by the scripting code. For example, it is possible to enforce a job naming convention by writing a bit of code to generate the names. To change the naming convention, I change that bit of code. If the names are already in YAML, then I'd have to "hack" the names manually with a text editor.

In any case, it might work quite well for the OP. It all depends on his exact set up.

John

[marouane zhani]

Hi Marcin,

I have the same needs actually, do you have a working solution for that ? 

I have tried a bit the Inheritance plugin (https://plugins.jenkins.io/project-inheritance/) but I am still having some problems with it. 

Looking forward to your reply.

Kind regards,

Marouane

[Nick Stolwijk]

We use the jobDSL to manage the 10 different jobs for the 100+ Git repositories we have. We are trying to reduce the number of jobs per repository by using a pipeline library in combination with a Jenkinsfile in each repository.

The original jobs:
- Build develop with deploy to Nexus

- Build feature branch with / without deploy to Nexus

- Update dependencies, test and commit.

- Start release

- Finish release

- OWASP build on master

- OWASP build on develop

- Sonar build on master

- Sonar build on develop

- Sonar build on featurebranch / PR

With new Maven Pipeline library:

- Start release

- Finish release
- Multibranch job based on Jenkinsfile

-- Runs:

* mvn process-test-classes

* mvn post-integration-test

* mvn verify -DskipTests

* Parallel OWASP and Sonar

* mvn deploy -DskipTests

If you want some more information, please let me know.

With regards,

Nick Stolwijk

~~~ Try to leave this world a little better than you found it and, when your turn comes to die, you can die happy in feeling that at any rate you have not wasted your time but have done your best ~~~

Lord Baden-Powell

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/0341ccd4-052b-4890-a49a-6510e50e79e6%40googlegroups.com.

[jeremy mordkoff]

I was thinking about this the other day. My plan is to create a simple jenkins job that manages all of my other jenkins jobs. Some of the other suggestions might feed into this. 

My goal is to able to bootstrap a new jenkins server in minutes for disaster recovery and for bringing up new development sites. 

roughly the steps would be 

1. deploy some docker servers (we use MAAS)
2. spin up a jenkins server instance using a Dockerfile in the devops repo
3. create this one special job and run it.  I am hoping I can define this job in my Dockerfile. This job will
1. survey MAAS for a list of jenkins build hosts and add them with executors and labels based on their size
2. pull the list of jobs from the devops repo (format TBD)
3. create the jobs
4. kick them off in some prescribed order (some have parameters or cron properties so they need to be run once manually)

I would be happy to share whatever I come up with although I can't promise it will be very soon. Any suggestions for projects I should look at for integration or inspiration?