How to take care of a large Jenkins installation and still keep your sanity

[Patrick Hund]

Hi all,

there's a new series of blog articles on the eBay developer blog which I hope you find useful and entertaining:

http://www.technology-ebay.de/the-teams/mobile-de/blog/taming-the-hydra-part-1.html

It's written by my colleague Marc Günther, who is responsible for maintaining our Jenkins CI systems.

We are mobile.de, Germany's biggest online marketplace and other vehicles.

Cheers,
Patrick

[Daniel Serodio]

Nice intro, I look forward to the continuation of this series.

Regards,
Daniel Serodio

[Patrick Hund]

Thanks, here's part II: http://www.technology-ebay.de/the-teams/mobile-de/blog/taming-the-hydra-part-2.html

[Tony Wallace]

I was looking at Jenkins Job Builder but had not yet taken a run at it.

[milki milk]

Now this is more interesting. Using the .jenkins.yml file from the repository to inform the job generator what jobs to create is similar to travis.

Of course, the details are in the implementation in the next post.

I'm curious. 

When a repo needs testing, does it always go through the job generator before going through the generated jobs every single time?

Does the job generator clean up any jobs that are no longer used when the yaml file changes?

Are the jobs generated unique per repository or shared?

Can you create nice views, folders, pipeline views as well?

[Khai Do]

Answers to your questions..

 

When a repo needs testing, does it always go through the job generator before going through the generated jobs every single time?

You can set it up to update jobs only when there are changes to the JJB definition files.  For example we use puppet to monitor changes to the JJB definition files then have puppet run JJB to regenerate the jobs.  JJB also has a local cache of existing jobs so the updates are idempotent.

 

Does the job generator clean up any jobs that are no longer used when the yaml file changes?

JJB includes a '--delete-old' option to remove old jobs. http://docs.openstack.org/infra/jenkins-job-builder/execution.html#deleting-jobs

 

Are the jobs generated unique per repository or shared?

Jobs are typically independent of a repo.  So for example, from one definition file, you can create multiple jobs that clones from multiple repositories and do similar things.  JJB was pretty much designed to support this use case.  

 

Can you create nice views, folders, pipeline views as well?

View support should land on the next release:  https://review.openstack.org/#/c/206178/

Folders support is in development: https://review.openstack.org/#/c/180185/

In terms of pipelines, i assume you mean something provided by the pipeline plugin and that seems like it's supported: http://docs.openstack.org/infra/jenkins-job-builder/publishers.html?highlight=pipeline#publishers.pipeline

JJB also supports many other Jenkins plugins.

[milki milk]

On Monday, August 31, 2015 at 9:11:54 AM UTC-7, Khai Do wrote:

Answers to your questions..

 

When a repo needs testing, does it always go through the job generator before going through the generated jobs every single time?

You can set it up to update jobs only when there are changes to the JJB definition files.  For example we use puppet to monitor changes to the JJB definition files then have puppet run JJB to regenerate the jobs.  JJB also has a local cache of existing jobs so the updates are idempotent.

Ah. So this is how I'm understanding this then:

* all jobs are generated out-of-band via jenkins-job-builder (presumably via http://docs.openstack.org/infra/jenkins-job-builder/builders.html#builders.dsl)

* each individual repo's jenkins.yml file are referenced by the DSL for configured values.

Going off of the summary of the article, it states: "developers to easily create and configure the jobs for their repositories, without the usual copy/paste problems, while still allowing"

If the jenkins.yml is only for configured values for already generated jobs, then how do developers "create" jobs for their repositories. It seems like they would need to work with the JJB definition files which is out-of-band of the repo.

In the case that jobs are shared, then developers can configure their repo for those shared and already created jobs. But, if a developer wants to have a convenient UI view of their project jobs, it would be difficult to isolate the relevant job history from shared jobs (e.g. shared job0 #4564 triggered shared job1 #545212 triggered shared job2 #421).  I believe that view support becomes necessary in this case.

How do you help developers view their relevant jobs?

[Khai Do]

Yes, your general assessment is correct.  You can, but do NOT necessarily need to use the jenkins DSL plugin to generate jobs since JJB has it's own DSL.  Not sure if 'out-of-band' mean separate executable, separate jenkins job definitions, or in repo vs out of repo.  JJB does have an executable that you run and it does read it's own yaml based definition files.  The JJB definition files will generate the actual jenkins xml files that define the jenkins job and deploy those to the jenkins master.   JJB provides a path[1] option so the yml definitions can live anywhere, in your project's repo or out of it.  I think the quick start guide[2] provides some good example of how all this works. 

JJB deploys all jobs onto jenkins which if you have lots and lots of jobs can make the jenkins UI not real user friendly.  JJB does not help with organizing jobs into views or folders at this time.  Again, those features are in development right now so hopefully with a little more contribution those will land sometime in the near future :)  

[1] http://docs.openstack.org/infra/jenkins-job-builder/execution.html#passing-multiple-paths

[2] http://docs.openstack.org/infra/jenkins-job-builder/quick-start.html

[milki milk]

On Monday, August 31, 2015 at 10:27:11 AM UTC-7, Khai Do wrote:

Yes, your general assessment is correct.  You can, but do NOT necessarily need to use the jenkins DSL plugin to generate jobs since JJB has it's own DSL.  Not sure if 'out-of-band' mean separate executable, separate jenkins job definitions, or in repo vs out of repo.  JJB does have an executable that you run and it does read it's own yaml based definition files.  The JJB definition files will generate the actual jenkins xml files that define the jenkins job and deploy those to the jenkins master.   JJB provides a path[1] option so the yml definitions can live anywhere, in your project's repo or out of it.  I think the quick start guide[2] provides some good example of how all this works. 

By out of band, I meant it is not related to the jenkins.yml configuration file in a project repo. A push to a project repository that triggers a job does not trigger JJB. However, if JJB can reference the project repos, then a project repo push could trigger JJB and create/delete jenkins jobs. Is this, in fact, what you have - jenkins.yml to configure the job DSL of the project job itself and another JJB yaml file to create/delete jobs?

Another thing that plagues my use of JJB is the fact that Security credentials, like those uses in the git scm configuration, must be hardcoded the Credential ID string. This string is unique to each Jenkins master instance though which means jobs with these strings must be uniquely defined per instance. Do you use Credentials in your JJB definition files? If so, how do you manage the hardcoded strings?

[Khai Do]

It depends on how you want to configure jjb to update your jobs.  Again we use puppet to execute jjb to update.  I've seen others use the same jenkins instance to update the jobs.  I mean they setup a jenkins job on the jjb project itself that triggers on every change to the jjb definitions which will update the other jobs.  

I see a few solutions to your credentials question.  you can maybe use the credentials plugin[1] or you can add a feature that reads the credentials from the JJB configuration file[2], look at the stash section.  To add this feature you will need to propose a change[3] similar to the one for stash config.

[1] http://docs.openstack.org/infra/jenkins-job-builder/wrappers.html?highlight=credential%20bind#wrappers.credentials-binding

[2] http://docs.openstack.org/infra/jenkins-job-builder/execution.html

[3] https://review.openstack.org/#/c/212004/

[Guenther, Marc(AWF)]

Hi all,

I think there has been a little bit of confusion here.

1. As described in my blog article, we are using Job-DSL-Plugin, not Jenkins Job Builder (JJB). Both try to solve similar problems (automatically generate jobs based on external descriptions) but there are two very important differences:
- JJB is an external process communicating with Jenkins via REST-API, whereas Job-DSL-Plugin runs inside of Jenkins itself.
- Job-DSL-Plugin uses Groovy DSL to describe their jobs, whereas JJB uses plain Yaml files. The huge advantage of having a complete programming language instead of just some templating engine was the main point of my last blog.

2. The 'jenkins.yml' files I describe in the blog are purely our own convention. Neither Jenkins Job Builder nor Job-DSL-Plugin use anything like that. In particular, they have nothing to do with JJB's yaml files. They also do not describe complete job definitions, but only choose and configure existing definitions (what I wrongly called template).

Those confusions being hopefully resolved, I would like to answer your questions.

On 28 Aug 2015, at 18:18, milki milk <nites...@gmail.com> wrote:

> Now this is more interesting. Using the .jenkins.yml file from the repository to inform the job generator what jobs to create is similar to travis.

Yes, exactly. Only in this case it is more flexible, as the job definition you choose through the 'jenkins.yml' file has the full power of Job-DSL-Plugin available, which means it can do everything you can possibly do in Jenkins. Travis afaik is not that flexible in what the jobs can do (on purpose, at least that's what I understood from my last chat with one of their developers).

> Of course, the details are in the implementation in the next post.

Oh, yes, reminds me that I still have to write that ;)

> I'm curious.
>
> When a repo needs testing, does it always go through the job generator before going through the generated jobs every single time?

I'm not sure what you mean by "when a repo needs testing". So when there has been a commit in the repo, which changes the 'jenkins.yml' file, and that commit gets pushed, then that change will be applied to the existing jobs before they run. So technically, yes, every push goes through the job generator, but it very quickly figures out when there is no change and will run the real job(s).

> Does the job generator clean up any jobs that are no longer used when the yaml file changes?

Yes, that's a feature of the Job-DSL-Plugin itself, it keeps track what it generated on the last run, and deletes everything that it didn't generate this time.

> Are the jobs generated unique per repository or shared?

Unique. Every repository has it's own job(s). Or, more clearly, every job always belongs to one and only one repository. You can trigger jobs of other repositories (aka upstream/downstream jobs), of course. (I wasn't really aware that you could have one job listen on multiple repositories in Jenkins, and if so, why one would do such a thing).

And regarding Khai Do's reply who answered "Shared", I guess what he was referring to was "job definitions". These are indeed shared, only the jobs generated from them are unique for each repo.

> Can you create nice views, folders, pipeline views as well?

Yes, all of them. We heavily use Views and Folders, as each repository gets it's own Folder in Jenkins, and inside there are Views for branches and forks, for example. We also generate Delivery Pipeline Views.

On Monday, August 31, 2015 at 10:50:59 AM UTC-7, milki milk wrote:
> Another thing that plagues my use of JJB is the fact that Security credentials, like those uses in the git scm configuration, must be hardcoded the Credential ID string. This string is unique to each Jenkins master instance though which means jobs with these strings must be uniquely defined per instance. Do you use Credentials in your JJB definition files? If so, how do you manage the hardcoded strings?

Recent version 1.21 of Credentials Plugin allows to specify meaningful IDs for your credentials, so you can reuse them over several Jenkins masters, which is veeeeery convenient. You could also use EnvInject-Plugin to inject credentials as build variables, but Credentials Plugin is more secure.

Hope this makes sense,
Marc
--
** Marc Guenther ** mgue...@team.mobile.de ** Engineering Support ** mobile.de **

[milki milk]

On Monday, August 31, 2015 at 1:25:41 PM UTC-7, Guenther, Marc(AWF) wrote:

Hi all,

I think there has been a little bit of confusion here.

1. As described in my blog article, we are using Job-DSL-Plugin, not Jenkins Job Builder (JJB). Both try to solve similar problems (automatically generate jobs based on external descriptions) but there are two very important differences:
- JJB is an external process communicating with Jenkins via REST-API, whereas Job-DSL-Plugin runs inside of Jenkins itself.
- Job-DSL-Plugin uses Groovy DSL to describe their jobs, whereas JJB uses plain Yaml files. The huge advantage of having a complete programming language instead of just some templating engine was the main point of my last blog.

2. The 'jenkins.yml' files I describe in the blog are purely our own convention. Neither Jenkins Job Builder nor Job-DSL-Plugin use anything like that. In particular, they have nothing to do with JJB's yaml files. They also do not describe complete job definitions, but only choose and configure existing definitions (what I wrongly called template).

Those confusions being hopefully resolved, I would like to answer your questions.

Oh. So Khai Do isn't responding as someone who knows how Ebay's workflow works. Yes, this was the disconnect I was feeling because there was no mention of JJB, only a "job generator." Thanks for clearing that up.

 

On 28 Aug 2015, at 18:18, milki milk <nites...@gmail.com> wrote:

> Now this is more interesting. Using the .jenkins.yml file from the repository to inform the job generator what jobs to create is similar to travis.

Yes, exactly. Only in this case it is more flexible, as the job definition you choose through the 'jenkins.yml' file has the full power of Job-DSL-Plugin available, which means it can do everything you can possibly do in Jenkins. Travis afaik is not that flexible in what the jobs can do (on purpose, at least that's what I understood from my last chat with one of their developers).

> Of course, the details are in the implementation in the next post.

Oh, yes, reminds me that I still have to write that ;)

Yes! Very interested. JJB is limited by Jenkins and its plugins and the fact that is is essentially just POSTing XML to configuration pages.

 

> I'm curious.
>
> When a repo needs testing, does it always go through the job generator before going through the generated jobs every single time?

I'm not sure what you mean by "when a repo needs testing". So when there has been a commit in the repo, which changes the 'jenkins.yml' file, and that commit gets pushed, then that change will be applied to the existing jobs before they run. So technically, yes, every push goes through the job generator, but it very quickly figures out when there is no change and will run the real job(s).

Yup that answers my question. My "testing", I meant the project repo triggers a Jenkins job. Since the jenkins.yml file configures a job, are all project initial jobs essentially the same - read the yml, configure the jobs, run the jobs?

 

> Can you create nice views, folders, pipeline views as well?

Yes, all of them. We heavily use Views and Folders, as each repository gets it's own Folder in Jenkins, and inside there are Views for branches and forks, for example. We also generate Delivery Pipeline Views.

This sounds great for users!

So, while you have a handle on job management, do you find yourself with any performance issues for creating/deleting jobs? Additionally, do you see performance issues when all the jobs? I find that jenkins slows down with lots of jobs because it likes to parse job history and artifacts all the time and stuff everything into memory.

How do you deal with the job history? When a job is deleted, all history is loss and links go stale. Do you archive history?

[milki milk]

On Monday, August 31, 2015 at 1:25:41 PM UTC-7, Guenther, Marc(AWF) wrote:

On Monday, August 31, 2015 at 10:50:59 AM UTC-7, milki milk wrote:
> Another thing that plagues my use of JJB is the fact that Security credentials, like those uses in the git scm configuration, must be hardcoded the Credential ID string. This string is unique to each Jenkins master instance though which means jobs with these strings must be uniquely defined per instance. Do you use Credentials in your JJB definition files? If so, how do you manage the hardcoded strings?

Recent version 1.21 of Credentials Plugin allows to specify meaningful IDs for your credentials, so you can reuse them over several Jenkins masters, which is veeeeery convenient. You could also use EnvInject-Plugin to inject credentials as build variables, but Credentials Plugin is more secure.

! This is awesome. meaningful ids is exactly solves my issue. Nice find.

-milki 

[milki milk]

On Friday, August 28, 2015 at 6:55:09 AM UTC-7, Patrick Hund wrote:

Thanks, here's part II: http://www.technology-ebay.de/the-teams/mobile-de/blog/taming-the-hydra-part-2.html

So the article only mentions 2 methods - Cloudbees Template Plugin and Jenkins Job DSL. There are at least two other alternatives I know about - Jenkins Job Builder and Workflow. Were these and possibly other alternatives explored as well?

-milki 

[Guenther, Marc(AWF)]

On 01 Sep 2015, at 05:02, milki milk <nites...@gmail.com> wrote:

> I'm not sure what you mean by "when a repo needs testing". So when there has been a commit in the repo, which changes the 'jenkins.yml' file, and that commit gets pushed, then that change will be applied to the existing jobs before they run. So technically, yes, every push goes through the job generator, but it very quickly figures out when there is no change and will run the real job(s).
>
> Yup that answers my question. My "testing", I meant the project repo triggers a Jenkins job. Since the jenkins.yml file configures a job, are all project initial jobs essentially the same - read the yml, configure the jobs, run the jobs?

Yes, exactly. The generator runs in the time span, that we always had for our Quiet Period anyway, which is 15sec.

> So, while you have a handle on job management, do you find yourself with any performance issues for creating/deleting jobs?

Actual job creation/deletion is quite fast. To my surprise, the bottleneck are all the github api calls we are doing. There is certainly room for improvement there.

> Additionally, do you see performance issues when all the jobs? I find that jenkins slows down with lots of jobs because it likes to parse job history and artifacts all the time and stuff everything into memory.

I don't really like to say this, but we do a daily restart of Jenkins master. Otherwise it wouldn't survive the week. Otoh, this was on the old legacy instance, and we never re-evaluated this, so maybe situation has improved with newer Jenkins versions. But given your statement, I guess not? ;(

> How do you deal with the job history? When a job is deleted, all history is loss and links go stale. Do you archive history?

Yes, when for example a feature branch is deleted, the corresponding sets of jobs for that branch are deleted as well, including their history. So far, no one was ever interested in that history.

Marc

[Guenther, Marc(AWF)]

Hi,

On 03 Sep 2015, at 01:36, milki milk <nites...@gmail.com> wrote:

> So the article only mentions 2 methods - Cloudbees Template Plugin and Jenkins Job DSL. There are at least two other alternatives I know about - Jenkins Job Builder and Workflow. Were these and possibly other alternatives explored as well?

Well, the article was already quite long anyway, didn't want to bore everyone away :)

Job DSL Plugin and Jenkins Job Builder both try to solve the same problem, but have different approaches. Jenkins Job Builder runs as an external process, which has the disadvantage that it does not know about current Jenkins state, so it has to keep it's own cache. Also, it uses Yaml for its templates, which limits the expressive power of what templates can do. They had to invent their own syntax for interpolating variables, and do inheritance, for example.

There are also similarities. Both can only do what you could already do manually with Jenkins, but now do it in an automated way. Both have to solve the problem to support all the different Jenkins plugins out there, and both provide a nice way to easily use not yet supported plugins yourself.

Although its Yaml files are not a programming language, one could still call Jenkins Job Builder from a programming language, so I guess we could have done some of the stuff that we did with Job DSL Plugin. But we already had an in-house mechanism using Rest API, which suffered from the limitation of not having access to current Jenkins internal state, so we didn't want to go down that road again.


Now, Workflow Plugin is an entirely different beast, which solves different problem. It defines a new type of job with new capabilities and as such is not really an alternative, but more an enhancement to the other two. It is perfectly valid to use Job-DSL Plugin to generate Workflow jobs for example, and we are currently investigating how to incorporate this in our existing jobs.

Marc

[milki milk]

On Thursday, September 3, 2015 at 4:03:50 AM UTC-7, Guenther, Marc(AWF) wrote:

On 01 Sep 2015, at 05:02, milki milk <nites...@gmail.com> wrote:

> I'm not sure what you mean by "when a repo needs testing". So when there has been a commit in the repo, which changes the 'jenkins.yml' file, and that commit gets pushed, then that change will be applied to the existing jobs before they run. So technically, yes, every push goes through the job generator, but it very quickly figures out when there is no change and will run the real job(s).
>
> Yup that answers my question. My "testing", I meant the project repo triggers a Jenkins job. Since the jenkins.yml file configures a job, are all project initial jobs essentially the same - read the yml, configure the jobs, run the jobs?

Yes, exactly. The generator runs in the time span, that we always had for our Quiet Period anyway, which is 15sec.

I see that Jenkins Job DSL has a queue command. Is this how you trigger jobs or do you have a subsequent build step that triggers a job? Queue doesn't seem to have any explicit support for parameters.

When you create a new repo, what do you use to create the initial Job DSL job for the repo? I'm thinking either have a single Job DSL that is responsible for all initial jobs or having JJB manage those initial jobs.

 

> Additionally, do you see performance issues when all the jobs? I find that jenkins slows down with lots of jobs because it likes to parse job history and artifacts all the time and stuff everything into memory.

I don't really like to say this, but we do a daily restart of Jenkins master. Otherwise it wouldn't survive the week. Otoh, this was on the old legacy instance, and we never re-evaluated this, so maybe situation has improved with newer Jenkins versions. But given your statement, I guess not? ;(

We haven't been able to evaluate this effectively because we are so afraid to do restarts which means its hard to do plugin updates or jenkins version updates. We also didn't like to do daily restarts because we pretty much have long and short jobs running all the time all day and all night long. Restarts of the master are disruptive and we can't do safe restarts because of the long running jobs.

Thanks for answering all my questions! This is the first time I've seen this much detail about successfully managing Jenkins and it sounds great.

-milki

[milki milk]

On Monday, August 31, 2015 at 1:25:41 PM UTC-7, Guenther, Marc(AWF) wrote:

2. The 'jenkins.yml' files I describe in the blog are purely our own convention. Neither Jenkins Job Builder nor Job-DSL-Plugin use anything like that. In particular, they have nothing to do with JJB's yaml files. They also do not describe complete job definitions, but only choose and configure existing definitions (what I wrongly called template).

So while waiting for the next blog post, I've tried implementing the same thing myself and I'm a fan of this method!

My project repos contain a .jenkins.json file because there didn't seem to be a Yaml parser included with the jenkins groovy. This file specifies the type of job should be run, what kind of pipelines I want to it to trigger, and includes all the variables needed by the respective jobs that are created.

I use jenkins-job-builder to define the "seed" job for the project that contains a checkout of both the project repo and a job-dsl repo placed at .jobdsl. This seed job can easily be replaced by another job-dsl job. In this job-dsl repo, I define a bunch of predefined pipelines and job templates which I can import into the groovy dsl script that gets run by the seed job.

When the seed job runs, it examines the .jenkins.json, imports the classes from the job-dsl repo, and then generates all the jobs needed for this project inside a Jenkins Cloudbees Folder. Finally, it triggers the "main job" for the project that was generated.

This appears to work well at the testing scale. We'll see how it handles everything at a larger scale.

I look forward to the next blog post that has more implementation details.

-milki

[milki milk]

Damien in JENKINS-31152 mentioned the Seed Plugin (https://github.com/jenkinsci/seed-plugin/wiki).

This sounds a lot like another formalized framework that is similar to Ebay's seed job. It takes a seed.groovy and/or a seed.properties as configuration. It has a concept of a "project seed" and a "branch seed". It uses groovy and Jenkins Job DSL under the hood to generate the rest of the jobs. It looks pretty good at first glance.

In particular, I like the generation configuration illustration in an image: https://raw.githubusercontent.com/wiki/jenkinsci/seed-plugin/Overview_Pipeline.png

-milki

[Indra Gunawan (ingunawa)]

I am interested in this topic to truly adopt "Configuration as Code” including Jenkins job configuration.  

What are the pros and cons of vs. Workflow plugin, DSL plugin, YAML Project Plugin and Literate Plugin?

Thank you

-Indra

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/977717f9-c41d-46a8-aaf6-332d2136c057%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Guenther, Marc(AWF)]

Hi,

here is part 3 of the series, where I describe the actual implementation that we came up with and are currently using:

http://www.technology-ebay.de/the-teams/mobile-de/blog/taming-the-hydra-part-3.html

Sorry for the long delay :)

Have fun,
Marc

On 28 Aug 2015, at 15:55, Patrick Hund wrote:

> Thanks, here's part II: http://www.technology-ebay.de/the-teams/mobile-de/blog/taming-the-hydra-part-2.html
>

[Guenther, Marc(AWF)]

Hi milki milk,

sorry for the very long delay... Currently trying to followup on all the stuff that accumulated...

On 03 Sep 2015, at 16:32, milki milk <nites...@gmail.com> wrote:

> On Thursday, September 3, 2015 at 4:03:50 AM UTC-7, Guenther, Marc(AWF) wrote:
>
>> On 01 Sep 2015, at 05:02, milki milk <nites...@gmail.com> wrote:
>>
>> > Yup that answers my question. My "testing", I meant the project repo triggers a Jenkins job. Since the jenkins.yml file configures a job, are all project initial jobs essentially the same - read the yml, configure the jobs, run the jobs?
>>
>> Yes, exactly. The generator runs in the time span, that we always had for our Quiet Period anyway, which is 15sec.
>>
> I see that Jenkins Job DSL has a queue command. Is this how you trigger jobs or do you have a subsequent build step that triggers a job?

At the moment we use the normal Jenkins mechanism (the Github Plugin) to trigger jobs through the web hooks, with the 15sec delay. This has several drawbacks:
- when the generator takes more than the 15sec to finish, the old versions of the jobs will be started.
- new jobs do not get triggered the first time, as they didn't exist when the web hook was triggered.

We are thinking to use the queue command instead, and turn off the builtin Jenkins polling/triggering mechanism completely.

> Queue doesn't seem to have any explicit support for parameters.

Parameters would not be needed, the web hooks do not have them either.

> When you create a new repo, what do you use to create the initial Job DSL job for the repo? I'm thinking either have a single Job DSL that is responsible for all initial jobs or having JJB manage those initial jobs.

Well, there is nothing special about new repositories. If they don't have a jenkins.yml file, they are simply ignored. If they have one, the generator job (= parameterized seed job) will do it's thing and create the relevant jobs.

>> > Additionally, do you see performance issues when all the jobs? I find that jenkins slows down with lots of jobs because it likes to parse job history and artifacts all the time and stuff everything into memory.
>>
>> I don't really like to say this, but we do a daily restart of Jenkins master. Otherwise it wouldn't survive the week. Otoh, this was on the old legacy instance, and we never re-evaluated this, so maybe situation has improved with newer Jenkins versions. But given your statement, I guess not? ;(
>>
> We haven't been able to evaluate this effectively because we are so afraid to do restarts which means its hard to do plugin updates or jenkins version updates. We also didn't like to do daily restarts because we pretty much have long and short jobs running all the time all day and all night long. Restarts of the master are disruptive and we can't do safe restarts because of the long running jobs.

Yeah, I miss TeamCity in that respect. You could restart the master when ever you wanted, and the slaves would continue building and simply reconnect once the master is back... We use Swarm Client Plugin in Jenkins, so here too the slaves reconnect automatically, but the builds are all aborted...

> Thanks for answering all my questions! This is the first time I've seen this much detail about successfully managing Jenkins and it sounds great.

Thanks :)

Marc