[JIRA] (JENKINS-43749) Support multiple Jenkinsfiles from the same repository

32 views
Skip to first unread message

mark@kakapo.co.nz (JIRA)

unread,
Mar 13, 2018, 5:24:03 PM3/13/18
to jenkinsc...@googlegroups.com
Mark Wright commented on Improvement JENKINS-43749
 
Re: Support multiple Jenkinsfiles from the same repository

Jens Beyer in your example, do you use githooks to trigger builds? If so, how do you prevent changes from one of your sub-projects from triggering building of another? E.g how do you prevent a change under /project-app from building /project2-app?

Or do you just build them all?

Add Comment Add Comment
 
This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e)
Atlassian logo

j.beyer@stoneball.de (JIRA)

unread,
Mar 19, 2018, 2:29:04 AM3/19/18
to jenkinsc...@googlegroups.com

Mark Wright: Indeed we have the problem that githooks trigger all builds, and this is some kind of issue, but not too big for us (rarely leads to intermediate build errors because the projects depend on each other). We thought about solutions once the issue would become bigger, like cancelling builds if their upstream builds are still in queue, or lockable resources) but at the moment we believe only the separation of the subprojects into own repositories would probably help in the long run. Except: If Git polling would honour sparse checkouts (maybe as additional option, since I can imagine this being quite expensive), this would not be necessary at all.

las@dbc.dk (JIRA)

unread,
Jul 12, 2018, 7:38:04 AM7/12/18
to jenkinsc...@googlegroups.com
larsskj commented on Improvement JENKINS-43749

Just found this thread - and I support the purpose as well.

However, maybe the thread should be renamed "Support multiple Jenkinsfiles from the same GIT repository" - as I'm doing quite a few things presently with Subversion that works very well, but just cannot be done with Git.

One usecase: I've setup a large system to do automatic updates on a lot of our Debian hosts. Some runs daily, some runs weekly. They're all scheduled to run between 5 and 6 o'clock in the morning, and as the update includes a reboot, they just cannot be run in normal working hours. From 6 o'clock our normal support staff is on guard to react to failed updates - quite often by rolling back the update by reverting to a VMware snapshot that is created as part of the automatic update. If they do not need to rollback, the snapshot is deleted again by another cron-like job running at 11 o'clock. This has worked quite well for several years.

Each host has it own Jenkinsfile that - among other things - run a special test after the upgrade tailored specifically for that host. The Jenkinsfiles are all stored in a single Subversion repository (Well: 25 - named A-Z - we have a LOT of hosts), and the Jenkins job is set up as a Multibranch Pipeline. Needless to say, the job has been set up to suppress automatic SCM triggering. Instead, they're triggered by the clock, and can be run manually on demand. The only gotcha so far is the fact that the job must be run manually the first time on a new host - otherwise Jenkins won't pick up the

properties([pipelineTriggers([cron('H 5 * * *')])])

statement from the Jenkinsfile.

When checked out from Subversion, a repository looks like this:

hosts
   |-   host-01.noone.net
        |-   Jenkinsfile
   |-   host-02.noone.net
        |-   Jenkinsfile
   |-   host-03.noone.net
        |-   Jenkinsfile
   |-   etc.

Adding a new host becomes as simple as copy/pasting an existing setup and making the necessary modifications. Because of the nature of Subversion, this structure can be used in a Jenkins multibranch pipeline which will create a pipeline for each host - named after the host - and automatically create new jobs as hosts are added - and disabling existing hosts as they're deleted from Subversion. Lots of Jenkinsfiles in a single repository - a solution that works very well for us.

But, unfortunately, we're slowly migrating to Git, and I haven't found a solution with Git that can do the same trick with Jenkins.

Another use case: We're hosting a lot of Drupal sites and we're building and deploying these as Docker containers. I've created a Jenkinsfile and a multibranch pipeline and the code is stored in Git. My colleagues use this setup the usual way, creating feature branches in Git, merging them into a develop branch, then merging into the master branch. We deploy all the Docker containers of all the branches for testing and approval by the product owner.

But the deployment jobs? These are pipelines as well - but I do not want deployment to part of the huge Jenkinsfile in the repo root. I've several reasons for that, but basically we don't want to wait hours for a long building job just to redeploy a few Docker containers. I would like very much to store the deployment jobs in the same Git repo - but with the current Jenkins pipeline functionality, I cannot - or at least I haven't figured out how to do it. So I have to maintain a separate repository for each deployment job, not an optimal situation.

With Subversion I could easily do it, but as Git doesn't allow checkout of subbranches - or rather, the Jenkins SCM plugins cannot do it - I just cannot do it with Git.

Just my thoughts.

This message was sent by Atlassian JIRA (v7.10.1#710002-sha1:6efc396)

p.leibiger@codecraft.de (JIRA)

unread,
Jul 12, 2018, 8:11:03 AM7/12/18
to jenkinsc...@googlegroups.com

larsskj The only way I found was using a different branch per host. It works me for now but it doesn't feel very nice.

las@dbc.dk (JIRA)

unread,
Jul 12, 2018, 8:21:03 AM7/12/18
to jenkinsc...@googlegroups.com
larsskj commented on Improvement JENKINS-43749

Peter Leibiger: That's actually what I'm doing. I do tell Jenkins that I have a Subversion branch per host - that's why the multibranch pipeline works so fine.

The real difference lies within the nature of Subversion vs. Git: With Subversion, you can check out a complete repository with all branches side-by-side, and it's very easy to copy code from one branch to another. With Git, you can only have a single branch active at the time, so it becomes very clumpsy to copy/paste from one branch to another. And you completely loose the overview at the same time.

djay.il@gmail.com (JIRA)

unread,
Jul 12, 2018, 8:55:03 AM7/12/18
to jenkinsc...@googlegroups.com

larsskj You kind of can checkout 2 branches at once using git worktrees. But that's still not ideal.

Supporting more than 1 Jenkinsfiles would be definitely better.

OTOH, if you don't mind creating jobs manually, then you can still create a pipeline and use a custom Jenkinsfile, which you manually set from that hierarchy above. That could also work in your case.

las@dbc.dk (JIRA)

unread,
Jul 12, 2018, 9:14:05 AM7/12/18
to jenkinsc...@googlegroups.com
larsskj commented on Improvement JENKINS-43749

Alex Lourie: I do mind creating jobs manually: The jobs are not maintained by me, they're maintained on a daily basis by our operations support staff, and they need the simplicity that they just has to maintain the list of jobs in Subversion.

I could create a Subversion post-commit hook that maintained the list of jobs in Jenkins - but that would be very ugly and add a lot of complexity to the solution.

It would be a lot better if Jenkins pipelines supported more than one Jenkinsfile in Git. They already do with Subversion.

p.leibiger@codecraft.de (JIRA)

unread,
Jul 12, 2018, 9:15:02 AM7/12/18
to jenkinsc...@googlegroups.com

Alex Lourie I just learned about git worktree 20 minutes ago , that makes it a lot easier for me.

I am not creating jobs manually, I use the Bitbucket Branch Source Plugin which automatically creates a job per branch that has a Jenkinsfile.

With git worktree I can now have all my hosts open in IntelliJ at once and copy from one to another.
Now Gitkraken just needs to support worktrees...

 

djay.il@gmail.com (JIRA)

unread,
Jul 12, 2018, 9:26:03 AM7/12/18
to jenkinsc...@googlegroups.com

Peter Leibiger well, today is not totally lost I guess.

larsskj yea, multi-Jenkinsfile support would be nice indeed. Sorry you're stuck with SVN

las@dbc.dk (JIRA)

unread,
Jul 12, 2018, 9:33:04 AM7/12/18
to jenkinsc...@googlegroups.com
larsskj commented on Improvement JENKINS-43749

Alex Lourie: I'm not sorry - I like SVN very much - and I'm not very fond of Git...

costincaraivan@gmail.com (JIRA)

unread,
Jul 20, 2018, 11:30:11 AM7/20/18
to jenkinsc...@googlegroups.com

I have to say something because I see developers commenting about the Pipeline "purity" and "goals". As a long time Jenkins user/admin/CI guy, I'd say that you need to know your clients.

 

I imagine that the shiny Pipeline sells Cloudbees contracts, and that's great, we want Cloudbees to be successful and be able to develop Jenkins in the future. However, the truth from the trenches is a bit different: Jenkins is both used for shiny pipelines but it's used just as much as a distributed cron with an execution history.

 

And because of missing features such as these, the declarative pipeline especially can't replace the freestyle jobs. And it should. Otherwise you're asking me (ok) and especially devs or ops guys who won't ever become Jenkins experts to learn:

 

  • Groovy (kind of a dead language outside of Jenkins, right now)
  • the scripted pipeline (custom Jenkins DSL, not used outside of Jenkins)
  • the declarative pipeline (same as the scripted one, but a bit nicer)

 

or to fall back to the clunky old manual configuration. Or the job DSL configuration, but that doesn't play well with Github/Bitbucket branch source plugins, for example.

 

Always keep in mind your users and their use cases

 

The distributed cron with history is actually Jenkins' main selling point. It can do what other less flexible tools such as Teamcity & co can't do. That's how I "sell" Jenkins to people, and that's why they "buy" it (not always with Cloudbees support contracts, true, but as they grow that can change ).

 

Anyway, sorry for the rant, but I really think this should be re-prioritized.

antti.turpeinen@cgi.com (JIRA)

unread,
Dec 28, 2018, 3:53:04 AM12/28/18
to jenkinsc...@googlegroups.com

Just noticed that JENKINS-50328 bug is related to this issue. If that bug would be fixed then we could create multiple multi-branch pipeline projects using same git repository but with different Jenkinsfile?

This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)

librannk@gmail.com (JIRA)

unread,
May 13, 2019, 7:19:24 AM5/13/19
to jenkinsc...@googlegroups.com

Hi

Any update on this bug ...

for mono-repositories in micro-services, this is very common scenario, 

where there are multiple jenkins files for multiple folder (microservice). 

github org webhook doesnt support this....and creating branch for each and every microservice is not feasible 

 

please fix this on priority if possible

admin@markeloff.net (JIRA)

unread,
May 29, 2019, 10:51:36 AM5/29/19
to jenkinsc...@googlegroups.com

Cool feature.

I really need the same.

 

admin@markeloff.net (JIRA)

unread,
May 29, 2019, 10:52:04 AM5/29/19
to jenkinsc...@googlegroups.com

alexeyjaga@gmail.com (JIRA)

unread,
Jun 27, 2019, 8:21:05 AM6/27/19
to jenkinsc...@googlegroups.com

This feature would allow to ease up configuration on many environments.

marcelo.carlos@simplybusiness.co.uk (JIRA)

unread,
Jul 1, 2019, 7:04:05 AM7/1/19
to jenkinsc...@googlegroups.com

Are there any plans to add/support this feature?

Our scenario is fairly similar to the others. We have a monorepo where we store different projects. To better illustrate our scenario and why this is important to us, here is one simplified use-case:

We have our repo (let's call it "monorepo"). In this repo we have several folders, such as:

  • toolOne (written in golang)
  • docker-images
    • base
    • imageTwo

Ideally, we'd have three pipelines in the example above, one for `toolOne`, one for `docker-images/base` and one for `docker-images/imageTwo`.

On top of that, an interesting additional use case is when we want to build `imageTwo`. That project contains a multi-stage Dockerfile which compiles part of the source code of `toolOne` and adds the compiled binary into the `imageTwo` image (that's one of the main reasons we choose the monorepo approach, this way we can easily and quickly handle cross-project/tools dependencies).

At the moment, we've been setting all of that up using a single Jenkinsfile at the root of the repo and handling builds with lots build parameters and conditionals, so we can run/skip stages according to the build we want to execute.

We looked into the "Job DSL Plugin", but we'd rather avoid since we've been successfully using Jenkinsfile in several of other projects and it is great to the same format for all projects and repos.

However, the scenario above is gradually becoming a blocker as the pipeline is growing too big due to all the conditionals and stages we had to define to work around the existing limitation of 1 Jenkinsfile per repo.

So, back to the original question, are there any plans you could share about adding/supporting this feature?

marcelo.carlos@simplybusiness.co.uk (JIRA)

unread,
Jul 1, 2019, 7:12:06 AM7/1/19
to jenkinsc...@googlegroups.com
Marcelo Carlos edited a comment on Improvement JENKINS-43749
Are there any plans to add/support this feature?

Our scenario is fairly similar to the others. We have a monorepo where we store different projects. To better illustrate our scenario and why this is important to us, here is one simplified use-case:

We have our repo (let's call it "monorepo"). In this repo we have several folders, such as:

* toolOne (written in golang)
* docker-images
** base
** imageTwo


Ideally, we'd have three pipelines in the example above, one for `toolOne`, one for `docker-images/base` and one for `docker-images/imageTwo`.

On top of that, an interesting additional use case is when we want to build `imageTwo`. That project contains a multi-stage Dockerfile which compiles part of the source code of `toolOne` and adds the compiled binary into the `imageTwo` image (that's one of the main reasons we choose the monorepo approach, this way we can easily and quickly handle cross-project/tools dependencies).

At the moment, we've been setting all of that up using a single Jenkinsfile at the root of the repo and handling builds with *lots* of build parameters and conditionals, so we can run/skip stages according to the build we want to execute.


We looked into the "Job DSL Plugin", but we'd rather avoid since we've been successfully using Jenkinsfile in several of other projects and it is great to the same format for all projects and repos.

However, the scenario above is gradually becoming a blocker as the pipeline is growing too big due to all the conditionals and stages we had to define to work around the existing limitation of 1 Jenkinsfile per repo.

So, back to the original question, are there any plans you could share about adding/supporting this feature?

fajran@kutu.nl (JIRA)

unread,
Jul 1, 2019, 7:33:06 AM7/1/19
to jenkinsc...@googlegroups.com

What I ended up doing to support our monorepo approach was the following

  • Create separate (multibranch pipeline) Jenkins job for every individual components in the monorepo
  • All Jenkins jobs will use the same repository URL but each has its own path to Jenkinsfile. This way each component can have its own distinct build pipeline
  • I disabled the build triggering function in Jenkins and instead create an additional service to handle build trigger
  • The build trigger service also manages the Jenkins jobs. Whenever it detects a new component, it will create the corresponding Jenkins job automatically, with the correct configuration

j3p0uk@hotmail.com (JIRA)

unread,
Jul 1, 2019, 7:36:05 AM7/1/19
to jenkinsc...@googlegroups.com

When creating multiple jobs off the same GitHub repository, I found we required this plugin to get sane PR checking: https://plugins.jenkins.io/github-scm-trait-notification-context

 

Default behaviour can play havoc with CI pipelines.

 

tzachyrm@gmail.com (JIRA)

unread,
Jul 3, 2019, 10:30:05 AM7/3/19
to jenkinsc...@googlegroups.com

Fajran Rusadi is your service creating jobs inside the multibranch folders? Can you please share some code for doing that?

fajran@kutu.nl (JIRA)

unread,
Jul 4, 2019, 10:24:07 AM7/4/19
to jenkinsc...@googlegroups.com

Tzach Yarimi to clarify, we had different multibranch pipeline job for different component. So what service does is create a new multibranch pipeline job when it detects a new component added to the repository. The new job will be configured to use the Jenkinsfile specific of that new component. 

If the job already exists, it will check if the branch is already registered otherwise it will update the job config to add a new branch and start the scanning process to make the branch job available. Once the branch job is available, the service will trigger the build of that.

Unfortunately I could not share the code.

ceztko@gmail.com (JIRA)

unread,
Dec 7, 2019, 7:56:05 PM12/7/19
to jenkinsc...@googlegroups.com

I'm relatively new to Jenkins but for me the reason this feature can't just be straightforwardly implemented in Jenkins is that it lack a concept of a first-citizen SCM resource, meaning that SCM repositories can be configured in jobs but they are not treated as shareable resources themselves. If repositories could be configured as shareable resources (with no build steps configurable in them) and job could be configured to source on these resources (instead of configuring a separate git/svn connection for each job) it would be easier to implement a single queue of jobs to trigger for SCM changes on a single repository and it would be easier to filter which pipeline to trigger in case Job1 is configured with Jenkinsfile1 and Job2 is configured with JenkinsFile2. The single queue and the filter can probably be implemented with scripting but to have UI support and more declarative approach in the upstream pipelines (as opposed to imperatively select which job to trigger in the downstream SCM projects) something like what I am suggesting could be needed.

This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f)
Atlassian logo

ceztko@gmail.com (JIRA)

unread,
Dec 7, 2019, 8:00:30 PM12/7/19
to jenkinsc...@googlegroups.com
Francesco Pretto edited a comment on Improvement JENKINS-43749
I'm relatively new to Jenkins but for me the reason this feature can't just be straightforwardly implemented in Jenkins is that it lack lacks a concept of a first-citizen SCM resource, meaning that SCM repositories can be configured in jobs but they are not treated as shareable resources themselves. If repositories could be configured as shareable resources (with no build steps configurable in them) and job could be configured to source on these resources (instead of configuring a separate git/svn connection for each job) it would be easier to implement a single queue of jobs to trigger for SCM changes on a single repository and it would be easier to filter which pipeline to trigger in case Job1 is configured with Jenkinsfile1 and Job2 is configured with JenkinsFile2. The single queue and the filter can probably be implemented with [scripting|https://stackoverflow.com/a/51237488/213871] but to have UI support and more declarative approach in the upstream pipelines (as opposed to imperatively select which job to trigger in the downstream SCM projects) something like what I am suggesting could be needed.

vaibhav.bnr87@gmail.com (JIRA)

unread,
Feb 10, 2020, 7:38:04 PM2/10/20
to jenkinsc...@googlegroups.com

Fajran Rusadi Is is possible to share the code removing your firms references? Seems this has been the issue with Jenkins since years and there is not a stable solution.

vaibhav.bnr87@gmail.com (JIRA)

unread,
Feb 10, 2020, 7:46:19 PM2/10/20
to jenkinsc...@googlegroups.com
victor paul edited a comment on Improvement JENKINS-43749
[~fajran] Is is it possible to share the code removing your firms references? Seems this has been the issue with Jenkins since years and there is not a stable solution.

ceztko@gmail.com (JIRA)

unread,
Feb 20, 2020, 5:04:05 AM2/20/20
to jenkinsc...@googlegroups.com

I don't know if this precisely answer the original issue here but I achieved to have multiple Jenkinsfile in the same repository/branch with the current feature set, I explained the procedure in this StackOverflow answer. The main idea is having separate Build and CSM projects configured in Jenkins, with the former that can be a simple Freestyle project with a polling on a repository and the latter a pipeline with a trigger on the CSM project. The trickier part was to add to add the skipDefaultCheckout directive in the Build pipeline, which is strange since I expected "Lightweight checkout" flag in the pipeline project to actually disable the checkout itself. Is this intended?

jbennett2091@yahoo.com (JIRA)

unread,
May 5, 2020, 2:34:06 AM5/5/20
to jenkinsc...@googlegroups.com

I cannot speak to all the various use-cases that people have brought up, but it seems like a fair-number of people looking for this would be satisfied if multi-branch pipelines just supported an 'exclude regions' flag.  You get that behavior with regular-pipeline, but not multi-branch pipeline.

In my situation (and seems like others have this as well), I have directoryA/Jenkinsfile and directoryB/Jenkinsfile, and I wish to trigger pipelineA when commits are made to DirectoryA, and pipelineB for DirectoryB.  There are a few solutions floating around that effectively have both pipelineA and pipelineB starting, doing a 'validate' step, and then pipeline logic to short-circuit, causing completing when the commit is on the other side.  But I'd really prefer that the build NEVER start in the first place for the wrong pipeline.  Commit to DirectoryA -> only PipelineA is triggered.   That amounts to PipelineA configuring "DirectoryB" as an excludes region (and vice versa).

That said, there are a bunch of use-cases all entwined herein, and this is only one of them.  But... it seems like a fairly well-defined one that might mitigate a lot of people's issues.

 

This message was sent by Atlassian Jira (v7.13.12#713012-sha1:6e07c38)
Atlassian logo

jbennett2091@yahoo.com (JIRA)

unread,
May 5, 2020, 2:36:05 AM5/5/20
to jenkinsc...@googlegroups.com
Jeffrey Bennett edited a comment on Improvement JENKINS-43749
I cannot speak to all the various use-cases that people have brought up, but it seems like a fair-number of people looking for this would be satisfied if multi-branch pipelines just supported an 'exclude regions' flag.  You get that behavior with regular-pipeline, but not multi-branch pipeline.

In my situation (and seems like others have this as well), I have directoryA/Jenkinsfile and directoryB/Jenkinsfile, and I wish to trigger pipelineA when commits are made to DirectoryA, and pipelineB for DirectoryB.  There are [a few solutions floating floading around|[https://stackoverflow.com/questions/49448029/multiple-jenkinsfile-in-one-repository/60316968#60316968]] that effectively have both pipelineA and pipelineB starting, doing a 'validate' step, and then pipeline logic to short-circuit, causing completing when the commit is on the other side.  But I'd really prefer that the build NEVER start in the first place for the wrong pipeline.  Commit to DirectoryA -> only PipelineA is triggered.   That amounts to PipelineA configuring "DirectoryB" as an excludes region (and vice versa).


That said, there are a bunch of use-cases all entwined herein, and this is only one of them.  But... it seems like a fairly well-defined one that might mitigate a lot of people's issues.

 

jbennett2091@yahoo.com (JIRA)

unread,
May 5, 2020, 2:37:05 AM5/5/20
to jenkinsc...@googlegroups.com
Jeffrey Bennett edited a comment on Improvement JENKINS-43749
I cannot speak to all the various use-cases that people have brought up, but it seems like a fair-number of people looking for this would be satisfied if multi-branch pipelines just supported an 'exclude regions' flag.  You get that behavior with regular-pipeline, but not multi-branch pipeline.

In my situation (and seems like others have this as well), I have directoryA/Jenkinsfile and directoryB/Jenkinsfile, and I wish to trigger pipelineA when commits are made to DirectoryA, and pipelineB for DirectoryB.  There are [ [ a few solutions floading floating around| |#60316968]] [https://stackoverflow.com/questions/49448029/multiple-jenkinsfile-in-one-repository/60316968 ] []| #60316968]] that effectively have both pipelineA and pipelineB starting, doing a 'validate' step, and then pipeline logic to short-circuit, causing completing when the commit is on the other side.  But I'd really prefer that the build NEVER start in the first place for the wrong pipeline.  Commit to DirectoryA -> only PipelineA is triggered.   That amounts to PipelineA configuring "DirectoryB" as an excludes region (and vice versa).


That said, there are a bunch of use-cases all entwined herein, and this is only one of them.  But... it seems like a fairly well-defined one that might mitigate a lot of people's issues.

 

jbennett2091@yahoo.com (JIRA)

unread,
May 5, 2020, 2:38:05 AM5/5/20
to jenkinsc...@googlegroups.com
Jeffrey Bennett edited a comment on Improvement JENKINS-43749
I cannot speak to all the various use-cases that people have brought up, but it seems like a fair-number of people looking for this would be satisfied if multi-branch pipelines just supported an 'exclude regions' flag.  You get that behavior with regular-pipeline, but not multi-branch pipeline.

In my situation (and seems like others have this as well), I have directoryA/Jenkinsfile and directoryB/Jenkinsfile, and I wish to trigger pipelineA when commits are made to DirectoryA, and pipelineB for DirectoryB.  There are [ [ a few solutions floating around| |#60316968]] [https://stackoverflow.com/questions/49448029/multiple-jenkinsfile-in-one-repository/60316968] [ ] |#60316968]]   that effectively have both pipelineA and pipelineB starting, doing a 'validate' step, and then pipeline logic to short-circuit, causing completing when the commit is on the other side.  But I'd really prefer that the build NEVER start in the first place for the wrong pipeline.  Commit to DirectoryA -> only PipelineA is triggered.   That amounts to PipelineA configuring "DirectoryB" as an excludes region (and vice versa).


That said, there are a bunch of use-cases all entwined herein, and this is only one of them.  But... it seems like a fairly well-defined one that might mitigate a lot of people's issues.

 
Reply all
Reply to author
Forward
0 new messages