Disposable Jenkins & GitOps do not make a good match

[domi]

Hi,

Since we now have JCasC and the job-dsl plugin it is easier then ever to treat jenkins as a disposable resource and just recreate it from scratch whenever you feel the need for it.
We do have build everything around this idea and reinstall our whole build infrastructure every time we need to update jenkins or even just a plugin (we create a docker image with all required plugins preinstalled).

But there are two issues which hurt us quite a bit:

1. Pipeline jobs do not listen on any commit webhooks before the job was not executed at least once (in our case Bitbucket, but the same is true for Github)
2. A multi branch pipeline (organizationFolder, *-branch-source-plugin) job will trigger a build for every single branch just discovered. This is quite a problem in case you have build your processes around git (some call it GitOps). e.g. if every time your master branch gets build something special happens: build a new release version of a library or even install something to some environment.

These two issues together cause us quite some problem, because every time we setup a new environment the following happens:

The branch source plugin (in our case Bitbucket) scans the whole organisation for branches, will trigger builds and therefore: new releases of libraries and artefacts get build and some stuff gets installed because the builds for many of our ‘master’ branches are triggered too.
Then I somehow have to trigger a build for all other pipelines (not managed by the branch-source-plugin) which should listen on any git webhooks, because otherwise they will never be triggered


Does anyone have the same experience? How do you work around them?

I have some ideas on how to solve/ workaround it:
- problem 1: write a plugin which is able to abort the build in case it is the first run and the job is configured to listen on webhooks (e.g. Bitbucket or GitHub), then after installing the fresh environment, just loop through all jobs (with groovy) and trigger all which have a hook configured and let the first run fail on purpose.
- problem 2: extends https://github.com/jenkinsci/basic-branch-build-strategies-plugin to some how skip some branches at the initial organisation scanning, I already created an issue for this: https://issues.jenkins-ci.org/browse/JENKINS-54861

But to be very honest, most of this sound some how hackisch and I hope someone else has a better idea on how to do this stuff properly.

Many thanks
Domi

[James Strachan]

we solved this in Jenkins X through the use of Prow for handling webhooks & orchestrating the serverless jenkins pod and then using kubernetes custom resources to maintain state (pipeline activities, environments, teams, releases etc)

https://jenkins-x.io/news/serverless-jenkins/

we then use an immutable docker image of jenkins that starts up (thanks to the custom war packager) and processes each pipeline execution in a separate container.

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/FA18D443-D4CA-4512-BAD1-255D6E3385C8%40fortysix.ch.
For more options, visit https://groups.google.com/d/optout.

--

James
-------
Twitter: @jstrachan

Blog: https://medium.com/@jstrachan/

Jenkins X: a CI/CD solution for modern cloud applications on Kubernetes

http://jenkins-x.io/

[Jesse Glick]

The simplest approach, starting from what you are doing now, would be
to keep the build records and other state files from your Jenkins
installation, recreating only the configuration files from Git.

[domi]

Thanks James, thats an interesting path, but it does look like we would have to build quite a lot around this to make it a good replacement of our current infrastructure. We also did have a look at jenkins-x, but to be honest, it is just way to much for us: first of all we don’t use k8s (we use cloud foundry) and I would say that running jenkins-x cost us more then our production environment, because of the number of pods (yes, we don’t have a micro service architecture) 

/Domi

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CALvxejm05axySHoqVEGrvX%3DrCPj4a4EfK3Q39oqDLrMwjrT-hQ%40mail.gmail.com.

[domi]

Thanks Jesse, it probably is, but even though - do you see a chance to fix the issues I mentioned?
e.g. why do I need to run a pipeline once before it accepts webhooks from GitHub or Bitbucket? I do understand that I can configure additional SCM sources from within a pipeline, but why does it not listen on the one I have defined on the I have defined to load the "Pipeline from SCM”?
/Domi

> --
> You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CANfRfr1W_8-%3DU8GJSuw_oazA0PSiXGyYDhv8mSKzLV0bScStFg%40mail.gmail.com.

[Tomas Bjerre]

Regarding configuration of webhooks with pipelines. This is why I always configure that with Job DSL. I configure the Jenkins job and the Git service (usually some rest-call) in the same Groovy (Job DSL) script.

[domi]

Hi Tomas,

I do this too, but this does not mean I don’t have to start the build at least once to make the webhooks work.

/Domi

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/d21bf102-f345-4c6b-a234-4ab8ad3c6995%40googlegroups.com.

[Tomas Bjerre]

But I don't configure the trigger part with a pipeline script.

I use Job DSL to create a pipeline job. In that pipeline job I configure the triggering-part with Job DSL and only keep the build process in the pipeline script.

Then I don't have to run the pipeline script once before it all works.

Like this:

pipelineJob("my pipeline job") {
 triggers {
  .... // Job DSL with triggering configuration
 }
 
 definition {
  cps {
   script(...) //Pipeline with build process
   sandbox()
  }
 }
}

You received this message because you are subscribed to a topic in the Google Groups "Jenkins Developers" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-dev/1gc8t6MAl4g/unsubscribe.
To unsubscribe from this group and all its topics, send an email to jenkinsci-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/A03F3239-D428-4EC3-8167-E7C3AD02CE7C%40fortysix.ch.

[Tony Noble]

This is something that can be annoying at times.

Would it not make sense to for Jenkins to at least attempt to parse the pipeline script when the job is created / amended?  Even if it ignores 'stages' and just looks at 'parameters' and 'triggers', that would at least get round not only this issue but also the need to write pipelines in such a way that they don't cause mayhem when running for the first time with no parameter definitions (or with parameter definitions that are no longer correct).

Tony

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CAK89W5K9YFGn%2BC4rELF12u68QOY%2Bye978a1SJCrQwJ%2Bs8h6N2g%40mail.gmail.com.

[domi]

Yes, I do the exact same thing, there is not trigger config in the pipeline script - but for me this does not work the way you describe it - the job at least has to run once. But you seem to use the cps block instead of the cpsScm block for you job-dsl configuration, not sure abut this...

pipelineJob("${folderName}/install") {

triggers {

bitbucketPush()

}

definition {

cpsScm {

scm {

git {

remote {

url('https://bitbucket.org/xxxxx')

credentials('BITBUCKET_CRED')

}

branches('develop')

scriptPath(’install.groovy')

lightweight(true)

extensions {

localBranch()

cleanCheckout()

}

}

}

}

}

}

/Domi

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CAK89W5K9YFGn%2BC4rELF12u68QOY%2Bye978a1SJCrQwJ%2Bs8h6N2g%40mail.gmail.com.

[Jesse Glick]

On Mon, Dec 3, 2018 at 7:49 AM Tony Noble <tony....@gmail.com> wrote:
> Would it not make sense to for Jenkins to at least attempt to parse the pipeline script when the job is created / amended? Even if it ignores 'stages' and just looks at 'parameters' and 'triggers', that would at least get round not only this issue but also the need to write pipelines in such a way that they don't cause mayhem when running for the first time with no parameter definitions (or with parameter definitions that are no longer correct).

There is discussion of this general issue in JENKINS-41929. Briefly,
for Declarative Pipelines (but not Scripted), we could in principle
parse the definition as soon as the job is created and set the
`List<JobProperty>` accordingly, including parameter definitions, and
probably also preconfigure SCM polling without relying on
`WorkflowRun.checkouts` from an initial build.

Note that for the special case of parameter definitions, the `params`
global variable is defined so that `defaultValue`s of parameters
defined in the job will be honored for the parameterless initial
build, which is a partial workaround.

As to Domi’s suggestion of having polling (on the “main repo”) be set
up automatically for the case of a job using `CpsScmFlowDefinition`,
yes this could work reliably for any Pipeline syntax I think, since
the SCM information is statically available. Feel free to file an RFE
in `workflow-cps-plugin` for it.

Anyway, the broad issue remains that Jenkins maintains a variety of
pieces of state—distinct from configuration—in `$JENKINS_HOME`, and so
if you blow that away, some functionality will be impaired. For the
particular case of SCM state as described here, you can work around
this by defining non-multibranch projects with a different build
trigger structure using the Job DSL plugin or similar job definition
tools, which amounts to much the same thing as Jenkins-X is doing in
`--prow` mode at least in this respect.

[domi]

Thanks for the explanation Jesse! I created an issue for the workflow-cps-plugin at https://issues.jenkins-ci.org/browse/JENKINS-54996
/Domi

> --
> You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CANfRfr33oP9JzwbVdvyuEPfs_EKwkUTGoF8kTvTL__9TQJCezA%40mail.gmail.com.

[James Strachan]

On Mon, Dec 3, 2018 at 8:11 AM domi <do...@fortysix.ch> wrote:

Thanks James, thats an interesting path, but it does look like we would have to build quite a lot around this to make it a good replacement of our current infrastructure. We also did have a look at jenkins-x, but to be honest, it is just way to much for us: first of all we don’t use k8s (we use cloud foundry) and I would say that running jenkins-x cost us more then our production environment, because of the number of pods (yes, we don’t have a micro service architecture) 

BTW CloudFoundry is pretty expensive compared to k8s ;). 

You should be able to host Jenkins X for a few hundred bucks a month with plenty of room for builds + preview environments. The footprint of Jenkins X is pretty small - other than a Nexus, we only run a few tiny pods when you're not running a build and we can turn many of those truly into serverless mode - so that you can use per second billing on some of the better cloud providers.

 

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/9B0B05A3-E884-419F-B85E-471F22B66AC6%40fortysix.ch.

For more options, visit https://groups.google.com/d/optout.