Jenkinsfile-runner + Configuration-as-Code

[nicolas de loof]

Hello there,

I'm working on Jenkinsfile-runner investigating possible use-cases. A major point I've noticed with this promising tool is that most Pipeline scripts will require some initial master setup (credentials, variables, node labels, tools, etc) so are not strictly portable. On the other hand, having worked on the Configuration-as-Code projet, I know a simple way to setup a jenkins master in a predictable and reproducible state. I'd like to combine both.

As a result, one could store in SCM 

• a Jenkinsfile to define the pipeline execution steps
  
• a jenkins.yaml file to define the required master setup
• a plugin.txt file to define the set of plugins required to run this pipeline (more on this later)

I think this could be a nice driver for JCasC to embrace some extra use-cases, typically: how to smoothly integrate with a developer's workflow to run arbitrary pipeline locally. 

Part of this use-case is the ability to dry-run a pipeline, which is a larger topic than just jenkinsfile-runner, but the later could be part of this story. Typically, it could offer simple ways to switch environment / credentials so that pipeline runs with test resources, and not production ones. Typically, I would expect we can run something like this :

jenkinsfile-runner --config jenkins-dev.yaml 

Doing so would setup the headless jenkins master with development-only credentials, and environment variables to rely on test resources (a local web server as deployment target for sample), while the actual CI/CD master would run the same on production infrastructure.

I also have in mind another possible usage : one-shot jenkins masters - let's call this "serverless jenkins" for the buzzword.

To avoid huge jenkins masters running thousand pipelines, we could rely on jenkinsfile-runner for pipeline execution, a front-end master being used for job configuration and web UI only. As a git event comes to jenkins infrastructure, a new serverless payload would be triggered (think : docker container running somewhere) to execute jenkinsfile-runner with adequate jenkins.yaml, so pipeline can run the same as if the front-end master had ran it by itself.

This jenkins.yaml could be generated directly from master (this isn't yet implemented in JCasC, but definitively in the roadmap) with the benefit it could be scoped to target job : this jenkins.yaml would only include credentials and other resources your job legitimately have access to, then when injected in transient, headless, jenkinsfile-runner's master, you gain physical security and can relax Pipeline's sandboxing.

On pipeline completion, jenkinsfile-runner would then publish the build status on front-end master (something comparable to buildpublisher-plugin).

About plugins management, we want this to be fully part of JCasC, and I have a pending PR on this topic. From JCasC this is a chicken-egg  issue, ad JCasC itself is a jenkins plugin. From jenkinsfile-runner this is way simpler to implement : as the setup take place before the headless master init sequence starts, we can download and install various plugins. 

As an initial state I've introduced support for plugins.txt file (the same one supported by jenkins official docker image). Storing such a plugins list in SCM will let jenkinsfile-runner install the adequate plugins set on headless master to run the pipeline. Major drawback is this only supports standard update center as plugin source. A tighter integration with JCasC could be used to parse the plugins section from yaml configuration and get plugins from any update sites with  adequate proxy settings.

So, many challenges to address and potential extension for those tools. For me to get into the right direction, I would welcome any feedback, either on this thread or on github issues.

[Jesse Glick]

On Fri, Apr 27, 2018 at 3:52 AM, nicolas de loof
<nicolas...@gmail.com> wrote:
> jenkinsfile-runner --config jenkins-dev.yaml

This sounds useful.

> I also have in mind another possible usage : one-shot jenkins masters -
> let's call this "serverless jenkins" for the buzzword.
>
> To avoid huge jenkins masters running thousand pipelines, we could rely on
> jenkinsfile-runner for pipeline execution, a front-end master being used for
> job configuration and web UI only. As a git event comes to jenkins
> infrastructure, a new serverless payload would be triggered (think : docker
> container running somewhere) to execute jenkinsfile-runner with adequate
> jenkins.yaml, so pipeline can run the same as if the front-end master had
> ran it by itself.

> […]

> On pipeline completion, jenkinsfile-runner would then publish the build
> status on front-end master (something comparable to buildpublisher-plugin).

This makes little sense to me. You still have the huge Jenkins master
rendering a web UI for thousands of pipelines, and you still have all
the drawbacks of a build expressed in a weird DSL, plus you have the
serious functional limitations of tools like Build Publisher, plus you
have to set up a bunch of infrastructure to make the “one-shot” master
be able to provision or link to build agents—unless it requires only
one node and no real interaction with anything else in the system, in
which case your Pipeline would better have been expressed as

node {
sh '. /thisjob/run.sh'
}

with the script and credentials coming from Kubernetes and there is no
need for any special runner.

As to avoiding Groovy sandboxing, this is best done in a different
way: by not using `workflow-cps`, and rather running the script in a
separate plain-Groovy process (yes, “serverless”) connecting to the
master for step implementations. Simpler, and far easier to migrate to
from existing configurations.

[nicolas de loof]

Le ven. 27 avr. 2018 à 15:46, Jesse Glick <jgl...@cloudbees.com> a écrit :

On Fri, Apr 27, 2018 at 3:52 AM, nicolas de loof
<nicolas...@gmail.com> wrote:
> jenkinsfile-runner --config jenkins-dev.yaml

This sounds useful.

> I also have in mind another possible usage : one-shot jenkins masters -
> let's call this "serverless jenkins" for the buzzword.
>
> To avoid huge jenkins masters running thousand pipelines, we could rely on
> jenkinsfile-runner for pipeline execution, a front-end master being used for
> job configuration and web UI only. As a git event comes to jenkins
> infrastructure, a new serverless payload would be triggered (think : docker
> container running somewhere) to execute jenkinsfile-runner with adequate
> jenkins.yaml, so pipeline can run the same as if the front-end master had
> ran it by itself.
> […]
> On pipeline completion, jenkinsfile-runner would then publish the build
> status on front-end master (something comparable to buildpublisher-plugin).

This makes little sense to me. You still have the huge Jenkins master
rendering a web UI for thousands of pipelines,

Right. Let's now consider this fronted is _not_ Jenkins but some UI which can expose build status ? Or maybe even no UI but just a GitHub commit status update ?

and you still have all
the drawbacks of a build expressed in a weird DSL,

Not my fault. Use of groovy for build-flow was a mistake. We should have learned from this initial "draft".

plus you have the
serious functional limitations of tools like Build Publisher,

Which in future I expect not to be required as build status, artifacts, logs... would be stored on some cloud-native storage and directly accessible without Jenkins in the middle to serve them.

plus you
have to set up a bunch of infrastructure to make the “one-shot” master
be able to provision or link to build agents

Right. Let's say we only support docker agents, and a descent plugin able to create such nodes (an idea I'd like to investigate is to have labels == docker image)

—unless it requires only
one node and no real interaction with anything else in the system, in
which case your Pipeline would better have been expressed as

node {
  sh '. /thisjob/run.sh'
}

Isn't this the sole reasonable Jenkinsfile one should write ? Kidding appart, 'stages' still make some sense for those interested in visualization or to spilt a huge build log into more focused sections. For anything else I agree a shell script is the best tool we have. (I wonder we named this "Jenkinsfile" and not "makefile" ;-) )

with the script and credentials coming from Kubernetes and there is no
need for any special runner.

As to avoiding Groovy sandboxing, this is best done in a different
way: by not using `workflow-cps`, and rather running the script in a
separate plain-Groovy process (yes, “serverless”) connecting to the
master for step implementations. Simpler, and far easier to migrate to
from existing configurations.

This sounds what I expected Jenkinsfile-runner to be, but probably not such a trivial think to implement and alternative workflow executor to replace CPS with a plain groovy runtime, is it ?

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CANfRfr37wA%3DQUC5GTSBy4AgSFocM%3DD7NFFjsYyvYqgF9WNB71w%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[Ewelina Wilkosz]

I am pretty sure you'll make many people happy with it - are you planning to write/is there already a JEP for that?

since you've mentioned JCasC couple of times I believe it would make sense to how exactly those two could work together (I'm assuming you have some ideas in mind, but we can try to have it somewhere else also :) )

[Calvin]

Great idea!

Part of this use-case is the ability to dry-run a pipeline, which is a larger topic than just jenkinsfile-runner, but the later could be part of this story. Typically, it could offer simple ways to switch environment / credentials so that pipeline runs with test resources, and not production ones. Typically, I would expect we can run something like this : 

 

jenkinsfile-runner --config jenkins-dev.yaml  

Doing so would setup the headless jenkins master with development-only credentials, and environment variables to rely on test resources (a local web server as deployment target for sample), while the actual CI/CD master would run the same on production infrastructure.

I do this today by running Jenkins master container without mounting the jenkins_home directory. This of course has all the drawbacks that Jesse listed. An officially thought out and supported method would be most welcome.

Right. Let's now consider this fronted is _not_ Jenkins but some UI which can expose build status ? Or maybe even no UI but just a GitHub commit status update ?

This is very interesting. How about a middle ground of splitting Jenkins master into multiple smaller (yet connected) Jenkins masters using namespaces?

One of the pros of a monolithic Jenkins instance is the ease of inter-project operability (think Parameterized Trigger Plugin). A major con is of course the heavy UI. It would be awesome to split projects with namespaces and have master render only a specific namespace in UI, all the while keeping the logical access to the rest of the Jenkins projects intact. Views achieves this for the most part but an accidental click to the Jenkins logo results in an unnecessary load of all the Jenkins projects. Supporting a partial render as a core Jenkins master concept will help us scale the monolithic Jenkins instance across larger orgs.

We can extend the namespace concept even further and consider a partial plugin availability. Why expose all the plugin options to all projects when each namespace has different minimal requirements? Why enforce the same plugin configurations across all the namespaces? I can imagine a system similar to Windows Group Policy where the corporate IT sets guidelines and namespace owners override within the IT's boundary. This will help reduce the # of rogue Jenkins masters that are configured insecurely.

I can't be the first person to think of this. Does anyone know an effort towards this direction that I can participate in?

[nicolas de loof]

I indeed think a JEP would make sense for jenkinsfile-runner, to describe how it interacts with other projects.

This is the next step imho.

This thread is more about brainstorming about some use-cases and check for interest about this.

To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-dev+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/5aedacdb-0c9d-4d3b-9085-a605abae7a75%40googlegroups.com.

[Jesse Glick]

On Sat, Apr 28, 2018 at 2:46 AM, nicolas de loof
<nicolas...@gmail.com> wrote:
>> you still have all
>> the drawbacks of a build expressed in a weird DSL,
>
> Not my fault.

Obviously not; half mine. :-) My point is that the approach you are
describing does not do anything to help resolve that. Whereas we *do*
have some ideas for running the script out of process, which solves
the DSL/sandbox issues, and should not require drastic changes to the
overall user experience.

>> you have the
>> serious functional limitations of tools like Build Publisher
>

> Which in future I expect not to be required as build status, artifacts,
> logs... would be stored on some cloud-native storage and directly accessible
> without Jenkins in the middle to serve them.

Definitely we plan for that. This is not what I was talking about.
There is a gigantic long tail of Jenkins features (mostly plugins)
which would not work in a Build Publisher-like environment; some of
that could be fixed with a lot of labor, some probably not at all
short of a total redesign.

> 'stages' still make some sense for those interested in visualization
> or to spilt a huge build log into more focused sections.

Yes; independently of all this discussion, I would love to see Blue
Ocean support something like the Collapsing Console Sections plugin,
so you would not be forced to “escape” from your native build process
into Pipeline script merely to demarcate (at least linear) stage
boundaries. For example,

https://github.com/jenkinsci/docker-workflow-plugin/pull/105

could have collapsed four scripts into one, making the Pipeline script
considerably shorter (and the master load correspondingly a bit
lighter).

> probably not such
> a trivial think to implement and alternative workflow executor to replace
> CPS with a plain groovy runtime, is it ?

Not trivial at all, but I believe feasible.

[nicolas de loof]

2018-05-01 14:59 GMT+02:00 Jesse Glick <jgl...@cloudbees.com>:

On Sat, Apr 28, 2018 at 2:46 AM, nicolas de loof
<nicolas...@gmail.com> wrote:
>> you still have all
>> the drawbacks of a build expressed in a weird DSL,
>
> Not my fault.

Obviously not; half mine. :-) My point is that the approach you are
describing does not do anything to help resolve that. Whereas we *do*
have some ideas for running the script out of process, which solves
the DSL/sandbox issues, and should not require drastic changes to the
overall user experience.

>> you have the
>> serious functional limitations of tools like Build Publisher
>
> Which in future I expect not to be required as build status, artifacts,
> logs... would be stored on some cloud-native storage and directly accessible
> without Jenkins in the middle to serve them.

Definitely we plan for that. This is not what I was talking about.
There is a gigantic long tail of Jenkins features (mostly plugins)
which would not work in a Build Publisher-like environment; some of
that could be fixed with a lot of labor, some probably not at all
short of a total redesign.

Not sure what you have in mind (if you have a concrete sample this could help). 

I can't imagine a plugin that couldn't work with a rsync-copy of a remote build dir. 

But I might miss something.

Anyway what I'd like to achieve is to fully remove Jenkins Web UI and only rely on Github commit status

for user interaction (read more on http://bit.ly/blind-jenkins). So plugins would only make sense 

during the build, and any UI contributing plugin would just be excluded.

 

> 'stages' still make some sense for those interested in visualization
> or to spilt a huge build log into more focused sections.

Yes; independently of all this discussion, I would love to see Blue
Ocean support something like the Collapsing Console Sections plugin,
so you would not be forced to “escape” from your native build process
into Pipeline script merely to demarcate (at least linear) stage
boundaries. For example,

https://github.com/jenkinsci/docker-workflow-plugin/pull/105

could have collapsed four scripts into one, making the Pipeline script
considerably shorter (and the master load correspondingly a bit
lighter).

+1

I'd like pipeline stages to be controlled from shell scripts, something like :

echo "entering stage foo" > /var/run/jenkins.sock

so I could rename my pipeline scripts as "Jenkinsfile.sh"

> probably not such
> a trivial think to implement and alternative workflow executor to replace
> CPS with a plain groovy runtime, is it ?

Not trivial at all, but I believe feasible.

ok, then when this becomes available we would be able to make jenkinsfile-runner way simpler :D

 

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-dev+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CANfRfr1eqa2a-9dzBkCwGibVgqyPHsb%2BGwxY0rJ3BmZU4eLCmw%40mail.gmail.com.

[Jesse Glick]

On Tue, May 1, 2018 at 10:42 AM, nicolas de loof
<nicolas...@gmail.com> wrote:
>> There is a gigantic long tail of Jenkins features (mostly plugins)
>> which would not work in a Build Publisher-like environment
>

> Not sure what you have in mind (if you have a concrete sample this could
> help).

The first thing that pops into my head is Throttle Concurrent Builds,
but I think if you start going through randomly selected plugins you
will find dozens if not hundreds of examples.

> what I'd like to achieve is to fully remove Jenkins Web UI and only
> rely on Github commit status

> for user interaction […]. So plugins

> would only make sense
> during the build, and any UI contributing plugin would just be excluded.

OK, this is definitely an interesting project, but I would consider it
a novel CI system with a modest degree of Jenkins compatibility.

> echo "entering stage foo" > /var/run/jenkins.sock

Or I was just thinking some kind of magic string to stdout/stderr. I
have seen some scripts using ANSI control characters, or the
(freestyle-compatible) plugin I mentioned before lets you configure
any regexp.

[Oleg Nenashev]

Hi all,

During last weeks I have made some experiments w.r.t integrating Jenkinsfile Runner and Configuration as Code.

As a PoC, I have updated Custom War Packager and added support of Jenkinsfile Runner Docker image bundling there.

You can find a demo here: https://github.com/jenkinsci/custom-war-packager/tree/master/demo/jenkinsfile-runner

Generally this PoC allows producing a Docker image, which includes Jenkinsfile Runner, arbitrary Jenkins plugins, and also self-configuration logic powered by JCasC or, if needed, System Groovy scripts. Everything i s bundled statically hence there is no performance overhead for downloading plugins when the Docker image starts. Then you can run this image in Docker or Kubernetes by a single command

So far it uses an old version of Jenkinsfile Runner, but my plan is replace it by the stock version once it is finalized. For now it is just a proof of concept, but it may be interesting for some cases.

Best regards,

Oleg

[James Rawlings]

We’ve been trying this out in Jenkins X and love it, great stuff!

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/49cc7f1c-ca41-4bd7-b0bf-639d1a1c803c%40googlegroups.com.

[Oleg Nenashev]

We had a conversation with Nicolas yesterday, and we agreed to discuss the the JCasC + Custom War Packager + Jenkinsfile Runner combination at the next JCasC meeting. Meeting date: Sep 12, 7AM UTC, Youtube link: https://www.youtube.com/watch?v=sxJxez8jsIw . We will also post the participant link in https://gitter.im/jenkinsci/configuration-as-code-plugin before the meeting starts.

BR, Oleg

P.S: Also, join our "Cloud Native Jenkins" talk at DevOps World | Jenkins World, we will present an demo based on these tools.

[Florian Wilhelm]

On Tuesday, May 1, 2018 at 4:42:30 PM UTC+2, nicolas de loof wrote:

2018-05-01 14:59 GMT+02:00 Jesse Glick <jgl...@cloudbees.com>:

On Sat, Apr 28, 2018 at 2:46 AM, nicolas de loof
<nicola...@gmail.com> wrote:

> probably not such
> a trivial think to implement and alternative workflow executor to replace
> CPS with a plain groovy runtime, is it ?

Not trivial at all, but I believe feasible.

ok, then when this becomes available we would be able to make jenkinsfile-runner way simpler :D

 

Im curious if anyone is working on that. Is there an issue for that in Jira or something? I suspect this would make JFR  run much faster as CPS transformation has quite an impact on speed, right?

 

[Oleg Nenashev]

AFAIK no. From my PoV it is not the biggest problem for JCasC performance.

Currently it is rather a startup time. I am working on improving it in the background, and there have been also serious improvements last winter, thanks to Evaristo Gutiérrez and Francisco Fernández.

There are still a lot of tweaks we could apply.