Run pipeline flyweight executors off of master

[Eli White]

We follow the Jenkins configuration best practices and have no executors on our master node and force everything to run on our agents. 

We are starting to work with pipeline jobs and are worried that bad Jenkinsfiles could cause problems on our master. 

How can we force the flyweight jobs to run on a designated machine *other* than master? 

How do other people handle this?

[Craig Rodrigues]

In your pipeline, the node parameter can take an argument of which node to run on.

See this example:
https://github.com/freebsd/freebsd-ci/blob/master/scripts/build/build-test.groovy#L100

In my job, I defined BUILD_NODE with the NodeLabel Parameter Plugin ( https://wiki.jenkins-ci.org/display/JENKINS/NodeLabel+Parameter+Plugin ).

--

Craig

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/d4ee67f4-ab75-4a1e-9883-69453bdd656b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Eli White]

My understanding is that Jenkinsfile execution runs as a flyweight node on master, but then uses heavyweight nodes on the given node label to execute.

Per this file: https://github.com/jenkinsci/pipeline-plugin/blob/master/TUTORIAL.md

Why are there two executors consumed by one Pipeline build?

• Every Pipeline build itself runs on the master, using a flyweight executor — an uncounted slot that is assumed to not take any significant computational power.
• This executor represents the actual Groovy script, which is almost always idle, waiting for a step to complete.
• Flyweight executors are always available.

[Thomas Zoratto]

Hello, 

I’m not sure one can do what you want.

Out of curiosity, what’s the use case for this ?

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/a41c28a9-fc93-459f-b990-87cb1210cb64%40googlegroups.com.

[Eli White]

By not running anything on master we don't have to worry about any sort of failure related to misconfiguration of jobs, or job related failures. For example, OOM, out of disk space, pausing when 'input' is run, etc. 

If a slave goes down you have 1 machine down. If master goes down, all CI for our company goes down. The need for us to protect and make behavior guarantees of the master node, we want it to do only what is absolutely necessary and keep user code from running on it. 

[Sverre Moe]

I don't think you are supposed to do any real work in a flyweight executor.

Steps need to be within a node{} which will allocate a heavyweight executor. I use the flyweight executors only to trigger downstream builds.

[Eli White]

Agreed, you are not supposed to do any real work in a flyweight node. 

However, in an organization where teams' repos contain Jenkinsfiles that they wrote, we can't guarantee the safety and correctness of those files and their following of best practices. 

We want to protect the master node against whatever user error we feasibly can. Messing up jenkins files seems like a really easy thing to do. 

It sounds like this ability might not exist. I'll open a JIRA for it.

[Sverre Moe]

Jenkins already does some of that for you by running the pipeline script in a sandbox. Certain methods/functions in the Jenkinsfile must be approved by a Jenkins administrator.

[Kevin Phillips]

In my opinion, even more problematic than possibly running unsafe code in a flyweight executor on the master is the fact that these flyweight pipeline threads perform a full checkout of the target repository - at least when using Git as the SCM. So if one has a relatively large code repository containing product code - say several gigabytes - and they want to store their Jenkinsfile with the code so they can leverage the MultiBranch options in Jenkins Pipeline, they end up with many copies of their entire codebase being checked out on the master, possibly causing the master to run out of disk space. Add to this the fact that these "temporary" workspaces created on the master by the flyweight tasks are not automatically deleted when not in use and there is no way to purge them from the Jenkins dashboard and you are just asking for problems. This isn't a case of whether this will become a problem as much as it is an issue of when it will become a problem.

In my opinion, Jenkinsfile's stored in SCM should be checked out on an agent as a typical freestyle job would do, and a default "node" allocated by the executor that launches the script contained therein for the node on which it is already running on. Then the rest of the Jenkinsfile could be processed in place without requiring any sort of flyweight executor at all, and avoiding the issues with having multiple checkouts of the same code spread across a build farm. Also, I think this would be inline with how some other more modern continuous integration servers work like travis-ci, so it's not without precidence.

[Spencer Malone]

We just encountered this very thing. Large repo with lots of PRs, threw in a very simple Jenkinsfile and there was an organization folder set up. It cloned a copy of the repo for each PR, since they all got triggered together, and very quickly ran out of space. We had kept the disk space relatively small on the master, since we didn't let it have any executors, but now we know. Has anyone figured out a long term fix for this?

[Joshua Noble]

i would have to agree, this is definitely an issue. I'm currently using Amazon EFS as the storage backend for my Jenkins master, who doesn't have any executors. The flyweight executors checking out repositories is definitely a concern of mine. EFS will automatically scale so storage isn't a huge issue for me, but duplication and clone times are. (for larger repos)

On Wednesday, August 10, 2016 at 2:31:45 PM UTC-4, Kevin Phillips wrote:

[cim...@gmail.com]

I have a similar issue. Among running pipelines we have also some matrix jobs. And allowing pipelines to even trigger requires allowing matrix parents on master. Matrix parents happen to clone SCM first and then they run sub-jobs. This can easily end up consuming all space on jenkins master. In my oppinion it is a bad approach forcing pipelines flyweight executor run on master.

I am not sure if this is bad approach of jenkins pipelines or matrix project plugin.

[Andreas Magnusson]

In some instance this checkout can be helped/removed by doing a lightweight checkout, but it don't work on our builds since we have ${TAGNAME} parameters on the checkouts. And parameters in the SCM URL are not compatible with lightweight checkout.