[JIRA] (JENKINS-41940) Pipeline stages should be configurable with a weight property, corresponding to the number of executors the stage occupies

[vitali.gontsharuk@reiterate.de (JIRA)]

Vitali Gontsharuk updated an issue

Jenkins / JENKINS-41940 Pipeline stages should be configurable with a weight property, corresponding to the number of executors the stage occupies Change By: Vitali Gontsharuk Summary: Pipeline stages should be configurable with a weight property, corresponding to the number of executors the stage occupies Add Comment

This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c)

[andrew.bayer@gmail.com (JIRA)]

Andrew Bayer updated an issue

Jenkins / JENKINS-41940 Pipeline stages should be configurable with a weight property, corresponding to the number of executors the stage occupies

Change By: Andrew Bayer Component/s: pipeline-stage-step-plugin Component/s: pipeline

Add Comment

[jglick@cloudbees.com (JIRA)]

Jesse Glick assigned an issue to Unassigned

stage is only for display. I think you are talking about node. Could perhaps be supported somehow in this plugin, TBD. I think the approach is misconceived, though. Better to have a QueueTaskDispatcher which blocks off executor slots dynamically based on measured system load or something.

Jenkins / JENKINS-41940 Pipeline stages should be configurable with a weight property, corresponding to the number of executors the stage occupies

Change By: Jesse Glick Component/s: heavy-job-plugin Component/s: pipeline-stage-step-plugin Labels: executor pipeline stage weight Assignee: Jesse Glick

Add Comment

[vitali.gontsharuk@reiterate.de (JIRA)]

Vitali Gontsharuk commented on JENKINS-41940

Re: Pipeline stages should be configurable with a weight property, corresponding to the number of executors the stage occupies Well, I was using the abstractions from Declarative Pipeline here. You obviously know better how this is mapped to Scripted Pipeline primitives. Concerning the approach: I have basically proposed the same functionality as currently exists for Freestyle jobs to be supported for Pipeline jobs. This is not a very fancy approach, but it is enough to get the job done in practice. If the same goal could be reached automatically, without any configuration from the user, that would be just perfect. What do you think, would it be a lot of effort to implement?

Add Comment

[vitali.gontsharuk@reiterate.de (JIRA)]

Vitali Gontsharuk commented on JENKINS-41940

Re: Pipeline stages should be configurable with a weight property, corresponding to the number of executors the stage occupies

Jesse Glick So who do you think would be an appropriate assignee for this? Thanks!

Add Comment

[jglick@cloudbees.com (JIRA)]

Jesse Glick commented on JENKINS-41940

Re: Pipeline stages should be configurable with a weight property, corresponding to the number of executors the stage occupies

Apparently there is no default assignee for this plugin.

Add Comment

[Thorsten@meinl.bnv-bamberg.de (JIRA)]

Thorsten Meinl commented on JENKINS-41940

Re: Pipeline stages should be configurable with a weight property, corresponding to the number of executors the stage occupies

As a workaround I'm currently using the following function: {{ def multiSlotNode(String label, int slots = 1, Closure body) { if (slots == 1) { node(label) { body() } } else if (slots > 1) { node(label) { multiSlotNode(env.NODE_NAME, slots - 1, body) } } else { throw new IllegalArgumentException("Number of slots must be greather than zero") } } }} Use as {{ multiSlotNode('desired-node-label', 2) { doSomething } }}

Add Comment

[Thorsten@meinl.bnv-bamberg.de (JIRA)]

Thorsten Meinl edited a comment on JENKINS-41940

Re: Pipeline stages should be configurable with a weight property, corresponding to the number of executors the stage occupies

As a workaround I'm currently using the following function:

{ { quote}

def multiSlotNode(String label, int slots = 1, Closure body) { if (slots == 1) {   node(label) { body() } } else if (slots > 1) {   node(label) { multiSlotNode(env.NODE_NAME, slots - 1, body) } } else {   throw new IllegalArgumentException("Number of slots must be greather than zero") } }

{quote } } Use as {{ multiSlotNode _multiSlotNode ('desired-node-label', 2) { doSomething } }} _

Add Comment

[Thorsten@meinl.bnv-bamberg.de (JIRA)]

Thorsten Meinl edited a comment on JENKINS-41940

Re: Pipeline stages should be configurable with a weight property, corresponding to the number of executors the stage occupies

As a workaround I'm currently using the following function:

{code:java}

def multiSlotNode(String label, int slots = 1, Closure body) { if (slots == 1) {   node(label) { body() } } else if (slots > 1) {   node(label) { multiSlotNode(env.NODE_NAME, slots - 1, body) } } else {   throw new IllegalArgumentException("Number of slots must be greather than zero") }

}{code} Use as {code:java}

multiSlotNode('desired-node-label', 2) { doSomething }

{code:java}

Add Comment

[Thorsten@meinl.bnv-bamberg.de (JIRA)]

Thorsten Meinl edited a comment on JENKINS-41940

Re: Pipeline stages should be configurable with a weight property, corresponding to the number of executors the stage occupies

As a workaround I'm currently using the following function:

{ quote code:java }

def multiSlotNode(String label, int slots = 1, Closure body) { if (slots == 1) {   node(label) { body() } } else if (slots > 1) {   node(label) { multiSlotNode(env.NODE_NAME, slots - 1, body) } } else {   throw new IllegalArgumentException("Number of slots must be greather than zero") }

}{ quote code } Use as _multiSlotNode {code:java} multiSlotNode ('desired-node-label', 2) { doSomething } _ {code:java}

Add Comment

[vitali.gontsharuk@reiterate.de (JIRA)]

Vitali Gontsharuk commented on JENKINS-41940

Re: Pipeline stages should be configurable with a weight property, corresponding to the number of executors the stage occupies

This seems like a very elegant workaround. Thanks for sharing it!

Add Comment

[jglick@cloudbees.com (JIRA)]

Jesse Glick commented on JENKINS-41940

Re: Pipeline stages should be configurable with a weight property, corresponding to the number of executors the stage occupies

That is deadlock-prone.

Add Comment

[jayspang@amazon.com (JIRA)]

Jay Spang commented on JENKINS-41940

Re: Pipeline stages should be configurable with a weight property, corresponding to the number of executors the stage occupies

Just commenting that this is a feature we want as well. We used to use the Heavy Job Plugin, which did exactly what we need. We had to leave it behind when moving to pipelines. Our Jenkins instance has ~30 jobs, and almost all of them are relatively lightweight and can run in parallel with other jobs. However, 2-3 of them are large C++ compiles, which are very resource intensive and should NOT share the node with other jobs. In this situation, my choices are... 1) Give my nodes only 1 executor. This results in lots of wasted capacity, as I have to spin up LOTS of nodes to ensure I can meet peak demand. 2) Give my nodes multiple executors. This results in frequent "slow builds" as Jenkins occasionally decides to run 2 compiles on the same node. Being able to say "this job is heavy and cannot share a node" would mitigate these pain points.

Add Comment

This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e)

[jayspang@amazon.com (JIRA)]

Jay Spang edited a comment on JENKINS-41940

Re: Pipeline stages should be configurable with a weight property, corresponding to the number of executors the stage occupies

Just commenting that this is a feature we want as well. We used to use the [Heavy Job Plugin|https://plugins.jenkins.io/heavy-job], which did exactly what we need. We had to leave it behind when moving to pipelines. :(

Our Jenkins instance has ~30 jobs, and almost all of them are relatively lightweight and can run in parallel with other jobs. However, 2-3 of them are large C++ compiles, which are very resource intensive and should NOT share the node with other jobs.

In this situation, my choices are... 1) Give my nodes only 1 executor. This results in lots of wasted capacity, as I have to spin up LOTS of nodes to ensure I can meet peak demand.   2) Give my nodes multiple executors. This results in frequent "slow builds" as Jenkins occasionally decides to run 2 compiles on the same node. Being able to say "this job is heavy and cannot share a node" would mitigate these pain points.

Add Comment

[wcmclen@sandia.gov (JIRA)]

William McLendon commented on JENKINS-41940

Re: Pipeline stages should be configurable with a weight property, corresponding to the number of executors the stage occupies

Just adding my comment that something like this would be really useful for us on our end. We have some large templated codes that eat a lot of ram when building, and it's bad if we get 2 or 3 make -j 30's queued up on the same build slave so being able to set the job weight in our pipeline jobs would enable us to move more of our testing environment into a pipeline system. If there was a way to do something like:   node(label: "node_label", weight: 30) { do-my-30-core-build }   life would be pretty good.

Add Comment

This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)

[wcmclen@sandia.gov (JIRA)]

William McLendon commented on JENKINS-41940

Re: Pipeline stages should be configurable with a weight property, corresponding to the number of executors the stage occupies

Jean-Paul G I will try out your hack, I'm not sure what happens though if your 'node' is really just a label that might include several machines.  For example, we often have labels such as "rhel7" and may have many red hat 7 systems in the build farm under that label so Jenkins can just schedule a build where there are available nodes. So if I have three 20-core machines under that label and four jobs with a build weight of 15 cores, what I'd want is for the first 3 jobs to run and the fourth to wait until it can fit on a machine. Do you know if your script would do that or would I get 3 machines with 15 executors taken for the first three jobs and then the 4th job taking 5 executors from each machine? Still would be nice if the pipeline plugin had a weight parameter to the node command

Add Comment

This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f)

[rocco.zanni@spreaker.com (JIRA)]

Rocco Zanni commented on JENKINS-41940

Re: Pipeline stages should be configurable with a weight property, corresponding to the number of executors the stage occupies

Same issue here. We have a job that makes heavy use of the CPU cores, and being able to declare a specific number of executors required to run this pipeline will make our life way easier.

Add Comment

[goganchic@gmail.com (JIRA)]

Egor Baranov commented on JENKINS-41940

Re: Pipeline stages should be configurable with a weight property, corresponding to the number of executors the stage occupies

Hello! It would be very interesting for me to implement such plugin. I've looked through source code of Heavy Job Plugin and some examples of Pipeline API like lockable-resources-plugin. Unfortunately Pipeline API differs from Freestyle Projects. For now it's clear how to create custom step and wrap another step with it but I can't understand how to create something like subtasks in Heavy Job plugin. I would be very happy if someone can explain me this moment a bit more or give me a useful link.

Add Comment

This message was sent by Atlassian Jira (v7.13.12#713012-sha1:6e07c38)