How to prevent concurrent execution of jobs on the same node, but allow a few parallel steps execution?

[Vitaly Karasik]

I have multibranch job, whole pipeline runs on certain node. 

Currently I have a slaves with single executor, so multiple jobs cannot run on the same node. I'd like to use parallel steps inside my pipeline, so I need to increase number of executors. 

What is the best way to prevent concurrent job execution? 

Such configuration was trivial with Jenkins freestyle job and Throttling plugin - I simple used "Maximum Concurrent Builds Per Node". But in pipeline it seems that I cannot use "throttle" for whole pipeline, I need to use it for each step. Is there more elegant solution?

TIA, Vitaly

[Den Slobozhanskyi]

Just to clarify - does it mean that Throttling plugin throttles not the job as a whole, but each step in parallel? My assumption was that throttling is only for the job, and nested threads have nothing to do with throttle. By your words it looks like setting throttling to "1" for the job reduce the possible parallel threads execution to a single thread at a time as well, is that true?

If this is not true, you just gather all the jobs under one throttling category and set the concurrent builds limit to 1.

BR,

Denys

среда, 30 января 2019 г., 14:17:15 UTC+1 пользователь Vitaly Karasik написал:

[Vitaly Karasik]

I'd like to mark *whole* pipeline job as throttling resource; and unlike freestyle jobs, Throttleling plugin does't know to do that. I.e. I cannot use "throttle" in pipeline options.

[Daniel Butler]

You could use the lockable resources plugin (v2.2+):

Set up a single resource for each node and give them all a common label

Then your Jenkinsfile would be:

lock(label: "common-label", variable: "LOCKED_NODE"){
    node(env.LOCKED_NODE){
        //build project
    }
}

Regards,

Daniel.

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/d6b76faf-f9e6-4960-8dc5-ab79c6c6ac0f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Vitaly Karasik]

>Set up a single resource for each node and give them all a common label

Thank you - I   thought about using lockable resources, but didn't know that I can mark each node as separate resource. 

I'll check it.

[Den Slobozhanskyi]

what plugin do you use precisely? I run jenkins 2.150.1 and this throttling plugin https://github.com/jenkinsci/throttle-concurrent-builds-plugin/blob/master/README.md . It clearly gives me an option to throttle a pipeline job in job configurations (yep, 100% pipeline job type)

четверг, 31 января 2019 г., 8:48:14 UTC+1 пользователь Vitaly Karasik написал:

[Vitaly Karasik]

My job is Multibranch job, for some reason such job, unlike 'regulal' pipeline job, doesn't provide this checkbox.

[Vitaly Karasik]

I started to play with the lockable resources plugin. 

I'd like to prevent parallel execution of my pipeline (multibranch pipeline) on the same Jenkins slave, so I tried to use "$NODE_NAME" as lockable resource, like this:

pipeline {

  agent { label 'slave-parallel' }

  options {

    lock ("${NODE_NAME}")

   }

stages() 

{

......

}

But unfortunately $NODE_NAME isn't available on this step. Is there simple workaround for thaT? 

On Thursday, January 31, 2019 at 11:43:30 AM UTC+2, Daniel Butler wrote:

[Brian Ray]

Can you put your lock step inside a step block? We still use Imperative Pipeline so I'm less familiar with Declarative Pipeline. But it seems like inside a step block that NODE_NAME should resolve correctly, because--I think--the block is running in a node context.