[JIRA] (JENKINS-45140) Add support of throttling of the entire build in Declarative Pipeline

24 views
Skip to first unread message

andrew.bayer@gmail.com (JIRA)

unread,
Feb 27, 2018, 11:58:02 AM2/27/18
to jenkinsc...@googlegroups.com
Andrew Bayer commented on New Feature JENKINS-45140
 
Re: Add support of throttling of the entire build in Declarative Pipeline

Jay Spang - the maxConcurrentPerNode property goes on the category you define in the Jenkins global config, and then you use that category in the Pipeline.

Add Comment Add Comment
 
This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e)
Atlassian logo

sharadholani@gmail.com (JIRA)

unread,
Sep 21, 2018, 8:51:03 AM9/21/18
to jenkinsc...@googlegroups.com
Sharad Holani updated an issue
 
Jenkins / New Feature JENKINS-45140
Change By: Sharad Holani
Attachment: image-2018-09-21-18-20-12-512.png
This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)

sharadholani@gmail.com (JIRA)

unread,
Sep 21, 2018, 8:52:05 AM9/21/18
to jenkinsc...@googlegroups.com
Sharad Holani commented on New Feature JENKINS-45140
 
Re: Add support of throttling of the entire build in Declarative Pipeline

Jay Spang

 

 

 

 

  • I tried the step you suggested. Added throttle settings in manage jenkins. Added  throttle(categories: ['test-category']) in options. Also added the code suggested by Kyle Walker. But the number of builds are not getting throttled.

sharadholani@gmail.com (JIRA)

unread,
Sep 21, 2018, 8:53:04 AM9/21/18
to jenkinsc...@googlegroups.com
Sharad Holani edited a comment on New Feature JENKINS-45140
[~jayspang]!image-2018-09-21-18-20-12-512.png |width=932,height=156 !

 

 

 

 

- I tried the step you suggested. Added throttle settings in manage jenkins. Added  throttle(categories: ['test-category']) in options. Also added the code suggested by [~kylejameswalker]. But the number of builds are not getting throttled.

metaphysical.intoxication@gmail.com (JIRA)

unread,
Apr 5, 2019, 7:16:04 AM4/5/19
to jenkinsc...@googlegroups.com

Hello, I'm having the same problem: 

options {
  throttle(['shared-workspace'])
}

 doesn't work for the entire declarative job.

As a workaround, I'm using shared libraries:
1) define the entire declarative pipeline in a shared library var, as per https://jenkins.io/blog/2017/10/02/pipeline-templates-with-shared-libraries/
2) surround pipeline invocation with throttle, e.g.

Jenkinsfile
throttle(['shared-workspace']) {
  myDeliveryPipeline {
      branch = 'master'
      scmUrl = 'ssh://g...@myScmServer.com/repos/myRepo.git'
      email = 'te...@example.com'
      serverPort = '8080'
      developmentServer = 'dev-myproject.mycompany.com'
      stagingServer = 'staging-myproject.mycompany.com'
      productionServer = 'production-myproject.mycompany.com'
  }
}

Hope this will be helpful to someone!

metaphysical.intoxication@gmail.com (JIRA)

unread,
Apr 5, 2019, 7:17:02 AM4/5/19
to jenkinsc...@googlegroups.com
Dmitry Mamchur edited a comment on New Feature JENKINS-45140
Hello, I'm having the same problem: 
{code:java}
options {
  throttle(['shared-workspace'])
}{code}
 doesn't work for the entire
build in a declarative job.


As a workaround, I'm using shared libraries:
1) define the entire declarative pipeline in a shared library var, as per [https://jenkins.io/blog/2017/10/02/pipeline-templates-with-shared-libraries/]
2) surround pipeline invocation with throttle, e.g.
{code:java}

Jenkinsfile
throttle(['shared-workspace']) {
  myDeliveryPipeline {
      branch = 'master'
      scmUrl = 'ssh://g...@myScmServer.com/repos/myRepo.git'
      email = 'te...@example.com'
      serverPort = '8080'
      developmentServer = 'dev-myproject.mycompany.com'
      stagingServer = 'staging-myproject.mycompany.com'
      productionServer = 'production-myproject.mycompany.com'
  }
}
{code}

Hope this will be helpful to someone!

medianick@gmail.com (JIRA)

unread,
Apr 18, 2019, 2:32:05 PM4/18/19
to jenkinsc...@googlegroups.com

I had to revert back to a scripted pipeline in order to get throttling to work as intended. The issue seems to be that the throttle option in a Declarative pipeline occurs on a particular agent node, and it's either too late – the executor is already consumed – or it's simply a no-op, but in any case, I'm not seeing any actual evidence of throttling, either overall or per node.

pipeline {
  agent { label 'foo' }
  options {
    throttle(categories: ['MyCategory']) // <-- doesn't do anything, even with MyCategory defined in the system configuration
  }
  stages {
    // ...
  }
}

What I would have wanted is either to have the throttle option take effect before the agent is selected, or some sort of declarative syntax that would let me have agent none at the top level of the pipeline (where the throttle option is specified), then the ability to specify a particular agent for all the actual stages to be executed (Build, Test, Publish, etc.). I tried this:

pipeline {
  agent none // <-- to avoid having an agent allocated before the throttle category is evaluated
  options {
    throttle(categories: ['MyCategory'])
  }
  stages {
    stage('Overall') {
      agent {
        label 'foo' // <-- to get the real agent I want for the real stages
      }
      stages {
        stage('Build') {
           // ...
        }
        stage('Test') {
          // ...
        }
      }
    }
  }
}

But it simply hung with no useful output. If there's some other way to have agent none at the top level, then a single, reused agent for the actual stages within, I'd be interested to hear about it. I know I could have an agent per stage (Build, Test, etc.), but that wouldn't work for me – it has to be the same machine for all of these, and I'd rather not stash and unstash all the artifacts I need between these stages.

guy.banay@zerto.com (JIRA)

unread,
Jul 29, 2019, 7:36:04 AM7/29/19
to jenkinsc...@googlegroups.com

Hi,

Any update with this issue?

The property "maxConcurrentPerNode" is not working

 

Thanks

 

medianick@gmail.com (JIRA)

unread,
Oct 11, 2019, 7:22:03 AM10/11/19
to jenkinsc...@googlegroups.com

I'm seeing what Guy Banay sees. I'm now testing a Declarative Pipeline workaround as Kyle Walker suggested – namely, putting the properties block outside the pipeline altogether – and it works, but it only honors the maxConcurrentTotal property, not the maxConcurrentNode one. Very simple repro (assuming a "ThrottleTest" category is configured globally):

properties([
  [
    $class: 'ThrottleJobProperty',
    categories: ['ThrottleTest'],
    throttleEnabled: true,
    throttleOption: 'category'
  ],
])

pipeline {
    agent any
    stages {
        stage('Long-running') {
            steps {
                input message: 'Shall we continue?'
                echo "Thanks! Continuing."
            }
        }
    }
}

If I configure a maximum of 5 concurrent builds across all nodes, but only 1 per node, the only limit enforced is the total limit; a single build agent can still run as many builds of this job as it has executors. Only the maxConcurrentTotal takes effect.

This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f)
Atlassian logo

marcus.a.philip@gmail.com (JIRA)

unread,
Apr 3, 2020, 5:35:04 AM4/3/20
to jenkinsc...@googlegroups.com

I tried Dmitry Mamchurs creative suggestion but on more recent versions this is explicitly prohibited:

WorkflowScript: 2: pipeline block must be at the top-level, not within another block. @ line 2, column 5.
       pipeline {
       ^

Demo pipeline:

 throttle(['myThrottle']) {
    pipeline {
        agent { label 'mylabel' }
        stages {
            stage('first') {
                steps {
                    sleep 60
                }
            }
        }
    }
}
This message was sent by Atlassian Jira (v7.13.12#713012-sha1:6e07c38)
Atlassian logo

metaphysical.intoxication@gmail.com (JIRA)

unread,
Apr 3, 2020, 5:51:03 AM4/3/20
to jenkinsc...@googlegroups.com

Marcus Philip The secret sauce is having your pipeline defined in a shared library.

vars/myPipeline.groovy

def call() {
    pipeline {
        agent { label 'mylabel' }
        stages {
            stage('first') {
                steps {
                    sleep 60
                }
            }
        }
    }
}

jobs/my-pipeline/Jenkinsfile

throttle(['myThrottle']) {
    myPipeline()
}

marcus.a.philip@gmail.com (JIRA)

unread,
Apr 3, 2020, 5:54:04 AM4/3/20
to jenkinsc...@googlegroups.com
Marcus Philip edited a comment on New Feature JENKINS-45140
I tried [~metamilk]s creative suggestion but on more recent versions this is explicitly prohibited:


{noformat}

WorkflowScript: 2: pipeline block must be at the top-level, not within another block. @ line 2, column 5.
       pipeline {
       ^
{noformat}

h3. Demo pipeline:
{code:
groovy java }
 throttle(['myThrottle']) {

    pipeline {
        agent { label 'mylabel' }
        stages {
            stage('first') {
                steps {
                    sleep 60
                }
            }
        }
    }
}
{code}
 

I've tested many other variants and I claim it is currently (with latest versions) impossible to get node throttling on a declarative pipeline. If someone has a counterexample I would appreciate that.

marcus.a.philip@gmail.com (JIRA)

unread,
Apr 3, 2020, 6:03:03 AM4/3/20
to jenkinsc...@googlegroups.com

Je-s F-ng C-st!

It's insane that that would make a difference but it does.

So the 'pipeline block must be at the top-level' check is just on a file (textual) basis, not on the actual code structure.

Thanks Dmitry Mamchur! You made my day!

jgrant216@gmail.com (JIRA)

unread,
Apr 3, 2020, 8:40:05 AM4/3/20
to jenkinsc...@googlegroups.com
Jeff G commented on New Feature JENKINS-45140

I honestly did not have much hope when I saw this issue having been updated in my e-mail this morning.

Marcus Philip, I'll second your comment/excitement on finally having a solution to this mystery. 

Dmitry Mamchur, thank you so much! 

 

Our team already has our declarative and scripted pipelines largely live in libraries.  I gave a couple attempts to implement the throttle as described by Dmitry Mamchur and here is a little more on how I think I'll `throttle` in our builds by renaming sharedPipeline to sharedPipelineInner and having sharedPipeline wrap the throttle category around the sharedPipeline() call.  This way, I don't need to update 100+ repositories * number of branches. 

The current Jenkinsfile looks like this more or less...
sharedPipeline()
vars/sharedPipeline.groovy
def call() {
throttle(['throttle-category'])

{ sharedPipelineInner() }

}
vars/sharedPipelineInner.groovy
def call() {
pipeline {
agent

{ label 'mylabel' }

stages {
stage('first') {
steps

{ sleep 60 }

}
}
}
}
 This way, no Jenkinsfile will need to be updated and they will all be throttled for free.

 

This defect asks for throttle within a step, which this solution does not provide, so I guess it will need to stay open.  The docs for the plugin should be updated, for sure, with an example like this.

jgrant216@gmail.com (JIRA)

unread,
Apr 3, 2020, 8:41:11 AM4/3/20
to jenkinsc...@googlegroups.com
Jeff G edited a comment on New Feature JENKINS-45140
I honestly did not have much hope when I saw this issue having been updated in my e-mail this morning.

[~marcus_phi], I'll second your comment/excitement on finally having a solution to this mystery. (flag)

[~metamilk], thank you so much! (y)

 

Our team already has our declarative and scripted pipelines largely live in libraries.  I gave a couple attempts to implement the throttle as described by [~metamilk] and here is a little more on how I think I'll `throttle` in our builds by renaming {{sharedPipeline}} to {{sharedPipelineInner}} and having {{sharedPipeline}} wrap the throttle category around the {{sharedPipeline()}} call.  This way, I don't need to update 100+ repositories * number of branches. 


The current {{Jenkinsfile}} looks like this more or less...
sharedPipeline()
vars/sharedPipeline.groovy
```

def call() {
    throttle(['throttle-category'])
{

{
        sharedPipelineInner()
    }

}
```

vars/sharedPipelineInner.groovy
def call() {
    pipeline {
        agent


{ label 'mylabel' }

        stages {
            stage('first')
\ {
                steps
{

{
                    sleep 60
                }

            }
        }
    }
}
 This way, no {{Jenkinsfile}} will need to be updated and they will all be throttled for free.

 

This defect asks for {{throttle}} within a step, which this solution does not provide, so I guess it will need to stay open.  The docs for the plugin should be updated, for sure, with an example like this.

jgrant216@gmail.com (JIRA)

unread,
Apr 3, 2020, 8:43:03 AM4/3/20
to jenkinsc...@googlegroups.com
Jeff G edited a comment on New Feature JENKINS-45140
I honestly did not have much hope when I saw this issue having been updated in my e-mail this morning.

[~marcus_phi], I'll second your comment/excitement on finally having a solution to this mystery. (flag)

[~metamilk], thank you so much! (y)

 

Our team already has our declarative and scripted pipelines largely live in libraries.  I gave a couple attempts to implement the throttle as described by [~metamilk] and here is a little more on how I think I'll `throttle` in our builds by renaming {{sharedPipeline}} to {{sharedPipelineInner}} and having {{sharedPipeline}} wrap the throttle category around the {{sharedPipeline()}} call.  This way, I don't need to update 100+ repositories * number of branches. 

The current {{Jenkinsfile}} looks like this more or less...
sharedPipeline()
vars/sharedPipeline.groovy
``` {code:java}

def call() {
throttle(['throttle-category'])

{ sharedPipelineInner() }
}
{code }
```

vars/sharedPipelineInner.groovy
{code:java}
def call() {
pipeline {
agent

{ label 'mylabel' }

stages {
stage('first') {
steps

{ sleep 60 }
      }
}
}
}
{code }

 This way, no {{Jenkinsfile}} will need to be updated and they will all be throttled for free.

 

This defect asks for {{throttle}} within a step, which this solution does not provide, so I guess it will need to stay open.  The docs for the plugin should be updated, for sure, with an example like this.

jgrant216@gmail.com (JIRA)

unread,
Apr 3, 2020, 8:44:05 AM4/3/20
to jenkinsc...@googlegroups.com

jgrant216@gmail.com (JIRA)

unread,
Apr 3, 2020, 8:44:23 AM4/3/20
to jenkinsc...@googlegroups.com

jgrant216@gmail.com (JIRA)

unread,
Apr 3, 2020, 8:45:04 AM4/3/20
to jenkinsc...@googlegroups.com

jgrant216@gmail.com (JIRA)

unread,
Apr 3, 2020, 8:45:11 AM4/3/20
to jenkinsc...@googlegroups.com
This defect asks for {{throttle}} within a step, which this solution does not provide, so I guess it will need to stay open.  The docs for the plugin should be updated, for sure, with an example like this.

jgrant216@gmail.com (JIRA)

unread,
Apr 3, 2020, 8:46:03 AM4/3/20
to jenkinsc...@googlegroups.com
Jeff G edited a comment on New Feature JENKINS-45140
I honestly did not have much hope when I saw this issue having been updated in my e-mail this morning.

[~marcus_phi], I'll second your comment/excitement on finally having a solution to this mystery. (flag)

[~metamilk], thank you so much! (y)

Our team already has our declarative and scripted pipelines largely live in libraries.  I gave a couple attempts to implement the throttle as described by [~metamilk] and here is a little more on how I think I'll `throttle` in our builds by renaming {{sharedPipeline}} to {{sharedPipelineInner}} and having {{sharedPipeline}} wrap the throttle category around the {{sharedPipeline()}} call.  This way, I don't need to update 100+ repositories * number of branches. 

The current {{Jenkinsfile}} looks like this more or less...
{code:java}
// library imports
sharedPipeline() {code}
vars/sharedPipeline.groovy
{code:java}
def call() {
  throttle(['throttle-category']) {
    sharedPipelineInner()
  }
}
{code}
vars/sharedPipelineInner.groovy
{code:java}
def call() {
  pipeline {
  agent { label 'mylabel' }
  stages {
    stage('first') {
      steps {
        sleep 60
      }
    }
  }
}
{code}
 

This defect asks for {{throttle}} within a step, which this solution does not provide, so I guess it will need to stay open.  The docs for the plugin should be updated, for sure, with an example like this.

jgrant216@gmail.com (JIRA)

unread,
Apr 3, 2020, 8:47:04 AM4/3/20
to jenkinsc...@googlegroups.com
  I did try to simply shift the pipeline config into a {{def}} within the original file, but that did not work.

This defect asks for {{throttle}} within a step, which this solution does not provide, so I guess it will need to stay open.  The docs for the plugin should be updated, for sure, with an example like this.

jgrant216@gmail.com (JIRA)

unread,
Apr 3, 2020, 9:23:03 AM4/3/20
to jenkinsc...@googlegroups.com


Edit - Then again, the title says _entire build_ so maybe this solution does apply and the example in the body does not. [~anthonymastrean], can you comment?
Reply all
Reply to author
Forward
0 new messages