[JIRA] (JENKINS-52500) cloud agents are not used if labels overlap with permanent agent

1 view
Skip to first unread message

kyle.mcgovern@cerner.com (JIRA)

unread,
Jul 11, 2018, 7:12:01 PM7/11/18
to jenkinsc...@googlegroups.com
Kyle McGovern created an issue
 
Jenkins / Bug JENKINS-52500
cloud agents are not used if labels overlap with permanent agent
Issue Type: Bug Bug
Assignee: Carlos Sanchez
Components: core, kubernetes-plugin
Created: 2018-07-11 23:11
Priority: Minor Minor
Reporter: Kyle McGovern

I've done my testing with the kubernetes plugin however this may apply to other cloud providers as well

When executing a job with a specific node label (IE linux) and a permanent agent shares the same label configured cloud providers are ignored.

example scripted pipeline

def test_builds = [:]
1.upto(36,{
  test_builds["${it}"] = {
    stage("${it}") {
      node('linux') {
        sh 'echo it worked'
        sh 'sleep 120'
      }
    }
  }
}) 

parallel(test_builds)

The follow log message is logged (after executing a few times to get the queue length up)

Consulting hudson.slaves.NodeProvisioner$StandardStrategyImpl@10a347a8 provisioning strategy with state StrategyState{label=linux, snapshot=LoadStatisticsSnapshot{definedExecutors=16, onlineExecutors=14, connectingExecutors=0, busyExecutors=14, idleExecutors=0, availableExecutors=0, queueLength=130}, plannedCapacitySnapshot=0, additionalPlannedCapacity=0}

Jul 11, 2018 5:59:25 PM FINER hudson.slaves.NodeProvisioner

Queue length 0 is less than the available capacity 0. No provisioning strategy required

Jul 11, 2018 5:59:25 PM FINER hudson.slaves.NodeProvisioner

Queue length 0 is less than the available capacity 0. No provisioning strategy required

Jul 11, 2018 5:59:25 PM FINER hudson.slaves.NodeProvisioner

Queue length 0 is less than the available capacity 0. No provisioning strategy required

Jul 11, 2018 5:59:25 PM FINER hudson.slaves.NodeProvisioner

Queue length 0 is less than the available capacity 0. No provisioning strategy required

Similar log messages repeat.

Running the same scripted pipeline but changing the node label to something unique to the cloud provider things work exactly as expected

Consulting hudson.slaves.NodeProvisioner$StandardStrategyImpl@10a347a8 provisioning strategy with state StrategyState{label=kubectl, snapshot=LoadStatisticsSnapshot{definedExecutors=9, onlineExecutors=4, connectingExecutors=5, busyExecutors=4, idleExecutors=0, availableExecutors=0, queueLength=32}, plannedCapacitySnapshot=0, additionalPlannedCapacity=6}

Jul 11, 2018 4:38:45 PM FINE hudson.slaves.NodeProvisioner

Excess workload 11.42 detected. (planned capacity=0,connecting capacity=5,Qlen=12.08,available=0.006&0,online=4,m=0.125)

Jul 11, 2018 4:38:47 PM INFO hudson.slaves.NodeProvisioner$StandardStrategyImpl apply

Started provisioning Kubernetes Pod Template from open-kc-k8s-corp-build-0 with 1 executors. Remaining excess workload: 10.42

Jul 11, 2018 4:38:47 PM INFO hudson.slaves.NodeProvisioner$StandardStrategyImpl apply

Started provisioning Kubernetes Pod Template from open-kc-k8s-corp-build-0 with 1 executors. Remaining excess workload: 9.42

Jul 11, 2018 4:38:47 PM INFO hudson.slaves.NodeProvisioner$StandardStrategyImpl apply

Started provisioning Kubernetes Pod Template from open-kc-k8s-corp-build-0 with 1 executors. Remaining excess workload: 8.42

Jul 11, 2018 4:38:47 PM INFO hudson.slaves.NodeProvisioner$StandardStrategyImpl apply

Started provisioning Kubernetes Pod Template from open-kc-k8s-corp-build-0 with 1 executors. Remaining excess workload: 7.42

Jul 11, 2018 4:38:47 PM INFO hudson.slaves.NodeProvisioner$StandardStrategyImpl apply

Started provisioning Kubernetes Pod Template from open-kc-k8s-corp-build-0 with 1 executors. Remaining excess workload: 6.42

Jul 11, 2018 4:38:47 PM INFO hudson.slaves.NodeProvisioner$StandardStrategyImpl apply

Started provisioning Kubernetes Pod Template from open-kc-k8s-corp-build-0 with 1 executors. Remaining excess workload: 5.42

I couldn't find an existing issue that described this specific issue so logged this one. Hoping it's something simple but I'm not seeing anything super obvious to me in the comment here or in the code underneath that would cause this behavior.

java runtime version: 1.8.0_162-b12
Jenkins version: 2.129
Kubernutes plugin version: 1.9.2

Add Comment Add Comment
 
This message was sent by Atlassian JIRA (v7.10.1#710002-sha1:6efc396)

jenkins-ci@carlossanchez.eu (JIRA)

unread,
Jul 12, 2018, 2:36:01 AM7/12/18
to jenkinsc...@googlegroups.com
Carlos Sanchez commented on Bug JENKINS-52500
 
Re: cloud agents are not used if labels overlap with permanent agent

how many executors do you have in the linux node ?
I would expect jenkins to use the permanent one up to the executors available, then start with cloud executors
Are you setting the overprovisioning flags for faster provisioning of cloud agents ? https://github.com/jenkinsci/kubernetes-plugin#over-provisioning-flags

kyle.mcgovern@cerner.com (JIRA)

unread,
Jul 12, 2018, 12:23:01 PM7/12/18
to jenkinsc...@googlegroups.com

Between 2 nodes there are 14 executors. 12 on one node (a physical) and 2 on a VM. A 3rd node is down for maintenance but it has 2 executors.

I did see those flags after digging through the code the standard provisioning strategy. With 100+ items in the queue would you have expected it to try and provision? When i was testing I got the build queue up to 540 without kube slave spinning up before I decided to log this jira.

kyle.mcgovern@cerner.com (JIRA)

unread,
Jul 12, 2018, 12:23:02 PM7/12/18
to jenkinsc...@googlegroups.com
Kyle McGovern edited a comment on Bug JENKINS-52500
Between 2 nodes there are 14 executors. 12 on one node (a physical) and 2 on a VM. A 3rd node is down for maintenance but it has 2 executors.

I did see those flags after digging through the code the standard provisioning strategy. With 100+ items in the queue would you have expected it to try and provision? When i was testing I got the build queue up to 540 without a kube slave spinning up before I decided to log this jira.

kyle.mcgovern@cerner.com (JIRA)

unread,
Jul 16, 2018, 11:39:01 AM7/16/18
to jenkinsc...@googlegroups.com

I added the suggested settings with no change in behavior

-Dhudson.slaves.NodeProvisioner.initialDelay=0 -Dhudson.slaves.NodeProvisioner.MARGIN=50 -Dhudson.slaves.NodeProvisioner.MARGIN0=0.85

jglick@cloudbees.com (JIRA)

unread,
Jul 16, 2019, 3:43:17 PM7/16/19
to jenkinsc...@googlegroups.com
Jesse Glick assigned an issue to Unassigned
 
Change By: Jesse Glick
Assignee: Carlos Sanchez
This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)
Reply all
Reply to author
Forward
0 new messages