[JIRA] (JENKINS-60657) EC2 plugins with multiple templates: fails to start an instance for job in the middle of the queue

[h35gao@edu.uwaterloo.ca (JIRA)]

Handi Gao created an issue

Jenkins / JENKINS-60657 EC2 plugins with multiple templates: fails to start an instance for job in the middle of the queue Issue Type: Bug Assignee: FABRIZIO MANFREDI Components: ec2-plugin Created: 2020-01-06 20:13 Environment: Jenkins: 2.204.1 Amazon EC2 plugin (ID: ec2): 1.47 Labels: ec2 ec2-plugin Priority: Minor Reporter: Handi Gao We have two templates for the ec2 plugin:   template1 label: test instance cap: 1   template2 label: report instance cap: 3     Jenkins doesn't create a new instance with template 2 when there are jobs in queue waiting for an available excutor on a running instance created with template 1, for example in our queue: 1. test_job (waiting for template1:test as all exectuors on template1:test are taken and we only want 1 instance with template1:test) 2. report_job (waiting for template2:report as it does not exist) Job2 will be blocked until job1 is scheduled as Jenkins keeps trying to schedule job1 and ignores the request of an instance with template2 from job2   Add Comment

This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f)

[h35gao@edu.uwaterloo.ca (JIRA)]

Handi Gao updated an issue

Jenkins / JENKINS-60657 EC2 plugins with multiple templates: fails to start an instance for job in the middle of the queue

Change By: Handi Gao

We have two templates for the ec2 plugin:

{code:java}

template1 label: test instance cap: 1

{code}   {code:java}

template2 label: report instance cap: 3

{code}

Jenkins doesn't create a new instance with template 2 when there are jobs in queue waiting for an available excutor on a running instance created with template 1, for example in our queue:

1. test_job (waiting for _template1:test_ as all exectuors on _template1:test_ are taken and we only want 1 instance with _template1:test_) 2. report_job (waiting for _template2:report_ as it does not exist) Job2 report_job will be blocked until job1 test_job is scheduled as Jenkins keeps trying to schedule job1 test_job and ignores the request of an instance with _template2_ from job2 report_job

Add Comment

[h35gao@edu.uwaterloo.ca (JIRA)]

Handi Gao updated an issue

Jenkins / JENKINS-60657 EC2 plugins with multiple templates: fails to start an instance for job in the middle of the queue Change By: Handi Gao

We have two templates for the ec2 plugin:   {code:java} template1 label: test instance cap: 1 {code}   {code:java} template2 label: report instance cap: 3 {code}

Jenkins doesn't create a new instance with template 2 when there are jobs in queue waiting for an available excutor on a running instance created with template 1, for example in our queue: 1. test_job (waiting for _template1:test_ as all exectuors on _template1:test_ are taken and we only want 1 instance with _template1:test_) 2. report_job (waiting for _template2:report_ as it does not exist)

report_job will be blocked until test_job is scheduled as Jenkins keeps trying to schedule test_job and ignores the request of an instance with _template2_ from report_job

Add Comment

[h35gao@edu.uwaterloo.ca (JIRA)]

Handi Gao updated an issue

Jenkins / JENKINS-60657 EC2 plugins with multiple templates: fails to start an instance for job in the middle of the queue Change By: Handi Gao

We have two templates for the ec2 plugin: {code:java} template1 label: test instance cap: 1 {code} {code:java} template2 label: report instance cap: 3 {code}    Jenkins doesn't create a new instance with template 2 when there are jobs in queue waiting for an available excutor on a running instance created with template 1, for example in our queue: 1. test_job (waiting for _template1:test_ as all exectuors on _template1:test_ are taken and we only want 1 instance with _template1:test_) 2. report_job (waiting for _template2:report_ as it does not exist) report_job will be blocked until test_job is scheduled as Jenkins keeps trying to schedule test_job and ignores the request of an instance with _template2_ from report_job

Seems like this only happens when there are a lot jobs in the queue (I don't know a definite number.. but I don't experience this issue when there are only two jobs in the queue)   And I believe the problem is from Jenkins core not the ec2 plugin as the NodeProvisioner doesn't not look for all the labels. Here is the log from hudson.slaves.NodeProvisioner class {code:java} Jan 16, 2020 9:38:08 PM FINER hudson.slaves.NodeProvisioner Consulting hudson.plugins.ec2.NoDelayProvisionerStrategy@1cfb55fa provisioning strategy with state StrategyState{label=null, snapshot=LoadStatisticsSnapshot{definedExecutors=6, onlineExecutors=6, connectingExecutors=0, busyExecutors=6, idleExecutors=0, availableExecutors=0, queueLength=127}, plannedCapacitySnapshot=0, additionalPlannedCapacity=0} Jan 16, 2020 9:38:09 PM FINER hudson.slaves.NodeProvisioner Consulting hudson.slaves.NodeProvisioner$StandardStrategyImpl@6a13a6e5 provisioning strategy with state StrategyState{label=null, snapshot=LoadStatisticsSnapshot{definedExecutors=6, onlineExecutors=6, connectingExecutors=0, busyExecutors=6, idleExecutors=0, availableExecutors=0, queueLength=127}, plannedCapacitySnapshot=0, additionalPlannedCapacity=0} Jan 16, 2020 9:38:09 PM FINE hudson.slaves.NodeProvisioner Excess workload 117.81 detected. (planned capacity=0,connecting capacity=0,Qlen=117.81,available=0&0,online=6,m=0.106) Jan 16, 2020 9:38:18 PM FINER hudson.slaves.NodeProvisioner Consulting hudson.plugins.ec2.NoDelayProvisionerStrategy@1cfb55fa provisioning strategy with state StrategyState{label=null, snapshot=LoadStatisticsSnapshot{definedExecutors=6, onlineExecutors=6, connectingExecutors=0, busyExecutors=6, idleExecutors=0, availableExecutors=0, queueLength=127}, plannedCapacitySnapshot=0, additionalPlannedCapacity=0} Jan 16, 2020 9:38:18 PM FINER hudson.slaves.NodeProvisioner Consulting hudson.slaves.NodeProvisioner$StandardStrategyImpl@6a13a6e5 provisioning strategy with state StrategyState{label=null, snapshot=LoadStatisticsSnapshot{definedExecutors=6, onlineExecutors=6, connectingExecutors=0, busyExecutors=6, idleExecutors=0, availableExecutors=0, queueLength=127}, plannedCapacitySnapshot=0, additionalPlannedCapacity=0} Jan 16, 2020 9:38:18 PM FINE hudson.slaves.NodeProvisioner Excess workload 118.729 detected. (planned capacity=0,connecting capacity=0,Qlen=118.729,available=0&0,online=6,m=0.106) {code}

Add Comment

[h35gao@edu.uwaterloo.ca (JIRA)]

Handi Gao updated an issue

Jenkins / JENKINS-60657 EC2 plugins with multiple templates: fails to start an instance for job in the middle of the queue Change By: Handi Gao

We have two templates for the ec2 plugin: {code:java} template1

label: test default instance cap: 1 usage: NORMAL {code}

{code:java} template2 label: report instance cap: 3

usage: EXCLUSIVE {code}

Jenkins doesn't create a new instance with template 2 when there are jobs in queue waiting for an available excutor on a running instance created with template 1, for example in our queue:

1. test_job ( waiting Restrict where this project can be run unchecked. Waiting for _template1:test_ as all exectuors on _template1:test_ are taken and we only want 1 instance with _template1:test_) 2. report_job ( waiting Restrict where this project can be run checked. Label expression: report. Waiting for _template2:report_ as it does not exist)

report_job will be blocked until test_job is scheduled as Jenkins keeps trying to schedule test_job and ignores the request of an instance with _template2_ from report_job

Seems like I suspect that this will only happens happen when there are a lot jobs in the queue (I don't know a definite number.. but I don't experience this issue when there are only two jobs first job in the queue ) has "Restrict where this project can be run" unchecked.   And I believe the problem is from Jenkins core not the ec2 plugin as the NodeProvisioner doesn't not look for all the labels as it should be ([https://github . com/jenkinsci/jenkins/blob/jenkins-2.204.1/core/src/main/java/hudson/slaves/NodeProvisioner.java#L815])

Here is the log from hudson.slaves.NodeProvisioner class {code:java} Jan 16, 2020 9:38:08 PM FINER hudson.slaves.NodeProvisioner Consulting hudson.plugins.ec2.NoDelayProvisionerStrategy@1cfb55fa provisioning strategy with state StrategyState{label=null, snapshot=LoadStatisticsSnapshot{definedExecutors=6, onlineExecutors=6, connectingExecutors=0, busyExecutors=6, idleExecutors=0, availableExecutors=0, queueLength=127}, plannedCapacitySnapshot=0, additionalPlannedCapacity=0} Jan 16, 2020 9:38:09 PM FINER hudson.slaves.NodeProvisioner Consulting hudson.slaves.NodeProvisioner$StandardStrategyImpl@6a13a6e5 provisioning strategy with state StrategyState{label=null, snapshot=LoadStatisticsSnapshot{definedExecutors=6, onlineExecutors=6, connectingExecutors=0, busyExecutors=6, idleExecutors=0, availableExecutors=0, queueLength=127}, plannedCapacitySnapshot=0, additionalPlannedCapacity=0} Jan 16, 2020 9:38:09 PM FINE hudson.slaves.NodeProvisioner Excess workload 117.81 detected. (planned capacity=0,connecting capacity=0,Qlen=117.81,available=0&0,online=6,m=0.106) Jan 16, 2020 9:38:18 PM FINER hudson.slaves.NodeProvisioner Consulting hudson.plugins.ec2.NoDelayProvisionerStrategy@1cfb55fa provisioning strategy with state StrategyState{label=null, snapshot=LoadStatisticsSnapshot{definedExecutors=6, onlineExecutors=6, connectingExecutors=0, busyExecutors=6, idleExecutors=0, availableExecutors=0, queueLength=127}, plannedCapacitySnapshot=0, additionalPlannedCapacity=0} Jan 16, 2020 9:38:18 PM FINER hudson.slaves.NodeProvisioner Consulting hudson.slaves.NodeProvisioner$StandardStrategyImpl@6a13a6e5 provisioning strategy with state StrategyState{label=null, snapshot=LoadStatisticsSnapshot{definedExecutors=6, onlineExecutors=6, connectingExecutors=0, busyExecutors=6, idleExecutors=0, availableExecutors=0, queueLength=127}, plannedCapacitySnapshot=0, additionalPlannedCapacity=0} Jan 16, 2020 9:38:18 PM FINE hudson.slaves.NodeProvisioner Excess workload 118.729 detected. (planned capacity=0,connecting capacity=0,Qlen=118.729,available=0&0,online=6,m=0.106) {code}

Add Comment

[h35gao@edu.uwaterloo.ca (JIRA)]

Handi Gao updated an issue

Jenkins / JENKINS-60657 EC2 plugins with multiple templates: fails to start an instance for job in the middle of the queue Change By: Handi Gao

We have two templates for the ec2 plugin: {code:java} template1 label: test default instance cap: 1 usage: NORMAL{code} {code:java} template2 label: report instance cap: 3 usage: EXCLUSIVE{code}    Jenkins doesn't create a new instance with template 2 when there are jobs in queue waiting for an available excutor on a running instance created with template 1, for example in our queue:

1. test_job (Restrict where this project can be run unchecked. Waiting for _template1:test_ as all exectuors on _template1:test_ are taken and we only want 1 instance with _template1:test_) 2. report_job (Restrict where this project can be run checked. Label expression: report. Waiting for _template2:report_ as it does not exist)

report_job will be blocked until test_job is scheduled as Jenkins keeps trying to schedule test_job and ignores the request of an instance with _template2_ from report_job

UPDATE: I suspect can confirm that this will only happen happens when the first job in the queue has   "Restrict where this project can be run" unchecked.   And I believe the problem is from Jenkins core not the ec2 plugin as the NodeProvisioner doesn't not look for all the labels as it should be ([https://github.com/jenkinsci/jenkins/blob/jenkins-2.204.1/core/src/main/java/hudson/slaves/NodeProvisioner.java#L815])

Here is the log from hudson.slaves.NodeProvisioner class {code:java} Jan 16, 2020 9:38:08 PM FINER hudson.slaves.NodeProvisioner Consulting hudson.plugins.ec2.NoDelayProvisionerStrategy@1cfb55fa provisioning strategy with state StrategyState{label=null, snapshot=LoadStatisticsSnapshot{definedExecutors=6, onlineExecutors=6, connectingExecutors=0, busyExecutors=6, idleExecutors=0, availableExecutors=0, queueLength=127}, plannedCapacitySnapshot=0, additionalPlannedCapacity=0} Jan 16, 2020 9:38:09 PM FINER hudson.slaves.NodeProvisioner Consulting hudson.slaves.NodeProvisioner$StandardStrategyImpl@6a13a6e5 provisioning strategy with state StrategyState{label=null, snapshot=LoadStatisticsSnapshot{definedExecutors=6, onlineExecutors=6, connectingExecutors=0, busyExecutors=6, idleExecutors=0, availableExecutors=0, queueLength=127}, plannedCapacitySnapshot=0, additionalPlannedCapacity=0} Jan 16, 2020 9:38:09 PM FINE hudson.slaves.NodeProvisioner Excess workload 117.81 detected. (planned capacity=0,connecting capacity=0,Qlen=117.81,available=0&0,online=6,m=0.106) Jan 16, 2020 9:38:18 PM FINER hudson.slaves.NodeProvisioner Consulting hudson.plugins.ec2.NoDelayProvisionerStrategy@1cfb55fa provisioning strategy with state StrategyState{label=null, snapshot=LoadStatisticsSnapshot{definedExecutors=6, onlineExecutors=6, connectingExecutors=0, busyExecutors=6, idleExecutors=0, availableExecutors=0, queueLength=127}, plannedCapacitySnapshot=0, additionalPlannedCapacity=0} Jan 16, 2020 9:38:18 PM FINER hudson.slaves.NodeProvisioner Consulting hudson.slaves.NodeProvisioner$StandardStrategyImpl@6a13a6e5 provisioning strategy with state StrategyState{label=null, snapshot=LoadStatisticsSnapshot{definedExecutors=6, onlineExecutors=6, connectingExecutors=0, busyExecutors=6, idleExecutors=0, availableExecutors=0, queueLength=127}, plannedCapacitySnapshot=0, additionalPlannedCapacity=0} Jan 16, 2020 9:38:18 PM FINE hudson.slaves.NodeProvisioner Excess workload 118.729 detected. (planned capacity=0,connecting capacity=0,Qlen=118.729,available=0&0,online=6,m=0.106) {code}

Add Comment

[h35gao@edu.uwaterloo.ca (JIRA)]

Handi Gao commented on JENKINS-60657

Re: EC2 plugins with multiple templates: fails to start an instance for job in the middle of the queue This may be solved by commit https://github.com/jenkinsci/jenkins/commit/1125b1b4d7a5280ce0bd0d4c3eb464e172bf94ec

Add Comment

[h35gao@edu.uwaterloo.ca (JIRA)]

Handi Gao edited a comment on JENKINS-60657

Re: EC2 plugins with multiple templates: fails to start an instance for job in the middle of the queue

This may be solved by commit [https://github.com/jenkinsci/jenkins/commit/1125b1b4d7a5280ce0bd0d4c3eb464e172bf94ec] I will close the ticket once we confirm that this is solved in version 2.209.

Add Comment

[h35gao@edu.uwaterloo.ca (JIRA)]

Handi Gao edited a comment on JENKINS-60657

Re: EC2 plugins with multiple templates: fails to start an instance for job in the middle of the queue

This may be solved by commit [https://github.com/jenkinsci/jenkins/commit/1125b1b4d7a5280ce0bd0d4c3eb464e172bf94ec]

I will close the ticket once we confirm that this is solved in version 2.209. Need to wait for Blueocean to release a new version before we can upgrade Jenkins to the latest version as we use jenkinsci/blueocean as base image.

Add Comment