[JIRA] (JENKINS-59959) The Concurrency Limit is not always respected.

Issue Type:	Bug
Assignee:	Unassigned
Components:	kubernetes-plugin
Created:	2019-10-28 10:16
Environment:	Jenkins ver. 2.176.3 Kubernets-plugin: 1.19.3
Priority:	Minor
Reporter:	Carl Dorbeus

I create a lot of jobs 300 of them using jobdsl, lot of jobs in a short time.

I have set the Kubernetes-plugin Concurrency Limit set to 20

Then Kubernetes plugin spins up a new node for almost every job. All but a few get in pending state due to resource limit in my Kubernetes cluster.

Sometimes the concurrency Limit is respected, i see a lot of this in my jenkins log. but it should never get to to 184 running or pending.

 
                                                                INFO: Maximum number of concurrently running agent pods (20) reached for Kubernetes Cloud kubernetes, not provisioning: 184 running or pending in namespace jenkins with Kubernetes labels {jenkins=slave}

This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f)

carl.dorbeus@gmail.com (JIRA)

unread,

Oct 28, 2019, 6:47:03 AM10/28/19

to jenkinsc...@googlegroups.com

Carl Dorbeus updated an issue

Jenkins /

Change By:	Carl Dorbeus

I create a lot of jobs 300 of them using jobdsl , lot of jobs in a short time .

I have set the Kubernetes-plugin Concurrency Limit set to 20

Then Kubernetes plugin spins up a new node /pod for almost every job. All but a few get in pending state due to resource limit in my Kubernetes cluster.

Sometimes the concurrency Limit is respected, i see a lot of this in my jenkins log. but it should never get to to 184 running or pending.

{code:java}

INFO: Maximum number of concurrently running agent pods (20) reached for Kubernetes Cloud kubernetes, not provisioning: 184 running or pending in namespace jenkins with Kubernetes labels {jenkins=slave}

{code}

carl.dorbeus@gmail.com (JIRA)

unread,

Oct 28, 2019, 6:49:03 AM10/28/19

to jenkinsc...@googlegroups.com

Carl Dorbeus updated an issue

Jenkins /

Change By:	Carl Dorbeus
Attachment:	image-2019-10-28-11-48-19-811.png

carl.dorbeus@gmail.com (JIRA)

unread,

Oct 28, 2019, 6:49:04 AM10/28/19

to jenkinsc...@googlegroups.com

Carl Dorbeus updated an issue

Jenkins /

Change By:	Carl Dorbeus

I create a lot of jobs 300 of them using jobdsl.

I have set the Kubernetes-plugin Concurrency Limit set to 20

Then Kubernetes plugin spins up a new node/pod for almost every job. All but a few get in pending state due to resource limit in my Kubernetes cluster , the pending pods is removed after a while and then recreated .

Sometimes the concurrency Limit is respected, i see a lot of this in my jenkins log. but it should never get to to 184 running or pending.
{code:java}
INFO: Maximum number of concurrently running agent pods (20) reached for Kubernetes Cloud kubernetes, not provisioning: 184 running or pending in namespace jenkins with Kubernetes labels {jenkins=slave}

{code}

codey.whitt@gmail.com (JIRA)

unread,

Jan 24, 2020, 6:21:03 PM1/24/20

to jenkinsc...@googlegroups.com

Codey Whitt commented on

From what I can tell, it has to do with how getActiveSlavePods relies on Kubernetes API to tell it what is currently running. There will always be an inherent delay between launch and this showing pods in the API so this approach will likely never be stable. IMO, a singleton that manages the instantiation of the slaves and their states would be the more appropriate solution. I plan to take a look at this next week as it's killing our Jenkins DR automation when our multibranch projects queue up 2k jobs on the first run after scanning.

sergio.merino.a@gmail.com (JIRA)

unread,

Jan 27, 2020, 11:25:04 AM1/27/20

to jenkinsc...@googlegroups.com

Sergio Merino commented on

I have a similar problem. The plugin execute as much pods as possible whenever it is under the limit. And stop when it is over the limit. But doesn't start just the proper amount of pods to don't exceed the limit. Example:

If the status is this:

Pods Limit: 30

Running: 30

Queued: 50

As soon as one running pod finish, Jenkins is going to execute the 50 queued ones at the same time, so you will finish with:

Pods Limit: 30

Running: 30 -1 (finished) + 50 = 79

Queued: 0

So, the limit is not respected at all

I was expecting that the plugin can manage the internal queue, even running the pods in the correct order

jonesbusy@gmail.com (JIRA)

unread,

Feb 29, 2020, 6:02:03 PM2/29/20

to jenkinsc...@googlegroups.com

Valentin Delaye commented on

Hi!

Same issue here

Plugin : 1.24.1
Core : 2.204.3

 
                                                                2020-02-29 22:57:35.179+0000 [id=30] INFO o.c.j.p.k.KubernetesCloud#addProvisionedSlave: Maximum number of concurrently running agent pods (10) reached for Kubernetes Cloud kubernetes, not provisioning: 23 running or pending in namespace jenkins with Kubernetes labels {jenkins=slave}
2020-02-29 22:57:35.179+0000 [id=30] INFO o.c.j.p.k.KubernetesCloud#provision: Excess workload after pending Kubernetes agents: 1

Thanks

This message was sent by Atlassian Jira (v7.13.12#713012-sha1:6e07c38)

aburdajewicz@cloudbees.com (JIRA)

unread,

Mar 9, 2020, 2:24:03 AM3/9/20

to jenkinsc...@googlegroups.com

Allan BURDAJEWICZ commented on

I have noticed that the limit are better respected when not using the NoDelayProvisioningStrategy (using the system property -Dio.jenkins.plugins.kubernetes.disableNoDelayProvisioning=true)