[JIRA] (JENKINS-55467) Long waiting for kubernetes executor

pastusenko@gmail.com (JIRA)

unread,

Jan 9, 2019, 3:16:04 AM1/9/19

to jenkinsc...@googlegroups.com

Yuri Pastushenko created an issue

Jenkins /

JENKINS-55467

Long waiting for kubernetes executor

Issue Type:	Bug
Assignee:	Carlos Sanchez
Components:	kubernetes-plugin
Created:	2019-01-09 08:15
Environment:	we tried on different versions of jenkins and kubernetes plugin. The behaviour is the same.
Priority:	Major
Reporter:	Yuri Pastushenko

We use kubernetes plugin for running freestyle projects.

We configured default pod template and labeled it, for example "k8s-runner".

We configured a freestyle project and restricted to run this project in "k8s-runner".

After start - the job wait up to 5 minutes if there is already running jobs on this label ("k8s-runner").

If there are no running jobs - job will start faster (1-2 minutes). But this is also very slow.

I made an experiments and understood that the problem is not in kubernetes.

I measured the time and get those results:

00:00 - I press on "Run job". I instantly see new build is created. And the status is "(pending—All nodes of label ‘cd-k8s-ansible’ are offline)"

03:24 - The build status changes to "(pending—-k8s-runner-job-t3f20 is offline)"

03:30 - The build is running.

I kubernetes logs i see that in 03:23 the pod "k8s-runner-job-t3f20" creation started.

My java experience is not so high, for debugging jenkins and kubernetes plugin. I cant find where is blocking place in the code.

We use this plugin for half of year, and by my observation, the jobs are started very slow.

We tried to change jenkins version, upgrade kubernetes plugin, play with configuration of plugin, try other kubernetes cluster.

Nothing helped us.

Add Comment

This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)

pastusenko@gmail.com (JIRA)

unread,

Jan 9, 2019, 3:30:02 AM1/9/19

to jenkinsc...@googlegroups.com

Yuri Pastushenko updated an issue

Jenkins /

JENKINS-55467

Long waiting for kubernetes executor

Change By:	Yuri Pastushenko

We use kubernetes plugin for running freestyle projects.

We configured default pod template and labeled it, for example "k8s-runner".

We configured a freestyle project and restricted to run this project in "k8s-runner".

After start - the job wait up to 5 minutes if there is already running jobs on this label ("k8s-runner").

If there are no running jobs - job will start faster (1-2 minutes). But this is also very slow.

I made an experiments and understood that the problem is not in kubernetes.

I started on long job on this runner (with label k8s-runner), and waited while the build is really started in pod. The pod name was - k8s-runner-job-zz1rv.

After this action i started another job with restriction on same label.

I measured the time of second job and get those results:

00:00 - I press on "Run job". I instantly see new build is created. And the status is "(pending—All nodes of label ‘[cd-k8s-ansible|https://ci.mobbtech.com/label/cd-k8s-ansible]’ are offline)"

00:10 - The build status changes to "(pending—Waiting for next available executor on `k8s-runner-job-zz1rv`)"

03: 01 - The build status changes to"(pending—All nodes of label ‘[cd-k8s-ansible|https://ci.mobbtech.com/label/cd-k8s-ansible]’ are offline)"

03: 24 - The build status changes to "(pending—-k8s-runner-job- t3f20 aq4c2 is offline)"

03:30 - The build is running.

I looked in kubernetes logs and i see that in 03:23 the pod "k8s-runner-job- t3f20 aq4c2 " creation started.

In this measurement we cant notice that jenkins is waiting ~3 minutes for pod thar already executes the code for other build. And after 3 minutes, it decides to create another pod.

By my observations - this time is not fixed. Sometimes is 1 minutes, sometimes 5 minutes.

My java experience is not so high, for debugging jenkins and kubernetes plugin. I cant find where is blocking place in the code.

We use this plugin for half of year, and by my observation, the jobs are started very slow.

We tried to change jenkins version, upgrade kubernetes plugin, play with configuration of plugin, try other kubernetes cluster.

Nothing helped us.

Add Comment

pastusenko@gmail.com (JIRA)

unread,

Jan 9, 2019, 3:35:03 AM1/9/19

to jenkinsc...@googlegroups.com

Yuri Pastushenko updated an issue

Jenkins /

JENKINS-55467

Long waiting for kubernetes executor

Change By:	Yuri Pastushenko

We use kubernetes plugin for running freestyle projects.

We configured default pod template and labeled it, for example "k8s-runner".

We configured a freestyle project and restricted to run this project in "k8s-runner".

After start - the job wait up to 5 minutes if there is already running jobs on this label ("k8s-runner").

If there are no running jobs - job will start faster (1-2 minutes). But this is also very slow.

I made an experiments experiment and understood that the problem is not in kubernetes found out where to start digging .

I started on long job on this runner (with label k8s-runner), and waited while the build is really started in pod. The pod name was - k8s-runner-job-zz1rv.

After this action i started another job with restriction on same label.

I measured the time (mm:ss) of second job and get those results:

00:00 - I press on "Run job". I instantly see new build is created. And the status is "(pending—All nodes of label ‘[cd-k8s-ansible|https://ci.mobbtech.com/label/cd-k8s-ansible]’ are offline)"

00:10 - The build status changes to "(pending—Waiting for next available executor on `k8s-runner-job-zz1rv`)"

03:01 - The build status changes to"(pending—All nodes of label ‘[cd-k8s-ansible|https://ci.mobbtech.com/label/cd-k8s-ansible]’ are offline)"

03:24 - The build status changes to "(pending—-k8s-runner-job-aq4c2 is offline)"

03:30 - The build is running.

I looked in kubernetes logs and i see that in 03:23 the pod "k8s-runner-job-aq4c2" creation started.

In this measurement we cant can notice that jenkins is waiting ~3 minutes for pod thar , that is already executes running the code for other build. And after 3 minutes, it jenkins decides to create another pod.

By my observations - this time is not fixed. Sometimes is 1 minutes, sometimes 5 minutes.

My I tried to debug the code section that is responsible for build running in jenkins and kubernetes plugin, but my java experience is not so high, for debugging jenkins and kubernetes plugin. I cant find where is blocking place in the code.

We use this plugin for half of year, and by my observation, the jobs are started very slow.

We tried to change jenkins version, upgrade kubernetes plugin, play with configuration of plugin, try other kubernetes cluster.

Nothing helped us.

Add Comment

jenkins-ci@carlossanchez.eu (JIRA)

unread,

Jan 9, 2019, 4:19:02 AM1/9/19

to jenkinsc...@googlegroups.com

Carlos Sanchez commented on

JENKINS-55467

Re: Long waiting for kubernetes executor

have you read https://github.com/jenkinsci/kubernetes-plugin/#over-provisioning-flags ?

Add Comment

pastusenko@gmail.com (JIRA)

unread,

Jan 10, 2019, 3:48:02 AM1/10/19

to jenkinsc...@googlegroups.com

Yuri Pastushenko commented on

JENKINS-55467

Re: Long waiting for kubernetes executor

Thank you a lot.

Ive tried today this provisioning flags.

The results are better. Bot not yet as expected.

I started 10 builds at 00:00.

00:35 started build #1;

00:47 started builds #2 and #3;

01:09 started builds #4, #5, #6;

01:27 started builds #7, #8, #9, #10.

The best way will be if all builds will start at one time (in my experiment - when first build starts).

Add Comment

jenkins-ci@carlossanchez.eu (JIRA)

unread,

Jan 10, 2019, 4:00:02 AM1/10/19

to jenkinsc...@googlegroups.com

Carlos Sanchez commented on

JENKINS-55467

Re: Long waiting for kubernetes executor

This is all based on how the jenkins cloud api spawns agents based on its algorithm. There's nothing the plugin can do until it is notified that an agent is needed.
There should be log entries that show when the plugin is called