[JIRA] (JENKINS-56347) Kubernetes plugin provisioning pods twice in 1.14.6

2 views
Skip to first unread message

filip@pytloun.cz (JIRA)

unread,
Mar 1, 2019, 11:54:03 AM3/1/19
to jenkinsc...@googlegroups.com
Filip Pytloun created an issue
 
Jenkins / Bug JENKINS-56347
Kubernetes plugin provisioning pods twice in 1.14.6
Issue Type: Bug Bug
Assignee: Carlos Sanchez
Components: kubernetes-plugin
Created: 2019-03-01 16:53
Priority: Major Major
Reporter: Filip Pytloun

Since 1.14.6 and it's update to kubernetes client 4.1.2, I started to observe weird behavior when plugin provisions two pods for one pod template. Second pod is exactly the same as first one but is spawned exactly 20 seconds after.

I tried version 1.14.5 and it's working well (also tested 1.14.2, 1.14.0).

Also it seems that SocketClosed exceptions caused by timeout on get operation became more frequent:
Caused: io.fabric8.kubernetes.client.KubernetesClientException: Operation: [get] for kind: [Pod] with name: [terraform-deploy-fabric-cluster-cec-az-246-hc9xs-2z6rf] in namespace: [jenkins] failed.
Add Comment Add Comment
 
This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)

filip@pytloun.cz (JIRA)

unread,
Mar 1, 2019, 11:56:02 AM3/1/19
to jenkinsc...@googlegroups.com
Filip Pytloun updated an issue
Change By: Filip Pytloun
Since 1.14.6 and it's update to kubernetes client 4.1.2, I started to observe weird behavior when plugin provisions two pods for one pod template. Second pod is exactly the same as first one but is spawned exactly 20 seconds after.

I tried version 1.14.5 and it's working well (also tested 1.14.2, 1.14.0).

Also it seems that SocketClosed exceptions caused by timeout on get operation became more frequent:
Caused: io.fabric8.kubernetes.client.KubernetesClientException: Operation: [get]  for kind: [Pod]  with name: [terraform-deploy-fabric-cluster-cec-az-246-hc9xs-2z6rf]  in namespace: [jenkins]  failed.
Add Comment Add Comment
 

filip@pytloun.cz (JIRA)

unread,
Mar 7, 2019, 6:26:01 AM3/7/19
to jenkinsc...@googlegroups.com
Filip Pytloun commented on Bug JENKINS-56347
 
Re: Kubernetes plugin provisioning pods twice in 1.14.6

Any update? This is serious issue and is also affecting latest version 1.14.8.

jenkins-ci@carlossanchez.eu (JIRA)

unread,
Mar 7, 2019, 6:51:02 AM3/7/19
to jenkinsc...@googlegroups.com

there's nobody else experiencing that behavior, so unless you can provide definitions, debug logs,... there's not much that can be done

ctryti@gmail.com (JIRA)

unread,
Mar 7, 2019, 8:07:01 AM3/7/19
to jenkinsc...@googlegroups.com

We had the same problem with 1.14.6 and 1.14.8. Each job we ran created one slave, then a second 20 seconds later. This quickly drained the available resources in our namespace.

We also avoided the problem by rolling back to 1.14.5.

Unfortunately we don't have any logs of it right now. But I'll try to find time to reproduce it and get some logs.

ctryti@gmail.com (JIRA)

unread,
Mar 7, 2019, 10:52:03 AM3/7/19
to jenkinsc...@googlegroups.com
Christian Tryti updated an issue
 
Change By: Christian Tryti
Attachment: jenkins.log

ctryti@gmail.com (JIRA)

unread,
Mar 7, 2019, 11:01:02 AM3/7/19
to jenkinsc...@googlegroups.com
 
Re: Kubernetes plugin provisioning pods twice in 1.14.6

Recreated it on Jenkins 2.150.3 with the following plugins (that i think might have any relevance):

  • Kubernetes-plugin: 1.14.8
  • Kubernetes-pipeline-plugin: 1.6
  • Kubernetes-credentials: 0.4.0

Our jobs run declarative pipelines. The following log jenkins.log shows 3 slaves being created.

With kubernetes-plugin 1.14.5, only 1 slave is created for each job.

ctryti@gmail.com (JIRA)

unread,
Mar 8, 2019, 3:39:02 AM3/8/19
to jenkinsc...@googlegroups.com

I can leave that jenkins-instance running, in case you want more logs. Shout out if you need more details about the rest of the setup as well.

jenkins-ci@carlossanchez.eu (JIRA)

unread,
Mar 8, 2019, 3:53:02 AM3/8/19
to jenkinsc...@googlegroups.com

what I see in the logs is that the KubernetesCloud.provision is called 3 times, something that is controlled by Jenkins, not the plugin.
Although it's suspicious that both of you say started happening in 1.14.6

Do you have the settings in https://github.com/jenkinsci/kubernetes-plugin/#over-provisioning-flags ?

ctryti@gmail.com (JIRA)

unread,
Mar 8, 2019, 4:00:03 AM3/8/19
to jenkinsc...@googlegroups.com

Not in that jenkins-instance.

vas.pothala@gmail.com (JIRA)

unread,
Mar 13, 2019, 2:31:03 PM3/13/19
to jenkinsc...@googlegroups.com

We are seeing the same problem as well in version 1.14.8

vas.pothala@gmail.com (JIRA)

unread,
Mar 13, 2019, 2:40:03 PM3/13/19
to jenkinsc...@googlegroups.com
Sriniva Pothala edited a comment on Bug JENKINS-56347
We are seeing the same problem as well in version 1.14.8 but it was intermittent. I am unable to reproduce it when its not happening.

stephan@blackbuild.com (JIRA)

unread,
Mar 28, 2019, 11:52:02 AM3/28/19
to jenkinsc...@googlegroups.com

Same here, downgrading to 1.14.5 solved it for now. Did not yet get around to setting up a test system.

rcruzper@gmail.com (JIRA)

unread,
Mar 29, 2019, 7:31:02 AM3/29/19
to jenkinsc...@googlegroups.com

We have the same problem here, it started to happend once we added this: https://github.com/jenkinsci/kubernetes-plugin/#over-provisioning-flags

Then we removed the flags but it was still happening, so we downgraded to 1.14.5 and now is not happening.

jenkins-ci@carlossanchez.eu (JIRA)

unread,
Apr 17, 2019, 7:09:07 AM4/17/19
to jenkinsc...@googlegroups.com

Can you try the PR in https://github.com/jenkinsci/kubernetes-plugin/pull/455 and post the logs ?

jenkins-ci@carlossanchez.eu (JIRA)

unread,
Apr 17, 2019, 7:09:08 AM4/17/19
to jenkinsc...@googlegroups.com
Carlos Sanchez started work on Bug JENKINS-56347
 
Change By: Carlos Sanchez
Status: Open In Progress

dtgames@kinggeek.org (JIRA)

unread,
Apr 30, 2019, 4:54:02 PM4/30/19
to jenkinsc...@googlegroups.com

Same issue here on 1.15.1 and 1.15.2 will see if I can try the PR and report back.

dtgames@kinggeek.org (JIRA)

unread,
Apr 30, 2019, 4:55:02 PM4/30/19
to jenkinsc...@googlegroups.com
Dax Games edited a comment on Bug JENKINS-56347
Same issue here on 1.15.1 and 1.15.2 will see if I can try the PR and report back.   I am seeing a slave spin every 10 seconds until one connects.

dtgames@kinggeek.org (JIRA)

unread,
May 1, 2019, 10:26:08 AM5/1/19
to jenkinsc...@googlegroups.com

Logs from PR in https://github.com/jenkinsci/kubernetes-plugin/pull/455.

 

This is a build of a single job that spins up 8 slaves:

May 01, 2019 2:13:50 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud provision
INFO: Excess workload after pending Kubernetes agents: 1
May 01, 2019 2:13:50 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud provision
INFO: Template for label pd-slave: Kubernetes Pod Template
May 01, 2019 2:13:50 PM okhttp3.internal.platform.Platform log
INFO: ALPN callback dropped: HTTP/2 is disabled. Is alpn-boot on the boot class path?
May 01, 2019 2:13:50 PM hudson.slaves.NodeProvisioner$StandardStrategyImpl apply
INFO: Started provisioning Kubernetes Pod Template from kubernetes with 1 executors. Remaining excess workload: 0
May 01, 2019 2:14:00 PM hudson.slaves.NodeProvisioner$2 run
INFO: Kubernetes Pod Template provisioning successfully completed. We have now 2 computer(s)
May 01, 2019 2:14:00 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch
INFO: Created Pod: pd-compute-platform/pd-slave-zcf1g
May 01, 2019 2:14:00 PM okhttp3.internal.platform.Platform log
INFO: ALPN callback dropped: HTTP/2 is disabled. Is alpn-boot on the boot class path?
May 01, 2019 2:14:02 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch
INFO: Pod is running: pd-compute-platform/pd-slave-zcf1g
May 01, 2019 2:14:10 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud provision
INFO: Excess workload after pending Kubernetes agents: 1
May 01, 2019 2:14:10 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud provision
INFO: Template for label pd-slave: Kubernetes Pod Template
May 01, 2019 2:14:10 PM hudson.slaves.NodeProvisioner$StandardStrategyImpl apply
INFO: Started provisioning Kubernetes Pod Template from kubernetes with 1 executors. Remaining excess workload: 0
May 01, 2019 2:14:20 PM hudson.slaves.NodeProvisioner$2 run
INFO: Kubernetes Pod Template provisioning successfully completed. We have now 3 computer(s)
May 01, 2019 2:14:20 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch
INFO: Created Pod: pd-compute-platform/pd-slave-txmw8
May 01, 2019 2:14:20 PM okhttp3.internal.platform.Platform log
INFO: ALPN callback dropped: HTTP/2 is disabled. Is alpn-boot on the boot class path?
May 01, 2019 2:14:21 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch
INFO: Pod is running: pd-compute-platform/pd-slave-txmw8
May 01, 2019 2:14:30 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud provision
INFO: Excess workload after pending Kubernetes agents: 1
May 01, 2019 2:14:30 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud provision
INFO: Template for label pd-slave: Kubernetes Pod Template
May 01, 2019 2:14:30 PM hudson.slaves.NodeProvisioner$StandardStrategyImpl apply
INFO: Started provisioning Kubernetes Pod Template from kubernetes with 1 executors. Remaining excess workload: 0
May 01, 2019 2:14:40 PM hudson.slaves.NodeProvisioner$2 run
INFO: Kubernetes Pod Template provisioning successfully completed. We have now 4 computer(s)
May 01, 2019 2:14:40 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch
INFO: Created Pod: pd-compute-platform/pd-slave-61lwt
May 01, 2019 2:14:40 PM okhttp3.internal.platform.Platform log
INFO: ALPN callback dropped: HTTP/2 is disabled. Is alpn-boot on the boot class path?
May 01, 2019 2:14:41 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch
INFO: Pod is running: pd-compute-platform/pd-slave-61lwt
May 01, 2019 2:14:50 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud provision
INFO: Excess workload after pending Kubernetes agents: 1
May 01, 2019 2:14:50 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud provision
INFO: Template for label pd-slave: Kubernetes Pod Template
May 01, 2019 2:14:50 PM hudson.slaves.NodeProvisioner$StandardStrategyImpl apply
INFO: Started provisioning Kubernetes Pod Template from kubernetes with 1 executors. Remaining excess workload: 0
May 01, 2019 2:15:00 PM hudson.slaves.NodeProvisioner$2 run
INFO: Kubernetes Pod Template provisioning successfully completed. We have now 5 computer(s)
May 01, 2019 2:15:00 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud provision
INFO: Excess workload after pending Kubernetes agents: 0
May 01, 2019 2:15:00 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud provision
INFO: Template for label pd-slave: Kubernetes Pod Template
May 01, 2019 2:15:00 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch
INFO: Created Pod: pd-compute-platform/pd-slave-x2gjp
May 01, 2019 2:15:00 PM okhttp3.internal.platform.Platform log
INFO: ALPN callback dropped: HTTP/2 is disabled. Is alpn-boot on the boot class path?
May 01, 2019 2:15:02 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch
INFO: Pod is running: pd-compute-platform/pd-slave-x2gjp
May 01, 2019 2:15:10 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud provision
INFO: Excess workload after pending Kubernetes agents: 1
May 01, 2019 2:15:10 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud provision
INFO: Template for label pd-slave: Kubernetes Pod Template
May 01, 2019 2:15:10 PM hudson.slaves.NodeProvisioner$StandardStrategyImpl apply
INFO: Started provisioning Kubernetes Pod Template from kubernetes with 1 executors. Remaining excess workload: 0
May 01, 2019 2:15:20 PM hudson.slaves.NodeProvisioner$2 run
INFO: Kubernetes Pod Template provisioning successfully completed. We have now 6 computer(s)
May 01, 2019 2:15:20 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud provision
INFO: Excess workload after pending Kubernetes agents: 0
May 01, 2019 2:15:20 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud provision
INFO: Template for label pd-slave: Kubernetes Pod Template
May 01, 2019 2:15:20 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch
INFO: Created Pod: pd-compute-platform/pd-slave-8n2hg
May 01, 2019 2:15:20 PM okhttp3.internal.platform.Platform log
INFO: ALPN callback dropped: HTTP/2 is disabled. Is alpn-boot on the boot class path?
May 01, 2019 2:15:22 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch
INFO: Pod is running: pd-compute-platform/pd-slave-8n2hg
May 01, 2019 2:15:30 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud provision
INFO: Excess workload after pending Kubernetes agents: 1
May 01, 2019 2:15:30 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud provision
INFO: Template for label pd-slave: Kubernetes Pod Template
May 01, 2019 2:15:30 PM hudson.slaves.NodeProvisioner$StandardStrategyImpl apply
INFO: Started provisioning Kubernetes Pod Template from kubernetes with 1 executors. Remaining excess workload: 0
May 01, 2019 2:15:40 PM hudson.slaves.NodeProvisioner$2 run
INFO: Kubernetes Pod Template provisioning successfully completed. We have now 7 computer(s)
May 01, 2019 2:15:40 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud provision
INFO: Excess workload after pending Kubernetes agents: 0
May 01, 2019 2:15:40 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud provision
INFO: Template for label pd-slave: Kubernetes Pod Template
May 01, 2019 2:15:40 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch
INFO: Created Pod: pd-compute-platform/pd-slave-lpv1t
May 01, 2019 2:15:40 PM okhttp3.internal.platform.Platform log
INFO: ALPN callback dropped: HTTP/2 is disabled. Is alpn-boot on the boot class path?
May 01, 2019 2:15:42 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch
INFO: Pod is running: pd-compute-platform/pd-slave-lpv1t
May 01, 2019 2:15:50 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud provision
INFO: Excess workload after pending Kubernetes agents: 1
May 01, 2019 2:15:50 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud provision
INFO: Template for label pd-slave: Kubernetes Pod Template
May 01, 2019 2:15:50 PM hudson.slaves.NodeProvisioner$StandardStrategyImpl apply
INFO: Started provisioning Kubernetes Pod Template from kubernetes with 1 executors. Remaining excess workload: 0
May 01, 2019 2:16:00 PM hudson.slaves.NodeProvisioner$2 run
INFO: Kubernetes Pod Template provisioning successfully completed. We have now 8 computer(s)
May 01, 2019 2:16:00 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud provision
INFO: Excess workload after pending Kubernetes agents: 0
May 01, 2019 2:16:00 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud provision
INFO: Template for label pd-slave: Kubernetes Pod Template
May 01, 2019 2:16:00 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch
INFO: Created Pod: pd-compute-platform/pd-slave-jpd7x
May 01, 2019 2:16:00 PM okhttp3.internal.platform.Platform log
INFO: ALPN callback dropped: HTTP/2 is disabled. Is alpn-boot on the boot class path?
May 01, 2019 2:16:04 PM hudson.TcpSlaveAgentListener$ConnectionHandler run
INFO: Accepted JNLP4-connect connection #1 from /10.234.12.72:32860
May 01, 2019 2:16:05 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch
INFO: Pod is running: pd-compute-platform/pd-slave-jpd7x
May 01, 2019 2:16:22 PM hudson.TcpSlaveAgentListener$ConnectionHandler run
INFO: Accepted JNLP4-connect connection #2 from /10.234.12.73:49150

 

 

dtgames@kinggeek.org (JIRA)

unread,
May 1, 2019, 10:44:03 AM5/1/19
to jenkinsc...@googlegroups.com

As a comparison this is from 1.14.5 plugin:

 

May 01, 2019 2:36:14 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud provision
INFO: Excess workload after pending Kubernetes agents: 1
May 01, 2019 2:36:14 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud provision
INFO: Template for label pd-slave: Kubernetes Pod Template
May 01, 2019 2:36:14 PM okhttp3.internal.platform.Platform log
INFO: ALPN callback dropped: HTTP/2 is disabled. Is alpn-boot on the boot class path?
May 01, 2019 2:36:14 PM hudson.slaves.NodeProvisioner$StandardStrategyImpl apply
INFO: Started provisioning Kubernetes Pod Template from kubernetes with 1 executors. Remaining excess workload: 0
May 01, 2019 2:36:24 PM hudson.slaves.NodeProvisioner$2 run
INFO: Kubernetes Pod Template provisioning successfully completed. We have now 2 computer(s)
May 01, 2019 2:36:24 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch
INFO: Created Pod: pd-slave-12gpw in namespace pd-pipedream
May 01, 2019 2:36:24 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch
INFO: Waiting for Pod to be scheduled (0/600): pd-slave-12gpw
May 01, 2019 2:36:25 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch
INFO: Container is waiting pd-slave-12gpw [jnlp]: ContainerStateWaiting(message=null, reason=ContainerCreating, additionalProperties={})
May 01, 2019 2:36:25 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch
INFO: Waiting for Pod to be scheduled (1/600): pd-slave-12gpw
May 01, 2019 2:36:26 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch
INFO: Waiting for agent to connect (0/250): pd-slave-12gpw
May 01, 2019 2:36:27 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch
INFO: Waiting for agent to connect (1/250): pd-slave-12gpw
May 01, 2019 2:36:28 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch
INFO: Waiting for agent to connect (2/250): pd-slave-12gpw
May 01, 2019 2:36:29 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch
INFO: Waiting for agent to connect (3/250): pd-slave-12gpw
May 01, 2019 2:36:30 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch
INFO: Waiting for agent to connect (4/250): pd-slave-12gpw
May 01, 2019 2:36:31 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch
INFO: Waiting for agent to connect (5/250): pd-slave-12gpw
May 01, 2019 2:36:32 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch
INFO: Waiting for agent to connect (6/250): pd-slave-12gpw
May 01, 2019 2:36:33 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch
INFO: Waiting for agent to connect (7/250): pd-slave-12gpw
May 01, 2019 2:36:34 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch
INFO: Waiting for agent to connect (8/250): pd-slave-12gpw
May 01, 2019 2:36:35 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch
INFO: Waiting for agent to connect (9/250): pd-slave-12gpw
May 01, 2019 2:36:36 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch
INFO: Waiting for agent to connect (10/250): pd-slave-12gpw
May 01, 2019 2:36:37 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch
INFO: Waiting for agent to connect (11/250): pd-slave-12gpw
May 01, 2019 2:36:38 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch
INFO: Waiting for agent to connect (12/250): pd-slave-12gpw
May 01, 2019 2:36:39 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch
INFO: Waiting for agent to connect (13/250): pd-slave-12gpw
May 01, 2019 2:36:40 PM hudson.model.AsyncPeriodicWork$1 run
INFO: Started DockerContainerWatchdog Asynchronous Periodic Work
May 01, 2019 2:36:40 PM com.nirima.jenkins.plugins.docker.DockerContainerWatchdog execute
INFO: Docker Container Watchdog has been triggered
May 01, 2019 2:36:40 PM com.nirima.jenkins.plugins.docker.DockerContainerWatchdog$Statistics writeStatisticsToLog
INFO: Watchdog Statistics: Number of overall executions: 1, Executions with processing timeout: 0, Containers removed gracefully: 0, Containers removed with force: 0, Containers removal failed: 0, Nodes removed successfully: 0, Nodes removal failed: 0, Container removal
 average duration (gracefully): 0 ms, Container removal average duration (force): 0 ms, Average overall runtime of watchdog: 0 ms, Average runtime of container retrieval: 0 ms
May 01, 2019 2:36:40 PM com.nirima.jenkins.plugins.docker.DockerContainerWatchdog loadNodeMap
INFO: We currently have 1 nodes assigned to this Jenkins instance, which we will check
May 01, 2019 2:36:40 PM com.nirima.jenkins.plugins.docker.DockerContainerWatchdog execute
INFO: Docker Container Watchdog check has been completed
May 01, 2019 2:36:40 PM hudson.model.AsyncPeriodicWork$1 run
INFO: Finished DockerContainerWatchdog Asynchronous Periodic Work. 2 ms
May 01, 2019 2:36:40 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch
INFO: Waiting for agent to connect (14/250): pd-slave-12gpw
May 01, 2019 2:36:41 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch
INFO: Waiting for agent to connect (15/250): pd-slave-12gpw
May 01, 2019 2:36:42 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch
INFO: Waiting for agent to connect (16/250): pd-slave-12gpw
May 01, 2019 2:36:43 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch
INFO: Waiting for agent to connect (17/250): pd-slave-12gpw
May 01, 2019 2:36:44 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch
INFO: Waiting for agent to connect (18/250): pd-slave-12gpw
May 01, 2019 2:36:45 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch
INFO: Waiting for agent to connect (19/250): pd-slave-12gpw
May 01, 2019 2:36:46 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch
INFO: Waiting for agent to connect (20/250): pd-slave-12gpw
May 01, 2019 2:36:47 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch
Omitted for brevity ...
INFO: Waiting for agent to connect (111/250): pd-slave-12gpw
May 01, 2019 2:38:19 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch
INFO: Waiting for agent to connect (112/250): pd-slave-12gpw
May 01, 2019 2:38:20 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch
INFO: Waiting for agent to connect (113/250): pd-slave-12gpw
May 01, 2019 2:38:20 PM hudson.TcpSlaveAgentListener$ConnectionHandler run
INFO: Accepted JNLP4-connect connection #1 from /10.234.12.76:49588
May 01, 2019 2:38:21 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch
INFO: Waiting for agent to connect (114/250): pd-slave-12gpw
May 01, 2019 2:38:22 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch
INFO: Waiting for agent to connect (115/250): pd-slave-12gpw
May 01, 2019 2:38:23 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch
INFO: Waiting for agent to connect (116/250): pd-slave-12gpw
May 01, 2019 2:38:42 PM org.jenkinsci.remoting.util.AnonymousClassWarnings warn

dtgames@kinggeek.org (JIRA)

unread,
May 1, 2019, 10:46:03 AM5/1/19
to jenkinsc...@googlegroups.com
Dax Games edited a comment on Bug JENKINS-56347
As a comparison this is from 1.14.5 plugin:

 
{code:java}

 Omitted for brevity ...
May 01, 2019 2:38:42 PM org.jenkinsci.remoting.util.AnonymousClassWarnings warn{code}

dtgames@kinggeek.org (JIRA)

unread,
May 1, 2019, 12:51:22 PM5/1/19
to jenkinsc...@googlegroups.com

It looks like the plugin is only waiting for the pod to be running and not waiting for it to actually connect back to jenkins.  The working plugin waits for the pod to connect back to Jenkins.

 

dtgames@kinggeek.org (JIRA)

unread,
May 1, 2019, 1:56:01 PM5/1/19
to jenkinsc...@googlegroups.com

It looks like there needs to be a wait for a connect back to Jenkins here

Like the working code here

 

But I am certainly no Java expert.

dtgames@kinggeek.org (JIRA)

unread,
May 1, 2019, 1:58:03 PM5/1/19
to jenkinsc...@googlegroups.com
Dax Games edited a comment on Bug JENKINS-56347

 

But I am certainly no Java expert.

dtgames@kinggeek.org (JIRA)

unread,
May 1, 2019, 4:17:02 PM5/1/19
to jenkinsc...@googlegroups.com

The commit that introduced this? 

dtgames@kinggeek.org (JIRA)

unread,
May 1, 2019, 4:18:02 PM5/1/19
to jenkinsc...@googlegroups.com
Dax Games edited a comment on Bug JENKINS-56347

dtgames@kinggeek.org (JIRA)

unread,
May 1, 2019, 4:19:03 PM5/1/19
to jenkinsc...@googlegroups.com

dtgames@kinggeek.org (JIRA)

unread,
May 1, 2019, 4:21:02 PM5/1/19
to jenkinsc...@googlegroups.com
Dax Games edited a comment on Bug JENKINS-56347

jenkins-ci@carlossanchez.eu (JIRA)

unread,
May 1, 2019, 5:11:06 PM5/1/19
to jenkinsc...@googlegroups.com
Change By: Carlos Sanchez
Status: In Progress Review

jenkins-ci@carlossanchez.eu (JIRA)

unread,
May 1, 2019, 5:19:02 PM5/1/19
to jenkinsc...@googlegroups.com
Carlos Sanchez commented on Bug JENKINS-56347
 
Re: Kubernetes plugin provisioning pods twice in 1.14.6

thanks for the diagnosis, that seems to be the cause. I have updated the PR at https://github.com/jenkinsci/kubernetes-plugin/pull/455

dtgames@kinggeek.org (JIRA)

unread,
May 2, 2019, 9:22:07 AM5/2/19
to jenkinsc...@googlegroups.com

No problem, just trying to help.

That fixes it!

May 02, 2019 1:18:54 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud provision
INFO: Excess workload after pending Kubernetes agents: 1
May 02, 2019 1:18:54 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud provision
INFO: Template for label pd-slave: Kubernetes Pod Template
May 02, 2019 1:18:54 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesClientProvider createClient
INFO: Created new Kubernetes client: kubernetes io.fabric8.kubernetes.client.DefaultKubernetesClient@5339d085
May 02, 2019 1:18:54 PM okhttp3.internal.platform.Platform log
INFO: ALPN callback dropped: HTTP/2 is disabled. Is alpn-boot on the boot class path?
May 02, 2019 1:18:54 PM hudson.slaves.NodeProvisioner$StandardStrategyImpl apply
INFO: Started provisioning Kubernetes Pod Template from kubernetes with 1 executors. Remaining excess workload: 0
May 02, 2019 1:19:04 PM hudson.slaves.NodeProvisioner$2 run
INFO: Kubernetes Pod Template provisioning successfully completed. We have now 2 computer(s)
May 02, 2019 1:19:04 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch
INFO: Created Pod: pd-pipedream/pd-slave-g6cwh
May 02, 2019 1:19:04 PM okhttp3.internal.platform.Platform log
INFO: ALPN callback dropped: HTTP/2 is disabled. Is alpn-boot on the boot class path?
May 02, 2019 1:19:07 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch
INFO: Pod is running: pd-pipedream/pd-slave-g6cwh
May 02, 2019 1:19:07 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch
INFO: Waiting for agent to connect (0/250): pd-slave-g6cwh
May 02, 2019 1:19:08 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch
INFO: Waiting for agent to connect (1/250): pd-slave-g6cwh
May 02, 2019 1:19:09 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch
INFO: Waiting for agent to connect (2/250): pd-slave-g6cwh

dtgames@kinggeek.org (JIRA)

unread,
May 2, 2019, 9:23:03 AM5/2/19
to jenkinsc...@googlegroups.com
Dax Games edited a comment on Bug JENKINS-56347
No problem, just trying to help.

That fixes it!


{code:java}

May 02, 2019 1:18:54 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud provision
INFO: Excess workload after pending Kubernetes agents: 1
May 02, 2019 1:18:54 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud provision
INFO: Template for label pd-slave: Kubernetes Pod Template
May 02, 2019 1:18:54 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesClientProvider createClient
INFO: Created new Kubernetes client: kubernetes io.fabric8.kubernetes.client.DefaultKubernetesClient@5339d085
May 02, 2019 1:18:54 PM okhttp3.internal.platform.Platform log
INFO: ALPN callback dropped: HTTP/2 is disabled. Is alpn-boot on the boot class path?
May 02, 2019 1:18:54 PM hudson.slaves.NodeProvisioner$StandardStrategyImpl apply
INFO: Started provisioning Kubernetes Pod Template from kubernetes with 1 executors. Remaining excess workload: 0
May 02, 2019 1:19:04 PM hudson.slaves.NodeProvisioner$2 run
INFO: Kubernetes Pod Template provisioning successfully completed. We have now 2 computer(s)
May 02, 2019 1:19:04 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch
INFO: Created Pod: pd-pipedream/pd-slave-g6cwh
May 02, 2019 1:19:04 PM okhttp3.internal.platform.Platform log
INFO: ALPN callback dropped: HTTP/2 is disabled. Is alpn-boot on the boot class path?
May 02, 2019 1:19:07 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch
INFO: Pod is running: pd-pipedream/pd-slave-g6cwh
May 02, 2019 1:19:07 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch
INFO: Waiting for agent to connect (0/250): pd-slave-g6cwh
May 02, 2019 1:19:08 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch
INFO: Waiting for agent to connect (1/250): pd-slave-g6cwh
May 02, 2019 1:19:09 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch
INFO: Waiting for agent to connect (2/250): pd-slave-g6cwh

{code}

Any idea how long before a new version is released?

jenkins-ci@carlossanchez.eu (JIRA)

unread,
May 7, 2019, 8:22:05 AM5/7/19
to jenkinsc...@googlegroups.com
Change By: Carlos Sanchez
Status: In Review Resolved
Resolution: Fixed
Reply all
Reply to author
Forward
0 new messages