[JIRA] (JENKINS-36263) Kubernetes pod deleted before BuildWrapper cleanup

1 view
Skip to first unread message

david.joaquim@gmail.com (JIRA)

unread,
Jun 28, 2016, 5:01:02 AM6/28/16
to jenkinsc...@googlegroups.com
David Joaquim created an issue
 
Jenkins / Bug JENKINS-36263
Kubernetes pod deleted before BuildWrapper cleanup
Issue Type: Bug Bug
Assignee: Carlos Sanchez
Components: kubernetes-plugin
Created: 2016/Jun/28 9:00 AM
Environment: Jenkins 2.3, kubernetes plugin 0.6, Cloudbees Docker Build environment Plugin 1.6.5
Priority: Critical Critical
Reporter: David Joaquim

Hello,

We use kubernetes plugin in conjunction with CloudBees Docker custom build environment Plugin.

There is bad interaction between those plugins generating major resource leak. Indeed we discovered that the Kubernetes Pods are destroyed when the build completed but before build wrappers have a chance to tear down properly. This leads to resource leak on any build wrapper plugin that release its resource during tear down.
In our specific case this mean all the docker containers launched by the custom environment plugin are never destroyed.

We think that the problem is that KubernetesSlave relies on the OnceRetentionStrategy that terminates the slave node on the taskCompleted*() callback. This event seems to be before any hudson.tasks.BuildWrapper.Environment#tearDown(hudson.model.AbstractBuild, hudson.model.BuildListener) can be called.

It was not clear for us how to solve this issue properly. Is there a way to extend hudson.model.listeners.RunListener and lookup/delete the pod from there ?

Best Regards,
David

Add Comment Add Comment
 
This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c)
Atlassian logo

tommywo@o2.pl (JIRA)

unread,
Sep 13, 2016, 5:22:01 AM9/13/16
to jenkinsc...@googlegroups.com
Tomasz Wojtun commented on Bug JENKINS-36263
 
Re: Kubernetes pod deleted before BuildWrapper cleanup

I can confirm that OnceRetentionStrategy terminates the slave node before all of the publishers finish.
Changing it to CloudRetentionStrategy helps, but it is not ideal as then slave can be reused by another build.

tommywo@o2.pl (JIRA)

unread,
Sep 13, 2016, 5:23:02 AM9/13/16
to jenkinsc...@googlegroups.com
Tomasz Wojtun edited a comment on Bug JENKINS-36263
I can confirm that OnceRetentionStrategy terminates the slave node before all of the publishers finish.
Changing it to CloudRetentionStrategy helps, but it is not ideal as then slave can be reused by another build.


kubernetes-plugin-0.8, jenkins 2.21

tommywo@o2.pl (JIRA)

unread,
Sep 16, 2016, 10:21:03 AM9/16/16
to jenkinsc...@googlegroups.com

Terminating a pod in a extension of hudson.model.listeners.RunListener on finalised event also can cause another RunListener to fail.
Maybe keeping slave idle for customizable amount of time, before killing it is a good idea.

jenkins-ci@carlossanchez.eu (JIRA)

unread,
May 25, 2018, 5:25:02 AM5/25/18
to jenkinsc...@googlegroups.com

can you try with the latest version?

This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e)
Atlassian logo

jglick@cloudbees.com (JIRA)

unread,
Jul 16, 2019, 3:44:04 PM7/16/19
to jenkinsc...@googlegroups.com
Jesse Glick assigned an issue to Unassigned
 
Change By: Jesse Glick
Assignee: Carlos Sanchez
This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)
Reply all
Reply to author
Forward
0 new messages