[JIRA] (JENKINS-58306) Kubernetes plugin upgrade breaks ownerReference usage

20 views
Skip to first unread message

deiwin.sarjas@gmail.com (JIRA)

unread,
Jul 2, 2019, 4:10:06 PM7/2/19
to jenkinsc...@googlegroups.com
Deiwin Sarjas created an issue
 
Jenkins / Bug JENKINS-58306
Kubernetes plugin upgrade breaks ownerReference usage
Issue Type: Bug Bug
Assignee: Carlos Sanchez
Components: kubernetes-plugin
Created: 2019-07-02 20:09
Environment: Jenkins 2.176.1
kubernetes-plugin 1.16.2
Priority: Minor Minor
Reporter: Deiwin Sarjas

Background

We create Kubernetes resources (Deployments) from within Kubernetes Pods created with podTemplate. We want these resources to be deleted when the Pod created with podTemplate is deleted. To do this we set ownerReferences for the created resources, using the generated Pod's name and uid. This worked well for us a for a while.

Problem

We upgraded the Kubernetes plugin from 1.15.2 to 1.16.2 and this stopped working. The manually created resources (Deployments) weren't being cleaned up by Kubernetes garbage collection any more. We had to revert the plugin upgrade.

Possible cause

Version 1.16.0 of the plugin upgraded kubernetes-client from 4.1.3 to 4.3.0.

The changelog for kubernetes-client includes only one change within that range that would be likely to cause this change in behavior: Fix #758: Deleting Deployments with `.cascading(true)` creates a new Replica Set.

Possible fix

I don't understand the plugin's code well, but it seems like adding .cascading(true) to this line could return the previous behavior.

Add Comment Add Comment
 
This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)

jglick@cloudbees.com (JIRA)

unread,
Jul 10, 2019, 10:13:02 AM7/10/19
to jenkinsc...@googlegroups.com
Jesse Glick commented on Bug JENKINS-58306
 
Re: Kubernetes plugin upgrade breaks ownerReference usage

Deiwin Sarjas do you have a simple way to reproduce this issue, such as a self-contained demo repository? In particular it is not obvious to me how your build pod is creating resources (I suppose using a service account?) and finding its own name/UID to link to. This seems like a rather unusual use case, certainly not one we normally test. We would ideally be able to write a functional test in kubernetes-plugin that fails in master (in the expected way—derivative resource remains after build completion) but passes when the client dependency is updated to include this proposed fix.

deiwin.sarjas@gmail.com (JIRA)

unread,
Jul 10, 2019, 10:25:03 AM7/10/19
to jenkinsc...@googlegroups.com

Yeah, I think I can put together a simple Jenkinsfile to demonstrate this. I'll post it here when I have one.

deiwin.sarjas@gmail.com (JIRA)

unread,
Jul 10, 2019, 10:28:02 AM7/10/19
to jenkinsc...@googlegroups.com

Just FYI, we use this to create on-deman Selenium clusters. The test pod has a container for the Selenium hub, and then we spawn a whole bunch of browsers. The browsers are garbage collected by k8s after the test finishes and the test pod is cleaned up by kubernetes-plugin.

deiwin.sarjas@gmail.com (JIRA)

unread,
Jul 12, 2019, 2:34:03 AM7/12/19
to jenkinsc...@googlegroups.com

Here's a pipeline snippet that checks for this behavior. I left out the auth things, because they're specific to the k8s cluster. We use a default service account in some cases and a kubectl config mounted from a k8s secret in other cases, for reference.

def kubectlYaml = """\
    apiVersion: v1
    kind: Pod
    spec:
      containers:
        - name: kubectl
          image: bitnami/kubectl:1.12
          stdin: true
          tty: true
          command:
            - '/bin/sh'
          args:
            - -c
            - cat
          env:
            - name: HUB_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: HUB_UID
              valueFrom:
                fieldRef:
                  fieldPath: metadata.uid
            - name: HUB_IP
              valueFrom:
                fieldRef:
                  fieldPath: status.podIP
      securityContext:
        runAsUser: 1000
        fsGroup: 1000
"""

def name = "test-${UUID.randomUUID()}"
podTemplate(
    cloud: '..',
    name: name,
    label: name,
    podRetention: never(),
    idleMinutes: 0,
    yaml: kubectlYaml,
) {
    node(name) {
        container('kubectl') {
            sh('''\
                deploy=$(
                kubectl create -o name -f - <<EOF
                apiVersion: apps/v1
                kind: Deployment
                metadata:
                  name: test-deploy
                  labels:
                    app: test-deploy
                  ownerReferences:
                  - apiVersion: v1
                    kind: Pod
                    controller: true
                    blockOwnerDeletion: true
                    name: $HUB_NAME
                    uid: $HUB_UID
                spec:
                  replicas: 2
                  selector:
                    matchLabels:
                      app: test-deploy
                  template:
                    metadata:
                      labels:
                        app: test-deploy
                    spec:
                      containers:
                      - name: ubuntu
                        image: ubuntu
                        command:
                          - '/bin/sh'
                        args:
                          - -c
                          - 'sleep infinity'
                EOF
                )
                kubectl rollout status "$deploy"
            '''.stripIndent())
        }
    }
}

def newName = "test-${UUID.randomUUID()}"
podTemplate(
    cloud: '..',
    name: newName,
    label: newName,
    podRetention: never(),
    idleMinutes: 0,
    yaml: kubectlYaml,
) {
    node(newName) {
        container('kubectl') {
            try {
                sh('sleep 60 && kubectl get deploy/test-deploy')
                error("Expected the deployment to be deleted, but it wasn't.")
            } catch(e) {
                echo("The deployment was deleted as expected.")
            } finally {
                try {
                    sh('kubectl delete deploy/test-deploy')
                } catch (e) {
                    // Ignore
                }
            }
        }
    }
}

jglick@cloudbees.com (JIRA)

unread,
Jul 15, 2019, 10:45:02 AM7/15/19
to jenkinsc...@googlegroups.com
Jesse Glick started work on Bug JENKINS-58306
 
Change By: Jesse Glick
Status: Open In Progress

jglick@cloudbees.com (JIRA)

unread,
Jul 15, 2019, 10:45:02 AM7/15/19
to jenkinsc...@googlegroups.com
Jesse Glick assigned an issue to Jesse Glick
 
Change By: Jesse Glick
Assignee: Carlos Sanchez Jesse Glick

jglick@cloudbees.com (JIRA)

unread,
Jul 16, 2019, 9:47:03 AM7/16/19
to jenkinsc...@googlegroups.com

vincent@latombe.net (JIRA)

unread,
Jul 16, 2019, 9:47:03 AM7/16/19
to jenkinsc...@googlegroups.com
Change By: Vincent Latombe
Status: In Review Fixed but Unreleased
Resolution: Fixed

jglick@cloudbees.com (JIRA)

unread,
Jul 16, 2019, 9:48:04 AM7/16/19
to jenkinsc...@googlegroups.com

jglick@cloudbees.com (JIRA)

unread,
Jul 16, 2019, 3:03:01 PM7/16/19
to jenkinsc...@googlegroups.com

deiwin.sarjas@gmail.com (JIRA)

unread,
Jul 17, 2019, 3:35:04 AM7/17/19
to jenkinsc...@googlegroups.com
 
Re: Kubernetes plugin upgrade breaks ownerReference usage

Thank you for the quick fix and release!

Reply all
Reply to author
Forward
0 new messages