[JIRA] (JENKINS-57116) Version 1.15.1 does not create pods

171 views
Skip to first unread message

hypery2k@web.de (JIRA)

unread,
Apr 19, 2019, 5:43:02 AM4/19/19
to jenkinsc...@googlegroups.com
Martin Reinhardt created an issue
 
Jenkins / Bug JENKINS-57116
Version 1.15.1 does not create pods
Issue Type: Bug Bug
Assignee: Carlos Sanchez
Components: kubernetes-plugin
Created: 2019-04-19 09:42
Priority: Minor Minor
Reporter: Martin Reinhardt

After upgrading to version 1.15.1 we're unable to create pods with this error:

WARNING: Error in provisioning; agent=KubernetesSlave name: jdk8-rvt2j, template=PodTemplate{inheritFrom='', name='jdk8', namespace='', instanceCap=8, idleMinutes=120, label='java java8', serviceAccount='jenkins-slaves', nodeSelector='', nodeUsageMode=NORMAL, workspaceVolume=EmptyDirWorkspaceVolume [memory=false], volumes=[SecretVolume [mountPath=/home/jenkins/nexus-jenkins-auth, secretName=nexus-jenkins-auth, defaultMode=null], SecretVolume [mountPath=/home/jenkins/sonar-jenkins-auth, secretName=sonar-jenkins-auth, defaultMode=null], EmptyDirVolume [mountPath=/tmp, memory=false]], containers=[ContainerTemplate{name='jnlp', image='docker-registry.default.svc:5000/ci-devtools/jenkins-slave-java8', alwaysPullImage=true, workingDir='/tmp', command='', args='${computer.jnlpmac} ${computer.name}', ttyEnabled=true, resourceRequestCpu='1', resourceRequestMemory='2Gi', resourceLimitCpu='1', resourceLimitMemory='4Gi', livenessProbe=org.csanchez.jenkins.plugins.kubernetes.ContainerLivenessProbe@23dd88f}], envVars=[KeyValueEnvVar [getValue()=/home/jenkins, getKey()=HOME], KeyValueEnvVar [getValue()=-Xmx1024m, getKey()=JAVA_OPTS], KeyValueEnvVar [getValue()=-Duser.timezone=Europe/Berlin -Djava.security.egd=file:/dev/urandom -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap -Xmx1024m -XX:+HeapDumpOnOutOfMemoryError -Dfile.encoding=UTF-8, getKey()=JAVA_TOOL_OPTIONS]]}
io.fabric8.kubernetes.client.KubernetesClientException: An error has occurred.
	at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:64)
	at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:53)
	at io.fabric8.kubernetes.client.utils.Serialization.unmarshal(Serialization.java:238)
	at io.fabric8.kubernetes.client.utils.Serialization.unmarshal(Serialization.java:191)
	at io.fabric8.kubernetes.client.utils.Serialization.unmarshal(Serialization.java:178)
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.unmarshal(OperationSupport.java:495)
	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.load(BaseOperation.java:331)
	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.load(BaseOperation.java:67)
	at org.csanchez.jenkins.plugins.kubernetes.PodTemplateUtils.parseFromYaml(PodTemplateUtils.java:509)
	at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
	at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382)
	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
	at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
	at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
	at org.csanchez.jenkins.plugins.kubernetes.PodTemplateBuilder.build(PodTemplateBuilder.java:201)
	at org.csanchez.jenkins.plugins.kubernetes.PodTemplate.build(PodTemplate.java:719)
	at org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher.launch(KubernetesLauncher.java:103)
	at hudson.slaves.SlaveComputer$1.call(SlaveComputer.java:294)
	at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
	at jenkins.security.ImpersonatingExecutorService$2.call(ImpersonatingExecutorService.java:71)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: com.fasterxml.jackson.databind.exc.MismatchedInputException: No content to map due to end-of-input
 at [Source: (BufferedInputStream); line: 1, column: 1]
	at com.fasterxml.jackson.databind.exc.MismatchedInputException.from(MismatchedInputException.java:59)
	at com.fasterxml.jackson.databind.ObjectMapper._initForReading(ObjectMapper.java:4145)
	at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4000)
	at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3077)
	at io.fabric8.kubernetes.client.utils.Serialization.unmarshal(Serialization.java:236)
	... 23 more

Switching back to 1.14.9 and works like charme
Add Comment Add Comment
 
This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)

jenkins-ci@carlossanchez.eu (JIRA)

unread,
Apr 19, 2019, 5:54:02 AM4/19/19
to jenkinsc...@googlegroups.com
Carlos Sanchez commented on Bug JENKINS-57116
 
Re: Version 1.15.1 does not create pods

have you tried 1.15.0 to narrow it down?

hypery2k@web.de (JIRA)

unread,
Apr 19, 2019, 6:48:01 AM4/19/19
to jenkinsc...@googlegroups.com

With 1.15.0 it's also working

rick@ricktbaker.com (JIRA)

unread,
Apr 19, 2019, 12:09:02 PM4/19/19
to jenkinsc...@googlegroups.com

I'm running jenkins within kubernetes and running into the same issue where the executor/slave pod is never started after upgrading  to 1.15.1.  Through the plugin manager I don't see an option for 1.15.0, I can only downgrade to 1.13.8, which I've tried but doesn't appear to work with the latest version of jenkins from the helm chart.

jenkins-ci@carlossanchez.eu (JIRA)

unread,
Apr 22, 2019, 6:08:03 AM4/22/19
to jenkinsc...@googlegroups.com

this must be caused by the same issue as JENKINS-57112 because it's the only change in 1.5.1

Can you provide your pod definition? is it using yaml ?

jenkins-ci@carlossanchez.eu (JIRA)

unread,
Apr 22, 2019, 6:13:01 AM4/22/19
to jenkinsc...@googlegroups.com
Carlos Sanchez edited a comment on Bug JENKINS-57116
this must be caused by the same issue as JENKINS-57112 because it's the only change in 1. 5 15 .1


Can you provide your pod definition? is it using yaml ?

jenkins-ci@carlossanchez.eu (JIRA)

unread,
Apr 22, 2019, 6:17:01 AM4/22/19
to jenkinsc...@googlegroups.com

try 1.15.2 just released

contact@martinreinhardt-online.de (JIRA)

unread,
Apr 22, 2019, 7:11:02 AM4/22/19
to jenkinsc...@googlegroups.com
Martin Reinhardt assigned an issue to Martin Reinhardt
 
Change By: Martin Reinhardt
Assignee: Carlos Sanchez Martin Reinhardt

contact@martinreinhardt-online.de (JIRA)

unread,
Apr 22, 2019, 7:11:03 AM4/22/19
to jenkinsc...@googlegroups.com
Martin Reinhardt started work on Bug JENKINS-57116
 
Change By: Martin Reinhardt
Status: Open In Progress

contact@martinreinhardt-online.de (JIRA)

unread,
Apr 22, 2019, 7:11:04 AM4/22/19
to jenkinsc...@googlegroups.com
Change By: Martin Reinhardt
Status: In Progress Review

contact@martinreinhardt-online.de (JIRA)

unread,
Apr 22, 2019, 7:12:02 AM4/22/19
to jenkinsc...@googlegroups.com
Martin Reinhardt commented on Bug JENKINS-57116
 
Re: Version 1.15.1 does not create pods

will give it tomorrow a try and provide feedback.

Thanks for the fast fix

eric_fontana@bose.com (JIRA)

unread,
Apr 22, 2019, 8:28:02 AM4/22/19
to jenkinsc...@googlegroups.com

I just tried 1.15.2 and it is broken for me, had to roll back 

eric_fontana@bose.com (JIRA)

unread,
Apr 22, 2019, 8:30:02 AM4/22/19
to jenkinsc...@googlegroups.com
Eric Fontana edited a comment on Bug JENKINS-57116
I just tried 1.15. 2 1 and it is broken for me, had to roll back 

eric_fontana@bose.com (JIRA)

unread,
Apr 22, 2019, 8:30:03 AM4/22/19
to jenkinsc...@googlegroups.com
Eric Fontana edited a comment on Bug JENKINS-57116
I just tried 1.15.1 and it is broken for me, had to roll back 

 

I don't see 1.15.2 as an option yet?

hypery2k@web.de (JIRA)

unread,
Apr 23, 2019, 11:05:02 PM4/23/19
to jenkinsc...@googlegroups.com

actually with 1.15.2 it's still not working:

WARNING: Error in provisioning; agent=KubernetesSlave name: nodejs-h53r0, template=PodTemplate{inheritFrom='', name='nodejs', namespace='', instanceCap=2, idleMinutes=120, label='nodejs nvm', serviceAccount='jenkins-slaves', nodeSelector='', nodeUsageMode=NORMAL, workspaceVolume=EmptyDirWorkspaceVolume [memory=false], volumes=[SecretVolume [mountPath=/home/jenkins/nexus-jenkins-auth, secretName=nexus-jenkins-auth, defaultMode=null], SecretVolume [mountPath=/home/jenkins/sonar-jenkins-auth, secretName=sonar-jenkins-auth, defaultMode=null], EmptyDirVolume [mountPath=/tmp, memory=false]], containers=[ContainerTemplate{name='jnlp', image='docker-registry.default.svc:5000/ci-devtools/jenkins-slave-nodejs', alwaysPullImage=true, workingDir='/tmp', command='', args='${computer.jnlpmac} ${computer.name}', ttyEnabled=true, resourceRequestCpu='200m', resourceRequestMemory='2Gi', resourceLimitCpu='1', resourceLimitMemory='4Gi', envVars=[KeyValueEnvVar [getValue()=/home/jenkins, getKey()=HOME]], livenessProbe=org.csanchez.jenkins.plugins.kubernetes.ContainerLivenessProbe@c599987}], yamls=[]}
io.fabric8.kubernetes.client.KubernetesClientException: An error has occurred.
	at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:64)
	at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:53)
	at io.fabric8.kubernetes.client.utils.Serialization.unmarshal(Serialization.java:238)
	at io.fabric8.kubernetes.client.utils.Serialization.unmarshal(Serialization.java:191)
	at io.fabric8.kubernetes.client.utils.Serialization.unmarshal(Serialization.java:178)
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.unmarshal(OperationSupport.java:495)
	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.load(BaseOperation.java:331)
	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.load(BaseOperation.java:67)
	at org.csanchez.jenkins.plugins.kubernetes.PodTemplateUtils.parseFromYaml(PodTemplateUtils.java:509)
	at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
	at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382)
	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
	at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
	at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
	at org.csanchez.jenkins.plugins.kubernetes.PodTemplateBuilder.build(PodTemplateBuilder.java:201)
	at org.csanchez.jenkins.plugins.kubernetes.PodTemplate.build(PodTemplate.java:718)
	at org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher.launch(KubernetesLauncher.java:103)
	at hudson.slaves.SlaveComputer$1.call(SlaveComputer.java:294)
	at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
	at jenkins.security.ImpersonatingExecutorService$2.call(ImpersonatingExecutorService.java:71)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: com.fasterxml.jackson.databind.exc.MismatchedInputException: No content to map due to end-of-input
 at [Source: (BufferedInputStream); line: 1, column: 1]
	at com.fasterxml.jackson.databind.exc.MismatchedInputException.from(MismatchedInputException.java:59)
	at com.fasterxml.jackson.databind.ObjectMapper._initForReading(ObjectMapper.java:4145)
	at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4000)
	at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3077)
	at io.fabric8.kubernetes.client.utils.Serialization.unmarshal(Serialization.java:236)
	... 23 more

hypery2k@web.de (JIRA)

unread,
Apr 23, 2019, 11:06:02 PM4/23/19
to jenkinsc...@googlegroups.com
Martin Reinhardt assigned an issue to Carlos Sanchez
 
Change By: Martin Reinhardt
Assignee: Martin Reinhardt Carlos Sanchez

hypery2k@web.de (JIRA)

unread,
Apr 23, 2019, 11:06:03 PM4/23/19
to jenkinsc...@googlegroups.com
Change By: Martin Reinhardt
Status: In Review Progress

jenkins-ci@carlossanchez.eu (JIRA)

unread,
Apr 24, 2019, 3:41:02 AM4/24/19
to jenkinsc...@googlegroups.com
Carlos Sanchez commented on Bug JENKINS-57116
 
Re: Version 1.15.1 does not create pods

Can you provide your pipeline and pod definition?

josh.chandler@golfchannel.com (JIRA)

unread,
Apr 29, 2019, 9:30:03 AM4/29/19
to jenkinsc...@googlegroups.com

I experienced this problem with 1.15.2; a rollback to 1.14.9 does fix the issue. My log output regarding the issue at hand is exactly the same as Martin Reinhardt. The plugin will try to create a pod, fail on that error, fail to delete the pod (as it never existed in the first place), and tries all of that over again. A log recorder on org.csanchez.jenkins.plugins.kubernetes provided no more logs than watching Jenkins' own output.

Versions, pod definition, and pipeline below. Not sure if this is what you're looking for exactly Carlos Sanchez - will be happy to provide more as you require.

Jenkins version: 2.162

Kubernetes Version: v1.12.6-gke.10

GKE OS Version: Container-Optimized OS (latest as of 2019-04-29)

Pod template:

$ kubectl get pods --export -o yaml jnlp-buildpod-cgpfm

apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  labels:
    jenkins: slave
  name: jnlp-buildpod-cgpfm
  selfLink: /api/v1/namespaces/ci/pods/jnlp-buildpod-cgpfm
spec:
  containers:
  - args:
    - cat
    command:
    - /bin/sh
    - -c
    env:
    - name: JENKINS_SECRET
      value: [redacted] 
    - name: JENKINS_AGENT_NAME
      value: jnlp-buildpod-cgpfm
    - name: JENKINS_NAME
      value: jnlp-buildpod-cgpfm
    - name: JENKINS_URL
      value: http://ops-jenkins.ci.svc.cluster.local:8080/
    - name: HOME
      value: /home/jenkins
    image: centos:7.6.1810
    imagePullPolicy: IfNotPresent
    name: cmdtools-centos
    resources: {}
    securityContext:
      privileged: false
      procMount: Default
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    tty: true
    volumeMounts:
    - mountPath: /var/run/docker.sock
      name: volume-0
    - mountPath: /home/jenkins
      name: workspace-volume
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: ops-jenkins-token-2dszv
      readOnly: true
    workingDir: /home/jenkins
  - args:
    - cat
    command:
    - /bin/sh
    - -c
    env:
    - name: JENKINS_SECRET
      value: [redacted]
    - name: JENKINS_AGENT_NAME
      value: jnlp-buildpod-cgpfm
    - name: JENKINS_NAME
      value: jnlp-buildpod-cgpfm
    - name: JENKINS_URL
      value: http://ops-jenkins.ci.svc.cluster.local:8080/
    - name: HOME
      value: /home/jenkins
    image: alpine:3.8
    imagePullPolicy: IfNotPresent
    name: cmdtools
    resources:
      limits:
        cpu: 100m
        memory: 100Mi
      requests:
        cpu: 100m
        memory: 100Mi
    securityContext:
      privileged: false
      procMount: Default
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    tty: true
    volumeMounts:
    - mountPath: /var/run/docker.sock
      name: volume-0
    - mountPath: /home/jenkins
      name: workspace-volume
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: ops-jenkins-token-2dszv
      readOnly: true
    workingDir: /home/jenkins
  - args:
    - cat
    command:
    - /bin/sh
    - -c
    env:
    - name: JENKINS_SECRET
      value: [redacted] 
    - name: JENKINS_AGENT_NAME
      value: jnlp-buildpod-cgpfm
    - name: JENKINS_NAME
      value: jnlp-buildpod-cgpfm
    - name: JENKINS_URL
      value: http://ops-jenkins.ci.svc.cluster.local:8080/
    - name: HOME
      value: /home/jenkins
    image: lachlanevenson/k8s-kubectl:v1.10.12
    imagePullPolicy: IfNotPresent
    name: kubectl
    resources:
      limits:
        cpu: 100m
        memory: 500Mi
      requests:
        cpu: 100m
        memory: 500Mi
    securityContext:
      privileged: false
      procMount: Default
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    tty: true
    volumeMounts:
    - mountPath: /var/run/docker.sock
      name: volume-0
    - mountPath: /home/jenkins
      name: workspace-volume
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: ops-jenkins-token-2dszv
      readOnly: true
    workingDir: /home/jenkins
  - args:
    - cat
    command:
    - /bin/sh
    - -c
    env:
    - name: JENKINS_SECRET
      value: [redacted] 
    - name: JENKINS_AGENT_NAME
      value: jnlp-buildpod-cgpfm
    - name: JENKINS_NAME
      value: jnlp-buildpod-cgpfm
    - name: JENKINS_URL
      value: http://ops-jenkins.ci.svc.cluster.local:8080/
    - name: HOME
      value: /home/jenkins
    image: python:3.6.7
    imagePullPolicy: IfNotPresent
    name: cmdtools-python
    resources: {}
    securityContext:
      privileged: false
      procMount: Default
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    tty: true
    volumeMounts:
    - mountPath: /var/run/docker.sock
      name: volume-0
    - mountPath: /home/jenkins
      name: workspace-volume
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: ops-jenkins-token-2dszv
      readOnly: true
    workingDir: /home/jenkins
  - args:
    - cat
    command:
    - /bin/sh
    - -c
    env:
    - name: JENKINS_SECRET
      value: [redacted]
    - name: JENKINS_AGENT_NAME
      value: jnlp-buildpod-cgpfm
    - name: JENKINS_NAME
      value: jnlp-buildpod-cgpfm
    - name: JENKINS_URL
      value: http://ops-jenkins.ci.svc.cluster.local:8080/
    - name: HOME
      value: /home/jenkins
    image: docker:18.09.1-dind
    imagePullPolicy: IfNotPresent
    name: docker
    resources:
      limits:
        cpu: "1"
        memory: 2Gi
      requests:
        cpu: 500m
        memory: 1Gi
    securityContext:
      privileged: true
      procMount: Default
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    tty: true
    volumeMounts:
    - mountPath: /var/run/docker.sock
      name: volume-0
    - mountPath: /home/jenkins
      name: workspace-volume
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: ops-jenkins-token-2dszv
      readOnly: true
    workingDir: /home/jenkins
  - env:
    - name: JENKINS_SECRET
      value: [redacted] 
    - name: JENKINS_AGENT_NAME
      value: jnlp-buildpod-cgpfm
    - name: JENKINS_NAME
      value: jnlp-buildpod-cgpfm
    - name: JENKINS_URL
      value: http://ops-jenkins.ci.svc.cluster.local:8080/
    - name: HOME
      value: /home/jenkins
    image: jenkins/jnlp-slave:alpine
    imagePullPolicy: IfNotPresent
    name: jnlp
    resources: {}
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /var/run/docker.sock
      name: volume-0
    - mountPath: /home/jenkins
      name: workspace-volume
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: ops-jenkins-token-2dszv
      readOnly: true
  dnsPolicy: ClusterFirst
  nodeName: [redacted] 
  priority: 0
  restartPolicy: Never
  schedulerName: default-scheduler
  securityContext: {}
  serviceAccount: ops-jenkins
  serviceAccountName: ops-jenkins
  terminationGracePeriodSeconds: 30
  tolerations:
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 300
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 300
  volumes:
  - hostPath:
      path: /var/run/docker.sock
      type: ""
    name: volume-0
  - emptyDir: {}
    name: workspace-volume
  - name: ops-jenkins-token-2dszv
    secret:
      defaultMode: 420
      secretName: ops-jenkins-token-2dszv
status:
  phase: Pending
  qosClass: Burstable

 

Pipeline (although any pipeline involving dynamic build agents provokes this behavior):

pipeline {
    agent {
        kubernetes {
            cloud 'build-cloud'
        }
    }
    stages {
        stage('Inform job start') {
            steps {
                slackSend color: '#FFFF00', message: "*[Cloud Reboot]* [<${env.BUILD_URL}|direct link>] Reboot job beginning.\n*Cause*: ${REBOOT_REASON}"
            }
        }
        stage('Reboot all pods') {
            steps {
                container('kubectl') {
                    withCredentials([kubeconfigContent(credentialsId: "${TARGET_CLOUD_ID}", variable: 'KUBECONFIG_CONTENT')]) {
                        sh """
                        mkdir -p ~/.kube
                        echo "${KUBECONFIG_CONTENT}" > ~/.kube/config
                        
                        commandSet=''
                        kubectl get namespaces | sed 1,1d | awk '{ print \$1 }' | while read namespace; do
                            kubectl get -n \$namespace -l "${LABELS}" pods | sed 1,1d | awk '{ print \$1 }' | shuf | while read podName; do
                                commandSet="\$commandSet\$podName "
                                if [ "\$(echo "\$commandSet" | sed 's/ \$//g' | tr ' ' '\n' | wc -l | tr -d ' ')" = "${BATCH_SIZE}" ]; then
                                    if [ "${DEBUG}" = 'true' ]; then
                                        echo "kubectl delete pods -n \$namespace \$(echo "\$commandSet" | sed 's/ \$//g')"
                                    else
                                        eval "kubectl delete pods -n \$namespace \$(echo "\$commandSet" | sed 's/ \$//g') || true"
                                        sleep ${BACKOFF_TIME}
                                    fi
                                    commandSet=''
                                fi
                            done
                            
                            if [ -n "\$commandSet" ]; then
                                if [ "${DEBUG}" = 'true' ]; then
                                    echo "kubectl delete pods -n \$namespace \$(echo "\$commandSet" | sed 's/ \$//g')"
                                else
                                    eval "kubectl delete pods -n \$namespace \$(echo "\$commandSet" | sed 's/ \$//g')"
                                fi
                            fi
                        done
                        """
                    }
                }
            }
            post {
                success {
                    slackSend color: '#00FF00', message: "*[Cloud Reboot]* [<${env.BUILD_URL}|direct link>]\nReboot job success! <${env.BUILD_URL}/console|Click here> for details."
                }
                failure {
                    slackSend color: '#FF0000', message: "*[Cloud Reboot]* [<${env.BUILD_URL}|direct link>]\nReboot job *FAILED* - <${env.BUILD_URL}/console|click here> for details."
                }
            }
        }
    }
}

josh.chandler@golfchannel.com (JIRA)

unread,
Apr 29, 2019, 10:15:03 AM4/29/19
to jenkinsc...@googlegroups.com
Josh Chandler edited a comment on Bug JENKINS-57116
I experienced this problem with 1.15.2; a rollback to 1.14.9 does fix the issue. My log output regarding the issue at hand is exactly the same as [~mreinhardt]. The plugin will try to create a pod, fail on that error, fail to delete the pod (as it never existed in the first place), and tries all of that over again. A log recorder on *org.csanchez.jenkins.plugins.kubernetes* provided no more logs than watching Jenkins' own output.

Versions, pod definition, and pipeline below. Not sure if this is what you're looking for exactly [~csanchez] - will be happy to provide more as you require.


Jenkins version: 2.162

Kubernetes Version: v1.12.6-gke.10

Kubernetes plugin version (since rolled back): 1.15.2
GKE OS Version: Container-Optimized OS (latest as of 2019-04-29)

Pod template:
{code:java}
  qosClass: Burstable {code}

 

Pipeline (although any pipeline involving dynamic build agents provokes this behavior):
{code:java}
} {code}

jenkins-ci@carlossanchez.eu (JIRA)

unread,
Apr 29, 2019, 10:46:02 AM4/29/19
to jenkinsc...@googlegroups.com

are you all:

  1. running declarative pipeline or scripted?
  2. defining pod templates in jenkins admin page or in Jenkinsfile?
  3. using yaml to define your pod templates?

josh.chandler@golfchannel.com (JIRA)

unread,
Apr 29, 2019, 11:01:04 AM4/29/19
to jenkinsc...@googlegroups.com

1) Declarative pipelines in general.

2) For me, Jenkins admin to define the pod template.

3) Not using yaml - all within the admin.

josh.chandler@golfchannel.com (JIRA)

unread,
Apr 29, 2019, 1:19:03 PM4/29/19
to jenkinsc...@googlegroups.com
Josh Chandler assigned an issue to Carlos Sanchez
 
Change By: Josh Chandler
Assignee: Josh Chandler Carlos Sanchez

josh.chandler@golfchannel.com (JIRA)

unread,
Apr 29, 2019, 1:19:03 PM4/29/19
to jenkinsc...@googlegroups.com
Josh Chandler assigned an issue to Josh Chandler
Change By: Josh Chandler
Assignee: Carlos Sanchez Josh Chandler

contact@martinreinhardt-online.de (JIRA)

unread,
Apr 30, 2019, 2:42:02 AM4/30/19
to jenkinsc...@googlegroups.com
Martin Reinhardt commented on Bug JENKINS-57116
 
Re: Version 1.15.1 does not create pods

1.) Using scripted

2) Using admin page

3. Also not using yaml

jenkins-ci@carlossanchez.eu (JIRA)

unread,
Apr 30, 2019, 5:24:02 AM4/30/19
to jenkinsc...@googlegroups.com

can you provide from your config.xml

the first line, something like <?xml version='1.1' encoding='UTF-8'?>
the yaml/yamls entries inside your podTemplate config <org.csanchez.jenkins.plugins.kubernetes.PodTemplate> paying attention to encoding

josh.chandler@golfchannel.com (JIRA)

unread,
Apr 30, 2019, 9:43:04 AM4/30/19
to jenkinsc...@googlegroups.com

First line of config.xml:

<?xml version='1.1' encoding='UTF-8'?>

 

I'm not seeing any yaml in my podTemplate; here's what I've got in config.xml:

<org.csanchez.jenkins.plugins.kubernetes.PodTemplate>
  <inheritFrom></inheritFrom>
  <name>jnlp-buildpod</name>
  <namespace>ci</namespace>
  <privileged>false</privileged>
  <capOnlyOnAlivePods>false</capOnlyOnAlivePods>
  <alwaysPullImage>false</alwaysPullImage>
  <instanceCap>10</instanceCap>
  <slaveConnectTimeout>600</slaveConnectTimeout>
  <idleMinutes>30</idleMinutes>
  <activeDeadlineSeconds>0</activeDeadlineSeconds>
  <label></label>
  <serviceAccount>ops-jenkins</serviceAccount>
  <nodeSelector></nodeSelector>
  <nodeUsageMode>NORMAL</nodeUsageMode>
  <customWorkspaceVolumeEnabled>false</customWorkspaceVolumeEnabled>
  <workspaceVolume class="org.csanchez.jenkins.plugins.kubernetes.volumes.workspace.EmptyDirWorkspaceVolume">
    <memory>false</memory>
  </workspaceVolume>
  <volumes>
    <org.csanchez.jenkins.plugins.kubernetes.volumes.HostPathVolume>
      <mountPath>/var/run/docker.sock</mountPath>
      <hostPath>/var/run/docker.sock</hostPath>
    </org.csanchez.jenkins.plugins.kubernetes.volumes.HostPathVolume>
  </volumes>
  <containers>
    <org.csanchez.jenkins.plugins.kubernetes.ContainerTemplate>
      <name>docker</name>
      <image>docker:18.09.1-dind</image>
      <privileged>true</privileged>
      <alwaysPullImage>false</alwaysPullImage>
      <workingDir>/home/jenkins</workingDir>
      <command>/bin/sh -c</command>
      <args>cat</args>
      <ttyEnabled>true</ttyEnabled>
      <resourceRequestCpu>500m</resourceRequestCpu>
      <resourceRequestMemory>1Gi</resourceRequestMemory>
      <resourceLimitCpu>1</resourceLimitCpu>
      <resourceLimitMemory>2Gi</resourceLimitMemory>
      <envVars/>
      <ports/>
      <livenessProbe>
        <execArgs></execArgs>
        <timeoutSeconds>0</timeoutSeconds>
        <initialDelaySeconds>0</initialDelaySeconds>
        <failureThreshold>0</failureThreshold>
        <periodSeconds>0</periodSeconds>
        <successThreshold>0</successThreshold>
      </livenessProbe>
    </org.csanchez.jenkins.plugins.kubernetes.ContainerTemplate>
    <org.csanchez.jenkins.plugins.kubernetes.ContainerTemplate>
      <name>kubectl</name>
      <image>lachlanevenson/k8s-kubectl:v1.10.12</image>
      <privileged>false</privileged>
      <alwaysPullImage>false</alwaysPullImage>
      <workingDir>/home/jenkins</workingDir>
      <command>/bin/sh -c</command>
      <args>cat</args>
      <ttyEnabled>true</ttyEnabled>
      <resourceRequestCpu>100m</resourceRequestCpu>
      <resourceRequestMemory>500Mi</resourceRequestMemory>
      <resourceLimitCpu>100m</resourceLimitCpu>
      <resourceLimitMemory>500Mi</resourceLimitMemory>
      <envVars/>
      <ports/>
      <livenessProbe>
        <execArgs></execArgs>
        <timeoutSeconds>0</timeoutSeconds>
        <initialDelaySeconds>0</initialDelaySeconds>
        <failureThreshold>0</failureThreshold>
        <periodSeconds>0</periodSeconds>
        <successThreshold>0</successThreshold>
      </livenessProbe>
    </org.csanchez.jenkins.plugins.kubernetes.ContainerTemplate>
    <org.csanchez.jenkins.plugins.kubernetes.ContainerTemplate>
      <name>cmdtools</name>
      <image>alpine:3.8</image>
      <privileged>false</privileged>
      <alwaysPullImage>false</alwaysPullImage>
      <workingDir>/home/jenkins</workingDir>
      <command>/bin/sh -c</command>
      <args>cat</args>
      <ttyEnabled>true</ttyEnabled>
      <resourceRequestCpu>100m</resourceRequestCpu>
      <resourceRequestMemory>100Mi</resourceRequestMemory>
      <resourceLimitCpu>100m</resourceLimitCpu>
      <resourceLimitMemory>100Mi</resourceLimitMemory>
      <envVars/>
      <ports/>
      <livenessProbe>
        <execArgs></execArgs>
        <timeoutSeconds>0</timeoutSeconds>
        <initialDelaySeconds>0</initialDelaySeconds>
        <failureThreshold>0</failureThreshold>
        <periodSeconds>0</periodSeconds>
        <successThreshold>0</successThreshold>
      </livenessProbe>
    </org.csanchez.jenkins.plugins.kubernetes.ContainerTemplate>
    <org.csanchez.jenkins.plugins.kubernetes.ContainerTemplate>
      <name>cmdtools-python</name>
      <image>python:3.6.7</image>
      <privileged>false</privileged>
      <alwaysPullImage>false</alwaysPullImage>
      <workingDir>/home/jenkins</workingDir>
      <command>/bin/sh -c</command>
      <args>cat</args>
      <ttyEnabled>true</ttyEnabled>
      <resourceRequestCpu></resourceRequestCpu>
      <resourceRequestMemory></resourceRequestMemory>
      <resourceLimitCpu></resourceLimitCpu>
      <resourceLimitMemory></resourceLimitMemory>
      <envVars/>
      <ports/>
      <livenessProbe>
        <execArgs></execArgs>
        <timeoutSeconds>0</timeoutSeconds>
        <initialDelaySeconds>0</initialDelaySeconds>
        <failureThreshold>0</failureThreshold>
        <periodSeconds>0</periodSeconds>
        <successThreshold>0</successThreshold>
      </livenessProbe>
    </org.csanchez.jenkins.plugins.kubernetes.ContainerTemplate>
    <org.csanchez.jenkins.plugins.kubernetes.ContainerTemplate>
      <name>cmdtools-centos</name>
      <image>centos:7.6.1810</image>
      <privileged>false</privileged>
      <alwaysPullImage>false</alwaysPullImage>
      <workingDir>/home/jenkins</workingDir>
      <command>/bin/sh -c</command>
      <args>cat</args>
      <ttyEnabled>true</ttyEnabled>
      <resourceRequestCpu></resourceRequestCpu>
      <resourceRequestMemory></resourceRequestMemory>
      <resourceLimitCpu></resourceLimitCpu>
      <resourceLimitMemory></resourceLimitMemory>
      <envVars/>
      <ports/>
      <livenessProbe>
        <execArgs></execArgs>
        <timeoutSeconds>0</timeoutSeconds>
        <initialDelaySeconds>0</initialDelaySeconds>
        <failureThreshold>0</failureThreshold>
        <periodSeconds>0</periodSeconds>
        <successThreshold>0</successThreshold>
      </livenessProbe>
    </org.csanchez.jenkins.plugins.kubernetes.ContainerTemplate>
  </containers>
  <envVars/>
  <annotations/>
  <imagePullSecrets/>
  <nodeProperties/>
  <podRetention class="org.csanchez.jenkins.plugins.kubernetes.pod.retention.Default"/>
</org.csanchez.jenkins.plugins.kubernetes.PodTemplate>

jenkins-ci@carlossanchez.eu (JIRA)

unread,
Apr 30, 2019, 1:12:02 PM4/30/19
to jenkinsc...@googlegroups.com
Change By: Carlos Sanchez
Status: In Progress Review

jenkins-ci@carlossanchez.eu (JIRA)

unread,
Apr 30, 2019, 1:14:02 PM4/30/19
to jenkinsc...@googlegroups.com
Carlos Sanchez commented on Bug JENKINS-57116
 
Re: Version 1.15.1 does not create pods

This error is caused by empty yaml fields but I don't see how it is triggered.
In any case I added some safeguards in a PR that should fix it
https://github.com/jenkinsci/kubernetes-plugin/pull/462

larguiadiegobern.cousinet@allianz.de (JIRA)

unread,
May 8, 2019, 8:57:01 AM5/8/19
to jenkinsc...@googlegroups.com

Problem seems to be that the yaml was changed from string to a string list.
https://github.com/jenkinsci/kubernetes-plugin/commit/c52ebdf5ad3fcd2dd8b53e2b8441d61d0a8693f3#diff-c1e2813295afdbf77e75d9606d614906
I diffed the pod template sections in config.xml from an affected jenkins instance against a healthy one:

< <yamls>
< <string></string>
< </yamls>
—
> <yamls class="empty-list"/>

 Hitting the save button on the jenkins configuration page fixes the problem. Unfortunately that solution doesn't scale well.

jenkins-ci@carlossanchez.eu (JIRA)

unread,
May 8, 2019, 9:45:03 AM5/8/19
to jenkinsc...@googlegroups.com
Carlos Sanchez assigned an issue to Vincent Latombe
 
Change By: Carlos Sanchez
Assignee: Carlos Sanchez Vincent Latombe

jenkins-ci@carlossanchez.eu (JIRA)

unread,
May 9, 2019, 3:19:02 AM5/9/19
to jenkinsc...@googlegroups.com
Carlos Sanchez assigned an issue to Carlos Sanchez
Change By: Carlos Sanchez
Assignee: Vincent Latombe Carlos Sanchez
Reply all
Reply to author
Forward
0 new messages