Jenkins Kubernetes Plugin not honoring activeDeadlineSeconds parameter

drpm

unread,

Oct 18, 2019, 5:00:18 AM10/18/19

to Jenkins Users

Hello,

For some reason, jenkins is not honoring activeDeadlineSeconds parameter when used in pipeline script. I'm using this kubernetes plugin for jenkins: https://wiki.jenkins.io/display/JENKINS/Kubernetes+Plugin

Sample Pipeline Script:

def label = "jenkins-test-build-${UUID.randomUUID().toString()}"
podTemplate(label: label, yaml: """
apiVersion: v1
kind: Pod
metadata:
labels:
slave: jenkins-slave-testing-build
spec:
activeDeadlineSeconds: 3600
containers:
- name: slave
image: '<ecr-repo>/jenkins-slave'
tty: true
command:
- cat
nodeSelector:
xyz.com/app: cibuild
tolerations:
- effect: NoSchedule
key: xyz.com/abc.xyz.com
operator: Equal
value: cibuild
"""
){
node (label) {
withEnv(['USER=jenkins']) {
container('slave') {
stage("test") {
sh "echo test"
}
}
}
}
}

I also checked the FINER logs for org.csanchez.jenkins.plugins.kubernetes and it appears to be getting the settings:

Combining pods, parent: Pod(apiVersion=v1, kind=Pod, metadata=ObjectMeta(annotations=null, clusterName=null, creationTimestamp=null, deletionGracePeriodSeconds=null, deletionTimestamp=null, finalizers=[], generateName=null, generation=null, labels={slave=jenkins-slave-testing-build}, managedFields=[], name=null, namespace=null, ownerReferences=[], resourceVersion=null, selfLink=null, uid=null, additionalProperties={}), spec=PodSpec(activeDeadlineSeconds=3600, affinity=null, automountServiceAccountToken=null, containers=[Container(args=[], command=[cat], env=[], envFrom=[], image=<ecrrepo>/jenkins-slave, imagePullPolicy=null, lifecycle=null, livenessProbe=null, name=slave, ports=[], readinessProbe=null, resources=null, securityContext=null, stdin=null, stdinOnce=null, terminationMessagePath=null, terminationMessagePolicy=null, tty=true, volumeDevices=[], volumeMounts=[VolumeMount(mountPath=null, mountPropagation=null, name=docker-sock, readOnly=null, subPath=null, subPathExpr=null, additionalProperties={})], workingDir=null, additionalProperties={})], dnsConfig=null, dnsPolicy=null, enableServiceLinks=null, hostAliases=[], hostIPC=null, hostNetwork=null, hostPID=null, hostname=null, imagePullSecrets=[], initContainers=[], nodeName=null, nodeSelector={xyz.com/app=cibuild}, preemptionPolicy=null, priority=null, priorityClassName=null, readinessGates=[], restartPolicy=null, runtimeClassName=null, schedulerName=null, securityContext=null, serviceAccount=null, serviceAccountName=null, shareProcessNamespace=null, subdomain=null, terminationGracePeriodSeconds=null, tolerations=[Toleration(effect=NoSchedule, key=xyz.com/abc.xyz.com, operator=Equal, tolerationSeconds=null, value=cibuild, additionalProperties={})], volumes=[Volume(awsElasticBlockStore=null, azureDisk=null, azureFile=null, cephfs=null, cinder=null, configMap=null, csi=null, downwardAPI=null, emptyDir=null, fc=null, flexVolume=null, flocker=null, gcePersistentDisk=null, gitRepo=null, glusterfs=null, hostPath=null, iscsi=null, name=null, nfs=null, persistentVolumeClaim=null, photonPersistentDisk=null, portworxVolume=null, projected=null, quobyte=null, rbd=null, scaleIO=null, secret=null, storageos=null, vsphereVolume=null, additionalProperties={})], additionalProperties={idleMinutes=null}), status=null, additionalProperties={})

but it's not working as expected since the pod is terminated right after the job is finished instead of waiting 3600s.

Oct 18, 2019 8:58:18 AM INFO org.csanchez.jenkins.plugins.kubernetes.pipeline.PodTemplateStepExecution$PodTemplateCallback finished
Removing pod template jenkins-test-build-87928862-54e0-44c3-9be4-c68d414d4427-w9jpl from cloud kubernetes
Oct 18, 2019 8:58:18 AM INFO org.csanchez.jenkins.plugins.kubernetes.pod.retention.Reaper eventReceived
cibuild/jenkins-test-build-87928862-54e0-44c3-9be4-c68d414d44-d2878 was just deleted, so removing corresponding Jenkins agent
Oct 18, 2019 8:58:18 AM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave deleteSlavePod
Terminated Kubernetes instance for agent cibuild/jenkins-mbx-test-build-87928862-54e0-44c3-9be4-c68d414d44-d2878
Oct 18, 2019 8:58:18 AM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave _terminate
Disconnected computer jenkins-test-build-87928862-54e0-44c3-9be4-c68d414d44-d2878

Thank you in advance for your help.

- drpm

Carlos Sanchez

unread,

Oct 18, 2019, 5:17:06 AM10/18/19

to Jenkins Users

the plugin will delete the kubernetes pod after the jenkins job is completed

Also I don't think activeDeadlineSeconds is what you want

activeDeadlineSeconds: Optional duration in seconds the pod may be active on the node relative to StartTime before the system will actively try to mark it failed and kill associated containers. Value must be a positive integer.

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/fd798b7c-39cf-41f4-b99b-8ebd5067dd6e%40googlegroups.com.

drpm

unread,

Oct 18, 2019, 5:37:41 AM10/18/19

to Jenkins Users

Hello,

Thank you for your reply.

It says here (https://github.com/jenkinsci/kubernetes-plugin) that pod will be deleted after the set activeDeadlineSeconds is passed or am I totally reading it wrong?

podRetention Controls the behavior of keeping slave pods. Can be 'never()', 'onFailure()', 'always()', or 'default()' - if empty will default to deleting the pod after activeDeadlineSeconds has passed.
activeDeadlineSeconds If podRetention is set to 'never()' or 'onFailure()', pod is deleted after this deadline is passed.

If this is totally wrong, can you recommend a right way to retain a pod even if the job is finished using this plugin?

To unsubscribe from this group and stop receiving emails from it, send an email to jenkins...@googlegroups.com.

Carlos Sanchez

unread,

Oct 18, 2019, 5:51:49 AM10/18/19

to Jenkins Users

Ah ok, for that one you need to set it at podTemplate level, not inside the yaml

To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/691690fb-e9bf-446e-a55f-8883e47cde66%40googlegroups.com.

drpm

unread,

Oct 18, 2019, 6:03:03 AM10/18/19

to Jenkins Users

I tried this one but the pod still got terminated.

def label = "jenkins-test-build-${UUID.randomUUID().toString()}"

podTemplate(label: label, activeDeadlineSeconds: 3600, yaml: """

apiVersion: v1
kind: Pod
metadata:
labels:
slave: jenkins-slave-testing-build
spec:

containers:

- name: slave
image: '<ecr-repo>/jenkins-slave'
tty: true
command:
- cat
nodeSelector:
xyz.com/app: cibuild
tolerations:
- effect: NoSchedule
key: xyz.com/abc.xyz.com
operator: Equal
value: cibuild
"""
){
node (label) {
withEnv(['USER=jenkins']) {
container('slave') {
stage("test") {
sh "echo test"
}
}
}
}
}

LOGS:

Oct 18, 2019 9:57:39 AM FINE org.csanchez.jenkins.plugins.kubernetes.KubernetesComputer
Computer KubernetesComputer name: jenkins-test-build-ae2e7dee-8f0f-48b8-8a3f-39fd1de717-0ct7q slave: null completed task PlaceholderExecutable:ExecutorStepExecution.PlaceholderTask{runId=testing#47,label=jenkins-test-build-ae2e7dee-8f0f-48b8-8a3f-39fd1de717-0ct7q,context=CpsStepContext[5:node]:Owner[testing/47:testing #47],cookie=2b863d15-f6f0-431f-8081-5eea211b992b,auth=null}
Oct 18, 2019 9:57:39 AM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave _terminate
Terminating Kubernetes instance for agent jenkins-test-build-ae2e7dee-8f0f-48b8-8a3f-39fd1de717-0ct7q
Oct 18, 2019 9:57:39 AM FINE org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud
Building connection to Kubernetes kubernetes URL https://kubernetes.default namespace cibuild
Oct 18, 2019 9:57:39 AM FINE org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud
Connected to Kubernetes kubernetes URL https://kubernetes.default/
Oct 18, 2019 9:57:39 AM INFO org.csanchez.jenkins.plugins.kubernetes.pipeline.PodTemplateStepExecution$PodTemplateCallback finished
Removing pod template jenkins-test-build-ae2e7dee-8f0f-48b8-8a3f-39fd1de7171c-npz46 from cloud kubernetes
Oct 18, 2019 9:57:39 AM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave deleteSlavePod
Terminated Kubernetes instance for agent cibuild/jenkin-test-build-ae2e7dee-8f0f-48b8-8a3f-39fd1de717-0ct7q
Oct 18, 2019 9:57:39 AM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave _terminate
Disconnected computer jenkins-test-build-ae2e7dee-8f0f-48b8-8a3f-39fd1de717-0ct7q
Oct 18, 2019 9:57:39 AM FINE org.csanchez.jenkins.plugins.kubernetes.pod.retention.Reaper
received deletion notice for cibuild/jenkins-test-build-ae2e7dee-8f0f-48b8-8a3f-39fd1de717-0ct7q which does not seem to correspond to any Jenkins agent

Terminated Kubernetes instance for agent cibuild/jenkins-test-build-87928862-54e0-44c3-9be4-c68d414d44-d2878

Oct 18, 2019 8:58:18 AM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave _terminate
Disconnected computer jenkins-test-build-87928862-54e0-44c3-9be4-c68d414d44-d2878

Thank you in advance for your help.

- drpm

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkins...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/fd798b7c-39cf-41f4-b99b-8ebd5067dd6e%40googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkins...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/691690fb-e9bf-446e-a55f-8883e47cde66%40googlegroups.com.

Reply all

Reply to author

Forward