Kubernetes plugin cannot start POD´s due to PVC creation error

[Torsten Reinhard]

Hi,

I´m running Jenkins Version 2.190.1 in an openShift 3.9 Cluster, Kubernetes plugin is at version 1.19.3

Since one of the last updates, I sometimes run into:

 [id=1597]    WARNING    o.c.j.p.k.KubernetesLauncher#launch: Error in provisioning; agent=KubernetesSlave name: b4dbc13f-6f01-42d5-a9d7-b31e9520adaa-7013x-lcc2j,
 template=PodTemplate{inheritFrom='',
 name='b4dbc13f-6f01-42d5-a9d7-b31e9520adaa-7013x',
 namespace='',
 label='b4dbc13f-6f01-42d5-a9d7-b31e9520adaa',
 nodeSelector='',
 nodeUsageMode=EXCLUSIVE,
 workspaceVolume=org.csanchez.jenkins.plugins.kubernetes.volumes.workspace.DynamicPVCWorkspaceVolume@79ebc880,
 containers=[ContainerTemplate{name='main', image='docker-registry-default.cnap-00-mp-prod.mycompanygroup.net:443/ci-next/jenkins-slave-oc:latest',
 alwaysPullImage=true, workingDir='/home/jenkins/agent', command='/bin/sh -c', args='cat', ttyEnabled=true, resourceRequestCpu='',
 resourceRequestMemory='',
 resourceLimitCpu='',
 resourceLimitMemory='',
 envVars=[KeyValueEnvVar [getValue()=https://rspsales-cinext.mycompanygroup.net , getKey()=LOCAL_URL],
 KeyValueEnvVar [getValue()=https://rspsales-cinext.mycompanygroup.net/nexus , getKey()=NEXUS_URL],
 KeyValueEnvVar [getValue()=default, getKey()=clusterName],
 KeyValueEnvVar [getValue()=rspsales-ci, getKey()=project],
 KeyValueEnvVar [getValue()=BuildConfig.yml, getKey()=buildConfigFile]]}],
 annotations=[org.csanchez.jenkins.plugins.kubernetes.PodAnnotation@9d4da4a8, org.csanchez.jenkins.plugins.kubernetes.PodAnnotation@aab9c821], yamls=[
apiVersion: v1
kind: Pod
metadata:
    labels:
        tier: ci
        cinextProject: null
        app: jenkins-slave
spec:
  containers:
  - name: jnlp
    image: 'jenkins/jnlp-slave:alpine'
    args: ['$(JENKINS_SECRET)', '$(JENKINS_NAME)']
    resources:
      limits:
        cpu: '200m'
        memory: '256Mi'
      requests:
        cpu: '200m'
        memory: '128Mi'
    env:
      - name: JAVA_OPTS
        value: '-Xmx128m'
]}
io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: POST at: https://10.221.128.1/api/v1/namespaces/rspsales-ci/persistentvolumeclaims . Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. persistentvolumeclaims "pvc-b4dbc13f-6f01-42d5-a9d7-b31e9520adaa-7013x-lcc2j" is forbidden: cannot set blockOwnerDeletion if an ownerReference refers to a resource you can't set finalizers on: User "system:serviceaccount:rspsales-ci:jenkins" cannot update pods/finalizers in project "rspsales-ci", <nil>.
    at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:510)
    at io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:447)
    at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:413)
    at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:372)
    at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleCreate(OperationSupport.java:241)
    at io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleCreate(BaseOperation.java:813)
    at io.fabric8.kubernetes.client.dsl.base.BaseOperation.create(BaseOperation.java:328)
    at io.fabric8.kubernetes.client.dsl.base.BaseOperation.create(BaseOperation.java:324)
    at org.csanchez.jenkins.plugins.kubernetes.volumes.workspace.DynamicPVCWorkspaceVolume.createVolume(DynamicPVCWorkspaceVolume.java:94)
    at org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher.launch(KubernetesLauncher.java:130)
    at hudson.slaves.SlaveComputer$1.call(SlaveComputer.java:297)
    at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
    at jenkins.security.ImpersonatingExecutorService$2.call(ImpersonatingExecutorService.java:71)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.ja)va:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)

I guess it´s related to the Dynamic PVC´s (see JENKINS-47591) introduced in 1.19.2 - but how can this be resolved ?

The strange thing about it is that after restarting Jenkins the POD launching works several times - and than suddenly starts to fail with the above message.

I´m running Jenkins with a dedicated service-account:jenkins at openShift, having either "edit" or now for testing "admin" role.

Thanx for any ideas,

Torsten

[Torsten Reinhard]

I´ve enabled a "kubernetes" Logger with Level.FINEST and got this output:

Combining pod templates, parent: PodTemplate{inheritFrom='', name='default', namespace='', label='', nodeSelector='', nodeUsageMode=EXCLUSIVE, workspaceVolume=org.csanchez.jenkins.plugins.kubernetes.volumes.workspace.DynamicPVCWorkspaceVolume@79ebc880, containers=[ContainerTemplate{name='main', image='docker-registry-default.cnap-00-mp-prod.mycompanygroup.net:443/jenkins-slave/base:latest', workingDir='/home/jenkins/agent', command='/bin/sh -c', args='cat', resourceRequestCpu='', resourceRequestMemory='', resourceLimitCpu='', resourceLimitMemory='', livenessProbe=org.csanchez.jenkins.plugins.kubernetes.ContainerLivenessProbe@16be1b19}]} Oct 17, 2019 1:55:40 PM FINEST org.csanchez.jenkins.plugins.kubernetes.PodTemplateUtils
Combining pod templates, template: PodTemplate{, name='9ccd91fa-0aba-46d6-b493-f48fb4136a68-60m1x', label='9ccd91fa-0aba-46d6-b493-f48fb4136a68', nodeUsageMode=EXCLUSIVE, containers=[ContainerTemplate{name='main', image='docker-registry-default.cnap-00-mp-prod.mycompanygroup.net:443/ci-next/jenkins-slave-java8-mvn:latest', alwaysPullImage=true, command='/bin/sh -c', args='cat', ttyEnabled=true, envVars=[KeyValueEnvVar [getValue()=https://rspsales-cinext.mycompanygroup.net, getKey()=LOCAL_URL], KeyValueEnvVar [getValue()=https://rspsales-cinext.mycompanygroup.net/nexus, getKey()=NEXUS_URL]]}], annotations=[org.csanchez.jenkins.plugins.kubernetes.PodAnnotation@9d4da4a8, org.csanchez.jenkins.plugins.kubernetes.PodAnnotation@aab9c821]}

Oct 17, 2019 1:55:40 PM FINEST org.csanchez.jenkins.plugins.kubernetes.PodTemplateUtils
Pod templates combined: PodTemplate{inheritFrom='', name='9ccd91fa-0aba-46d6-b493-f48fb4136a68-60m1x', namespace='', label='9ccd91fa-0aba-46d6-b493-f48fb4136a68', nodeSelector='', nodeUsageMode=EXCLUSIVE, workspaceVolume=org.csanchez.jenkins.plugins.kubernetes.volumes.workspace.DynamicPVCWorkspaceVolume@79ebc880, containers=[ContainerTemplate{name='main', image='docker-registry-default.cnap-00-mp-prod.mycompanygroup.net:443/ci-next/jenkins-slave-java8-mvn:latest', alwaysPullImage=true, workingDir='/home/jenkins/agent', command='/bin/sh -c', args='cat', ttyEnabled=true, resourceRequestCpu='', resourceRequestMemory='', resourceLimitCpu='', resourceLimitMemory='', envVars=[KeyValueEnvVar [getValue()=https://rspsales-cinext.mycompanygroup.net, getKey()=LOCAL_URL], KeyValueEnvVar [getValue()=https://rspsales-cinext.mycompanygroup.net/nexus, getKey()=NEXUS_URL]]}], annotations=[org.csanchez.jenkins.plugins.kubernetes.PodAnnotation@9d4da4a8, org.csanchez.jenkins.plugins.kubernetes.PodAnnotation@aab9c821], yamls=

I´m wondering why the parent PodTemplate has workspaceVolume=DynamicPVCWorkspaceVolume? 
Is this the new default?

I´ll try to enable a custom Workspace (EmptyDir) in Kubernetes plugin configuration - which should prevent from creation of DynamicPVC´s causing the error.