| We observed this in version 1.17.2 of the kubernetes-plugin. We had reached our maximum pod limit (100) in our kubernetes cluster, all of which had failing JNLP containers. They all had the same error: the master had refused the connection as it had an 'unknown client name' Agent JNLP container logs:
Aug 14, 2019 9:51:40 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Protocol JNLP4-connect encountered an unexpected exception
java.util.concurrent.ExecutionException: org.jenkinsci.remoting.protocol.impl.ConnectionRefusalException: Unknown client name: proteus-apply-central-v95mf
at org.jenkinsci.remoting.util.SettableFuture.get(SettableFuture.java:223)
at hudson.remoting.Engine.innerRun(Engine.java:614)
at hudson.remoting.Engine.run(Engine.java:474)
Caused by: org.jenkinsci.remoting.protocol.impl.ConnectionRefusalException: Unknown client name: proteus-apply-central-v95mf
at org.jenkinsci.remoting.protocol.impl.ConnectionHeadersFilterLayer.newAbortCause(ConnectionHeadersFilterLayer.java:378)
at org.jenkinsci.remoting.protocol.impl.ConnectionHeadersFilterLayer.onRecvClosed(ConnectionHeadersFilterLayer.java:433)
at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecvClosed(ProtocolStack.java:816)
at org.jenkinsci.remoting.protocol.FilterLayer.onRecvClosed(FilterLayer.java:287)
at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.onRecvClosed(SSLEngineFilterLayer.java:172)
at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecvClosed(ProtocolStack.java:816)
at org.jenkinsci.remoting.protocol.NetworkLayer.onRecvClosed(NetworkLayer.java:154)
at org.jenkinsci.remoting.protocol.impl.BIONetworkLayer.access$1500(BIONetworkLayer.java:48)
at org.jenkinsci.remoting.protocol.impl.BIONetworkLayer$Reader.run(BIONetworkLayer.java:247)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:93)
at java.lang.Thread.run(Thread.java:748)
Suppressed: java.nio.channels.ClosedChannelException
... 7 more
"Local headers refused by remote: Unknown client name:"
Here is the jenkins master log of a single one of these failing pods (ui-pipeline-shared-7sdt7). Apologies I did not discover[ the debug logging settings|https://github.com/jenkinsci/kubernetes-plugin#debugging] during our incident so I do not have much logs:
k logs jenkins-icecream-68c6d8497d-xhl9x -n cicd | grep ui-pipeline-shared-7sdt7 (central/ping-services)
INFO: Terminating Kubernetes instance for agent ui-pipeline-shared-7sdt7
WARNING: Failed to delete pod for agent cicd/ui-pipeline-shared-7sdt7: not found
INFO: Disconnected computer ui-pipeline-shared-7sdt7
ERROR: Failed to delete pod for agent cicd/ui-pipeline-shared-7sdt7: not found
Disconnected computer ui-pipeline-shared-7sdt7
INFO: Created Pod: cicd/ui-pipeline-shared-7sdt7
INFO: Pod is running: cicd/ui-pipeline-shared-7sdt7
WARNING: Error in provisioning; agent=KubernetesSlave name: ui-pipeline-shared-7sdt7, template=PodTemplate{inheritFrom='', name='ui-pipeline-shared', namespace='cicd', slaveConnectTimeout=1000, label='ui-pipeline-shared', nodeSelector='', nodeUsageMode=EXCLUSIVE, workspaceVolume=EmptyDirWorkspaceVolume [memory=false], yamls=[apiVersion: v1
INFO: Terminating Kubernetes instance for agent ui-pipeline-shared-7sdt7
SEVERE: Computer for agent is null: ui-pipeline-shared-7sdt7
FATAL: Computer for agent is null: ui-pipeline-shared-7sdt7
INFO: [JNLP4-connect connection from 100.96.0.1/100.96.0.1:23036] Refusing headers from remote: Unknown client name: ui-pipeline-shared-7sdt7
As you can see here the termination of the ui-pipeline-shared-7sdt7 pod happens before the pod is 'created' or 'running'. Here are full master logs, ungrepped:
Aug 14, 2019 11:09:26 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud provision
INFO: Template for label ui-pipeline-shared: Kubernetes Pod Template
Aug 14, 2019 11:09:26 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave deleteSlavePod
WARNING: Failed to delete pod for agent cicd/proteus-apply-central-8h7bv: not found
Aug 14, 2019 11:09:26 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave _terminate
INFO: Disconnected computer proteus-apply-central-8h7bv
ERROR: Failed to delete pod for agent cicd/proteus-apply-central-8h7bv: not found
Disconnected computer proteus-apply-central-8h7bv
Aug 14, 2019 11:09:26 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave _terminate
INFO: Terminating Kubernetes instance for agent ui-pipeline-shared-kvc2j
Aug 14, 2019 11:09:26 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave deleteSlavePod
WARNING: Failed to delete pod for agent null/ui-pipeline-shared-kvc2j: not found
Aug 14, 2019 11:09:26 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave _terminate
INFO: Disconnected computer ui-pipeline-shared-kvc2j
ERROR: Failed to delete pod for agent null/ui-pipeline-shared-kvc2j: not found
Disconnected computer ui-pipeline-shared-kvc2j
Aug 14, 2019 11:09:26 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch
INFO: Created Pod: cicd/ui-pipeline-shared-kvc2j
WARNING: Error in provisioning; agent=KubernetesSlave name: proteus-apply-central-8h7bv, template=PodTemplate{inheritFrom='', name='proteus-apply-central', namespace='cicd', slaveConnectTimeout=1000, label='proteus-apply-central', nodeSelector='', nodeUsageMode=EXCLUSIVE, workspaceVolume=EmptyDirWorkspaceVolume [memory=false], yamls=[apiVersion: v1 kind: Pod metadata: labels: jenkins: runners annotations: iam.amazonaws.com/role: arn:aws:iam::208980577242:role/devtools_jenkins spec: containers: - name: proteus-apply image: docker.corp.pingidentity.com:5000/devtools/proteus/pipeline-images/proteus-apply:stable imagePullPolicy: Always command: - cat tty: true ]}java.lang.IllegalStateException: Node was deleted, computer is null
at org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher.launch(KubernetesLauncher.java:149)
at hudson.slaves.SlaveComputer$1.call(SlaveComputer.java:294)
at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
at jenkins.security.ImpersonatingExecutorService$2.call(ImpersonatingExecutorService.java:71)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)Aug 14, 2019 11:09:33 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch
INFO: Pod is running: cicd/ui-pipeline-shared-kvc2j
Aug 14, 2019 11:09:33 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch
WARNING: Error in provisioning; agent=KubernetesSlave name: ui-pipeline-shared-kvc2j, template=PodTemplate{inheritFrom='', name='ui-pipeline-shared', namespace='cicd', slaveConnectTimeout=1000, label='ui-pipeline-shared', nodeSelector='', nodeUsageMode=EXCLUSIVE, workspaceVolume=EmptyDirWorkspaceVolume [memory=false], yamls=[apiVersion: v1 kind: Pod metadata: labels: jenkins: runners annotations: #kube2iam iam.amazonaws.com/role: arn:aws:iam::208980577242:role/devtools_jenkins spec: containers: - command: - cat image: docker.corp.pingidentity.com:5000/ping-base/node-builder:10 imagePullPolicy: Always name: node-builder tty: true - command: - cat # This uses the image built on the 'icecream' feature branch here: https://gitlab.corp.pingidentity.com/platform-pipeline/platform-js-static-analysis-service/tree/icecream. image: docker.corp.pingidentity.com:5000/platform-pipeline/platform-js-static-analysis-service:icecream imagePullPolicy: Always name: platform-js-static-analysis-service tty: true env: - name: PING_SONAR_PASSWORD valueFrom: secretKeyRef: name: sonarqube key: password - command: - cat # This uses the image built on the https://gitlab.corp.pingidentity.com/devtools/icecream/cdn_deployer image: docker.corp.pingidentity.com:5000/devtools/icecream/cdn_deployer:stable imagePullPolicy: Always name: cdn-deploy tty: true ]}java.lang.IllegalStateException: Node was deleted, computer is null
at org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher.launch(KubernetesLauncher.java:149)
at hudson.slaves.SlaveComputer$1.call(SlaveComputer.java:294)
at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
at jenkins.security.ImpersonatingExecutorService$2.call(ImpersonatingExecutorService.java:71)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)Aug 14, 2019 11:09:33 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave _terminate
INFO: Terminating Kubernetes instance for agent ui-pipeline-shared-kvc2j
Aug 14, 2019 11:09:33 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave _terminate
SEVERE: Computer for agent is null: ui-pipeline-shared-kvc2j
FATAL: Computer for agent is null: ui-pipeline-shared-kvc2j
Aug 14, 2019 11:09:34 PM org.jenkinsci.remoting.protocol.impl.ConnectionHeadersFilterLayer onRecv
INFO: [JNLP4-connect connection from 100.96.0.1/100.96.0.1:28564] Refusing headers from remote: Unknown client name: ui-pipeline-shared-kvc2j
|
