| After upgrade to 1.14.3 from 1.12.7 everything was working fine but after 24 hours no new slave pods were created and following exception was visible in the log: Feb 11, 2019 3:53:01 PM okhttp3.internal.platform.Platform log
INFO: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401} Feb 11, 2019 3:53:01 PM okhttp3.internal.platform.Platform log
INFO: <-- END HTTP (129-byte body)
Feb 11, 2019 3:53:01 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud provision
WARNING: Failed to count the # of live instances on Kubernetes
io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: GET at: https://b22.jonqe.lab.eng.bos.redhat.com:8443/api/v1/namespaces/jenkins-slaves/pods?labelSelector=jenkins%3Dslave. Message: Unauthorized. Received status: Status(apiVersion=v1, code=401, details=null, kind=Status, message=Unauthorized, metadata=ListMeta(_continue=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=Unauthorized, status=Failure, additionalProperties={}).
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:478)
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:417)
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:381)
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:344)
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:328)
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.listRequestHelper(BaseOperation.java:193)
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.list(BaseOperation.java:618)
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.list(BaseOperation.java:68)
at org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud.addProvisionedSlave(KubernetesCloud.java:505)
at org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud.provision(KubernetesCloud.java:458)
at hudson.slaves.NodeProvisioner$StandardStrategyImpl.apply(NodeProvisioner.java:715)
at hudson.slaves.NodeProvisioner.update(NodeProvisioner.java:320)
at hudson.slaves.NodeProvisioner.access$000(NodeProvisioner.java:61)
at hudson.slaves.NodeProvisioner$NodeProvisionerInvoker.doRun(NodeProvisioner.java:809)
at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:72)
at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748) When the jenkins master is restarted, everything is working fine again but after 24 hours it's failing again. Following is visible in openshift log: Feb 14 07:23:45 b22 atomic-openshift-master-api: E0214 07:23:45.537998 2177 authentication.go:64] Unable to authenticate the request due to an error: [invalid bearer token, [invalid bearer token, oauthaccesstokens.oauth.openshift.io "7BM2kmQ6wu8GZx9vOFVupO8W-a5Wc9Unf2ltJtogg2c" not found]] It seems that new version of plugin is using one client with one token which expires by default in 24 hours (OS config: accessTokenMaxAgeSeconds: 86400). When the jenkins is restarted, new client with new token is created and it's working for another 24 hours and then it expires again. It's possible to increase accessTokenMaxAgeSeconds on OS side but it's not a good workaround. It still requires to restart jenkins when the token expires. |
