[JIRA] (JENKINS-60105) Kubernetes plugin fails with EKS

18 views
Skip to first unread message

jniebuhr@cfire.de (JIRA)

unread,
Nov 8, 2019, 6:44:03 AM11/8/19
to jenkinsc...@googlegroups.com
Jochen Niebuhr created an issue
 
Jenkins / Bug JENKINS-60105
Kubernetes plugin fails with EKS
Issue Type: Bug Bug
Assignee: Unassigned
Components: kubernetes-plugin
Created: 2019-11-08 11:43
Priority: Minor Minor
Reporter: Jochen Niebuhr

I'm using a Jenkins installation which is run on a standard ec2 instance. This Jenkins instance is supposed to connect to an EKS cluster. EKS cluster auth is done via aws-iam-authenticator which is correctly configured on the machine. I've put all the config in the .kube/config for the jenkins user.
The kubernetes plugin will pick up the k8s config just fine and succeed through all connection tests. It will also run builds just fine for about 15 minutes. At that point, the token issues from aws iam authenticator will be expired and no more pods can be created for my builds. Configuration tests do still work after that, so I guess the plugin creates a new client for those.

The plugin will print the following error message:

```
2019-11-08 11:32:50.552+0000 [id=24] WARNING o.c.j.p.k.KubernetesCloud#provision: Failed to count the # of live instances on Kubernetes2019-11-08 11:32:50.552+0000 [id=24] WARNING o.c.j.p.k.KubernetesCloud#provision: Failed to count the # of live instances on Kubernetesio.fabric8.kubernetes.client.KubernetesClientException: Failure executing: GET at: https://296cb72614fca8d61a1b611020af335c.yl4.eu-west-1.eks.amazonaws.com/api/v1/namespaces/jenkins/pods?labelSelector=app.kubernetes.io%2Fpart-of%3Djenkins%2Capp.kubernetes.io%2Fcomponent%3Dslave. Message: Unauthorized! Token may have expired! Please log-in again. Unauthorized. at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:510) at io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:447) at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:413) at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:372) at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:354) at io.fabric8.kubernetes.client.dsl.base.BaseOperation.listRequestHelper(BaseOperation.java:153) at io.fabric8.kubernetes.client.dsl.base.BaseOperation.list(BaseOperation.java:620) at io.fabric8.kubernetes.client.dsl.base.BaseOperation.list(BaseOperation.java:69) at org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud.getActiveSlavePods(KubernetesCloud.java:615) at org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud.addProvisionedSlave(KubernetesCloud.java:590) at org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud.provision(KubernetesCloud.java:542) at hudson.slaves.NodeProvisioner$StandardStrategyImpl.apply(NodeProvisioner.java:729) at hudson.slaves.NodeProvisioner.update(NodeProvisioner.java:332) at hudson.slaves.NodeProvisioner.access$900(NodeProvisioner.java:63) at hudson.slaves.NodeProvisioner$NodeProvisionerInvoker.doRun(NodeProvisioner.java:823) at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:72) at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)
```

Add Comment Add Comment
 
This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f)
Atlassian logo

jniebuhr@cfire.de (JIRA)

unread,
Nov 8, 2019, 6:45:02 AM11/8/19
to jenkinsc...@googlegroups.com
Jochen Niebuhr updated an issue
Change By: Jochen Niebuhr
I'm using a Jenkins installation which is run on a standard ec2 instance. This Jenkins instance is supposed to connect to an EKS cluster. EKS cluster auth is done via aws-iam-authenticator which is correctly configured on the machine. I've put all the config in the .kube/config for the jenkins user.
The kubernetes plugin will pick up the k8s config just fine and succeed through all connection tests. It will also run builds just fine for about 15 minutes. At that point, the token issues from aws iam authenticator will be expired and no more pods can be created for my builds. Configuration tests do still work after that, so I guess the plugin creates a new client for those.

The plugin will print the following error message:

{code :java }
2019-11-08 11:32:50.552+0000 [id=24] WARNING o.c.j.p.k.KubernetesCloud#provision: Failed to count the # of live instances on Kubernetes2019-11-08 11:32:50.552+0000 [id=24] WARNING o.c.j.p.k.KubernetesCloud#provision: Failed to count the # of live instances on Kubernetesio.fabric8.kubernetes.client.KubernetesClientException: Failure executing: GET at: [https://
296cb72614fca8d61a1b611020af335c . yl4 . .. eu-west-1.eks.amazonaws.com/api/v1/namespaces/jenkins/pods?labelSelector=app.kubernetes.io%2Fpart-of%3Djenkins%2Capp.kubernetes.io%2Fcomponent%3Dslave]. Message: Unauthorized! Token may have expired! Please log-in again. Unauthorized. at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:510) at io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:447) at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:413) at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:372) at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:354) at io.fabric8.kubernetes.client.dsl.base.BaseOperation.listRequestHelper(BaseOperation.java:153) at io.fabric8.kubernetes.client.dsl.base.BaseOperation.list(BaseOperation.java:620) at io.fabric8.kubernetes.client.dsl.base.BaseOperation.list(BaseOperation.java:69) at org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud.getActiveSlavePods(KubernetesCloud.java:615) at org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud.addProvisionedSlave(KubernetesCloud.java:590) at org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud.provision(KubernetesCloud.java:542) at hudson.slaves.NodeProvisioner$StandardStrategyImpl.apply(NodeProvisioner.java:729) at hudson.slaves.NodeProvisioner.update(NodeProvisioner.java:332) at hudson.slaves.NodeProvisioner.access$900(NodeProvisioner.java:63) at hudson.slaves.NodeProvisioner$NodeProvisionerInvoker.doRun(NodeProvisioner.java:823) at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:72) at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)

{code}

jniebuhr@cfire.de (JIRA)

unread,
Nov 8, 2019, 6:45:03 AM11/8/19
to jenkinsc...@googlegroups.com
Jochen Niebuhr updated an issue
I'm using a Jenkins installation which is run on a standard ec2 instance. This Jenkins instance is supposed to connect to an EKS cluster. EKS cluster auth is done via aws-iam-authenticator which is correctly configured on the machine. I've put all the config in the .kube/config for the jenkins user.
The kubernetes plugin will pick up the k8s config just fine and succeed through all connection tests. It will also run builds just fine for about 15 minutes. At that point, the token issues from aws iam authenticator will be expired and no more pods can be created for my builds. Configuration tests do still work after that, so I guess the plugin creates a new client for those.

The plugin will print the following error message:

``` {code}

2019-11-08 11:32:50.552+0000 [id=24] WARNING o.c.j.p.k.KubernetesCloud#provision: Failed to count the # of live instances on Kubernetes2019-11-08 11:32:50.552+0000 [id=24] WARNING o.c.j.p.k.KubernetesCloud#provision: Failed to count the # of live instances on Kubernetesio.fabric8.kubernetes.client.KubernetesClientException: Failure executing: GET at:
[ https://296cb72614fca8d61a1b611020af335c.yl4.eu-west-1.eks.amazonaws.com/api/v1/namespaces/jenkins/pods?labelSelector=app.kubernetes.io%2Fpart-of%3Djenkins%2Capp.kubernetes.io%2Fcomponent%3Dslave ] . Message: Unauthorized! Token may have expired! Please log-in again. Unauthorized. at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:510) at io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:447) at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:413) at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:372) at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:354) at io.fabric8.kubernetes.client.dsl.base.BaseOperation.listRequestHelper(BaseOperation.java:153) at io.fabric8.kubernetes.client.dsl.base.BaseOperation.list(BaseOperation.java:620) at io.fabric8.kubernetes.client.dsl.base.BaseOperation.list(BaseOperation.java:69) at org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud.getActiveSlavePods(KubernetesCloud.java:615) at org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud.addProvisionedSlave(KubernetesCloud.java:590) at org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud.provision(KubernetesCloud.java:542) at hudson.slaves.NodeProvisioner$StandardStrategyImpl.apply(NodeProvisioner.java:729) at hudson.slaves.NodeProvisioner.update(NodeProvisioner.java:332) at hudson.slaves.NodeProvisioner.access$900(NodeProvisioner.java:63) at hudson.slaves.NodeProvisioner$NodeProvisionerInvoker.doRun(NodeProvisioner.java:823) at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:72) at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)
```
{code}

calebmayeux@hotmail.com (JIRA)

unread,
Dec 3, 2019, 7:14:05 PM12/3/19
to jenkinsc...@googlegroups.com
Caleb Mayeux commented on Bug JENKINS-60105
 
Re: Kubernetes plugin fails with EKS

Jochen Niebuhr I think this is covered in the plugin readme under the EKS section (https://plugins.jenkins.io/kubernetes)

Copy/pasting from that readme for convenience:

Running with a remote Kubernetes Cloud in AWS EKS

EKS enforces authentication to the cluster through aws-iam-authenticator. The token expires after 15 minutes so the kubernetes client cache needs to be set to something below this by setting a java argument, like so:

 

JAVA_ARGS="-Dorg.csanchez.jenkins.plugins.kubernetes.clients.cacheExpiration=60"

vincent@latombe.net (JIRA)

unread,
Jan 24, 2020, 10:19:05 AM1/24/20
to jenkinsc...@googlegroups.com

Caleb is correct, but as this is a recurring problem, I believe the plugin could use a shorter timeout to avoid special tuning when using EKS.

vincent@latombe.net (JIRA)

unread,
Jan 24, 2020, 10:19:07 AM1/24/20
to jenkinsc...@googlegroups.com
Vincent Latombe assigned an issue to Vincent Latombe
 
Change By: Vincent Latombe
Assignee: Vincent Latombe

vincent@latombe.net (JIRA)

unread,
Jan 24, 2020, 10:19:07 AM1/24/20
to jenkinsc...@googlegroups.com

vincent@latombe.net (JIRA)

unread,
Jan 24, 2020, 10:19:09 AM1/24/20
to jenkinsc...@googlegroups.com
Vincent Latombe started work on Bug JENKINS-60105
 
Change By: Vincent Latombe
Status: Open In Progress

vincent@latombe.net (JIRA)

unread,
Feb 19, 2020, 4:46:03 AM2/19/20
to jenkinsc...@googlegroups.com
Change By: Vincent Latombe
Status: In Review Resolved
Resolution: Fixed
Released As: kubernetes 1.24.0
Reply all
Reply to author
Forward
0 new messages