Re: Integrating Jenkins with EKS Cluster

[Junaid Mukhtar]

Hi Team

We are using Jenkins to run build, test and deploy jobs to our KOPS Cluster successfully without any issues. Jenkins is running on a dedicated EC2 instance

We are now planning to switch over to EKS Cluster and want to point Jenkins to the new EKS cluster to perform the same actions.

We are using Kubernetes Plugin for the KOPS cluster which uses certificate and username/password for authentication. But for EKS AWS uses aws-iam-authenticator that goes into the kubeconfig to generate on the fly tokens and so far we haven't been able to come up with a way to use aws-iam-authenticator to integrate Jenkins with EKS cluster.

I would be really greateful if you could point me in a right direction or share with me the steps to integrate Jenkins with EKS cluster.

--------
Regards,
Junaid

[Vincent Latombe]

You can provide a kubeconfig file as credentials. You'll need aws-iam-authenticator set up on the EC2 instance you have Jenkins installed on, as well as running your EC2 instance with an instance profile with the ability to log in to your EKS cluster.

Out of curiosity, why don't you run your Jenkins instance directly in your EKS cluster? Then it would use directly service account credentials.

Vincent

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/CAPO_fvoE9RnNETYENS%3DfRdmsgbh-hktj7fQm3Q_mp_7S0UG%2BFw%40mail.gmail.com.

[Junaid Mukhtar]

Hi Vincent

We are currently running Jenkins (EC2) and Kubernetes (EC2) and are trying to migrate from Kubernetes to EKS for ease of management. I was hoping to do one step at a time and reuse as much as possible in Jenkins. But if moving Jenkins into EKS would make life a lot easier then i can definitely do that.

With the first approach, i will have to run the deployments by running the kubectl apply commands right?

--------
Regards,
Junaid

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/CAH-zGCjQARuJkvoUkUJ33_gfj2A1tAgg%2B7%2BN4XfCuG_-CPBByA%40mail.gmail.com.

[Vincent Latombe]

No, you just need to configure the kubernetes cloud by providing credentials type secret file, and you provide a kubeconfig that you validated works using kubectl. It replaces the configuration completely (including host).

One more gotcha with EKS: https://github.com/jenkinsci/kubernetes-plugin#running-with-a-remote-kubernetes-cloud-in-aws-eks

Vincent

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/CAPO_fvrWSa9KYniO8VybwoA6gW2GDRDsQF49bAHVNiZ4Y7gECQ%40mail.gmail.com.

[Junaid Mukhtar]

I have made the necessary configurations but I am getting following exception; it looks like Jenkins is not sending the configured username properly

io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: GET at: https://xxxxx.sk1.eu-west-1.eks.amazonaws.com/api/v1/namespaces/kube-system/pods?labelSelector=jenkins%3Dslave. Message: pods is forbidden: User "syste
m:anonymous" cannot list resource "pods" in API group "" in the namespace "kube-system". Received status: Status(apiVersion=v1, code=403, details=StatusDetails(causes=[], group=null, kind=pods, name=null, retryAfterSeconds=null, uid=null, additionalProperties={
}), kind=Status, message=pods is forbidden: User "system:anonymous" cannot list resource "pods" in API group "" in the namespace "kube-system", metadata=ListMeta(_continue=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=Forbidden, st
atus=Failure, additionalProperties={}).
        at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:476)
        at io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:415)
        at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:381)
        at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:344)
        at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:328)
        at io.fabric8.kubernetes.client.dsl.base.BaseOperation.list(BaseOperation.java:598)
        at io.fabric8.kubernetes.client.dsl.base.BaseOperation.list(BaseOperation.java:63)
        at org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud.addProvisionedSlave(KubernetesCloud.java:494)
        at org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud.provision(KubernetesCloud.java:447)
        at hudson.slaves.NodeProvisioner$StandardStrategyImpl.apply(NodeProvisioner.java:715)
        at hudson.slaves.NodeProvisioner.update(NodeProvisioner.java:320)
        at hudson.slaves.NodeProvisioner.access$000(NodeProvisioner.java:62)
        at hudson.slaves.NodeProvisioner$NodeProvisionerInvoker.doRun(NodeProvisioner.java:809)
        at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:72)
        at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

--------
Regards,
Junaid

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/CAH-zGCjKAkuxVvZdRYgHJBarDPETyfc2xwmbi%3D_28Xy27rJPgA%40mail.gmail.com.

[Junaid Mukhtar]

Made some progress, Jenkins on EC2 is able to connect to the EKS Cluster via the kubeconfig and test connection is fine. I also tested the kubectl command from the jenkins servers and it also works fine

But when i try to run a pipeline i get followin

io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: GET at: https://xxxxx.sk1.eu-west-1.eks.amazonaws.com/api/v1/namespaces/jenkins/pods?labelSelector=jenkins%3Dslave. Message: Unauthorized! Token may have expired! Please log-in again. Unauthorized.
        at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:503)
        at io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:440)
        at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:406)
        at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:365)
        at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:347)
        at io.fabric8.kubernetes.client.dsl.base.BaseOperation.listRequestHelper(BaseOperation.java:145)
        at io.fabric8.kubernetes.client.dsl.base.BaseOperation.list(BaseOperation.java:612)
        at io.fabric8.kubernetes.client.dsl.base.BaseOperation.list(BaseOperation.java:63)
        at org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud.getActiveSlavePods(KubernetesCloud.java:581)
        at org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud.addProvisionedSlave(KubernetesCloud.java:556)
        at org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud.provision(KubernetesCloud.java:508)

        at hudson.slaves.NodeProvisioner$StandardStrategyImpl.apply(NodeProvisioner.java:715)
        at hudson.slaves.NodeProvisioner.update(NodeProvisioner.java:320)
        at hudson.slaves.NodeProvisioner.access$000(NodeProvisioner.java:62)
        at hudson.slaves.NodeProvisioner$NodeProvisionerInvoker.doRun(NodeProvisioner.java:809)
        at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:72)
        at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

Any idea guys?

--------

Regards,
Junaid

[Junaid Mukhtar]

Hi Vincent

Do you know how to set the JAVA Arg for the Jenkins running in the EKS and is installed by the helm install command?

--------
Regards,
Junaid

On Fri, Sep 6, 2019 at 12:55 PM Vincent Latombe <vincent...@gmail.com> wrote:

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/CAH-zGCjKAkuxVvZdRYgHJBarDPETyfc2xwmbi%3D_28Xy27rJPgA%40mail.gmail.com.