[JIRA] (JENKINS-52593) Support EKS authentication, or specifying a kubeconfig directly

15 views
Skip to first unread message

marcinromaszewicz@deepmap.ai (JIRA)

unread,
Jul 16, 2018, 4:58:01 PM7/16/18
to jenkinsc...@googlegroups.com
Marcin Romaszewicz created an issue
 
Jenkins / Improvement JENKINS-52593
Support EKS authentication, or specifying a kubeconfig directly
Issue Type: Improvement Improvement
Assignee: Carlos Sanchez
Components: kubernetes-plugin
Created: 2018-07-16 20:57
Priority: Minor Minor
Reporter: Marcin Romaszewicz

The kubernetes-plugin currently supports authenticating into the cluster via the server certificate and username/password.

Amazon's EKS works differently, where you need to use an authentication plugin to generate tokens. The instructions are here (https://docs.aws.amazon.com/eks/latest/userguide/configure-kubectl.html)

The short version is that you need to have an executable in your path, which generates auth tokens on demand. The kubectl file for connecting to EKS looks like so:

apiVersion: v1
clusters:
- cluster:
    server: https://SOME-LONG-ID.yl4.us-west-2.eks.amazonaws.com
    certificate-authority-data: <ca cert goes here>
  name: eks-cluster
contexts:
- context:
    cluster: eks-cluster
    user: eks-user
  name: marcin
current-context: eks
kind: Config
preferences: {}
users:
- name: eks-user
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1alpha1
      command: heptio-authenticator-aws
      args:
        - "token"
        - "-i"
        - "eks-cluster-name"

This configuration is described here:
https://docs.aws.amazon.com/eks/latest/userguide/getting-started.html#eks-configure-kubectl

I wonder if you guys could extend the current plugin to accept a kubeconfig file instead of the resources within it. This way, I could generate whatever kubeconfig is necessary myself, and created an image derived from yours which would contain heptio-authenticator-aws. This would allow for more generalized authentication without you having to support every conceivable alternative.
Add Comment Add Comment
 
This message was sent by Atlassian JIRA (v7.10.1#710002-sha1:6efc396)

jenkins-ci@carlossanchez.eu (JIRA)

unread,
Jul 17, 2018, 6:00:02 AM7/17/18
to jenkinsc...@googlegroups.com
Carlos Sanchez commented on Improvement JENKINS-52593
 
Re: Support EKS authentication, or specifying a kubeconfig directly

I don't think that's the way to configure a service as it still requires your AWS credentials for the binary to create a token.
You should create a service account and use those credentials

marcinromaszewicz@deepmap.ai (JIRA)

unread,
Jul 17, 2018, 7:10:02 PM7/17/18
to jenkinsc...@googlegroups.com

Yeah, you're right, service accounts are a better idea.

syaramada-c@scrippsnetworks.com (JIRA)

unread,
Oct 5, 2018, 10:05:02 AM10/5/18
to jenkinsc...@googlegroups.com

Hi, Can I know if we need to pass access keys and secret keys here to get authenticated with EKS clusters?
This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)

n.c.alebregtse@gmail.com (JIRA)

unread,
Oct 30, 2018, 6:14:02 AM10/30/18
to jenkinsc...@googlegroups.com

I just wanted to give a heads-up here that the kubernetes-client library has been updated to use the aws-iam-authenticator configuration from the kubeconfig file (see [here|https://github.com/fabric8io/kubernetes-client/pull/1224|https://github.com/fabric8io/kubernetes-client/pull/1224]).] ). So the easiest fix now would be to upgrade to a more recent version of the kubernetes-client (e.g. version 4.1.0).

n.c.alebregtse@gmail.com (JIRA)

unread,
Oct 30, 2018, 6:15:01 AM10/30/18
to jenkinsc...@googlegroups.com
Niels Alebregtse edited a comment on Improvement JENKINS-52593

n.c.alebregtse@gmail.com (JIRA)

unread,
Oct 30, 2018, 6:15:01 AM10/30/18
to jenkinsc...@googlegroups.com
Niels Alebregtse edited a comment on Improvement JENKINS-52593
I just wanted to give a heads-up here that the kubernetes-client library has been updated to use the aws-iam-authenticator configuration from the kubeconfig file (see [here|https://github.com/fabric8io/kubernetes-client/pull/1224]). ] ). So the easiest fix now would be to upgrade to a more recent version of the kubernetes-client (e.g. version 4.1.0).

jenkins-ci@carlossanchez.eu (JIRA)

unread,
Oct 30, 2018, 11:58:02 AM10/30/18
to jenkinsc...@googlegroups.com

 the kubernetes-client library has been upgraded in the latest version

jenkins-ci@carlossanchez.eu (JIRA)

unread,
Feb 26, 2019, 4:10:02 AM2/26/19
to jenkinsc...@googlegroups.com
Change By: Carlos Sanchez
Status: Open Closed
Resolution: Fixed

jmukhtar@divaaco.com (JIRA)

unread,
Sep 3, 2019, 6:00:03 AM9/3/19
to jenkinsc...@googlegroups.com
junaid mukhtar commented on Improvement JENKINS-52593
 
Re: Support EKS authentication, or specifying a kubeconfig directly

Carlos Sanchez do you have any howto or a guide on how to do it?

 

I am struggling to connect Jenkins with EKS cluster via the kubernetes plugin. Any help would be highly appreciated

papercrane@reversefold.com (JIRA)

unread,
Oct 23, 2019, 12:50:03 PM10/23/19
to jenkinsc...@googlegroups.com

The documentation does not explain how to configure the kubernetes plugin to use EKS. It just mentions aws-iam-authenticator, nothing about where it needs to be or what values need to be put into the cloud configuration. I've assigned an IAM role with full EKS permissions to the server but the credentials dropdown won't let me select a credential with the IAM role and won't allow me to select an AWS access/secret key.

Turning off the https certificate check gives me an error about the system:anonymous user not having permissions. What do I need to do to get the kubernetes plugin to authenticate properly to EKS?

Error testing connection https://<ID>.gr7.us-west-2.eks.amazonaws.com: Failure executing: GET at: https://<ID>.gr7.us-west-2.eks.amazonaws.com/api/v1/namespaces/default/pods. Message: pods is forbidden: User "system:anonymous" cannot list resource "pods" in API group "" in the namespace "default". Received status: Status(apiVersion=v1, code=403, details=StatusDetails(causes=[], group=null, kind=pods, name=null, retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status, message=pods is forbidden: User "system:anonymous" cannot list resource "pods" in API group "" in the namespace "default", metadata=ListMeta(_continue=null, remainingItemCount=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=Forbidden, status=Failure, additionalProperties={}).
This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f)
Atlassian logo

syaramada-c@scrippsnetworks.com (JIRA)

unread,
Oct 23, 2019, 1:07:02 PM10/23/19
to jenkinsc...@googlegroups.com
suryatej yaramada updated an issue
 
Change By: suryatej yaramada
Attachment: Screenshot from 2019-10-23 13-05-42.png

syaramada-c@scrippsnetworks.com (JIRA)

unread,
Oct 23, 2019, 1:07:03 PM10/23/19
to jenkinsc...@googlegroups.com
suryatej yaramada commented on Improvement JENKINS-52593
 
Re: Support EKS authentication, or specifying a kubeconfig directly

Right now we are creating a service account and allowing access to EKS from Jenkins to configure kubernetes-plugin

papercrane@reversefold.com (JIRA)

unread,
Oct 23, 2019, 2:35:03 PM10/23/19
to jenkinsc...@googlegroups.com

suryatej yaramada Can you give some details about this? What is the configuration of the service account? How is it connected to jenkins? How are you setting up that Credential? Are you also setting up a kubeconfig somewhere?

calebmayeux@hotmail.com (JIRA)

unread,
Dec 3, 2019, 7:12:03 PM12/3/19
to jenkinsc...@googlegroups.com

Justin Patrin and junaid mukhtar jI'm in the same boat as y'all. I also haven't gotten it to work, but based on spending entirely too much time googling and looking through code changes, I think the general idea is to do something like this (this example is if you're using the public jenkins docker image):

FROM jenkins/jenkins:2.176.4
USER root
RUN apt-get update && apt-get install -y python3-pip vim
RUN pip3 install awscli
USER jenkins

 

Then inside the container you exec in and run the command "aws configure" and put in the access and secret key from IAM.

Then you create a file ~/.kube/config and fill it out as per https://docs.aws.amazon.com/eks/latest/userguide/create-kubeconfig.html

Test out that the command as per that config file works, i.e. run "aws eks get-token --cluster-name whateverYouNamedYourCluster" and make sure it spits out a json that has a token in it.

 

That's what I've cobbled together based on looking at this change https://github.com/fabric8io/kubernetes-client/pull/1224/commits/ef2c87472d87e144da09190e1896a9dcbf6208c4

and looking at the readme for the kubernetes client here: https://github.com/fabric8io/kubernetes-client 

 

Like I said, this hasn't worked for me yet, but I feel like I'm close. If this helps you solve it, please post how to do it here for me and anyone else who runs into this issue. We should probably open an issue to update the plugin readme on this as well. If we can figure out how to get it to work we could even make that PR.

 

Also if suryatej yaramada or Carlos Sanchez wanted to weigh in on how to configure using this I'd be much obliged. Thanks!

kmscherer@gmail.com (JIRA)

unread,
Dec 5, 2019, 3:32:04 PM12/5/19
to jenkinsc...@googlegroups.com

I recently got Jenkins working with EKS and I feel your pain. Here is what I ended up cobbling together.

I also installed python3, pip and awscli into my Jenkins container. I put the aws credentials directly in /home/jenkins/.aws/config though I read later you can have a Role on the EC2 instance and awscli can be configured to retrieve temp credentials for that role so no credentials need to stored on the instance.

I was able to create the kubeconfig file and run the 'aws eks get-token' command but I couldn't figure out how to configure Jenkins. I was saved by a mailing list post and Google: https://groups.google.com/d/msg/jenkinsci-users/8YpQL3eG-Zg/tznSHvkYAQAJ

I also added "-Dorg.csanchez.jenkins.plugins.kubernetes.clients.cacheExpiration=60" to the Jenkins Master startup options as mentioned in https://github.com/jenkinsci/kubernetes-plugin#running-with-a-remote-kubernetes-cloud-in-aws-eks

The kubeconfig needs to added to Jenkins as a "File Secret" in the Credentials section. Then in the Kubernetes plugin configuration, the credentials option must be set to that secret. The rest of the fields can be ignored. I hope this will helps

kmscherer@gmail.com (JIRA)

unread,
Dec 5, 2019, 4:10:02 PM12/5/19
to jenkinsc...@googlegroups.com
Konrad Scherer edited a comment on Improvement JENKINS-52593

talliskane@gmail.com (JIRA)

unread,
Dec 13, 2019, 4:37:02 PM12/13/19
to jenkinsc...@googlegroups.com

Thanks for the summary Konrad, I was able to follow along with my setup.

My Jenkins master is trying to authenticate with the entire kube config tucked away in a secret. Also, I can see that my jenkins user on the master is able to run the token 'aws eks get-token' successfully.

I am however running into the "Message: Unauthorized! Token may have expired! " issue, even with my Jenkins master configured to use the correct java opt

/etc/alternatives/java -Dcom.sun.akuma.Daemon=daemonized -Dorg.csanchez.jenkins.plugins.kubernetes.clients.cacheExpiration=60 -DJENKINS_HOME=/var/lib/jenkins -jar /usr/lib/jenkins/jenkins.war ...

I'm wondering what else can be involved, and how I can verify that the cache is actually the issue. Do you have any ideas?

Some info:

openjdk version "1.8.0_222"
ami linux: 4.14.152-127.182.amzn2.x86_64
jenkins kube plugin: org.csanchez.jenkins.plugins:kubernetes:1.22.0
Jenkins: 2.208 
kubectl: v1.17.0 / v1.14.8-eks-b8860f
aws-cli/1.16.303 Python/2.7.16 Linux/4.14.152-127.182.amzn2.x86_64 botocore/1.13.39

Thanks in advance.

talliskane@gmail.com (JIRA)

unread,
Dec 13, 2019, 4:38:02 PM12/13/19
to jenkinsc...@googlegroups.com
Tallis Vanek edited a comment on Improvement JENKINS-52593
Thanks for the summary Konrad [~kscherer] , I was able to follow along with my setup.


My Jenkins master is trying to authenticate with the entire kube config tucked away in a secret. Also, I can see that my jenkins user on the master is able to run the token 'aws eks get-token' successfully.

I am however running into the "Message: Unauthorized! Token may have expired! " issue, even with my Jenkins master configured to use the correct java opt
{code:java}

/etc/alternatives/java -Dcom.sun.akuma.Daemon=daemonized -Dorg.csanchez.jenkins.plugins.kubernetes.clients.cacheExpiration=60 -DJENKINS_HOME=/var/lib/jenkins -jar /usr/lib/jenkins/jenkins.war ...
{code}

I'm wondering what else can be involved, and how I can verify that the cache is actually the issue. Do you have any ideas?

Some info:
{code:java}

openjdk version "1.8.0_222"
ami linux: 4.14.152-127.182.amzn2.x86_64
jenkins kube plugin: org.csanchez.jenkins.plugins:kubernetes:1.22.0
Jenkins: 2.208 
kubectl: v1.17.0 / v1.14.8-eks-b8860f
aws-cli/1.16.303 Python/2.7.16 Linux/4.14.152-127.182.amzn2.x86_64 botocore/1.13.39
{code}
Thanks in advance.

marcinromaszewicz@deepmap.ai (JIRA)

unread,
Dec 13, 2019, 5:07:03 PM12/13/19
to jenkinsc...@googlegroups.com

Since originally filing this bug, I've been using service account based kubeconfig files without issue against EKS. It's easier than using the aws-iam-authenticator, since all you do is bake in a user token into your kubeconfig, and you can constrain it to namespaces or whatever. Doing the same with IAM credentials is more annoying. Try that route instead of directly using your IAM role for EKS.

kmscherer@gmail.com (JIRA)

unread,
Dec 13, 2019, 8:00:02 PM12/13/19
to jenkinsc...@googlegroups.com

Something to check is the EKS user configuration and make sure the credentials available to the Jenkins user are properly configured with a Kubernetes role in EKS:

https://docs.aws.amazon.com/eks/latest/userguide/add-user-role.html

Only the credentials that created the EKS cluster have a proper Kubernetes role by default.
Reply all
Reply to author
Forward
0 new messages