[JIRA] (JENKINS-57352) Test connection fails and also unable to get K8s pods via Jenkins using service account

1 view
Skip to first unread message

harsha_gv061@ymail.com (JIRA)

unread,
May 7, 2019, 6:55:03 AM5/7/19
to jenkinsc...@googlegroups.com
Harsha GV created an issue
 
Jenkins / Bug JENKINS-57352
Test connection fails and also unable to get K8s pods via Jenkins using service account
Issue Type: Bug Bug
Assignee: Carlos Sanchez
Components: kubernetes-plugin
Created: 2019-05-07 10:54
Environment: Jenkins in K8s
Labels: kubernetes kubernetes-plugin
Priority: Minor Minor
Reporter: Harsha GV

Provisioning of K8s pods from Kubernetes Jenkins plugin using the my cloud credentials is failing, hence created service account using below command:

PS: Jenkins is deployed in K8s 

kubectl  -n mynamespace create serviceaccount jenkins

Then tried Test Connection from K8s-plugin, but connection failed with below error:

Error testing connection https://api.k8s2.apac.cloud.net:6443: Failure executing: GET at: https://api.k8s2.apac.cloud.net:6443/api/v1/namespaces/mynamespace/pods. Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. pods is forbidden: User "system:serviceaccount:mynamespace:default" cannot list resource "pods" in API group "" in the namespace "mynamespace": Unexpected user-id: system:serviceaccount:mynamespace:default.

Also when jenkins triggered to run in labeled k8s pods, fails with below error:

Failed to count the # of live instances on Kubernetes
io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: GET at: https://api.k8s2.apac.cloud.net:6443/api/v1/namespaces/mynamespace/pods?labelSelector=jenkins%3Dslave. Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. pods is forbidden: User "system:anonymous" cannot list resource "pods" in API group "" in the namespace "mynamespace": Unexpected user-id: system:anonymous.
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:472)
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:409)
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:381)
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:344)
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:328)
	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.list(BaseOperation.java:584)
	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.list(BaseOperation.java:49)
	at org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud.addProvisionedSlave(KubernetesCloud.java:493)
	at org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud.provision(KubernetesCloud.java:448)
	at hudson.slaves.NodeProvisioner$StandardStrategyImpl.apply(NodeProvisioner.java:715)
	at hudson.slaves.NodeProvisioner.update(NodeProvisioner.java:320)
	at hudson.slaves.NodeProvisioner.access$000(NodeProvisioner.java:61)
	at hudson.slaves.NodeProvisioner$NodeProvisionerInvoker.doRun(NodeProvisioner.java:809)
	at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:72)
	at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Add Comment Add Comment
 
This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)

harsha_gv061@ymail.com (JIRA)

unread,
May 7, 2019, 6:58:02 AM5/7/19
to jenkinsc...@googlegroups.com
Harsha GV updated an issue
Change By: Harsha GV
*PS: Jenkins is deployed in K8s and not using openshift [doe not intend to]*

Provisioning of K8s pods from Kubernetes Jenkins plugin using the my cloud credentials is failing, hence created service account using below command:
PS: Jenkins is deployed in K8s

{code :java }

kubectl  -n mynamespace create serviceaccount jenkins
{code}
Then tried Test Connection from K8s-plugin, but connection failed with below error:
{code :java }

Error testing connection https://api.k8s2.apac.cloud.net:6443: Failure executing: GET at: https://api.k8s2.apac.cloud.net:6443/api/v1/namespaces/mynamespace/pods. Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. pods is forbidden: User "system:serviceaccount:mynamespace:default" cannot list resource "pods" in API group "" in the namespace "mynamespace": Unexpected user-id: system:serviceaccount:mynamespace:default.
{code}
Also when jenkins triggered to run in labeled k8s pods, fails with below error:
{code :java }

Failed to count the # of live instances on Kubernetes
io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: GET at: https://api.k8s2.apac.cloud.net:6443/api/v1/namespaces/mynamespace/pods?labelSelector=jenkins%3Dslave. Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. pods is forbidden: User "system:anonymous" cannot list resource "pods" in API group "" in the namespace "mynamespace": Unexpected user-id: system:anonymous.
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:472)
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:409)
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:381)
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:344)
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:328)
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.list(BaseOperation.java:584)
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.list(BaseOperation.java:49)
at org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud.addProvisionedSlave(KubernetesCloud.java:493)
at org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud.provision(KubernetesCloud.java:448)
at hudson.slaves.NodeProvisioner$StandardStrategyImpl.apply(NodeProvisioner.java:715)
at hudson.slaves.NodeProvisioner.update(NodeProvisioner.java:320)
at hudson.slaves.NodeProvisioner.access$000(NodeProvisioner.java:61)
at hudson.slaves.NodeProvisioner$NodeProvisionerInvoker.doRun(NodeProvisioner.java:809)
at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:72)
at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
{code}

harsha_gv061@ymail.com (JIRA)

unread,
May 7, 2019, 6:58:03 AM5/7/19
to jenkinsc...@googlegroups.com
Harsha GV updated an issue
*PS: Jenkins is deployed in K8s and not using openshift [ doe do not intend to]*


Provisioning of K8s pods from Kubernetes Jenkins plugin using the my cloud credentials is failing, hence created service account using below command:

harsha_gv061@ymail.com (JIRA)

unread,
May 7, 2019, 6:59:03 AM5/7/19
to jenkinsc...@googlegroups.com
Harsha GV updated an issue
*PS: Jenkins is deployed in K8s and not using openshift [do not intend to]*


Provisioning of K8s pods from Kubernetes Jenkins plugin using the my cloud credentials is failing, hence created service account using below command:
{code:java}
kubectl  -n mynamespace create serviceaccount jenkins
{code}
Then after selecting service account authentication credentials tried Test Connection from *_Test Connection_* in K8s-plugin cloud configuration page , but connection failed with below error:

jenkins-ci@carlossanchez.eu (JIRA)

unread,
May 7, 2019, 7:46:02 AM5/7/19
to jenkinsc...@googlegroups.com
Carlos Sanchez commented on Bug JENKINS-57352
 
Re: Test connection fails and also unable to get K8s pods via Jenkins using service account

looks like it is picking the default service account, but I've never seen the "Unexpected user-id" error
How do you configure the service account in jenkins?

harsha_gv061@ymail.com (JIRA)

unread,
May 8, 2019, 2:15:02 PM5/8/19
to jenkinsc...@googlegroups.com

Using secret text by adding token string of service account

jenkins-ci@carlossanchez.eu (JIRA)

unread,
May 9, 2019, 4:26:02 AM5/9/19
to jenkinsc...@googlegroups.com

what kubernetes version? distro? cloud?

harsha_gv061@ymail.com (JIRA)

unread,
May 9, 2019, 11:28:01 AM5/9/19
to jenkinsc...@googlegroups.com

K8s:
Client v1.11.5
Server v1.13.5

Jenkins deployed in K8s.

harsha_gv061@ymail.com (JIRA)

unread,
May 9, 2019, 11:34:02 AM5/9/19
to jenkinsc...@googlegroups.com
Harsha GV edited a comment on Bug JENKINS-57352
K8s:
Client v1.11.5
Server v1.13.5

Jenkins deployed in K8s.


And also could you please point me to any documentation to get the minimum roles and rolebindings needs to be assigned to a Jenkins service account - used by k8s plugin to to dynamically provision a Jenkins agent, run a single build, then tear-down that agent.

harsha_gv061@ymail.com (JIRA)

unread,
May 9, 2019, 11:34:02 AM5/9/19
to jenkinsc...@googlegroups.com

K8s:
Client v1.11.5
Server v1.13.5

Jenkins deployed in K8s.

Also could you please point me to any documentation to get the minimum roles and rolebindings needs to be assigned to a Jenkins service account - used by k8s plugin to to dynamically provision a Jenkins agent, run a single build, then tear-down that agent?

harsha_gv061@ymail.com (JIRA)

unread,
May 9, 2019, 11:34:04 AM5/9/19
to jenkinsc...@googlegroups.com
Harsha GV updated an issue
Change By: Harsha GV
Comment:
K8s:
Client v1.11.5
Server v1.13.5

Jenkins deployed in K8s.

And also could you please point me to any documentation to get the minimum roles and rolebindings needs to be assigned to a Jenkins service account - used by k8s plugin to to dynamically provision a Jenkins agent, run a single build, then tear-down that agent.

jenkins-ci@carlossanchez.eu (JIRA)

unread,
May 9, 2019, 11:40:02 AM5/9/19
to jenkinsc...@googlegroups.com

harsha_gv061@ymail.com (JIRA)

unread,
May 13, 2019, 7:21:02 AM5/13/19
to jenkinsc...@googlegroups.com

GKE kubernetes

harsha_gv061@ymail.com (JIRA)

unread,
May 13, 2019, 8:24:03 AM5/13/19
to jenkinsc...@googlegroups.com

Service account is created and its roles are applied as defined in the https://github.com/jenkinsci/kubernetes-plugin/blob/master/src/main/kubernetes/service-account.yml.

But test connection still throws the error:

Error testing connection https://api.k8s2.apac.cloud.net:6443: Failure executing: GET at: https://api.k8s2.apac.cloud.net:6443/api/v1/namespaces/mynamespace/pods. Message: Unauthorized! Configured service account doesn't have access. Service account may have been revoked. Unauthorized.

harsha_gv061@ymail.com (JIRA)

unread,
May 13, 2019, 9:19:01 AM5/13/19
to jenkinsc...@googlegroups.com
Harsha GV edited a comment on Bug JENKINS-57352
GKE kubernetes

Service account is created and its roles are applied as defined in the https://github.com/jenkinsci/kubernetes-plugin/blob/master/src/main/kubernetes/service-account.yml.

But test connection still throws the error:
{code}

Error testing connection https://api.k8s2.apac.cloud.net:6443: Failure executing: GET at: https://api.k8s2.apac.cloud.net:6443/api/v1/namespaces/mynamespace/pods. Message: Unauthorized! Configured service account doesn't have access. Service account may have been revoked. Unauthorized.
{code}

harsha_gv061@ymail.com (JIRA)

unread,
May 13, 2019, 9:19:02 AM5/13/19
to jenkinsc...@googlegroups.com
Harsha GV updated an issue
Change By: Harsha GV
Comment:
Service account is created and its roles are applied as defined in the https://github.com/jenkinsci/kubernetes-plugin/blob/master/src/main/kubernetes/service-account.yml.

But test connection still throws the error:
{code}
Error testing connection https://api.k8s2.apac.cloud.net:6443: Failure executing: GET at: https://api.k8s2.apac.cloud.net:6443/api/v1/namespaces/mynamespace/pods. Message: Unauthorized! Configured service account doesn't have access. Service account may have been revoked. Unauthorized.
{code}

harsha_gv061@ymail.com (JIRA)

unread,
May 13, 2019, 9:20:02 AM5/13/19
to jenkinsc...@googlegroups.com
Harsha GV edited a comment on Bug JENKINS-57352

Service account is created and its roles are applied as defined in the https://github.com/jenkinsci/kubernetes-plugin/blob/master/src/main/kubernetes/service-account.yml.

But test connection still throws the error:
{code}
Error testing connection https://api.k8s2.apac.cloud.net:6443: Failure executing: GET at: https://api.k8s2.apac.cloud.net:6443/api/v1/namespaces/mynamespace/pods. Message: Unauthorized! Configured service account doesn't have access. Service account may have been revoked. Unauthorized.
{code}

harsha_gv061@ymail.com (JIRA)

unread,
May 13, 2019, 10:59:01 AM5/13/19
to jenkinsc...@googlegroups.com
Harsha GV edited a comment on Bug JENKINS-57352
Using GKE kubernetes.

Service account is created and its roles are were applied as defined in the https://github.com/jenkinsci/kubernetes-plugin/blob/master/src/main/kubernetes/service-account.yml.


But test connection still throws the error:
{code}
Error testing connection https://api.k8s2.apac.cloud.net:6443: Failure executing: GET at: https://api.k8s2.apac.cloud.net:6443/api/v1/namespaces/mynamespace/pods. Message: Unauthorized! Configured service account doesn't have access. Service account may have been revoked. Unauthorized.
{code}

jglick@cloudbees.com (JIRA)

unread,
Jul 8, 2019, 2:30:03 PM7/8/19
to jenkinsc...@googlegroups.com
Jesse Glick resolved as Incomplete
 

Probably either a cluster or Jenkins credentials misconfiguration. Unclear how to reproduce from scratch.
Change By: Jesse Glick
Status: Open Resolved
Assignee: Carlos Sanchez
Resolution: Incomplete

jmukhtar@divaaco.com (JIRA)

unread,
Sep 9, 2019, 8:20:06 AM9/9/19
to jenkinsc...@googlegroups.com
junaid mukhtar assigned an issue to junaid mukhtar
Change By: junaid mukhtar
Assignee: junaid mukhtar
This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f)
Atlassian logo

jmukhtar@divaaco.com (JIRA)

unread,
Sep 9, 2019, 8:21:09 AM9/9/19
to jenkinsc...@googlegroups.com
junaid mukhtar assigned an issue to Harsha GV
Change By: junaid mukhtar
Assignee: junaid mukhtar Harsha GV

jmukhtar@divaaco.com (JIRA)

unread,
Sep 9, 2019, 8:22:02 AM9/9/19
to jenkinsc...@googlegroups.com
junaid mukhtar commented on Bug JENKINS-57352
 
Re: Test connection fails and also unable to get K8s pods via Jenkins using service account

I was able to reproduce the issue by using an older version of Kubernetes-plugin running on dedicated EC2 instance and trying to connect to the EKS cluster.

 

Kubernetes Plugin: 1.13.5

EKS: 1.13

 

```WARNING: Failed to count the # of live instances on Kubernetes
io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: GET at: https://XXXXX.sk1.eu-west-1.eks.amazonaws.com/api/v1/namespaces/kube-system/pods?labelSelector=jenkins%3Dslave. Message: pods is forbidden: User "system:anonymous" cannot list resource "pods" in API group "" in the namespace "kube-system". Received status: Status(apiVersion=v1, code=403, details=StatusDetails(causes=[], group=null, kind=pods, name=null, retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status, message=pods is forbidden: User "system:anonymous" cannot list resource "pods" in API group "" in the namespace "kube-system", metadata=ListMeta(_continue=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=Forbidden, status=Failure, additionalProperties={}).```

jmukhtar@divaaco.com (JIRA)

unread,
Sep 9, 2019, 8:23:02 AM9/9/19
to jenkinsc...@googlegroups.com
junaid mukhtar assigned an issue to Carlos Sanchez
 
Change By: junaid mukhtar
Assignee: Harsha GV Carlos Sanchez
Reply all
Reply to author
Forward
0 new messages