[JIRA] (JENKINS-39867) Message: Unauthorized from OpenShift API after some time

18 views
Skip to first unread message

jenkins-ci@carlossanchez.eu (JIRA)

unread,
Nov 22, 2016, 3:43:01 AM11/22/16
to jenkinsc...@googlegroups.com
Carlos Sanchez updated an issue
 
Jenkins / Bug JENKINS-39867
Message: Unauthorized from OpenShift API after some time
Change By: Carlos Sanchez
Summary: Message: Unauthorized from OpenShift API after some time
Add Comment Add Comment
 
This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c)
Atlassian logo

jenkins-ci@carlossanchez.eu (JIRA)

unread,
Nov 22, 2016, 3:44:01 AM11/22/16
to jenkinsc...@googlegroups.com
Carlos Sanchez commented on Bug JENKINS-39867
 
Re: Message: Unauthorized from OpenShift API after some time

how are you authenticating to openshift? using a service account, user/password, certs,...?

albertvipe@gmail.com (JIRA)

unread,
Nov 22, 2016, 4:59:02 AM11/22/16
to jenkinsc...@googlegroups.com
Albert V commented on Bug JENKINS-39867

A user/password.
And even with this error we can still logging in to Openshift with this credentials.

jenkins-ci@carlossanchez.eu (JIRA)

unread,
Nov 23, 2016, 1:24:01 PM11/23/16
to jenkinsc...@googlegroups.com

iocanel@gmail.com (JIRA)

unread,
Nov 23, 2016, 1:32:03 PM11/23/16
to jenkinsc...@googlegroups.com

The jenkins pod needs to use a service account and that service account needs to be allowed to list, create, delete pods.

For example the following command `oc adm policy who-can create pods` needs to return a list that contains the service account assigned to jenkins.

This might explain things better than me:
https://docs.openshift.com/enterprise/3.0/admin_guide/manage_authorization_policy.html

iocanel@gmail.com (JIRA)

unread,
Nov 23, 2016, 1:39:02 PM11/23/16
to jenkinsc...@googlegroups.com

My comment above applies to setups where jenkins is deployed as pod. From the description above I understand that this is not the case and Jenkins lives outside of openshift, right?

If this is the case, you may be hitting a bug in the openshift client, which was fixed recently: https://github.com/fabric8io/kubernetes-client/commit/97990ab51fe8d8bc307b7494098f0ebe3a3422d6

So, we need to update to a more recent version.

albertvipe@gmail.com (JIRA)

unread,
Nov 24, 2016, 4:28:01 AM11/24/16
to jenkinsc...@googlegroups.com
Albert V commented on Bug JENKINS-39867

Ioannis Canellos You're right. I've an external Jenkins installation.
Great news if this bug was fixed!
So Carlos Sanchez, the next version of Kubernetes-plugin when is expected to be released?

agustin_burguete@hotmail.com (JIRA)

unread,
Nov 29, 2016, 2:34:02 PM11/29/16
to jenkinsc...@googlegroups.com

I do not know if it is related to this but I am getting an error like the one described in here: https://github.com/fabric8io/kubernetes-client/issues/558

I am using Jenkins 2.19.3 and kubernetes-plugin 0.9

If the pod template I use does NOT have volumes, everything goes fine. But if in the same template, I add an NFS volume is when I get this error:

SEVERE: Error in provisioning; slave=KubernetesSlave name: jenkins-slave-template-12849497fadef4, template=org.csanchez.jenkins.plugins.kubernetes.PodTemplate@4687fe6a
io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: POST at: https://10.49.160.125:8443/api/v1/namespaces/jenkins-slaves/pods. Message: Forbidden! User admin/10-49-160-125:8443 doesn't have permission..

albertvipe@gmail.com (JIRA)

unread,
Dec 1, 2016, 6:27:01 AM12/1/16
to jenkinsc...@googlegroups.com
Albert V commented on Bug JENKINS-39867

agustin burguete:

  • Which is the default user-id and group-id for the NFS?
  • Which is the user-id and group-id of your jenkins user?
  • Which directories do you have in the NFS?

I think that this problem is not related with the plugin. Anyway it would be better if you open a new issue for your problem and we will try to help you

iocanel@gmail.com (JIRA)

unread,
Dec 1, 2016, 9:23:04 AM12/1/16
to jenkinsc...@googlegroups.com

agustin burguete You need to make sure that the service account you are using has permission to use volumes.

agustin_burguete@hotmail.com (JIRA)

unread,
Dec 1, 2016, 7:43:02 PM12/1/16
to jenkinsc...@googlegroups.com

Thank you for your comments, I am bound to make some mistakes since I do not have that much experience. Nevertheless, it is when using NFS volume or Hosted volume (aprox. name, cannot check it right now) that I get the error. With no volume or Empty dir volume, it works fine, the plugin does start-up a pod in Openshift. I am using a Vagrant all-in-one version of Openshift Origin as kubernetes host, and therefore I just use a system-admin login for all purposes, so I would not expect permission issues. I am replicating the setup described in here https://blog.openshift.com/jenkins-slaves-in-openshift-using-an-external-jenkins-environment/ where two ways are used to have jenkins slaves, one is Swarm plugin (preallocated slave pods selfdiscover the master) the other kubernetes-plugin (the master actively launches slaves on kubernetes, that would be within openshift in this case). For swarm plugin I found that v2.2 does have regression in terms of honoring the NO_PROXY env. var, which is important on my setup, so it works fine if I stay in 2.0. For kubernetes-plugin I am finding the problem already stated, it launches slaves OK only if they have no volumes or empty dir volumes. I am wildguessing it might have to do with some bug in kubernetes-client which is used for the plugin?, Since it is a jenkins for experimenting, I am using an 'admin' account in jenkins too. The pod slaves preinstantiated (the ones that connect to master jenkins via swarm) do also have the same NFS volume defined (via openshift deployment config in this case) and they work just fine. NFS directory has an nfsnobody owner and group, that volumes is accessed ok for the preallocated pods launched via openshift, and anyway the other option via 'host volume' does not work either (same error). NFS directories, Openshift origin and master jenkins all are located in the same machine. Master jenkins cohabits with Openshift Origin installation (so master jenkins is not a pod within Openshift, the slaves are meant to be deployed in Openshift). The whole point of the setup is to have all slaves share a persistent volume to keep there maven's repository, so that when maven 'downloads half of internet' upon a build, is kept there and all those jars do not disappear and have to be downloaded again for a next build. Slaves preallocated with swarm do fine on this, the ones with kubernetes-plugin... seemingly cannot have a persistent volume for now...Sorry for the lengthy comment! Your suggestions are very welcome. Should this be a separate issue?

jenkins-ci@carlossanchez.eu (JIRA)

unread,
Dec 2, 2016, 2:51:02 PM12/2/16
to jenkinsc...@googlegroups.com

jenkins-ci@carlossanchez.eu (JIRA)

unread,
Dec 18, 2016, 2:30:01 PM12/18/16
to jenkinsc...@googlegroups.com
Carlos Sanchez closed an issue as Fixed
 

Should be fixed in 0.10

Change By: Carlos Sanchez
Status: Open Closed
Assignee: Carlos Sanchez
Resolution: Fixed

albertvipe@gmail.com (JIRA)

unread,
Dec 29, 2016, 6:33:01 AM12/29/16
to jenkinsc...@googlegroups.com
Albert V commented on Bug JENKINS-39867
 
Re: Message: Unauthorized from OpenShift API after some time

Suddenly I'm experiencing the same issue again in the 0.10 version of Kubernetes Plugin:

Dec 29, 2016 11:58:55 AM WARNING org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud provision
Failed to count the # of live instances on Kubernetes
io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: GET at: https://myopenshift.url/api/v1/namespaces/jenkins/pods?labelSelector=jenkins%3Dslave. Message: Unauthorized
.
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:315)
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:268)
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:237)
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:230)
	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.list(BaseOperation.java:506)
	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.list(BaseOperation.java:62)
	at org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud.addProvisionedSlave(KubernetesCloud.java:651)
	at org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud.provision(KubernetesCloud.java:513)
	at hudson.slaves.NodeProvisioner$StandardStrategyImpl.apply(NodeProvisioner.java:701)
	at hudson.slaves.NodeProvisioner.update(NodeProvisioner.java:307)
	at hudson.slaves.NodeProvisioner.access$000(NodeProvisioner.java:60)
	at hudson.slaves.NodeProvisioner$NodeProvisionerInvoker.doRun(NodeProvisioner.java:798)
	at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:50)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

Ioannis Canellos Could be another Kubernetes (Openshift) Client issue?

albertvipe@gmail.com (JIRA)

unread,
Dec 29, 2016, 6:37:01 AM12/29/16
to jenkinsc...@googlegroups.com
Albert V reopened an issue
 

I missed to re-open the issue. More info in this comment.

Change By: Albert V
Resolution: Fixed
Status: Closed Reopened

albertvipe@gmail.com (JIRA)

unread,
Jan 16, 2017, 4:54:01 AM1/16/17
to jenkinsc...@googlegroups.com

iocanel@gmail.com (JIRA)

unread,
Jan 16, 2017, 5:52:02 AM1/16/17
to jenkinsc...@googlegroups.com

Albert V Can you please tell me what you mean by "suddendly"? What I get from it is that it worked, but at some point it stopped.

Also can you please paste the output of:

oc adm policy who-can list pods -n jenkins

albertvipe@gmail.com (JIRA)

unread,
Jan 16, 2017, 6:52:05 AM1/16/17
to jenkinsc...@googlegroups.com
Albert V commented on Bug JENKINS-39867

I've used the term "suddenly" because after the plugin update everything had working well but without any reason (or the same one than causes this error in the 0.9 plugin version) this happened again. I could not find any pattern to reproduce this error.
The output that you asked is this:

$ oc adm policy who-can list pods -n jenkins
Namespace: jenkins
Verb:      list
Resource:  pods

Users:  myUser1
        myUser2
        system:admin
        system:serviceaccount:jenkins:deployer
        system:serviceaccount:management-infra:management-admin
        system:serviceaccount:openshift-infra:build-controller
        system:serviceaccount:openshift-infra:daemonset-controller
        system:serviceaccount:openshift-infra:deployment-controller
        system:serviceaccount:openshift-infra:deploymentconfig-controller
        system:serviceaccount:openshift-infra:endpoint-controller
        system:serviceaccount:openshift-infra:gc-controller
        system:serviceaccount:openshift-infra:hpa-controller
        system:serviceaccount:openshift-infra:job-controller
        system:serviceaccount:openshift-infra:namespace-controller
        system:serviceaccount:openshift-infra:pet-set-controller
        system:serviceaccount:openshift-infra:pv-attach-detach-controller
        system:serviceaccount:openshift-infra:pv-binder-controller
        system:serviceaccount:openshift-infra:pv-recycler-controller
        system:serviceaccount:openshift-infra:replicaset-controller
        system:serviceaccount:openshift-infra:replication-controller

Groups: GL_APP_OSHIFT_Admins
        system:cluster-admins
        system:cluster-readers
        system:masters
        system:nodes

Restarting the jenkins isntance it still repair this error.

Thank you for helping me with this

iocanel@gmail.com (JIRA)

unread,
Jan 16, 2017, 6:57:03 AM1/16/17
to jenkinsc...@googlegroups.com

So I am assuming that you are either connection with myUser1, myUser2 or admin, right?

Could it be possible that each time jenkins starts, it authenticates but after 24 hours, the token expires and you start getting these errors until, you restart again?
If this is the case, then we need to make sure in the kubernetes-client that we handle properlly token expiration.

albertvipe@gmail.com (JIRA)

unread,
Jan 16, 2017, 8:26:01 AM1/16/17
to jenkinsc...@googlegroups.com
Albert V commented on Bug JENKINS-39867

Yes, I'm using one to those myUser1 user.

It seems that could be a token expiration problem because (I can't tell you surely) this error happened once per day.

albertvipe@gmail.com (JIRA)

unread,
Mar 29, 2017, 10:52:05 AM3/29/17
to jenkinsc...@googlegroups.com
Albert V commented on Bug JENKINS-39867

Is there any update of this? It is still happening even in the latest (0.11) version.

Thanks!

This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e)
Atlassian logo
Reply all
Reply to author
Forward
0 new messages