[JIRA] (JENKINS-53757) Exec Failure: Expected HTTP 101 response but was '403 Forbidden'

737 views
Skip to first unread message

brett.cashman@ayasdi.com (JIRA)

unread,
Sep 24, 2018, 5:41:03 PM9/24/18
to jenkinsc...@googlegroups.com
Brett Cashman created an issue
 
Jenkins / Bug JENKINS-53757
Exec Failure: Expected HTTP 101 response but was '403 Forbidden'
Issue Type: Bug Bug
Assignee: Carlos Sanchez
Components: kubernetes-plugin
Created: 2018-09-24 21:40
Priority: Major Major
Reporter: Brett Cashman

I have a bare-metal Jenkins master (v2.141, running kubernetes-plugin v1.12.6) and a bare-metal Kubernetes cluster (v1.11.0). I'm trying to run a simplified version of the multi-container example described in the "Container Group Support" section of the README.

My Jenkinsfile is as follows:

def label = "pod-${env.JOB_NAME}-${env.BUILD_NUMBER}".replace('_', '-').replace('/', '-')

podTemplate(label: label, cloud: 'kubernetes2', namespace: 'default',
    containers: [
        containerTemplate(name: 'maven', image: 'maven:3.5-jdk-8-alpine', ttyEnabled: true, command: 'cat')
    ])
{   
   node(label) {
      echo "#1"
      stage('Fetch Maven') {
         git 'https://github.com/jenkinsci/kubernetes-plugin.git'
         echo "#2"
         container('maven') {
            stage ('Build Maven') {
               echo "#3"
               sh 'mvn -B clean install'
            }
         }
      }
   }
}

Everything seems copacetic until the `sh` command in the final stage. Log fragment:

[Pipeline] stage
[Pipeline] { (Build Maven)
[Pipeline] echo
#3
[Pipeline] sh
[experimental_pipeline_k8s] Running shell script
java.net.ProtocolException: Expected HTTP 101 response but was '403 Forbidden'
	at okhttp3.internal.ws.RealWebSocket.checkResponse(RealWebSocket.java:219)
	at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:186)
	at okhttp3.RealCall$AsyncCall.execute(RealCall.java:153)
	at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

 

I've set up a Kubernetes-specific log recorder on the master and it's not capturing anything untoward – the pod appears to be starting and terminating properly.

Any thoughts?

Add Comment Add Comment
 
This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)

jenkins-ci@carlossanchez.eu (JIRA)

unread,
Sep 25, 2018, 7:12:01 AM9/25/18
to jenkinsc...@googlegroups.com
Carlos Sanchez commented on Bug JENKINS-53757
 
Re: Exec Failure: Expected HTTP 101 response but was '403 Forbidden'

If you add a okhttp logger as explained in readme you will see which request is forbidden

brett.cashman@ayasdi.com (JIRA)

unread,
Sep 25, 2018, 3:05:03 PM9/25/18
to jenkinsc...@googlegroups.com

Thanks for the reply, Carlos Sanchez. I see it, but I don't see anything in it that seems amiss.

{"kind":"Pod","apiVersion":"v1","metadata":{"name":"jenkins-slave-v3b2j-bqflp","namespace":"default","selfLink":"/api/v1/namespaces/default/pods/jenkins-slave-v3b2j-bqflp","uid":"ed83d74e-c0f3-11e8-a8e0-00259088de1e","resourceVersion":"7335389","creationTimestamp":"2018-09-25T18:50:52Z","labels":{"jenkins":"slave","jenkins/pod-Experimental-experimental-pipeline-k8s-13":"true"}},"spec":{"volumes":[{"name":"workspace-volume","emptyDir":{}},{"name":"default-token-ctr95","secret":{"secretName":"default-token-ctr95","defaultMode":420}}],"containers":[{"name":"maven","image":"maven:3.5-jdk-8-alpine","command":["cat"],"workingDir":"/home/jenkins","env":[{"name":"JENKINS_SECRET","value":"3c0b2cddbead92f507a4a10a32e8bd12b38f3f763e1457468fc459f8637a073d"},{"name":"JENKINS_NAME","value":"jenkins-slave-v3b2j-bqflp"},{"name":"JENKINS_URL","value":"http://10.168.4.88:8080/"},{"name":"HOME","value":"/home/jenkins"}],"resources":{},"volumeMounts":[{"name":"workspace-volume","mountPath":"/home/jenkins"},{"name":"default-token-ctr95","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent","securityContext":{"privileged":false},"tty":true},{"name":"jnlp","image":"jenkins/jnlp-slave:alpine","env":[{"name":"JENKINS_SECRET","value":"3c0b2cddbead92f507a4a10a32e8bd12b38f3f763e1457468fc459f8637a073d"},{"name":"JENKINS_NAME","value":"jenkins-slave-v3b2j-bqflp"},{"name":"JENKINS_URL","value":"http://10.168.4.88:8080/"},{"name":"HOME","value":"/home/jenkins"}],"resources":{},"volumeMounts":[{"name":"workspace-volume","mountPath":"/home/jenkins"},{"name":"default-token-ctr95","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent"}],"restartPolicy":"Never","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","serviceAccountName":"default","serviceAccount":"default","nodeName":"kube01.eng.ayasdi.com","securityContext":{},"schedulerName":"default-scheduler","tolerations":[{"key":"node.kubernetes.io/not-ready","operator":"Exists","effect":"NoExecute","tolerationSeconds":300},{"key":"node.kubernetes.io/unreachable","operator":"Exists","effect":"NoExecute","tolerationSeconds":300}]},"status":{"phase":"Running","conditions":[{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2018-09-25T18:50:53Z"},{"type":"Ready","status":"True","lastProbeTime":null,"lastTransitionTime":"2018-09-25T18:51:05Z"},{"type":"ContainersReady","status":"True","lastProbeTime":null,"lastTransitionTime":null},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2018-09-25T18:50:52Z"}],"hostIP":"10.168.4.46","podIP":"10.244.0.81","startTime":"2018-09-25T18:50:53Z","containerStatuses":[{"name":"jnlp","state":{"running":{"startedAt":"2018-09-25T18:51:04Z"}},"lastState":{},"ready":true,"restartCount":0,"image":"docker.io/jenkins/jnlp-slave:alpine","imageID":"docker-pullable://docker.io/jenkins/jnlp-slave@sha256:7a826ce43cb7eeb59b0dc5000f89e00060ba31e51524d87ea811ba23453185ed","containerID":"docker://b11319580fa9243ebff8afffabb5f771c6c828776a83526cc20f19cfd77d786e"},{"name":"maven","state":{"running":{"startedAt":"2018-09-25T18:51:04Z"}},"lastState":{},"ready":true,"restartCount":0,"image":"docker.io/maven:3.5-jdk-8-alpine","imageID":"docker-pullable://docker.io/maven@sha256:b6cfa57a53eec51c9c7ef68701dc6951896ff5ac8eeca19d27d52457035b6491","containerID":"docker://96b0098c478b54478bb6edcc0f9efe4ae0a2deb8284a67188f558b7a927b5a6a"}],"qosClass":"BestEffort"}}

That's the JSON payload for the request immediately before the `Executing shell script inside container [maven] of pod [jenkins-slave-v3b2j-bqflp]` log entry from the Kubernetes plugin itself. Then I see an onFailure event, and a subsequent JSON payload for the teardown request.

 

brett.cashman@ayasdi.com (JIRA)

unread,
Oct 22, 2018, 4:38:02 PM10/22/18
to jenkinsc...@googlegroups.com

Hi, any update on this? Still running into the same issue, and based on what I'm seeing in the logs it's unclear what the problem might be.

jenkins-ci@carlossanchez.eu (JIRA)

unread,
Dec 7, 2018, 1:07:01 PM12/7/18
to jenkinsc...@googlegroups.com

you would need to check in your kubernetes logs why it is denying the request

nickbr23@hotmail.co.uk (JIRA)

unread,
Mar 5, 2019, 11:13:02 AM3/5/19
to jenkinsc...@googlegroups.com

Any update on this?

 

I've just ran into what appears to be the same issue.

 

I've checked the kubernetes api logs to see why the exec request is unauthorised and I can't see any information in there at all (I can't even see anything remotely related to exec). There is a singular get request for the pod which is used before for the 'checkout scm'.

 

The only relevant exception from the pipeline is:

Exception:
hudson.remoting.ProxyException: io.fabric8.kubernetes.client.KubernetesClientException: <h3>Unauthorized</h3>
at io.fabric8.kubernetes.client.dsl.internal.ExecWebSocketListener.onFailure(ExecWebSocketListener.java:228)

 

Any ideas would be greatly appreciated.

jenkins-ci@carlossanchez.eu (JIRA)

unread,
Mar 5, 2019, 12:04:02 PM3/5/19
to jenkinsc...@googlegroups.com

Please provide the okhttp logs as requested above, there should be a request failing with that error

If this happens all the time I suggest to check if you have something between Jenkins and the kubernetes API that does not support websockets

nickbr23@hotmail.co.uk (JIRA)

unread,
Mar 5, 2019, 12:55:02 PM3/5/19
to jenkinsc...@googlegroups.com

When I have put the okhttp3 logger on I don't see any extra log messages apart from an error about a leaked connection when the pod is deleted after the failure.

Does this suggest that the websocket connection cannot be made? Would that come back as a 403 Unauthorized?

 

I'll have to look into whether there's a corporate firewall or similar in between.

 

Thanks!

jenkins-ci@carlossanchez.eu (JIRA)

unread,
Mar 5, 2019, 1:46:01 PM3/5/19
to jenkinsc...@googlegroups.com

In the okhttp you must see all the requests together with response codes, can you double check that you configured the logger correctly?

Yes, there is some issue with the websocket connection, particularly If all the other connections work. exec uses websockets

abermudez1718@gmail.com (JIRA)

unread,
Mar 7, 2019, 3:25:02 PM3/7/19
to jenkinsc...@googlegroups.com

I also ran into the same issue. 
[Pipeline] stage[Pipeline] { (Run Maven)[Pipeline] container[Pipeline] {[Pipeline] sh*13:35:20* [kubernetes-for-dummies] Running shell script*13:35:21* java.net.ProtocolException: Expected HTTP 101 response but was '403 Forbidden'13:35:21 at okhttp3.internal.ws.RealWebSocket.checkResponse(RealWebSocket.java:219)13:35:21 at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:186)13:35:21 at okhttp3.RealCall$AsyncCall.execute(RealCall.java:153)13:35:21 at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)13:35:21 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)13:35:21 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)13:35:21 at java.lang.Thread.run(Thread.java:745)
 

I followed the instructions Carlos Sanchez provides in the kubernetes-plugin README.md on setting up the okhttp3 logger: https://github.com/carlossg/jenkins-kubernetes-plugin#debugging to the best of my knowledge (I did not set Log Level DEBUG so I went with ALL). I re-ran the job but did not see any responses containing '403 Forbidden' in the loggers log.

abermudez1718@gmail.com (JIRA)

unread,
Mar 7, 2019, 3:30:02 PM3/7/19
to jenkinsc...@googlegroups.com
Angel Bermudez edited a comment on Bug JENKINS-53757
I also ran into the same issue.  

[Pipeline] stage
[Pipeline] { (Run Maven)
[Pipeline] container
[Pipeline] {
[Pipeline] sh *
13:35:20 * [kubernetes-for-dummies] Running shell script *
13:35:21 * java.net.ProtocolException: Expected HTTP 101 response but was '403 Forbidden' *
13:35:21 *       at okhttp3.internal.ws.RealWebSocket.checkResponse(RealWebSocket.java:219) *
13:35:21 *       at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:186) *
13:35:21 *       at okhttp3.RealCall$AsyncCall.execute(RealCall.java:153) *
13:35:21 *       at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32) *
13:35:21 *       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) *
13:35:21 *   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) *
13:35:21 *   at java.lang.Thread.run(Thread.java:745)
 

I followed the instructions [~csanchez] provides in the kubernetes-plugin README.md on setting up the okhttp3 logger: [https://github.com/carlossg/jenkins-kubernetes-plugin#debugging] to the best of my knowledge (I did not set Log Level DEBUG so I went with ALL). I re-ran the job but did not see any responses containing '403 Forbidden' in the loggers log.

nickbr23@hotmail.co.uk (JIRA)

unread,
Mar 8, 2019, 7:45:04 AM3/8/19
to jenkinsc...@googlegroups.com

I managed to fix my issue.

 

Turns out the plugin was configured to look at a proxy rather than the api server. Once the url was changed to look at the api-server and a clusterrolebinding setup for the jenkins user, it all worked.

abermudez1718@gmail.com (JIRA)

unread,
Mar 8, 2019, 9:00:02 AM3/8/19
to jenkinsc...@googlegroups.com

For got to mention I'm on kubernetes-plugin 1.13.5, Jenkins Master 2.138.2.

I mimicked the idea Nick Bromfield had for the "clusterrolebinding setup for the jenkins user" and it all worked!

brett.cashman@ayasdi.com (JIRA)

unread,
Mar 8, 2019, 3:42:01 PM3/8/19
to jenkinsc...@googlegroups.com

Nick Bromfield, Angel Bermudez, could you guys describe what you actually did in detail? I mean, I could Google around for it, and spend time trying to parse threads on Stack Overflow and Github, but if either or both of you could document what you did concisely, it'd be a big help.

Thanks!

jglick@cloudbees.com (JIRA)

unread,
Jul 16, 2019, 3:43:20 PM7/16/19
to jenkinsc...@googlegroups.com
Jesse Glick assigned an issue to Unassigned
 
Change By: Jesse Glick
Assignee: Carlos Sanchez

rene.schoenlein@cdi-ag.de (JIRA)

unread,
Aug 23, 2019, 4:45:03 AM8/23/19
to jenkinsc...@googlegroups.com
Rene Schönlein commented on Bug JENKINS-53757
 
Re: Exec Failure: Expected HTTP 101 response but was '403 Forbidden'

After doing some google research, it seems to me that the issue is related to:

https://github.com/fabric8io/kubernetes-client/issues/1667

https://github.com/fabric8io/kubernetes-client/pull/1669

As a workaround, I changed the plugin configuration so that the kubernetes URL includes a port (e.g. from "https://kubernetes.default" to "https://kubernetes.default:443").

rene.schoenlein@cdi-ag.de (JIRA)

unread,
Aug 23, 2019, 4:46:03 AM8/23/19
to jenkinsc...@googlegroups.com
Rene Schönlein updated an issue
 
Change By: Rene Schönlein
Comment:
After doing some google research, it seems to me that the issue is related to:

[https://github.com/fabric8io/kubernetes-client/issues/1667]

[https://github.com/fabric8io/kubernetes-client/pull/1669]

As a workaround, I changed the plugin configuration so that the kubernetes URL includes a port (e.g. from "https://kubernetes.default" to "https://kubernetes.default:443").

rene.schoenlein@cdi-ag.de (JIRA)

unread,
Aug 23, 2019, 4:46:04 AM8/23/19
to jenkinsc...@googlegroups.com

After doing some google research, it seems to me that the issue is related to:

https://github.com/fabric8io/kubernetes-client/issues/1667

https://github.com/fabric8io/kubernetes-client/pull/1669

As a workaround, I changed the plugin configuration so that the kubernetes URL includes a port (e.g. from "https://kubernetes.default" to "https://kubernetes.default:443").

Add Comment Add Comment
 

rene.schoenlein@cdi-ag.de (JIRA)

unread,
Aug 23, 2019, 4:49:04 AM8/23/19
to jenkinsc...@googlegroups.com
Rene Schönlein updated an issue
Change By: Rene Schönlein
Comment:
After doing some google research, it seems to me that the issue is related to:

[https://github.com/fabric8io/kubernetes-client/issues/1667]

[https://github.com/fabric8io/kubernetes-client/pull/1669]

As a workaround, I changed the plugin configuration so that the kubernetes URL includes a port (e.g. from "https://kubernetes.default" to "https://kubernetes.default:443").

jenkins-ci@carlossanchez.eu (JIRA)

unread,
Aug 24, 2019, 5:28:05 AM8/24/19
to jenkinsc...@googlegroups.com
Carlos Sanchez closed an issue as Duplicate
Change By: Carlos Sanchez
Status: Open Closed
Resolution: Duplicate
Reply all
Reply to author
Forward
0 new messages