[JIRA] (JENKINS-59000) KubernetesClientException onFailure(WatchConnectionManager.java:198) on k8s v1.15.3

286 views
Skip to first unread message

dghubble@gmail.com (JIRA)

unread,
Aug 19, 2019, 9:27:02 PM8/19/19
to jenkinsc...@googlegroups.com
Dalton Hubble created an issue
 
Jenkins / Bug JENKINS-59000
KubernetesClientException onFailure(WatchConnectionManager.java:198) on k8s v1.15.3
Issue Type: Bug Bug
Assignee: Unassigned
Components: kubernetes-plugin
Created: 2019-08-20 01:26
Environment: Kubernetes v1.15.3
Priority: Minor Minor
Reporter: Dalton Hubble

I've upgraded a Kubernetes cluster from v1.15.2 to v1.15.3 and started noticing kubernetes-plugin cannot provision any pods. I was able to rollback to fix the issue, roll forward again to cause it, and generally isolate it to the specific cluster version bump. I can change flip just the apiserver version back and forth to cause and fix the issue. I've tried both kubernetes-plugin v1.16.4 and v1.18.1 (both affected).

What may be relevant is that Kubernetes v1.13.10, v1.14.6, and v1.15.3 was published today with CVE fixes for Go net/http vulnerabilities. Maybe one of those fixes interferes with kubernetes-plugin operation (that plugin code seems to concern the websocket and HTTP watch). I haven't confirmed the issue is present on Kubernetes v1.13.10 or v1.14.6, but suspect its on all the patched releases.

Error in provisioning; agent=KubernetesSlave name: python-pod-0ppnb-qnvc7, template=PodTemplate{, name='python-pod-0ppnb', namespace='jenkins', label='python-pod', nodeUsageMode=EXCLUSIVE, containers=[ContainerTemplate{name='python-agent', image='quay.io/dghubble/python-agent:4462978780179489a5608c480d9c64f8cb61bc7f', workingDir='/home/jenkins/agent', command='cat', ttyEnabled=true}], annotations=[org.csanchez.jenkins.plugins.kubernetes.PodAnnotation@aab9c821]}
io.fabric8.kubernetes.client.KubernetesClientException: 
	at io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager$1.onFailure(WatchConnectionManager.java:198)
	at okhttp3.internal.ws.RealWebSocket.failWebSocket(RealWebSocket.java:571)
	at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:198)
	at okhttp3.RealCall$AsyncCall.execute(RealCall.java:206)
	at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

https://github.com/fabric8io/kubernetes-client/blob/master/kubernetes-client/src/main/java/io/fabric8/kubernetes/client/dsl/internal/WatchConnectionManager.java

https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.15.md#changelog-since-v1152

 
Add Comment Add Comment
 
This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)

olivier@omary.fr (JIRA)

unread,
Aug 21, 2019, 6:40:04 AM8/21/19
to jenkinsc...@googlegroups.com
MARY Olivier commented on Bug JENKINS-59000
 
Re: KubernetesClientException onFailure(WatchConnectionManager.java:198) on k8s v1.15.3

Exactly same problem for us. Rollback to 1.15.2 in progress...

 

bjornmagnusson1984@gmail.com (JIRA)

unread,
Aug 21, 2019, 7:50:02 AM8/21/19
to jenkinsc...@googlegroups.com

Same thing for 1.14.6

olivier@omary.fr (JIRA)

unread,
Aug 21, 2019, 9:54:03 AM8/21/19
to jenkinsc...@googlegroups.com

In complement,

Watch on pods (kubectl get pod -w) to show creation of pod and directly Terminated in 1sec, so API/Yaml seems good, just plugin delete it just after apply pod template.

dakotapollard@gmail.com (JIRA)

unread,
Aug 21, 2019, 5:24:03 PM8/21/19
to jenkinsc...@googlegroups.com

Our K8s clusters were just upgraded to v1.15.3 and we are also experiencing the same issue. Our Jenkins uses this plugin for agents so all of our pipelines are blocked and we aren't able to roll back our cluster. Does anyone have an idea of a workaround for this? Any idea what needs to be fixed in the plugin or if someone is already looking into it?

asidhu11702@gmail.com (JIRA)

unread,
Aug 21, 2019, 10:34:02 PM8/21/19
to jenkinsc...@googlegroups.com

I am having the same issue on aks 1.13.10 version. Any one looking at it yet

ivan.lefkate@gmail.com (JIRA)

unread,
Aug 21, 2019, 10:38:02 PM8/21/19
to jenkinsc...@googlegroups.com

I got the same issue and also confirmed that rolling back to 1.15.2 fixes it.

asidhu11702@gmail.com (JIRA)

unread,
Aug 22, 2019, 12:51:38 AM8/22/19
to jenkinsc...@googlegroups.com

I dont have an option to roll back as azure kubernetes doesnt let me do it. Is the community doing something about this issue? carlos canchez is the person who wrote that plugin, please message him as well

asidhu11702@gmail.com (JIRA)

unread,
Aug 22, 2019, 12:52:02 AM8/22/19
to jenkinsc...@googlegroups.com

hayden@haydenball.me.uk (JIRA)

unread,
Aug 22, 2019, 4:06:02 AM8/22/19
to jenkinsc...@googlegroups.com
Hayden Ball updated an issue
 
Change By: Hayden Ball
Priority: Minor Major

christiani@netcompany.com (JIRA)

unread,
Aug 22, 2019, 6:53:03 AM8/22/19
to jenkinsc...@googlegroups.com
Christian Ihle commented on Bug JENKINS-59000
 
Re: KubernetesClientException onFailure(WatchConnectionManager.java:198) on k8s v1.15.3

Stumbled upon the same issue when testing an upgrade to 1.13.10. Currently this blocks us from implementing security fixes to our clusters.

olivier@omary.fr (JIRA)

unread,
Aug 22, 2019, 7:46:05 AM8/22/19
to jenkinsc...@googlegroups.com
MARY Olivier updated an issue
 
Change By: MARY Olivier
Priority: Major Critical

jenkins-ci@carlossanchez.eu (JIRA)

unread,
Aug 22, 2019, 8:37:02 AM8/22/19
to jenkinsc...@googlegroups.com
Carlos Sanchez commented on Bug JENKINS-59000
 
Re: KubernetesClientException onFailure(WatchConnectionManager.java:198) on k8s v1.15.3

I have run the tests against kind 1.15.3 and they pass

In the logs you should have an error Exec Failure: HTTP... can you post it?
https://github.com/jenkinsci/kubernetes-plugin#debugging

jenkins-ci@carlossanchez.eu (JIRA)

unread,
Aug 22, 2019, 8:40:02 AM8/22/19
to jenkinsc...@googlegroups.com

It would help if you describe what kubernetes are you running: EKS, GKE, AKS, openshift,...

olivier@omary.fr (JIRA)

unread,
Aug 22, 2019, 8:41:02 AM8/22/19
to jenkinsc...@googlegroups.com

Can't log it in k8s pod is created then deleted all in less than 1sec... so cant describe/logs..

 

Just Jenkins logs like the description of this issue.

asidhu11702@gmail.com (JIRA)

unread,
Aug 22, 2019, 8:45:02 AM8/22/19
to jenkinsc...@googlegroups.com

Carlos I am using AKS 1.3.10. The error I am seeing in the master logs as follows:

 

Aug 22, 2019 12:43:11 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch
WARNING: Error in provisioning; agent=KubernetesSlave name: worker-8b08130f-837c-4a7a-9d51-d81cb28d75c4-f4jt1-12ck1, template=PodTemplate{, name='worker-8b08130f-837c-4a7a-9d51-d81cb28d75c4-f4jt1', namespace='development', label='worker-8b08130f-837c-4a7a-9d51-d81cb28d75c4', nodeUsageMode=EXCLUSIVE, volumes=[HostPathVolume [mountPath=/var/run/docker.sock, hostPath=/var/run/docker.sock], HostPathVolume [mountPath=/root/.m2/repository, hostPath=/root/.m2/repository]], containers=[ContainerTemplate

{name='docker', image='wsibimagerepo.azurecr.io/docker:stable-git', workingDir='/home/jenkins/agent', command='cat', ttyEnabled=true}

, ContainerTemplate{name='kubectl', image='lachlanevenson/k8s-kubectl:v1.14.6', workingDir='/home/jenkins/agent', command='cat', ttyEnabled=true}], annotations=[org.csanchez.jenkins.plugins.kubernetes.PodAnnotation@aab9c821]}


io.fabric8.kubernetes.client.KubernetesClientException:
at io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager$1.onFailure(WatchConnectionManager.java:198)
at okhttp3.internal.ws.RealWebSocket.failWebSocket(RealWebSocket.java:571)
at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:198)
at okhttp3.RealCall$AsyncCall.execute(RealCall.java:206)
at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

 

Aug 22, 2019 12:43:11 PM io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager$1 onFailure
WARNING: Exec Failure: HTTP 403, Status: 403 -
java.net.ProtocolException: Expected HTTP 101 response but was '403 Forbidden'
at okhttp3.internal.ws.RealWebSocket.checkResponse(RealWebSocket.java:229)
at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:196)


at okhttp3.RealCall$AsyncCall.execute(RealCall.java:206)
at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

 

 
Add Comment Add Comment
 

asidhu11702@gmail.com (JIRA)

unread,
Aug 22, 2019, 8:46:02 AM8/22/19
to jenkinsc...@googlegroups.com

All my pipelines that run on 1.13.10 are broken, we are screwed. I cant go back now

asidhu11702@gmail.com (JIRA)

unread,
Aug 22, 2019, 8:48:06 AM8/22/19
to jenkinsc...@googlegroups.com

I opened ticket with aks and they are saying this is jenkins plugin problem, they are saying nothing wrong with aks code base. They are saying this plugin was exploiting security loophole and now since kubernetes tightened thats why we are getting error.

 

Aug 22, 2019 12:41:11 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch
WARNING: Error in provisioning; agent=KubernetesSlave name: worker-8b08130f-837c-4a7a-9d51-d81cb28d75c4-f4jt1-7chls, template=PodTemplate{, name='worker-8b08130f-837c-4a7a-9d51-d81cb28d75c4-f4jt1', namespace='development', label='worker-8b08130f-837c-4a7a-9d51-d81cb28d75c4', nodeUsageMode=EXCLUSIVE, volumes=[HostPathVolume [mountPath=/var/run/docker.sock, hostPath=/var/run/docker.sock], HostPathVolume [mountPath=/root/.m2/repository, hostPath=/root/.m2/repository]], containers=[ContainerTemplate

{name='docker', image='wsibimagerepo.azurecr.io/docker:stable-git', workingDir='/home/jenkins/agent', command='cat', ttyEnabled=true}

, ContainerTemplate{name='kubectl', image='lachlanevenson/k8s-kubectl:v1.14.6', workingDir='/home/jenkins/agent', command='cat', ttyEnabled=true}], annotations=[org.csanchez.jenkins.plugins.kubernetes.PodAnnotation@aab9c821]}
io.fabric8.kubernetes.client.KubernetesClientException:
at io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager$1.onFailure(WatchConnectionManager.java:198)
at okhttp3.internal.ws.RealWebSocket.failWebSocket(RealWebSocket.java:571)
at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:198)
at okhttp3.RealCall$AsyncCall.execute(RealCall.java:206)
at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

Aug 22, 2019 12:41:11 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave _terminate
INFO: Terminating Kubernetes instance for agent worker-8b08130f-837c-4a7a-9d51-d81cb28d75c4-f4jt1-7chls
Terminated Kubernetes instance for agent development/worker-8b08130f-837c-4a7a-9d51-d81cb28d75c4-f4jt1-7chls
Disconnected computer worker-8b08130f-837c-4a7a-9d51-d81cb28d75c4-f4jt1-7chls
Aug 22, 2019 12:41:12 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave deleteSlavePod
INFO: Terminated Kubernetes instance for agent development/worker-8b08130f-837c-4a7a-9d51-d81cb28d75c4-f4jt1-7chls
Aug 22, 2019 12:41:12 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave _terminate
INFO: Disconnected computer worker-8b08130f-837c-4a7a-9d51-d81cb28d75c4-f4jt1-7chls
Aug 22, 2019 12:41:21 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud provision
INFO: Excess workload after pending Kubernetes agents: 1
Aug 22, 2019 12:41:21 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud provision
INFO: Template for label worker-8b08130f-837c-4a7a-9d51-d81cb28d75c4: Kubernetes Pod Template
Aug 22, 2019 12:41:21 PM hudson.slaves.NodeProvisioner$StandardStrategyImpl apply
INFO: Started provisioning Kubernetes Pod Template from kubernetes with 1 executors. Remaining excess workload: 0
Aug 22, 2019 12:41:31 PM hudson.slaves.NodeProvisioner$2 run
INFO: Kubernetes Pod Template provisioning successfully completed. We have now 2 computer(s)
Aug 22, 2019 12:41:31 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud provision
INFO: Excess workload after pending Kubernetes agents: 0
Aug 22, 2019 12:41:31 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud provision
INFO: Template for label worker-8b08130f-837c-4a7a-9d51-d81cb28d75c4: Kubernetes Pod Template
Aug 22, 2019 12:41:32 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch
INFO: Created Pod: development/worker-8b08130f-837c-4a7a-9d51-d81cb28d75c4-f4jt1-3cdm6
Aug 22, 2019 12:41:32 PM io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager$1 onFailure


WARNING: Exec Failure: HTTP 403, Status: 403 -
java.net.ProtocolException: Expected HTTP 101 response but was '403 Forbidden'
at okhttp3.internal.ws.RealWebSocket.checkResponse(RealWebSocket.java:229)
at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:196)
at okhttp3.RealCall$AsyncCall.execute(RealCall.java:206)
at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

Aug 22, 2019 12:41:32 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch
WARNING: Error in provisioning; agent=KubernetesSlave name: worker-8b08130f-837c-4a7a-9d51-d81cb28d75c4-f4jt1-3cdm6, template=PodTemplate{, name='worker-8b08130f-837c-4a7a-9d51-d81cb28d75c4-f4jt1', namespace='development', label='worker-8b08130f-837c-4a7a-9d51-d81cb28d75c4', nodeUsageMode=EXCLUSIVE, volumes=[HostPathVolume [mountPath=/var/run/docker.sock, hostPath=/var/run/docker.sock], HostPathVolume [mountPath=/root/.m2/repository, hostPath=/root/.m2/repository]], containers=[ContainerTemplate

{name='docker', image='wsibimagerepo.azurecr.io/docker:stable-git', workingDir='/home/jenkins/agent', command='cat', ttyEnabled=true}

, ContainerTemplate{name='kubectl', image='lachlanevenson/k8s-kubectl:v1.14.6', workingDir='/home/jenkins/agent', command='cat', ttyEnabled=true}], annotations=[org.csanchez.jenkins.plugins.kubernetes.PodAnnotation@aab9c821]}
io.fabric8.kubernetes.client.KubernetesClientException:
at io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager$1.onFailure(WatchConnectionManager.java:198)
at okhttp3.internal.ws.RealWebSocket.failWebSocket(RealWebSocket.java:571)
at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:198)
at okhttp3.RealCall$AsyncCall.execute(RealCall.java:206)
at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

Aug 22, 2019 12:41:32 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave _terminate
INFO: Terminating Kubernetes instance for agent worker-8b08130f-837c-4a7a-9d51-d81cb28d75c4-f4jt1-3cdm6
Terminated Kubernetes instance for agent development/worker-8b08130f-837c-4a7a-9d51-d81cb28d75c4-f4jt1-3cdm6
Disconnected computer worker-8b08130f-837c-4a7a-9d51-d81cb28d75c4-f4jt1-3cdm6
Aug 22, 2019 12:41:32 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave deleteSlavePod
INFO: Terminated Kubernetes instance for agent development/worker-8b08130f-837c-4a7a-9d51-d81cb28d75c4-f4jt1-3cdm6
Aug 22, 2019 12:41:32 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave _terminate
INFO: Disconnected computer worker-8b08130f-837c-4a7a-9d51-d81cb28d75c4-f4jt1-3cdm6
Aug 22,

christiani@netcompany.com (JIRA)

unread,
Aug 22, 2019, 8:53:02 AM8/22/19
to jenkinsc...@googlegroups.com
Christian Ihle edited a comment on Bug JENKINS-59000
Stumbled upon the same issue when testing an upgrade to 1.13.10. Currently this blocks us from implementing security fixes to our clusters.


Clusters are on premise, kubeadm-configured.

jenkins-ci@carlossanchez.eu (JIRA)

unread,
Aug 22, 2019, 8:56:03 AM8/22/19
to jenkinsc...@googlegroups.com

you are getting a 403, have you checked that you have the right permissions to exec into a pod?
https://stackoverflow.com/questions/48118125/kubernetes-rbac-role-verbs-to-exec-to-pod

olivier@omary.fr (JIRA)

unread,
Aug 22, 2019, 9:08:03 AM8/22/19
to jenkinsc...@googlegroups.com

I think it's not a 403 cause when you watch the pods `kubectl get -a pods --watch` you can see the pod instance created and then deleted. If 403 then no pod should appear... no ?

hayden@haydenball.me.uk (JIRA)

unread,
Aug 22, 2019, 9:28:06 AM8/22/19
to jenkinsc...@googlegroups.com

Carlos Sanchez I'm seeing the same issue having applied the sample RBAC config (https://github.com/jenkinsci/kubernetes-plugin/blob/master/src/main/kubernetes/service-account.yml).

As far as I can tell, that should allow exec.

ivan.lefkate@gmail.com (JIRA)

unread,
Aug 22, 2019, 9:28:06 AM8/22/19
to jenkinsc...@googlegroups.com

Hi Carlos,

This is what I have, it works fine on 1.15.2 and was also receiving 403 on 1.15.3 (I also thought it was an RBAC configuration issue):

apiVersion: v1
kind: ServiceAccount
metadata:
name: jenkins
namespace: jenkins
—
kind: Role
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: jenkins
namespace: jenkins
rules:

  • apiGroups: [""]
    resources: ["pods"]
    verbs: ["create","delete","get","list","patch","update","watch"]
  • apiGroups: [""]
    resources: ["pods/exec"]
    verbs: ["create","delete","get","list","patch","update","watch"]
  • apiGroups: [""]
    resources: ["pods/log"]
    verbs: ["get","list","watch"]
  • apiGroups: [""]
    resources: ["secrets"]
    verbs: ["get"]
    —
    apiVersion: rbac.authorization.k8s.io/v1beta1
    kind: RoleBinding
    metadata:
    name: jenkins
    namespace: jenkins
    roleRef:
    apiGroup: rbac.authorization.k8s.io
    kind: Role
    name: jenkins
    subjects:
  • kind: ServiceAccount
    name: jenkins
    namespace: jenkins

 

Regards,

Ivan

 

 

christiani@netcompany.com (JIRA)

unread,
Aug 22, 2019, 9:31:04 AM8/22/19
to jenkinsc...@googlegroups.com

I also get the 403 in the jenkins logs. Using the jenkins serviceaccount I can exec into pods with kubectl. The 403 also appears in the apiserver logs like this:

{"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"810feb2c-b0f2-48f7-8517-d59329cd230f","stage":"ResponseComplete","requestURI":"/","verb":"get","user":{"username":"system:serviceaccount:itasbuild:jenkins-sa","uid":"a7822545-bdb6-11e9-b23e-0050569b9a86","groups":["system:serviceaccounts","system:serviceaccounts:itasbuild","system:authenticated"]},"sourceIPs":["10...."],"userAgent":"okhttp/3.12.0","responseStatus":{"metadata":{},"status":"Failure","reason":"Forbidden","code":403},"requestReceivedTimestamp":"2019-08-22T12:53:50.902256Z","stageTimestamp":"2019-08-22T12:53:50.902405Z","annotations":{"authorization.k8s.io/decision":"forbid","authorization.k8s.io/reason":""}}

Not sure why it's trying to get "/" on the apiserver.

dave@schile.com (JIRA)

unread,
Aug 22, 2019, 1:00:04 PM8/22/19
to jenkinsc...@googlegroups.com

Seeing the same thing

dave@schile.com (JIRA)

unread,
Aug 22, 2019, 2:17:02 PM8/22/19
to jenkinsc...@googlegroups.com
David Schile edited a comment on Bug JENKINS-59000
Seeing the same thing : 403 from the server when attempting an `exec`  The api server audit logs also show the `requestURI` as `/`

 

dave@schile.com (JIRA)

unread,
Aug 22, 2019, 2:18:03 PM8/22/19
to jenkinsc...@googlegroups.com
David Schile edited a comment on Bug JENKINS-59000
Seeing the same thing: 403 from the server when attempting an `

{code:java}
 exec ` {code}
  The api server audit logs also show the `

{code:java}
 requestURI ` {code}
 as `/`
{code:java}
/{code}
  

dave@schile.com (JIRA)

unread,
Aug 22, 2019, 3:42:02 PM8/22/19
to jenkinsc...@googlegroups.com

fabric8 error:

Aug 22, 2019 7:37:49 PM FINE io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager runWatch
Connecting websocket ... io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@6d1c3fb4
Aug 22, 2019 7:37:49 PM WARNING io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager$1 onFailure
Exec Failure: HTTP 403, Status: 403 - 
java.net.ProtocolException: Expected HTTP 101 response but was '403 Forbidden'
	at okhttp3.internal.ws.RealWebSocket.checkResponse(RealWebSocket.java:229)
	at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:196)
	at okhttp3.RealCall$AsyncCall.execute(RealCall.java:206)
	at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread
.java:748)

Aug 22, 2019 7:37:49 PM FINE io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager scheduleReconnect
Submitting reconnect task to the executor
Aug 22, 2019 7:37:49 PM FINE io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager close
Force closing the watch io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@6d1c3fb4
Aug 22, 2019 7:37:49 PM FINE io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager$2 execute
Scheduling reconnect task
Aug 22, 2019 7:37:49 PM FINE io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager nextReconnectInterval
Current reconnect backoff is 1000 milliseconds (T0)

Please advise if there are any further logs or info needed. We are blocked at this point.

asidhu11702@gmail.com (JIRA)

unread,
Aug 22, 2019, 4:04:02 PM8/22/19
to jenkinsc...@googlegroups.com

carlos its not a permission issue, i can do exec into the slave pods no issue. please help we are screwed

jenkins-ci@carlossanchez.eu (JIRA)

unread,
Aug 22, 2019, 5:01:03 PM8/22/19
to jenkinsc...@googlegroups.com

in https://github.com/jenkinsci/kubernetes-plugin#debugging you can see how to enable the okhttp logs to see what urls are being requested and which are returning the 403

dave@schile.com (JIRA)

unread,
Aug 22, 2019, 5:05:06 PM8/22/19
to jenkinsc...@googlegroups.com

All I get from the okhttp3 log is:

Aug 22, 2019 9:02:28 PM INFO okhttp3.internal.platform.Platform log
ALPN callback dropped: HTTP/2 is disabled. Is alpn-boot on the boot class path?
Aug 22, 2019 9:02:58 PM INFO okhttp3.internal.platform.Platform log
ALPN callback dropped: HTTP/2 is disabled. Is alpn-boot on the boot class path?

I think it's because the errors are coming from okhttp3.internal ?

dave@schile.com (JIRA)

unread,
Aug 22, 2019, 5:06:02 PM8/22/19
to jenkinsc...@googlegroups.com
David Schile edited a comment on Bug JENKINS-59000
All I get from the okhttp3 log is:

{noformat}

Aug 22, 2019 9:02:28 PM INFO okhttp3.internal.platform.Platform log
ALPN callback dropped: HTTP/2 is disabled. Is alpn-boot on the boot class path?
Aug 22, 2019 9:02:58 PM INFO okhttp3.internal.platform.Platform log
ALPN callback dropped: HTTP/2 is disabled. Is alpn-boot on the boot class path?
{noformat}

I think it's because the errors are coming from okhttp3.internal ?

azolo1089@gmail.com (JIRA)

unread,
Aug 22, 2019, 6:13:02 PM8/22/19
to jenkinsc...@googlegroups.com
Justin Baker updated an issue
 
Change By: Justin Baker
Attachment: log.out

azolo1089@gmail.com (JIRA)

unread,
Aug 22, 2019, 6:18:06 PM8/22/19
to jenkinsc...@googlegroups.com
Justin Baker commented on Bug JENKINS-59000
 
Re: KubernetesClientException onFailure(WatchConnectionManager.java:198) on k8s v1.15.3

Carlos Sanchez Looks like it's making a request failed request before first? Either way, here are some logs that I got:  Unable to embed resource: log.out of type application/octet-stream

This seems to be the response that destroys it all (You can see the entire context in the log):

<-- 403 https://kubernetes.default.svc.cluster.local/ (1ms)

{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"forbidden: User \"system:serviceaccount:jenkins:jenkins\" cannot get path \"/\"","reason":"Forbidden","details":{},"code":403}

Hope that helps!

azolo1089@gmail.com (JIRA)

unread,
Aug 22, 2019, 6:18:09 PM8/22/19
to jenkinsc...@googlegroups.com
Justin Baker edited a comment on Bug JENKINS-59000
[~csanchez] Looks like it's making a request failed request before first pod request ? Either way, here are some logs that I got:   ! Unable to embed resource: log.out |width=7,height=7,align=absmiddle! of type application/octet-stream

This seems to be the response that destroys it all (You can see the entire context in the log):

{{^<-- 403 [ https://kubernetes.default.svc.cluster.local/ ] (1ms)^}}

{{^{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"forbidden: User \"system:serviceaccount:jenkins:jenkins\" cannot get path \"/\"","reason":"Forbidden","details":{},"code":403}^}}

Hope that helps!

azolo1089@gmail.com (JIRA)

unread,
Aug 22, 2019, 6:18:09 PM8/22/19
to jenkinsc...@googlegroups.com
Justin Baker edited a comment on Bug JENKINS-59000
[~csanchez] Looks like it's making a request failed request before the first pod request? Either way, here are some logs that I got:  Unable to embed resource: log.out of type application/octet-stream


This seems to be the response that destroys it all (You can see the entire context in the log):

{{^<-- 403 [https://kubernetes.default.svc.cluster.local/] (1ms)^}}

{{^{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"forbidden: User \"system:serviceaccount:jenkins:jenkins\" cannot get path \"/\"","reason":"Forbidden","details":{},"code":403}^}}

Hope that helps!

azolo1089@gmail.com (JIRA)

unread,
Aug 22, 2019, 6:20:05 PM8/22/19
to jenkinsc...@googlegroups.com
Justin Baker edited a comment on Bug JENKINS-59000
[~csanchez] Looks like it's making a request failed request before the first pod request? Either way, here are some logs that I got:   Unable to embed resource [https : log //gist . out of type application github.com / octet-stream Azolo/375f72a7e4b77b338c5b675fc3675662]

This seems to be the response that destroys it all (You can see the entire context in the log):

{{^<-- 403 [https://kubernetes.default.svc.cluster.local/] (1ms)^}}

{{^{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"forbidden: User \"system:serviceaccount:jenkins:jenkins\" cannot get path \"/\"","reason":"Forbidden","details":{},"code":403}^}}

Hope that helps!

azolo1089@gmail.com (JIRA)

unread,
Aug 22, 2019, 7:21:03 PM8/22/19
to jenkinsc...@googlegroups.com
Justin Baker updated an issue
Change By: Justin Baker
Attachment: log.out

rene.schoenlein@cdi-ag.de (JIRA)

unread,
Aug 23, 2019, 4:49:04 AM8/23/19
to jenkinsc...@googlegroups.com
Rene Schönlein commented on Bug JENKINS-59000
 
Re: KubernetesClientException onFailure(WatchConnectionManager.java:198) on k8s v1.15.3

After doing some google research, it seems to me that the issue is related to:

https://github.com/fabric8io/kubernetes-client/issues/1667

https://github.com/fabric8io/kubernetes-client/pull/1669

As a workaround, I changed the plugin configuration so that the kubernetes URL includes a port (e.g. from "https://kubernetes.default" to "https://kubernetes.default:443").

christiani@netcompany.com (JIRA)

unread,
Aug 23, 2019, 4:54:02 AM8/23/19
to jenkinsc...@googlegroups.com

I already tested that, but it didn't make any difference. Did you get everything working again?

rene.schoenlein@cdi-ag.de (JIRA)

unread,
Aug 23, 2019, 4:58:02 AM8/23/19
to jenkinsc...@googlegroups.com

Christian Ihle
Yes, for me this was the only change needed to get it working again. I am on Kubernetes 1.13.10 by the way.

rene.schoenlein@cdi-ag.de (JIRA)

unread,
Aug 23, 2019, 4:59:01 AM8/23/19
to jenkinsc...@googlegroups.com
Rene Schönlein edited a comment on Bug JENKINS-59000
[~christiani]
Yes, for me this was the only change needed to get it working again. I am on Kubernetes 1 v1 .13.10 by the way.

christiani@netcompany.com (JIRA)

unread,
Aug 23, 2019, 5:39:03 AM8/23/19
to jenkinsc...@googlegroups.com

Ah, figured it out. We changed the configmap, but seems Jenkins didn't reload the value during restart. Changing it in the user interface made it work. Needs some testing before we roll out 1.13.10 to our users, but it looks promising.

hayden@haydenball.me.uk (JIRA)

unread,
Aug 23, 2019, 5:57:03 AM8/23/19
to jenkinsc...@googlegroups.com

Rene Schönlein Thank you so much - that's sorted it for us.

asidhu11702@gmail.com (JIRA)

unread,
Aug 23, 2019, 8:35:02 AM8/23/19
to jenkinsc...@googlegroups.com

carlos, is this the right fix?

NARESHKUMAR1861@GMAIL.COM (JIRA)

unread,
Aug 23, 2019, 1:40:02 PM8/23/19
to jenkinsc...@googlegroups.com

 Hey, we upgraded kube cluster to v 1.13.10, then started getting this error. Tried this workaround "changed the plugin configuration so that the kubernetes URL includes a port (e.g. from "https://kubernetes.default" to "https://kubernetes.default:443")."

But this didn't work for me.

Now it's throwing "java.net.SocketException: Connection reset" and POD is failed to launch.

Any help ?

NARESHKUMAR1861@GMAIL.COM (JIRA)

unread,
Aug 23, 2019, 2:05:02 PM8/23/19
to jenkinsc...@googlegroups.com

Correction to the above comment. Mentioned workaround fixed the issue. Thank you.

jenkins-ci@carlossanchez.eu (JIRA)

unread,
Aug 24, 2019, 5:28:05 AM8/24/19
to jenkinsc...@googlegroups.com
Carlos Sanchez started work on Bug JENKINS-59000
 
Change By: Carlos Sanchez
Status: Open In Progress

jenkins-ci@carlossanchez.eu (JIRA)

unread,
Aug 24, 2019, 5:28:06 AM8/24/19
to jenkinsc...@googlegroups.com
Change By: Carlos Sanchez
Status: In Progress Review

jenkins-ci@carlossanchez.eu (JIRA)

unread,
Aug 24, 2019, 5:28:07 AM8/24/19
to jenkinsc...@googlegroups.com
Carlos Sanchez assigned an issue to Carlos Sanchez
Change By: Carlos Sanchez
Assignee: Carlos Sanchez

jenkins-ci@carlossanchez.eu (JIRA)

unread,
Aug 24, 2019, 5:29:02 AM8/24/19
to jenkinsc...@googlegroups.com

asidhu11702@gmail.com (JIRA)

unread,
Aug 24, 2019, 11:05:05 AM8/24/19
to jenkinsc...@googlegroups.com

Hi Carlos, what is this pull request going to do for us? The work around mentioned in the thread currently works for us which is https://kubernetes.default:443 in the plugin.

jenkins-ci@carlossanchez.eu (JIRA)

unread,
Aug 24, 2019, 11:10:02 AM8/24/19
to jenkinsc...@googlegroups.com

The pr avoids the need for the workaround

vincent@latombe.net (JIRA)

unread,
Aug 26, 2019, 9:32:04 AM8/26/19
to jenkinsc...@googlegroups.com
Change By: Vincent Latombe
Status: In Review Fixed but Unreleased
Resolution: Fixed

vincent@latombe.net (JIRA)

unread,
Aug 26, 2019, 10:46:05 AM8/26/19
to jenkinsc...@googlegroups.com
Change By: Vincent Latombe
Status: Fixed but Unreleased Closed
Released As: 1.18.2

asidhu11702@gmail.com (JIRA)

unread,
Aug 31, 2019, 12:52:02 AM8/31/19
to jenkinsc...@googlegroups.com
Ahuhu uhuhuh commented on Bug JENKINS-59000
 
Re: KubernetesClientException onFailure(WatchConnectionManager.java:198) on k8s v1.15.3

carlos is the problem fixed? can we download the latest plugin that has the problems fixed?

kmushegian@gmail.com (JIRA)

unread,
Sep 6, 2019, 4:42:03 PM9/6/19
to jenkinsc...@googlegroups.com

Was having the same issue this morning, upgrading the plug-in to the most recent version (1.18.3) fixed the issue and doesn't require the port workaround!
Reply all
Reply to author
Forward
0 new messages