[JIRA] (JENKINS-55123) Kubernetes plugin does not respect 'no_proxy' configurations when calling Kubernetes API

207 views
Skip to first unread message

dtgames@kinggeek.org (JIRA)

unread,
May 9, 2019, 4:51:03 PM5/9/19
to jenkinsc...@googlegroups.com
Dax Games updated an issue
 
Jenkins / Bug JENKINS-55123
Kubernetes plugin does not respect 'no_proxy' configurations when calling Kubernetes API
Change By: Dax Games
Summary: Kubernetes plugin does not respect 'no_proxy' configurations when calling Kubernetes API
Add Comment Add Comment
 
This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)

dtgames@kinggeek.org (JIRA)

unread,
May 9, 2019, 4:55:03 PM5/9/19
to jenkinsc...@googlegroups.com
Dax Games commented on Bug JENKINS-55123
 
Re: Kubernetes plugin does not respect 'no_proxy' configurations when calling Kubernetes API

I found this code that passes the master env vars related to proxy settings to the slave.

The problem is the pod never starts because the plugin can't talk to the Kubernetes API because it ignores the no_proxy settings.

dtgames@kinggeek.org (JIRA)

unread,
May 9, 2019, 4:57:03 PM5/9/19
to jenkinsc...@googlegroups.com
Dax Games updated an issue
Change By: Dax Games
Comment: Carlos/Plugin Maintainers,

Can we have a way to pass proxy and no_proxy settings from the Jenkins master to the Kubernetes slave?

Thanks.

jenkins-ci@carlossanchez.eu (JIRA)

unread,
May 10, 2019, 2:55:03 AM5/10/19
to jenkinsc...@googlegroups.com
Carlos Sanchez commented on Bug JENKINS-55123
 
Re: Kubernetes plugin does not respect 'no_proxy' configurations when calling Kubernetes API

If you are talking about the communication between Jenkins and Kubernetes API the following system properties are used to configure proxy

  • http.proxy
  • https.proxy
  • no.proxy
  • proxy.username
  • proxy.password

https://github.com/fabric8io/kubernetes-client/blob/9e850ad769134566e32f8c41498e5e64ecf8b799/kubernetes-client/src/main/java/io/fabric8/kubernetes/client/Config.java#L106

You can see a test that is passing here https://github.com/jenkinsci/kubernetes-plugin/pull/477

jenkins-ci@carlossanchez.eu (JIRA)

unread,
May 10, 2019, 3:07:02 AM5/10/19
to jenkinsc...@googlegroups.com

Also see issue in kubernetes-client library https://github.com/fabric8io/kubernetes-client/issues/1335

dtgames@kinggeek.org (JIRA)

unread,
May 10, 2019, 2:46:06 PM5/10/19
to jenkinsc...@googlegroups.com

I am now running Jenkins using the following java properties:

-Dhttp.proxy=http://x.x.x.x:xxxx -Dhttps.proxy=https://x.x.x.x:xxxx -Dno.Proxy=xxx -Dhttp.proxyHost=x.x.x.x -Dhttp.proxyPort=xxxx -Dhttps.proxyHost=x.x.x.x -Dhttps.proxyPort=xxxx -Dhttp.nonProxyHosts=xxx

Testing the connection from the Jenkins server running in Kubernetes through the Kubernetes plugin using the 'test connection' results in:

Error testing connection https://kubernetes.default: java.io.IOException: Unexpected response code for CONNECT: 502

and the following log entry on the proxy server:

1557513166.881     67 192.168.1.73 TCP_TUNNEL/502 0 CONNECT kubernetes.default.svc:443 - FIRSTUP_PARENT/192.168.1.100 -

Testing the connection from the Jenkins server running in Kubernetes through the 'Manage Jenkins->Manage Plugins->Advanced' tab using the 'validate proxy button with the test URL of 'https://kubernetes.default.svc' results in the following expected failure since we are not authenticated:

Failed to connect to https://kubernetes.default.svc.
javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated
	at sun.security.ssl.SSLSessionImpl.getPeerCertificates(SSLSessionImpl.java:450)
	at org.apache.commons.httpclient.protocol.SSLProtocolSocketFactory.verifyHostName(SSLProtocolSocketFactory.java:257)
	at org.apache.commons.httpclient.protocol.SSLProtocolSocketFactory.createSocket(SSLProtocolSocketFactory.java:115)
	at org.apache.commons.httpclient.protocol.SSLProtocolSocketFactory.createSocket(SSLProtocolSocketFactory.java:156)
	at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:714)
	at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:394)
	at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:178)
	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:404)
	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:330)

From the command line of the same Jenkins server that has the exact same proxy settings defined in the java properties above set in environment variables. I can issue the following and retrieve a json object containing all pods in the namespace:

curl $APISERVER/api/v1/namespaces/pd-dgames-test/pods/ --header "Authorization: Bearer $TOKEN" --cacert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt

Kubernetes plugin spinning up a slave repeatedly gives me this:

May 10, 2019 6:24:53 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud provision
INFO: Excess workload after pending Kubernetes agents: 1
May 10, 2019 6:24:53 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud provision
INFO: Template for label pd-slave: Kubernetes Pod Template
May 10, 2019 6:24:53 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud provision
WARNING: Failed to count the # of live instances on Kubernetes
java.io.IOException: Unexpected response code for CONNECT: 502
        at okhttp3.internal.connection.RealConnection.createTunnel(RealConnection.java:389)
        at okhttp3.internal.connection.RealConnection.connectTunnel(RealConnection.java:211)
        at okhttp3.internal.connection.RealConnection.connect(RealConnection.java:152)
        at okhttp3.internal.connection.StreamAllocation.findConnection(StreamAllocation.java:256)
        at okhttp3.internal.connection.StreamAllocation.findHealthyConnection(StreamAllocation.java:134)
        at okhttp3.internal.connection.StreamAllocation.newStream(StreamAllocation.java:113)
        at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:42)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
        at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
        at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
        at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:125)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
        at io.fabric8.kubernetes.client.utils.BackwardsCompatibilityInterceptor.intercept(BackwardsCompatibilityInterceptor.java:119)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
        at io.fabric8.kubernetes.client.utils.ImpersonatorInterceptor.intercept(ImpersonatorInterceptor.java:68)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
        at io.fabric8.kubernetes.client.utils.HttpClientUtils$2.intercept(HttpClientUtils.java:107)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
        at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:200)
        at okhttp3.RealCall.execute(RealCall.java:77)
        at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:379)
        at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:344)
        at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:328)
        at io.fabric8.kubernetes.client.dsl.base.BaseOperation.listRequestHelper(BaseOperation.java:193)
        at io.fabric8.kubernetes.client.dsl.base.BaseOperation.list(BaseOperation.java:618)
        at io.fabric8.kubernetes.client.dsl.base.BaseOperation.list(BaseOperation.java:68)
        at org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud.addProvisionedSlave(KubernetesCloud.java:505)
        at org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud.provision(KubernetesCloud.java:458)
        at hudson.slaves.NodeProvisioner$StandardStrategyImpl.apply(NodeProvisioner.java:715)
        at hudson.slaves.NodeProvisioner.update(NodeProvisioner.java:320)
        at hudson.slaves.NodeProvisioner.access$000(NodeProvisioner.java:61)
        at hudson.slaves.NodeProvisioner$NodeProvisionerInvoker.doRun(NodeProvisioner.java:809)
        at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:72)
        at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

dtgames@kinggeek.org (JIRA)

unread,
May 10, 2019, 2:47:03 PM5/10/19
to jenkinsc...@googlegroups.com
Dax Games edited a comment on Bug JENKINS-55123
Yeah been there looked at that also.
I am now running Jenkins using the following java properties:

{code:java}

-Dhttp.proxy=http://x.x.x.x:xxxx -Dhttps.proxy=https://x.x.x.x:xxxx -Dno.Proxy=xxx -Dhttp.proxyHost=x.x.x.x -Dhttp.proxyPort=xxxx -Dhttps.proxyHost=x.x.x.x -Dhttps.proxyPort=xxxx -Dhttp.nonProxyHosts=xxx
{code}


Testing the connection from the Jenkins server running in Kubernetes through the Kubernetes plugin using the 'test connection' results in:

{code:java}

Error testing connection https://kubernetes.default: java.io.IOException: Unexpected response code for CONNECT: 502
{code}


and the following log entry on the proxy server:

{code:bash}

1557513166.881     67 192.168.1.73 TCP_TUNNEL/502 0 CONNECT kubernetes.default.svc:443 - FIRSTUP_PARENT/192.168.1.100 -
{code}


Testing the connection from the Jenkins server running in Kubernetes through the 'Manage Jenkins->Manage Plugins->Advanced' tab using the 'validate proxy button with the test URL of 'https://kubernetes.default.svc' results in the following expected failure since we are not authenticated:

{code:java}

Failed to connect to https://kubernetes.default.svc.
javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated
at sun.security.ssl.SSLSessionImpl.getPeerCertificates(SSLSessionImpl.java:450)
at org.apache.commons.httpclient.protocol.SSLProtocolSocketFactory.verifyHostName(SSLProtocolSocketFactory.java:257)
at org.apache.commons.httpclient.protocol.SSLProtocolSocketFactory.createSocket(SSLProtocolSocketFactory.java:115)
at org.apache.commons.httpclient.protocol.SSLProtocolSocketFactory.createSocket(SSLProtocolSocketFactory.java:156)
at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:714)
at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:394)
at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:178)
at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:404)
at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:330)
{code}


From the command line of the same Jenkins server that has the exact same proxy settings defined in the java properties above set in environment variables.  I can issue the following and retrieve a json object containing all pods in the namespace:

{code:bash}

curl $APISERVER/api/v1/namespaces/pd-dgames-test/pods/ --header "Authorization: Bearer $TOKEN" --cacert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
{code}


Kubernetes plugin spinning up a slave repeatedly gives me this:

{code:java}
{code}

dtgames@kinggeek.org (JIRA)

unread,
May 10, 2019, 2:48:05 PM5/10/19
to jenkinsc...@googlegroups.com

dtgames@kinggeek.org (JIRA)

unread,
May 10, 2019, 2:50:02 PM5/10/19
to jenkinsc...@googlegroups.com

jenkins-ci@carlossanchez.eu (JIRA)

unread,
May 10, 2019, 2:59:02 PM5/10/19
to jenkinsc...@googlegroups.com

so you don't want https://kubernetes.default.svc to go through the proxy, correct?
do you have -Dno.Proxy=kubernetes.default.svc,somethingelse ?

dtgames@kinggeek.org (JIRA)

unread,
May 10, 2019, 3:01:04 PM5/10/19
to jenkinsc...@googlegroups.com

yes that is in my -Dno.Proxy kubernetes.default.svc,... that starts Kenkins

dtgames@kinggeek.org (JIRA)

unread,
May 10, 2019, 3:01:05 PM5/10/19
to jenkinsc...@googlegroups.com
Dax Games edited a comment on Bug JENKINS-55123
yes that is in my {{-Dno.Proxy kubernetes.default.svc,...}} that starts Kenkins Jenkins

dtgames@kinggeek.org (JIRA)

unread,
May 10, 2019, 3:05:02 PM5/10/19
to jenkinsc...@googlegroups.com
Dax Games edited a comment on Bug JENKINS-55123
yes Yes that is in my {{-Dno.Proxy kubernetes.default.svc,...}} that starts Jenkins .

{{APISERVER=https://kubernetes.default.svc}} in the curl command.

jglick@cloudbees.com (JIRA)

unread,
Jul 16, 2019, 3:43:43 PM7/16/19
to jenkinsc...@googlegroups.com
Jesse Glick assigned an issue to Unassigned
 
Change By: Jesse Glick
Assignee: Carlos Sanchez

davidkarlsen@java.net (JIRA)

unread,
Aug 29, 2019, 9:17:02 AM8/29/19
to jenkinsc...@googlegroups.com
davidkarlsen commented on Bug JENKINS-55123
 
Re: Kubernetes plugin does not respect 'no_proxy' configurations when calling Kubernetes API

what's strange also is that this only happens when declaring a yaml/yamlFile in a declarative pipeline, not if it is defined as a podTemplate on jenkins master.
Reply all
Reply to author
Forward
0 new messages