[JIRA] (JENKINS-39695) Shell stop in a pipeline job: Java.net.UnknownHostException: kubernetes.default.svc: unknown error

350 views
Skip to first unread message

seb@nmeos.net (JIRA)

unread,
Nov 12, 2016, 11:42:03 PM11/12/16
to jenkinsc...@googlegroups.com
Sébastien Douche created an issue
 
Jenkins / Bug JENKINS-39695
Shell stop in a pipeline job: Java.net.UnknownHostException: kubernetes.default.svc: unknown error
Issue Type: Bug Bug
Assignee: Carlos Sanchez
Components: kubernetes-plugin
Created: 2016/Nov/13 4:41 AM
Environment: Jenkins: 2.19.2
kubernetes-plugin: 0.9
Priority: Minor Minor
Reporter: Sébastien Douche

Hi,
I'm trying the job example, but the job fails on the shell step:

{{{
[Pipeline] container
[Pipeline] {
[Pipeline] stage
[Pipeline]

{ (Build a Go project) [Pipeline] sh [k8s-test2] Running shell script Executing shell script inside container [golang] of pod [jenkins-slave-7be4b22b164184] java.net.UnknownHostException: kubernetes.default.svc: unknown error at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method) at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:928) at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1323) at java.net.InetAddress.getAllByName0(InetAddress.java:1276) at java.net.InetAddress.getAllByName(InetAddress.java:1192) at java.net.InetAddress.getAllByName(InetAddress.java:1126) at com.squareup.okhttp.Dns$1.lookup(Dns.java:39) at com.squareup.okhttp.internal.http.RouteSelector.resetNextInetSocketAddress(RouteSelector.java:175) at com.squareup.okhttp.internal.http.RouteSelector.nextProxy(RouteSelector.java:141) at com.squareup.okhttp.internal.http.RouteSelector.next(RouteSelector.java:83) at com.squareup.okhttp.internal.http.StreamAllocation.findConnection(StreamAllocation.java:174) at com.squareup.okhttp.internal.http.StreamAllocation.findHealthyConnection(StreamAllocation.java:126) at com.squareup.okhttp.internal.http.StreamAllocation.newStream(StreamAllocation.java:95) at com.squareup.okhttp.internal.http.HttpEngine.connect(HttpEngine.java:281) at com.squareup.okhttp.internal.http.HttpEngine.sendRequest(HttpEngine.java:224) at com.squareup.okhttp.Call.getResponse(Call.java:286) at com.squareup.okhttp.Call$ApplicationInterceptorChain.proceed(Call.java:243) at com.squareup.okhttp.Call.getResponseWithInterceptorChain(Call.java:205) at com.squareup.okhttp.Call.access$100(Call.java:35) at com.squareup.okhttp.Call$AsyncCall.execute(Call.java:171) at com.squareup.okhttp.internal.NamedRunnable.run(NamedRunnable.java:33) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) [Pipeline] }

[Pipeline] // stage
[Pipeline] }
[Pipeline] // container
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
}}}

But kube-dns is up and running :

{{@kubectl run -i --tty busybox --image=busybox --generator="run-pod/v1"
Waiting for pod default/busybox to be running, status is Pending, pod ready: false

Hit enter for command prompt

/ # nslookup kubernetes.default.svc
Server: 172.18.200.10
Address 1: 172.18.200.10 kube-dns.kube-system.svc.cluster.local

Name: kubernetes.default.svc
Address 1: 172.18.200.1 kubernetes.default.svc.cluster.local
}}

Any ideas?

Add Comment Add Comment
 
This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c)
Atlassian logo

kgraham@businessinsider.com (JIRA)

unread,
Nov 17, 2016, 3:02:03 PM11/17/16
to jenkinsc...@googlegroups.com
kevin graham commented on Bug JENKINS-39695
 
Re: Shell stop in a pipeline job: Java.net.UnknownHostException: kubernetes.default.svc: unknown error

I too am having this same issue, here is the stack trace

java.net.UnknownHostException: kubernetes.default.svc: Name or service not known
	at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
	at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:928)
	at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1323)
	at java.net.InetAddress.getAllByName0(InetAddress.java:1276)
	at java.net.InetAddress.getAllByName(InetAddress.java:1192)
	at java.net.InetAddress.getAllByName(InetAddress.java:1126)
	at com.squareup.okhttp.Dns$1.lookup(Dns.java:39)
	at com.squareup.okhttp.internal.http.RouteSelector.resetNextInetSocketAddress(RouteSelector.java:175)
	at com.squareup.okhttp.internal.http.RouteSelector.nextProxy(RouteSelector.java:141)
	at com.squareup.okhttp.internal.http.RouteSelector.next(RouteSelector.java:83)
	at com.squareup.okhttp.internal.http.StreamAllocation.findConnection(StreamAllocation.java:174)
	at com.squareup.okhttp.internal.http.StreamAllocation.findHealthyConnection(StreamAllocation.java:126)
	at com.squareup.okhttp.internal.http.StreamAllocation.newStream(StreamAllocation.java:95)
	at com.squareup.okhttp.internal.http.HttpEngine.connect(HttpEngine.java:281)
	at com.squareup.okhttp.internal.http.HttpEngine.sendRequest(HttpEngine.java:224)
	at com.squareup.okhttp.Call.getResponse(Call.java:286)
	at com.squareup.okhttp.Call$ApplicationInterceptorChain.proceed(Call.java:243)
	at com.squareup.okhttp.Call.getResponseWithInterceptorChain(Call.java:205)
	at com.squareup.okhttp.Call.access$100(Call.java:35)
	at com.squareup.okhttp.Call$AsyncCall.execute(Call.java:171)
	at com.squareup.okhttp.internal.NamedRunnable.run(NamedRunnable.java:33)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

......
java.io.IOException: Pipe not connected
	at java.io.PipedOutputStream.write(PipedOutputStream.java:140)
	at java.io.OutputStream.write(OutputStream.java:75)
	at org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1.launch(ContainerExecDecorator.java:110)
	at hudson.Launcher$ProcStarter.start(Launcher.java:381)
	at org.jenkinsci.plugins.durabletask.BourneShellScript.launchWithCookie(BourneShellScript.java:147)
	at org.jenkinsci.plugins.durabletask.FileMonitoringTask.launch(FileMonitoringTask.java:61)
	at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.start(DurableTaskStep.java:158)
	at org.jenkinsci.plugins.workflow.cps.DSL.invokeStep(DSL.java:184)
	.......

and here is my jenkinsfile


stage('SCM - Checkout Archive and Stash') {

    node('master') {
        git branch: 'develop', credentialsId: 'ssh_credentials', url: 'g...@github.com:foo-biz/myapp.git'

        stash name: 'src', includes: 'src/**/*'

    }
}
    
stage('Application - Update Dependencies') {

    node('jnlp-slave-pod') {
        container('build-env') {
            sh "ls -l"
        }
    }
}

I have my setup under settings working, and it does start up a slave, but it always bombs out trying to resolve that address. When I run the very same kind of pod and exec 'nslookup', i to am able to resolve it.

So this is a very confusing bug

Does the resolution attempt happen on the jenkins master? or on the api server/node?

jenkins-ci@carlossanchez.eu (JIRA)

unread,
Nov 24, 2016, 12:46:01 PM11/24/16
to jenkinsc...@googlegroups.com

the resolution should happen in the jenkins agent, the jnlp continer inside the pod

jenkins-ci@carlossanchez.eu (JIRA)

unread,
Dec 6, 2016, 6:28:01 AM12/6/16
to jenkinsc...@googlegroups.com
Carlos Sanchez updated an issue
 
Change By: Carlos Sanchez
Hi,
I'm trying the job example, but the job fails on the shell step:

{ {{ code}
[Pipeline] container
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Build a Go project)
[Pipeline] sh
[k8s-test2] Running shell script
Executing shell script inside container [golang] of pod [jenkins-slave-7be4b22b164184]
java.net.UnknownHostException: kubernetes.default.svc: unknown error
at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:928)
at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1323)
at java.net.InetAddress.getAllByName0(InetAddress.java:1276)
at java.net.InetAddress.getAllByName(InetAddress.java:1192)
at java.net.InetAddress.getAllByName(InetAddress.java:1126)
at com.squareup.okhttp.Dns$1.lookup(Dns.java:39)
at com.squareup.okhttp.internal.http.RouteSelector.resetNextInetSocketAddress(RouteSelector.java:175)
at com.squareup.okhttp.internal.http.RouteSelector.nextProxy(RouteSelector.java:141)
at com.squareup.okhttp.internal.http.RouteSelector.next(RouteSelector.java:83)
at com.squareup.okhttp.internal.http.StreamAllocation.findConnection(StreamAllocation.java:174)
at com.squareup.okhttp.internal.http.StreamAllocation.findHealthyConnection(StreamAllocation.java:126)
at com.squareup.okhttp.internal.http.StreamAllocation.newStream(StreamAllocation.java:95)
at com.squareup.okhttp.internal.http.HttpEngine.connect(HttpEngine.java:281)
at com.squareup.okhttp.internal.http.HttpEngine.sendRequest(HttpEngine.java:224)
at com.squareup.okhttp.Call.getResponse(Call.java:286)
at com.squareup.okhttp.Call$ApplicationInterceptorChain.proceed(Call.java:243)
at com.squareup.okhttp.Call.getResponseWithInterceptorChain(Call.java:205)
at com.squareup.okhttp.Call.access$100(Call.java:35)
at com.squareup.okhttp.Call$AsyncCall.execute(Call.java:171)
at com.squareup.okhttp.internal.NamedRunnable.run(NamedRunnable.java:33)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // container
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
{code } }}


But kube-dns is up and running :

{ { code}
@kubectl run -i --tty busybox --image=busybox --generator="run-pod/v1"
Waiting for pod default/busybox to be running, status is Pending, pod ready: false

Hit enter for command prompt

/ # nslookup kubernetes.default.svc
Server:    172.18.200.10
Address 1: 172.18.200.10 kube-dns.kube-system.svc.cluster.local

Name:      kubernetes.default.svc
Address 1: 172.18.200.1 kubernetes.default.svc.cluster.local
{code } }

Any ideas?

jenkins-ci@carlossanchez.eu (JIRA)

unread,
Dec 6, 2016, 6:31:02 AM12/6/16
to jenkinsc...@googlegroups.com
 
Re: Shell stop in a pipeline job: Java.net.UnknownHostException: kubernetes.default.svc: unknown error

this is happening when jenkins master is running outside of kubernetes. Some code in the k8s api client must be using some default endpoint kubernetes.default.svc , any idea where could that be Ioannis Canellos ?

iocanel@gmail.com (JIRA)

unread,
Dec 6, 2016, 6:50:01 AM12/6/16
to jenkinsc...@googlegroups.com

The kubernetes-client is using kubernetes.default.svc unless explicitly specified.

When the client is used from the kubernetes-plugin it will use whatever value, has been configured in the Kubernetes Cloud.

iocanel@gmail.com (JIRA)

unread,
Dec 6, 2016, 6:51:01 AM12/6/16
to jenkinsc...@googlegroups.com

In other words, if you use the kubernetes-plugin outside of kubernetes, you need to properlly configure it to talk to the remote kubernetes.

inbarajan.prabhu@gmail.com (JIRA)

unread,
Dec 7, 2016, 1:50:02 AM12/7/16
to jenkinsc...@googlegroups.com

Ioannis Canellos - Can you please clarify the comment , my setup is somewhat similar. I am running my jenkins master in docker swarm, but have configured it to run my slave workloads on remote kubernetes.
i have looked at the kubernetes-plugin code somewhat superficially , but dont see how the stack trace above correlates with any logic from the kubernetes-client plugin. Interestingly enough, this problem seems to impact all the shell commands in pipeline DSL , the other jvm calls seems okay.

" if you use the kubernetes-plugin outside of kubernetes, you need to properlly configure it to talk to the remote kubernetes."

what configuration are you referring to?

inbarajan.prabhu@gmail.com (JIRA)

unread,
Dec 7, 2016, 1:55:01 AM12/7/16
to jenkinsc...@googlegroups.com
Prabhu Inbarajan edited a comment on Bug JENKINS-39695
[~iocanel] - Can you please clarify the comment , my setup is somewhat similar. I am running my jenkins master in docker swarm, but have configured it to run my slave workloads on remote kubernetes.

i have looked at the kubernetes-plugin code somewhat superficially , but dont see how the stack trace above correlates with any logic from the kubernetes-client plugin. Interestingly enough, this problem seems to impact all the shell commands in pipeline DSL , the other jvm calls seems okay.

" if you use the kubernetes-plugin outside of kubernetes, you need to properlly configure it to talk to the remote kubernetes."
what configuration are you referring to?

inbarajan.prabhu@gmail.com (JIRA)

unread,
Dec 7, 2016, 3:21:05 AM12/7/16
to jenkinsc...@googlegroups.com
Prabhu Inbarajan edited a comment on Bug JENKINS-39695
[~iocanel] - Can you please clarify the comment , my setup is somewhat similar. I am running my jenkins master in docker swarm, but have configured it to run my slave workloads on remote kubernetes.
i have looked at the kubernetes-plugin code somewhat superficially , but dont see how the stack trace above correlates with any logic from the kubernetes-client plugin. Interestingly enough, this problem seems to impact all the shell commands in pipeline DSL , the other jvm calls seems okay.

{code:java}

https://github.com/jenkinsci/kubernetes-plugin/blob/52c0af572ef091399fa76657be34471c033d1464/src/main/java/org/csanchez/jenkins/plugins/kubernetes/pipeline/ContainerExecDecorator.java#L92

{code}


" if you use the kubernetes-plugin outside of kubernetes, you need to properlly configure it to talk to the remote kubernetes."
what configuration are you referring to (or) perhaps you are referring to the usage of the kubernetes-client within the kubernetes-plugin ?

i tried setting the kubernetes_master (env) or kubernetes.master (system property) for jenkins master - but there is no change to the behavior

iocanel@gmail.com (JIRA)

unread,
Dec 7, 2016, 3:32:01 AM12/7/16
to jenkinsc...@googlegroups.com

Prabhu Inbarajan Thanks for investigating this, your feedback is really helpful.

It seems that the "containaier" exec part is using a default instance of the client, rather than obtaining one through the Kubernetes Cloud.
And this is why it ignores your change. I'll fix that asap.

Thanks again!

iocanel@gmail.com (JIRA)

unread,
Dec 7, 2016, 5:54:01 AM12/7/16
to jenkinsc...@googlegroups.com

iocanel@gmail.com (JIRA)

unread,
Dec 7, 2016, 5:54:01 AM12/7/16
to jenkinsc...@googlegroups.com

inbarajan.prabhu@gmail.com (JIRA)

unread,
Dec 7, 2016, 1:26:01 PM12/7/16
to jenkinsc...@googlegroups.com

Thanks Ioannis Canellos . That was super fast. Appreciate the quick turn around.

Can you point out where the fix is ? seems like it is in the kubernetes-plugin? if it is in a branch , I would be happy to test it out. I looked at the commit histories , but couldnt locate the fix branch.

Thanks
prabhu

iocanel@gmail.com (JIRA)

unread,
Dec 7, 2016, 1:29:02 PM12/7/16
to jenkinsc...@googlegroups.com

It was merged to master! So if you could try out the latest master that would be great!

inbarajan.prabhu@gmail.com (JIRA)

unread,
Dec 7, 2016, 2:07:01 PM12/7/16
to jenkinsc...@googlegroups.com
Prabhu Inbarajan edited a comment on Bug JENKINS-39695
Thanks [~iocanel]  . That was super fast. Appreciate the quick turn around.

Can you point out where the fix is ? seems like it is in the kubernetes-plugin? if it is in a branch , I would be happy to test it out. I looked at the commit histories ,
but couldnt locate seems like this is the fix branch one .
{code}
https://github.com/jenkinsci/kubernetes-plugin/commit/2be70ba43db9c00ab872e8d2c1eac7db27f084e0
{code}
Thanks
prabhu

inbarajan.prabhu@gmail.com (JIRA)

unread,
Dec 7, 2016, 3:51:01 PM12/7/16
to jenkinsc...@googlegroups.com

the fix works as advertised. i dont see the original issue anymore. however, a couple of observations - please validate:
1. now each container step is required to declare which cloud it needs to be launched in. would be great to have this inherit this info from the pod template, but seems like the design challenge is each pod templates can be inherited and each pod template can bind to a cloud, we cant derive the intent clean.
2. if the containerstep fails to declare the cloud, that results in a null pointer exception, we may need to handle it in a clean way.

test snippet below for reference (modified slightly from the original for build time):
Note:
1. that my jenkins instance runs from docker swarm , where as the slave workloads are scheduled on minikube
2. built with latest from master (0.10-SNAPSHOT)

podTemplate(
     cloud: 'minikube',
     label: 'docker',
     containers: [
        //containerTemplate(name: 'jnlp', image: 'jenkinsci/jnlp-slave:2.62-alpine', args: '${computer.jnlpmac} ${computer.name}'),
        containerTemplate(name: 'maven', image: 'maven:3.3.9-jdk-8-alpine', ttyEnabled: true, command: 'cat'),
        containerTemplate(name: 'golang', image: 'golang:1.6.3-alpine', ttyEnabled: true, command: 'cat')
      ],
      volumes: [secretVolume(secretName: 'shared-secret', mountPath: '/etc/shared-secrets')],
      ) {
        node ('docker') {
            stage 'Get a Maven project'
            git 'https://github.com/pgrimard/spring-boot-hello-world.git'
            container(name:'maven', cloud:'minikube') {
                stage 'Build a Maven project'
                try {
                sh 'mvn clean install'
                }catch (Exception ex) {
                    ex.printStackTrace()
                }
            }
    
            stage 'Get a Golang project'
            git url: 'https://github.com/golang/example.git'
            container(name:'golang', cloud:'minikube') {
                stage 'Build a Go project'
                sh """
                mkdir -p /go/src/github.com/golang/
                ln -s `pwd` /go/src/github.com/golang/example
                export GOPATH=/go
                cd /go/src/github.com/golang/example/hello && go build
                """
            }
        }   
}

aleksejs.cibinogins@gmail.com (JIRA)

unread,
Dec 29, 2016, 1:45:02 AM12/29/16
to jenkinsc...@googlegroups.com

Prabhu Inbarajan , can you provide an example for " each container step is required to declare which cloud it needs to be launched in".

Cause, I am getting NPE and have tried different ways do declare cloud for container, but nothing helps.
If I am downgrading to version 0,9, I am getting an issue from the ticket.
(My Jenkins is outside the cloud)

inbarajan.prabhu@gmail.com (JIRA)

unread,
Dec 29, 2016, 1:53:03 AM12/29/16
to jenkinsc...@googlegroups.com

Aleksejs Cibinogins -
please see the comment above for the code snippet in context:
I refer to the container step as follows, where each container (step) needs to be bound to the cloud. the full snippet is posted above. the NPE is very likely because the containerstep (https://github.com/jenkinsci/kubernetes-plugin/blob/master/src/main/java/org/csanchez/jenkins/plugins/kubernetes/pipeline/ContainerStepExecution.java#L51) the KubernetesCloud object is initialized from the ContainerStep's cloud name. Please let me know if this is unclear and i will add more.

container(name:'golang', cloud:'minikube')

{ stage 'Build a Go project' sh """ mkdir -p /go/src/github.com/golang/ ln -s `pwd` /go/src/github.com/golang/example export GOPATH=/go cd /go/src/github.com/golang/example/hello && go build """ }

aleksejs.cibinogins@gmail.com (JIRA)

unread,
Dec 29, 2016, 2:02:01 AM12/29/16
to jenkinsc...@googlegroups.com
Reply all
Reply to author
Forward
0 new messages