K8s jenkins dynamic agent fails & shows - tcpSlaveAgentListener error
66 views
Skip to first unread message
Mk
Aug 28, 2020, 12:46:45 AM8/28/20
to Jenkins Users
Dear Team,
I have Jenkins Master & K8s-Master running on same server(CentOS-8), Configured Jenkins Kubernetes Plugin version - 1.26.4 But while running pipeline job i always getting an error, Below is K8s cluster jenkins agent pod log.
[root@K8s-Master /]# kubectl logs -f pipeline-test-33-sj6tl-r0clh-g559d -c jnlp
Aug 08, 2020 8:37:21 AM hudson.remoting.jnlp.Main createEngine
INFO: Setting up agent: pipeline-test-33-sj6tl-r0clh-g559d
Aug 08, 2020 8:37:21 AM hudson.remoting.jnlp.Main$CuiListener <init>
INFO: Jenkins agent is running in headless mode.
Aug 08, 2020 8:37:21 AM hudson.remoting.Engine startEngine
INFO: Using Remoting version: 4.3
Aug 08, 2020 8:37:21 AM org.jenkinsci.remoting.engine.WorkDirManager initializeWorkDir
INFO: Using /home/jenkins/agent/remoting as a remoting work directory
Aug 08, 2020 8:37:21 AM org.jenkinsci.remoting.engine.WorkDirManager setupLogging
INFO: Both error and output logs will be printed to /home/jenkins/agent/remoting
Aug 08, 2020 8:37:21 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Locating server among [http://jenkins-serverjenkins/]
Aug 08, 2020 8:37:41 AM hudson.remoting.jnlp.Main$CuiListener error
SEVERE: Failed to connect to http://jenkins-server/jenkins/tcpSlaveAgentListener/: jenkins-server
java.io.IOException: Failed to connect to http://jenkins-serverjenkins/tcpSlaveAgentListener/: jenkins-server
at org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver.resolve(JnlpAgentEndpointResolver.java:217)
at hudson.remoting.Engine.innerRun(Engine.java:693)
at hudson.remoting.Engine.run(Engine.java:518)
Caused by: java.net.UnknownHostException: jenkins-server
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:184)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:607)
at sun.net.NetworkClient.doConnect(NetworkClient.java:175)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:463)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:558)
at sun.net.www.http.HttpClient.<init>(HttpClient.java:242)
at sun.net.www.http.HttpClient.New(HttpClient.java:339)
at sun.net.www.http.HttpClient.New(HttpClient.java:357)
at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1226)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1162)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1056)
at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:990)
at org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver.resolve(JnlpAgentEndpointResolver.java:214)
... 2 more
Below settings configuration already enabled glocal security section.
Manage Jenkins --> Configure Global Security --> Agents Random [Enabled]
I am successfully able to communicate from my Jenkins to the K8s master cluster(Verified in Jenkins Cloud section).
Even in K8s master server all the namespace pods are running. Flannel CNI installed, Not sure what is causing problem with agent provisioning through Jenkins.
Even in K8s master server all the namespace pods are running. Flannel CNI installed, Not sure what is causing problem with agent provisioning through Jenkins.
Would appreciate any suggestions.
Mk
Aug 28, 2020, 2:29:36 AM8/28/20
to Jenkins Users
Further tried by adding below entries in /etc/hosts file on both Jenkins/K8s Master and K8s-Worker node system as well. But still result is same.
# cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
75.76.77.5 jenkins-server jenkins-server.company.domain.com
75.76.77.6 k8s-node-1 k8s-node-1.company.domain.com
Excuted below on K8s-Worker node.
# curl -I http://jenkins-server/jenkins/tcpSlaveAgentListener/
HTTP/1.1 200
Server: nginx/1.14.1
Date: Fri, 28 Aug 2020 06:13:34 GMT
Content-Type: text/plain;charset=UTF-8
Connection: keep-alive
Cache-Control: private
Expires: Thu, 01 Jan 1970 00:00:00 GMT
X-Content-Type-Options: nosniff
X-Hudson-JNLP-Port: 40021
X-Jenkins-JNLP-Port: 40021
X-Instance-Identity: MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAnkgz8Av2x8R9R2KZDzWm1K11O01r7VDikW48rCNQlgw/pUeNSPJu9pv7kH884tOE65GkMepNdtJcOFQFtY1qZ0sr5y4GF5TOc7+U/TqfwULt60r7OQlKcrsQx/jJkF0xLjR+xaJ64WKnbsl0AiZhd8/ynk02UxFXKcgwkEP2PGpGyQ1ps5t/yj6ueFiPAHX2ssK8aI7ynVbf3YyVrtFOlqhnTy11mJFoLAZnpjYRCJsrX5z/xciVq5c2XmEikLzMpjFl0YBAsDo7JL4eBUwiBr64HPcSKrsBBB9oPE4oI6GkYUCAni8uOLfzoNr9B1eImaETYSdVPdSKW/ez/OeHjQIDAQAB
X-Jenkins-Agent-Protocols: JNLP4-connect, Ping
X-Remoting-Minimum-Version: 3.14# curl http://jenkins-server:40021/
Jenkins-Agent-Protocols: JNLP4-connect, Ping
Jenkins-Version: 2.235.3
Jenkins-Session: 4455fd45
Client: 75.76.77.6
Server: 75.76.77.5
Remoting-Minimum-Version: 3.14
Richard Bywater
Aug 29, 2020, 5:06:06 AM8/29/20
to jenkins...@googlegroups.com
What is the K8s-Worker node that you are referring to? You didn't mention that in the first email so I'm not sure where that fits in. Also which entries did you actually put into the /etc/hosts file as you don't say what it looked like before.
I'm fairly new to K8s myself but my main question would be whether or not you have a service defined in the namespace fronting the Jenkins master called 'jenkins-server'?
Richard.
--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/c155bb2a-d143-4825-9406-59f066e47385o%40googlegroups.com.
Mk
Aug 29, 2020, 9:30:09 AM8/29/20
to Jenkins Users
Hi Richard,
Thanks for the reply. Sorry i didn't give my environment details before. Please find my environment details below.
Server:-1
Jenkins Master & Kubernetes Cluster Master Server Environment (Both deployed in same physical server).
OS - Centos-8.2
Jenkins Version - Jenkins 2.235.3 (Deployed through Tomcat Apache) - Jenkins not on K8s cluster.
Jenkins Kubernetes Plugin version - 1.26.4
Jenkins URL - http://jenkins-server/jenkins/
kubectl version --short
Client Version: v1.18.8
Server Version: v1.18.8
Server:-2
Kubernetes Worker-Node Server Environment:-
OS - Centos-8.2
K8s-Worker Node I am referring my Kubernetes cluster node(Currently i have single K8s-Worker-Node) and my Jenkins master pod template doesn't have any namespace (it is just empty) in my Jenkins cloud configuration section GUI.
Earlier my /etc/hosts file in both server was like below. Addtionally i tried adding the following (jenkins-server.company.domain.com & k8s-node-1.company.domain.com)
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
75.76.77.6 k8s-node-1
Since from my K8s-Worker-Node server getting response through curl commands to Jenkins master. so i feel it looks there is no issue with Kubernetes Jenkins plugin confiugration. it would be mostly DNS resolution issue with Kubernetes cluster.
In my K8s Master server all namespaces pods are running. Also below is my K8s master & Worker node's
/etc/resolv.conf file. For K8s DNS resolving any other configuration required?
# cat /etc/resolv.conf
# Generated by NetworkManager
nameserver 61.62.63.41
nameserver 61.62.63.42What are configuration that I need to do to resolve this problem? Please help me. Thanks.
What is the K8s-Worker node that you are referring to? You didn't mention that in the first email so I'm not sure where that fits in. Also which entries did you actually put into the /etc/hosts file as you don't say what it looked like before.I'm fairly new to K8s myself but my main question would be whether or not you have a service defined in the namespace fronting the Jenkins master called 'jenkins-server'?Richard.
On Fri, 28 Aug 2020 at 18:29, Mk <moha...@gmail.com> wrote:
Further tried by adding below entries in /etc/hosts file on both Jenkins/K8s Master and K8s-Worker node system as well. But still result is same.# cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
75.76.77.5 jenkins-server jenkins-server.company.domain.com
75.76.77.6 k8s-node-1 k8s-node-1.company.domain.com
Executed below on K8s-Worker node.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkins...@googlegroups.com.
Mk
Aug 30, 2020, 5:12:37 AM8/30/20
to Jenkins Users
Team, It was an kubernetes DNS resolution issue. With the help of following link - https://kubernetes.io/docs/tasks/administer-cluster/dns-debugging-resolution/ created (dnsutils.yaml) pod and found that my K8s cluster pods was returning following error "connection timed out; no servers could be reached" for below command.
kubectl exec -i -t dnsutils -- nslookup kubernetes.default
So i have uninstalled and re-installed Kubernetes version - v1.19.0. Now everything working fine. Thanks.!!!
On Friday, August 28, 2020 at 10:16:45 AM UTC+5:30, Mk wrote:
Reply all
Reply to author
Forward
0 new messages