| hi, when running multiple concurrent builds of specific job (with declarative definition of the build pods jenkins seem to freak out and deny the connection from build slaves to the master:
(from the jnlp container):
client name: brandhub-container-builder-834z6-3bmbk
at org.jenkinsci.remoting.util.SettableFuture.get(SettableFuture.java:223)
at hudson.remoting.Engine.innerRun(Engine.java:614)
at hudson.remoting.Engine.run(Engine.java:474)
Caused by: org.jenkinsci.remoting.protocol.impl.ConnectionRefusalException: Unknown client name: brandhub-container-builder-834z6-3bmbk
at org.jenkinsci.remoting.protocol.impl.ConnectionHeadersFilterLayer.newAbortCause(ConnectionHeadersFilterLayer.java:378)
at org.jenkinsci.remoting.protocol.impl.ConnectionHeadersFilterLayer.onRecvClosed(ConnectionHeadersFilterLayer.java:433)
at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecvClosed(ProtocolStack.java:816)
at org.jenkinsci.remoting.protocol.FilterLayer.onRecvClosed(FilterLayer.java:287)
at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.onRecvClosed(SSLEngineFilterLayer.java:172)
at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecvClosed(ProtocolStack.java:816)
at org.jenkinsci.remoting.protocol.NetworkLayer.onRecvClosed(NetworkLayer.java:154)
at org.jenkinsci.remoting.protocol.impl.BIONetworkLayer.access$1500(BIONetworkLayer.java:48)
at org.jenkinsci.remoting.protocol.impl.BIONetworkLayer$Reader.run(BIONetworkLayer.java:247)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:93)
at java.lang.Thread.run(Thread.java:748)
Suppressed: java.nio.channels.ClosedChannelException
... 7 more
when running only one build "it works".
the multiple concurrent builds may get finished after some time.
one need to be patient and on the lucky side. i searched the internet (and this jira) already and for quite some time.
could not find help yet. ACHTUNG: when putting some other pod templates inline/declarative i get the same error. but if i define them in the configuration ui part in jenkins the builds are more stable and i got more reliable and satisfying results. tried it also with this builds but that trick did not help.
... here are some logs and content of related files, also some version: k8s: 1.11.x
jenkins: jenkins/jenkins:2.84 (10Gi Memory, 2000m cpu)
kubernetes:1.16.4 THE JENKINSFILE:
pipeline {
agent {
kubernetes {
label 'brandhub-container-builder'
yamlFile 'api/.build.yaml'
}
}
stages {
stage('build and push container') {
steps {
container(name: 'kaniko', shell: '/busybox/sh') {
withEnv(['PATH+EXTRA=/busybox:/kaniko']) {
sh '''#!/busybox/sh
/kaniko/executor -f `pwd`/api/Dockerfile -c `pwd`/api --verbosity=warn \
--destination=reg.cicd.intra.xxxxxxx.xxxxxx.io/ow-apps-mbbrandhub/cicd-php-node-x:latest
'''
}
}
}
}
}
}
RELATED BUILD YAML:
apiVersion: v1
kind: Pod
metadata:
name: kaniko
spec:
containers:
- name: kaniko
image: gcr.io/kaniko-project/executor:debug
imagePullPolicy: Always
command:
- /busybox/cat
tty: true
volumeMounts:
- name: jenkins-docker-cfg
mountPath: /kaniko/.docker
volumes:
- name: jenkins-docker-cfg
secret:
secretName: regcreds
items:
- key: config.json
path: config.json
ALL THESE PODS ARE FAILING:
NAME READY STATUS RESTARTS AGE
brandhub-container-builder-nlvvh-26hcb 1/2 Error 0 29s
brandhub-container-builder-nlvvh-6wqbn 1/2 Error 0 1m
brandhub-container-builder-nlvvh-703pq 1/2 Error 0 29s
brandhub-container-builder-nlvvh-d7jxd 1/2 Error 0 1m
brandhub-container-builder-nlvvh-gn1pp 1/2 Error 0 49s
brandhub-container-builder-nlvvh-s383r 1/2 Error 0 1m
brandhub-container-builder-nlvvh-tn9nk 1/2 Error 0 49s
brandhub-container-builder-nlvvh-v5tck 1/2 Error 0 29s
brandhub-container-builder-nlvvh-w9939 1/2 Error 0 49s
efs-provisioner-75dcb94b57-s4smt 1/1 Running 0 14h
jenkins-non-prod-6555477944-mfj7t 0/1 Pending 0 15s
nodejs-8-3fmm3 4/4 Running 0 44m
nodejs-8-g2cpq 3/4 Running 0 26m
pcf-19574 1/1 Running 0 11m
pcf-1fxx7 1/1 Running 0 1d
pcf-4q0n3 1/1 Running 0 11m
pcf-8k17d 1/1 Running 0 4h
pcf-gr87l 1/1 Running 0 52m
pcf-xxtzw 1/1 Running 0 11m
remove-faulty-pods-1562694000-dqfzk 0/1 Completed 0 1m
GET LOGS:
$ k logs -f brandhub-container-builder-nlvvh-g9m2h -c jnlp
JNLP POD LOGS:
Warning: JnlpProtocol3 is disabled by default, use JNLP_PROTOCOL_OPTS to alter the behavior
Jul 09, 2019 5:37:59 PM hudson.remoting.jnlp.Main createEngine
INFO: Setting up agent: brandhub-container-builder-nlvvh-g9m2h
Jul 09, 2019 5:37:59 PM hudson.remoting.jnlp.Main$CuiListener <init>
INFO: Jenkins agent is running in headless mode.
Jul 09, 2019 5:38:00 PM hudson.remoting.Engine startEngine
INFO: Using Remoting version: 3.29
Jul 09, 2019 5:38:00 PM hudson.remoting.Engine startEngine
WARNING: No Working Directory. Using the legacy JAR Cache location: /home/jenkins/.jenkins/cache/jars
Jul 09, 2019 5:38:00 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Locating server among [http://jenkins-non-prod:8080/]
Jul 09, 2019 5:38:00 PM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver resolve
INFO: Remoting server accepts the following protocols: [JNLP4-connect, Ping, JNLP3-connect]
Jul 09, 2019 5:38:00 PM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver resolve
INFO: Remoting TCP connection tunneling is enabled. Skipping the TCP Agent Listener Port availability check
Jul 09, 2019 5:38:00 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Agent discovery successful
Agent address: jenkins-non-prod-agent
Agent port: 50004
Identity: 40:0f:dc:fe:7d:f1:f8:51:94:83:cc:2e:3e:c7:f5:7f
Jul 09, 2019 5:38:00 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Handshaking
Jul 09, 2019 5:38:00 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Connecting to jenkins-non-prod-agent:50004
Jul 09, 2019 5:38:00 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Trying protocol: JNLP4-connect
Jul 09, 2019 5:38:00 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Remote identity confirmed: 40:0f:dc:fe:7d:f1:f8:51:94:83:cc:2e:3e:c7:f5:7f
Jul 09, 2019 5:38:00 PM org.jenkinsci.remoting.protocol.impl.ConnectionHeadersFilterLayer onRecv
INFO: [JNLP4-connect connection to jenkins-non-prod-agent/100.65.158.111:50004] Local headers refused by remote: Unknown client name: brandhub-container-builder-nlvvh-g9m2h
Jul 09, 2019 5:38:00 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Protocol JNLP4-connect encountered an unexpected exception
java.util.concurrent.ExecutionException: org.jenkinsci.remoting.protocol.impl.ConnectionRefusalException: Unknown client name: brandhub-container-builder-nlvvh-g9m2h
at org.jenkinsci.remoting.util.SettableFuture.get(SettableFuture.java:223)
at hudson.remoting.Engine.innerRun(Engine.java:614)
at hudson.remoting.Engine.run(Engine.java:474)
Caused by: org.jenkinsci.remoting.protocol.impl.ConnectionRefusalException: Unknown client name: brandhub-container-builder-nlvvh-g9m2h
at org.jenkinsci.remoting.protocol.impl.ConnectionHeadersFilterLayer.newAbortCause(ConnectionHeadersFilterLayer.java:378)
at org.jenkinsci.remoting.protocol.impl.ConnectionHeadersFilterLayer.onRecvClosed(ConnectionHeadersFilterLayer.java:433)
at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecvClosed(ProtocolStack.java:816)
at org.jenkinsci.remoting.protocol.FilterLayer.onRecvClosed(FilterLayer.java:287)
at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.onRecvClosed(SSLEngineFilterLayer.java:172)
at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecvClosed(ProtocolStack.java:816)
at org.jenkinsci.remoting.protocol.NetworkLayer.onRecvClosed(NetworkLayer.java:154)
at org.jenkinsci.remoting.protocol.impl.BIONetworkLayer.access$1500(BIONetworkLayer.java:48)
at org.jenkinsci.remoting.protocol.impl.BIONetworkLayer$Reader.run(BIONetworkLayer.java:247)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:93)
at java.lang.Thread.run(Thread.java:748)
Suppressed: java.nio.channels.ClosedChannelException
... 7 more
Jul 09, 2019 5:38:00 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Connecting to jenkins-non-prod-agent:50004
Jul 09, 2019 5:38:00 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Server reports protocol JNLP4-plaintext not supported, skipping
Jul 09, 2019 5:38:00 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Protocol JNLP3-connect is not enabled, skipping
Jul 09, 2019 5:38:00 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Server reports protocol JNLP2-connect not supported, skipping
Jul 09, 2019 5:38:00 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Server reports protocol JNLP-connect not supported, skipping
Jul 09, 2019 5:38:00 PM hudson.remoting.jnlp.Main$CuiListener error
SEVERE: The server rejected the connection: None of the protocols were accepted
java.lang.Exception: The server rejected the connection: None of the protocols were accepted
at hudson.remoting.Engine.onConnectionRejected(Engine.java:682)
at hudson.remoting.Engine.innerRun(Engine.java:639)
at hudson.remoting.Engine.run(Engine.java:474)
does anyone has an idea what's going on here?
or what could be tweaked to get it more stable? any help is pretty much appreciated. |
