Hi,
We are trying to use the Kubernetes Plugin to run tests in a different container - but *almost* every time we try and execute a command in the second container, the build fails immediately. I say "almost" because the build always succeeds after restarting the Jenkins master (or possibly after it's been idle for a long time).
Here's the Jenkinsfile we're using to test:
def label = "k8s-test-${UUID.randomUUID().toString()}"
podTemplate(
cloud: 'kubernetes-test.k8s.local',
namespace: 'mynamespace',
label: label,
yaml: """
apiVersion: v1
kind: Pod
spec:
containers:
- name: busybox
image: busybox
command: ['cat']
tty: true
"""
) {
node (label) {
stage ('Get Agent Info') {
sh "java -version"
}
stage ('Run tests') {
container('busybox') {
sh "echo foo"
sh "netstat -tln"
sh "df -h"
}
}
stage ('Grab Logs') {
containerLog('jnlp')
}
}
}
This fails at the "echo foo" step. The error reported by the pipeline is:
Task okhttp3.RealCall$AsyncCall@b5277fa rejected from java.util.concurrent.ThreadPoolExecutor@6c383ec7[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 3]
What we know so far:
* Both containers are successfully deployed to the same pod, as expected.
* If we don't try and run the shell commands on the 'busybox' container, the build completes successfully.
* If Jenkins is restarted, the build passes. But subsequent builds fail.
* The Agent logs contain similar exceptions whether the builds pass or fail, so it's hard to pinpoint a cause.
The ThreadPool exception may imply that *something* has run out of Executors, but it's not clear what, or how to rectify the situation.
Has anyone seen anything like this? Any ideas how to get to the bottom of it? As per the plugin's README, I've created Jenkins log recorders for org.csanchez.jenkins.plugins.kubernetes and okhttp3, but there are no obvious problems reported in the logs.
Software Versions:
Jenkins: v2.150.1
Kubernetes Plugin: 1.13.7
Kubernetes: 1.10.11 (same behaviour observed on 1.8.7)
All suggestions appreciated!
Thanks,
Simon.