| My theory about the issue: Jenkins has `Text file busy` error coming from `durable` which is used by `pipeline nodes and processes` plugin.
This issue is easily reproducible by running any command e.g. `sh "echo 'Hello'"` in any jenkins job many times in parallel.
This hang is caused when running a docker container using the docker plugin, but when we run `docker` without using `sh` it never hangs but that `Text file busy` error still appears because we run a lot of jobs at the same time. The container in this case will stay running forever To reproduce
error, please follow these steps: 1. Run Jenkins locally `docker run -p 8080:8080 -p 50000:50000 jenkins/jenkins:lts` ver. 2.89.4
2. install just the recommended plugins on it
3. create new pipeline projects called `hello` and `hello2`
4. put this code in ‘hello’ :
node{
sh "echo 'hello'"
}
and this in `hello2`:
node{
sh "echo 'hello'"
sleep(2)
}
5. Create a new pipeline project called `runner` and put this inside:
COUNTER = 0
node{
def jobs = [:]
// add 24 instances of the same test to run them later in parallel
24.times {
jobs[('runner' + COUNTER++)] = {triggerProject('hello')()}
}
// add 24 instances of the same test to run them later in parallel
24.times {
jobs[('runner' + COUNTER++)] = {triggerProject('hello2')()}
}
// run them all 20 times in parallel
20.times {
parallel jobs
}
}
def triggerProject(jobName) {
return {
try{
build job: jobName, parameters: [string(name: 'VALUE', value: String.valueOf(COUNTER++))]
} catch (ex){
println ex
}
}
}
6. Go to your jenkins configurations and change the executors to `50`
7. try to run the runner once and if you got some sandbox exception, go to the in-procces Script Approval in the Manage Jenkins page and click approve for all commands
8. run runner again and you will see that some of `hello` and `hello2` has `text file busy` error The logs you get afterwards:
Running in Durability level: MAX_SURVIVABILITY
[Pipeline] node
Running on Jenkins in /var/jenkins_home/workspace/hello@6
[Pipeline] {
[Pipeline] sh
[hello@6] Running shell script
sh: 1: /var/jenkins_home/workspace/hello@6@tmp/durable-a771d7dd/script.sh: Text file busy
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
ERROR: script returned exit code 2
Finished: FAILURE
|
