From what I see, the issue happens because Jenkins tries to unexport ProxyInputStream objects from two sides. My repro steps: 1) Create a test slave with 20 executors 2) Run the Pipeline script below 3) Interrupt build once it starts performing short stashes
// The map we'll store the parallel steps in before executing hem.
def stepsForParallel = [:]
stage "Prepare stashed file"
node("test") {
sh "cp /Users/nenashev/Documents/jenkins/demo/jenkins-2.7.1/jenkins.war ."
for (int i=0; i<10; i++) {
sh "cp jenkins.war jenkins_${i}.war"
}
writeFile file: 'foo.txt', text: 'Hello'
stash includes: 'foo.txt', name: 'foo_stashed'
stash includes: '*', name: 'jenkins_stashed'
}
stage "Parallel unstash"
for (int i = 0; i < 19; i++) {
def step = {
node("test") {
ws("ws_$i") {
unstash includes: '*', name: 'jenkins_stashed'
}
}
}
stepsForParallel["unstash_$i"] = step
}
stepsForParallel["fast_unstash"] = {
node("test") {
ws("ws_fast") {
for (int i=0; i<2000; i++) {
unstash includes: '*', name: 'foo_stashed'
}
}
}
}
parallel stepsForParallel
Side 1. Remote call releases exports at the end of the call. E.g. if it gets interrupted
Released at Thu Aug 04 13:12:16 MSK 2016
at hudson.remoting.ExportTable$Entry.release(ExportTable.java:131)
at hudson.remoting.ExportTable$ExportList.release(ExportTable.java:247)
at hudson.remoting.UserRequest.releaseExports(UserRequest.java:224)
at hudson.remoting.UserRequest.releaseExports(UserRequest.java:220)
at hudson.remoting.Channel.call(Channel.java:798)
at hudson.FilePath.act(FilePath.java:1007)
at hudson.FilePath.act(FilePath.java:996)
at hudson.FilePath.untarFrom(FilePath.java:728)
at org.jenkinsci.plugins.workflow.flow.StashManager.unstash(StashManager.java:114)
Side 2. When ProxyInputStream gets closed, it sends asynchronous EOF command to the remote side. This EOF also tries to release the allocated object (FileInputStream in my case). By the time the request gets to the remote executor, UserRequest.releaseExports() unexports the object. ProxyInputStream is nested into RemoteInputStream, which is actually being used for many cases like stdin/stdout propagation in remote calls. So technically any remote call operation may overlap with UserRequest termination handler and cause an overlap in such case. So the solution would be to either make ProxyInputStream EOF command synchronous or to make it tolerant against race conditions. Reproducing the issue in tests |