| We've got hit by this one as well after upgrading core and a bunch of plugins, most notably we upgraded:
Jenkins core: 2.176.2 => 2.190.1
pipeline-build-step:2.7 => 2.9
workflow-cps:2.71 => 2.74
workflow-durable-task-step:2.32 => 2.34
workflow-job:2.33 => 2.35
credentials-binding-plugin was not updated and we are running 1.18 (for some reason we have any explicit inclusion of this plugin, instead it is included as dependency to other plugins). I've been able to reproduce it with the following code:
node {
for (int i = 0; i < 10000; i++) {
withCredentials([]) {
echo "At ${i}"
}
}
}
and it doesn't seem that this is bug is restricted to only credentials step, e.g. the following also fails for us:
node {
for (int i = 0; i < 10000; i++) {
wrap([$class: 'BuildUser']) {
echo "At ${i}"
}
}
}
Rather it seems like there is some sort of concurrency problem related to GeneralNonBlockingStepExecution which is used by both steps above. So I would suggest changing component for this issue to workflow-cps (and try to get attention from one of the plugin maintainers). During debugging of this problem I also managed to get a stacktrace during execution of the following:
node {
for (int i = 0; i < 10000; i++) {
withEnv(['A=b']) {
withCredentials([]) {
sh "echo hello ${i}"
}
}
}
}
Console output with stacktrace:
...
[Pipeline] {
[Pipeline] sh
+ echo hello 34
hello 34
[Pipeline] }
[Pipeline] // withCredentials
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] withEnv
[Pipeline] {
[Pipeline] withCredentials
[Pipeline] End of Pipeline
java.lang.ArrayIndexOutOfBoundsException: 0
at org.jenkinsci.plugins.workflow.cps.DSL$ThreadTaskImpl.invokeBody(DSL.java:647)
at org.jenkinsci.plugins.workflow.cps.DSL$ThreadTaskImpl.eval(DSL.java:615)
at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:196)
at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:370)
at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$200(CpsThreadGroup.java:93)
at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:282)
at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:270)
at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:66)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:131)
at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Finished: FAILURE
This looks very related as it occurred during start of withCredentials step, and looking at the code in DSL.java it is obviously some sort of race-condition problem. |
