[workflow] Generating parallel branches causes job to hang

[Andres Rodriguez]

Hi Everyone,

First of all, I want to say that I'm really enjoying the workflow plugin. Thanks for all the effort in bringing this feature to production. Specially loading the DSL file from a git repository allows us to auto-test our changes to a job through the same mechanism that we test changes to the code (using Gerrit).

I'm currently encountering some weird behavior when generating a list of tests parallel branches dynamically. I've boiled it down to the following small test case that you can paste into the workflow script box:

@NonCPS
def generateTestBranches (testList) {
    def branches = [:]
    for (testName in testList.split()) {
        branches[testName] = {
            echo testName
            sleep 20
        }
    }
    return branches
}

def runTests (testList) {
    def testBranches = generateTestBranches(testList)
    parallel testBranches
}

runTests "test1 test2"

The output of this job (it can't be aborted):

Running: Execute sub-workflows in parallel : Start
[test1] Running: Parallel branch: test1
Aborted by anonymous
Aborted by anonymous
Aborted by anonymous

I've also tried replacing the "parallel testBranches" with a simple loop to execute the closures in the map manually (my understanding is that map.each is broken atm so I used a loop instead):

    for ( test in testBranches) {
         echo "executing ${test.key}"
         test.value()
    }

And the output is as follows (note that it prints test2 instead of test1 from inside the closure):

Started by user anonymous
Running: Print Message
executing test1
Running: Print Message
test2
Running: Print Message
executing test2
Running: Print Message
test2
Running: End of Workflow
Finished: SUCCESS

I'm not a Groovy expert (I just started learning for the workflow plugin), so I might be doing something dumb here. I'm trying to read the groovy docs to figure out if the way I'm generating the closure is an issue. But overall the state the system gets into seems to be pretty bad (can't abort the job). So I wanted to send out an email here just in case this is a problem with the Jenkins groovy core.

Thanks again for the constant improvements to the workflow plugin.

Regards,
Andres

[Michael Štědrý]

Hi Andres,

I found myself needing the same functionality, that is to run a few parallel branches that initiate work elsewhere and wait for notification. After all the branches finish I would continue the workflow. It fails with the same error for me and the build could not be aborted, except after Jenkins restart when it tried to resume.

This problem would be best brought to the Jenkins Developers (jenkin...@googlegroups.com) as it is frequented by Cloudbees employees such as Jesse Glick who is most helpful in his answers.

What I can help you with is the closure problem where your closure references the testName variable and ends up with the last value. You could write the assignment like this (a closure with a parameter which is local to the closure):

@NonCPS
def generateTestBranches (testList) {
    def branches = [:]
    for (testName in testList.split()) {

        branches[testName] = { name ->

            { it ->

                echo name

                sleep 20

            }

        }.call(testName)

    }
    return branches
}

or define it beforehand and call in in the assignment

@NonCPS
def generateTestBranches (testList) {

    def createBranch = { name ->

        { it ->

            echo name
            sleep 20

        }

    }

    def branches = [:]
    for (testName in testList.split()) {

        branches[testName] = createBranch(testName)

    }

    return branches

}

[Michael Štědrý]

Hi Andres,

I've got one more interesting thing for you. I managed to get your sample working even though I had to remove the functions:

def testList = "test1 test2".split()

def branches = [:]

for (int i = 0; i < testList.length; i++) {

    def testName = testList[i];

    branches[testName] = {

        echo testName

        sleep 20

    }

}

parallel branches

What I found out is that I couldn't use the nice for loop because it fails with:

java.io.NotSerializableException: java.util.AbstractList$Itr

...

Caused by: an exception which occurred:
	in field itr
	in field target
	in field continue_
	in field parent
	in field capture
	in field def
	in field closures
	in object org.jenkinsci.plugins.workflow.cps.CpsThreadGroup@8b36066

And when I put the creation of the branches in a function it started to behave strangely.

On Sunday, September 27, 2015 at 10:25:47 PM UTC+2, Andres Rodriguez wrote:

[Brian Ray]

See this earlier thread for the Iterator serialization issue:

https://groups.google.com/d/msg/jenkinsci-users/LGRv7Jq60YI/ZN-68zuw2loJ

[Andres Rodriguez]

Thanks for the feedback everyone, it was quite helpful.

Regards,

Andres