[JIRA] (JENKINS-60801) Variable changes are shared between matrix (parallel) stages.

14 views
Skip to first unread message

jakub.krysl@gmail.com (JIRA)

unread,
Jan 16, 2020, 1:06:07 PM1/16/20
to jenkinsc...@googlegroups.com
Jakub Krysl created an issue
 
Jenkins / Bug JENKINS-60801
Variable changes are shared between matrix (parallel) stages.
Issue Type: Bug Bug
Assignee: Unassigned
Components: pipeline
Created: 2020-01-16 18:05
Environment: Jenkins ver. 2.190.2
Pipeline: API 2.83
Pipeline: Basic Steps 2.19
Pipeline: Declarative 1.5.0
Labels: matrix pipeline
Priority: Major Major
Reporter: Jakub Krysl

New matrix step provides ability to run the same code in parallel cells. But if that code assigns ANY variable (not just env), it gets overwritten later by the next cell. This is huge deal, it basically prevents from assigning any variable withing this kind of pipeline which makes it nearly useless.

The condition is to set variable, have a piece of code that takes some time to execute and use the already set variable after. This can be simulated with sleep().

I boiled down my code to easily reproduce it, here is the pipeline:

pipeline {
    parameters {
        booleanParam(name: 'RHEL5', defaultValue: true)
        booleanParam(name: 'RHEL6', defaultValue: true)
        booleanParam(name: 'RHEL7', defaultValue: true)
    }
    stages {
        stage('Matrix') {
            matrix {
                when {
                    anyOf {
                        expression { params.RHEL5 && env.RHEL == '5' }
                        expression { params.RHEL6 && env.RHEL == '6' }
                        expression { params.RHEL7 && env.RHEL == '7' }
                    }
                }
                axes {
                    axis {
                        name 'RHEL'
                        values '5', '6', '7'
                    }
                }
                stages {
                    stage('Build single'){
                        steps {
                            script {
                                echo "RHEL version: " + env.RHEL
                                myVar = "something.something.RHEL-${env.RHEL}"
                                echo "myVar: " + myVar
                                sleep(10)
                                echo "AFTER RHEL version: " + env.RHEL 
                                echo "AFTER myVar: " + myVar
                            }
                        }
                    }
                }
            }
        }
    }
}

The result of this code:

 

RHEL version: 5
myVar: something.something.RHEL-5
Sleeping for 10 sec
RHEL version: 6
myVar: something.something.RHEL-6
Sleeping for 10 sec
RHEL version: 7
myVar: something.something.RHEL-7
Sleeping for 10 sec
No need to sleep any longer
RHEL version: 5
myVar: something.something.RHEL-7
No need to sleep any longer
RHEL version: 6
myVar: something.something.RHEL-7
No need to sleep any longer
RHEL version: 7
myVar: something.something.RHEL-7

As you can see, myVar gets overwritten by the last run cell even for the other cells.

 

 

Add Comment Add Comment
 
This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f)
Atlassian logo

jakub.krysl@gmail.com (JIRA)

unread,
Jan 16, 2020, 1:07:16 PM1/16/20
to jenkinsc...@googlegroups.com
Jakub Krysl updated an issue
Change By: Jakub Krysl
New matrix step provides ability to run the same code in parallel cells. But if that code assigns ANY variable (not just env), it gets overwritten later by the next cell. This is huge deal, it basically prevents from assigning any variable withing in this kind of pipeline which makes it nearly useless.


The condition is to set variable, have a piece of code that takes some time to execute and use the already set variable after. This can be simulated with sleep().

I boiled down my code to easily reproduce it, here is the pipeline:
{code:java}
{code}

The result of this code:

 
{code:java}

RHEL version: 5
myVar: something.something.RHEL-5
Sleeping for 10 sec
RHEL version: 6
myVar: something.something.RHEL-6
Sleeping for 10 sec
RHEL version: 7
myVar: something.something.RHEL-7
Sleeping for 10 sec
No need to sleep any longer
RHEL version: 5
myVar: something.something.RHEL-7
No need to sleep any longer
RHEL version: 6
myVar: something.something.RHEL-7
No need to sleep any longer
RHEL version: 7
myVar: something.something.RHEL-7{code}

As you can see, myVar gets overwritten by the last run cell even for the other cells.

 

 

jakub.krysl@gmail.com (JIRA)

unread,
Jan 17, 2020, 5:52:02 AM1/17/20
to jenkinsc...@googlegroups.com

jakub.krysl@gmail.com (JIRA)

unread,
Jan 17, 2020, 5:52:03 AM1/17/20
to jenkinsc...@googlegroups.com

jakub.krysl@gmail.com (JIRA)

unread,
Jan 17, 2020, 5:53:06 AM1/17/20
to jenkinsc...@googlegroups.com
Jakub Krysl edited a comment on Bug JENKINS-60801
 
Re: Variable changes are shared between matrix (parallel) stages.
Here is a script boiled down to simplest reproduces reproducer for both Matrix and Parallel pipeline:
{code:java}
def getTasks() {
    def tasks = [:]
    //['5', '6', '7'].each { RHEL ->
    versions = ['5', '6', '7']
    for (int i=0; i<versions.size(); i++) {
        def RHEL = versions[i]
        tasks["${RHEL}"] = {
            node('slave') {
                echo "in PARALLEL"
                TEST = "in_variable_" + RHEL
                echo "RHEL: " + RHEL + ", TEST: " + TEST
                sleep(5)
                echo "RHEL: " + RHEL + ", TEST: " + TEST
            }
        }
    }
    return tasks
}

pipeline {
    agent { label 'slave' }
    stages {
        stage('Build Matrix') {
            matrix {

                axes {
                    axis {
                     name 'RHEL'
                     values '5', '6', '7'
                    }
                }
                stages('RHEL') {

                    stage('Build single'){
                     steps {
                     script {
                     echo "in MATRIX"
                     TEST = "in_variable_" + RHEL
                     echo "RHEL: " + RHEL + ", TEST: " + TEST
                     sleep(5)
                     echo "RHEL: " + RHEL + ", TEST: " + TEST
                     }
                     }
                    }
                }
            }
        }
        stage('Parallel build') {
            steps {
                script {
                    parallel getTasks()
                }
            }
        }
    }
}

{code}
Output:
{code:java}
in MATRIX
RHEL: 5, TEST: in_variable_5
Sleeping for 5 sec
in MATRIX
RHEL: 6, TEST: in_variable_6
Sleeping for 5 sec
in MATRIX
RHEL: 7, TEST: in_variable_7
Sleeping for 5 sec
RHEL: 5, TEST: in_variable_7
RHEL: 6, TEST: in_variable_7
RHEL: 7, TEST: in_variable_7

Running on jenkins-slave2 in /home/jenkins/workspace/TEST_PIPELINE
Running on jenkins-slave2 in /home/jenkins/workspace/TEST_PIPELINE@2
Running on jenkins-slave1 in /home/jenkins/workspace/TEST_PIPELINE@2
in PARALLEL
RHEL: 5, TEST: in_variable_5
Sleeping for 5 sec
in PARALLEL
RHEL: 6, TEST: in_variable_6
Sleeping for 5 sec
in PARALLEL
RHEL: 7, TEST: in_variable_7
Sleeping for 5 sec
RHEL: 5, TEST: in_variable_7
RHEL: 6, TEST: in_variable_7
RHEL: 7, TEST: in_variable_7
Finished: SUCCESS
{code}
Setting this issue to blocker as it prevents me from using any cell/branch specific variables.

jakub.krysl@gmail.com (JIRA)

unread,
Jan 17, 2020, 5:53:06 AM1/17/20
to jenkinsc...@googlegroups.com

Here is a script boiled down to simplest reproduces for both Matrix and Parallel pipeline:

Output:

Setting this issue to blocker as it prevents me from using any cell/branch specific variables.

prakash.gandhi.be@outlook.com (JIRA)

unread,
Apr 10, 2020, 2:53:03 AM4/10/20
to jenkinsc...@googlegroups.com
prakash raj assigned an issue to prakash raj
 
Change By: prakash raj
Assignee: prakash raj
This message was sent by Atlassian Jira (v7.13.12#713012-sha1:6e07c38)
Atlassian logo

prakash.gandhi.be@outlook.com (JIRA)

unread,
Apr 10, 2020, 2:54:02 AM4/10/20
to jenkinsc...@googlegroups.com
prakash raj assigned an issue to Unassigned

corey.woodcox@gmail.com (JIRA)

unread,
Apr 20, 2020, 4:24:02 PM4/20/20
to jenkinsc...@googlegroups.com
Corey Woodcox commented on Bug JENKINS-60801
 
Re: Variable changes are shared between matrix (parallel) stages.

We also see this in a similar build script. We have a job that loops through a list of versions and fires off several docker image builds in parallel and at the end of the script the image name variables have overwritten each other.

I can try and grab more information if you think it would help troubleshoot.

Reply all
Reply to author
Forward
0 new messages