[JIRA] (JENKINS-53734) Declarative pipeline - Support for parallel stages inside sequential stages

121 views
Skip to first unread message

prettymama@centrum.cz (JIRA)

unread,
Sep 23, 2018, 8:16:02 AM9/23/18
to jenkinsc...@googlegroups.com
Martin Tee created an issue
 
Jenkins / Improvement JENKINS-53734
Declarative pipeline - Support for parallel stages inside sequential stages
Issue Type: Improvement Improvement
Assignee: Andrew Bayer
Components: pipeline-model-definition-plugin
Created: 2018-09-23 12:15
Environment: jenkins 2.138.1, pipeline-model-definition-plugin 1.3.2
Labels: plugin pipeline
Priority: Major Major
Reporter: Martin Tee

Yesterday, I tried using sequential stages. I have a pipeline consisting of 2 distinct parts, let's say JAVA PART and DOCKER PART, each with a different agent and several stages. Thus, I grouped them into two stages, assigned an agent and well, it worked as expected. Things went south when I wanted to use parallel stages inside JAVA PART:

pipeline {
    agent none
    
    stages {
        stage("JAVA PART") {
            agent A

            stages {
                stage("Java one") {
                    steps {
                        echo "Java one"
                    }
                }
                
                stage("Java two") {
                    parallel {
                        stage("Java parallel one") {
                            steps {
                                echo "Java parallel one"
                            }
                        }
                        
                        stage("Java parallel two") {
                            steps {
                                echo "Java parallel two"
                            }
                        }
                    }
                }
                
                stage("Java three") {
                    steps {
                        echo "Java three"
                    }
                }
            }
        }
        
        stage("DOCKER PART") {
            agent B
            
            steps {
                echo "Docker one"
            }
        }
    }
}

I ended up with 

Parallel stages or branches can only be included in a top-level stage.

I wonder what the rationale behind this restriction is, because when I switch from parallel to stages and make it serial, everything is fine.

pipeline {
    agent none
    
    stages {
        stage("JAVA PART") {
            agent A

            stages {
                stage("Java one") {
                    steps {
                        echo "Java one"
                    }
                }
                
                stage("Java two") {
                    stages {
                        stage("Java parallel one") {
                            steps {
                                echo "Java parallel one"
                            }
                        }
                        
                        stage("Java parallel two") {
                            steps {
                                echo "Java parallel two"
                            }
                        }
                    }
                }
                
                stage("Java three") {
                    steps {
                        echo "Java three"
                    }
                }
            }
        }
        
        stage("DOCKER PART") {
            agent B
            
            steps {
                echo "Docker one"
            }
        }
    }
}
Add Comment Add Comment
 
This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)

prettymama@centrum.cz (JIRA)

unread,
Sep 26, 2018, 5:55:06 AM9/26/18
to jenkinsc...@googlegroups.com
Martin Tee commented on Improvement JENKINS-53734
 
Re: Declarative pipeline - Support for parallel stages inside sequential stages

Found a workaround via scripted pipelines, but it is kinda ugly:

pipeline {
    agent none
    
    stages {
        stage("JAVA PART") {
            agent A

            stages {
                stage("Java one") {
                    steps {
                        echo "Java one"
                    }
                }
                
                stage("Java two"
) {
                    steps {
                        script {
                            def parallelStepsMap = [:]
                            parallelStepsMap["Java parallel one"] = {
                                echo "Java parallel one"
                            }
                            parallelStepsMap["Java parallel two"] = {
                                echo "Java parallel two"
                            }

                            parallel parallelStepsMap
                        }
                    }
                }
                
                stage("Java three") {
                    steps {
                        echo "Java three"
                    }
                }
            }
        }
        
        stage("DOCKER PART") {
            agent B
            
            steps {
                echo "Docker one"
            }
        }
    }
}
 
                                                            

I believe declarative pipelines should isolate me from having to do such things.

andrew.bayer@gmail.com (JIRA)

unread,
Oct 16, 2018, 10:13:02 AM10/16/18
to jenkinsc...@googlegroups.com
Andrew Bayer started work on Improvement JENKINS-53734
 
Change By: Andrew Bayer
Status: Open In Progress

andrew.bayer@gmail.com (JIRA)

unread,
Oct 16, 2018, 10:15:02 AM10/16/18
to jenkinsc...@googlegroups.com

andrew.bayer@gmail.com (JIRA)

unread,
Oct 16, 2018, 10:18:02 AM10/16/18
to jenkinsc...@googlegroups.com
Andrew Bayer commented on Improvement JENKINS-53734
 
Re: Declarative pipeline - Support for parallel stages inside sequential stages

So the motivation was that we don't want to allow arbitrarily deep parallel nesting - nested parallels won't be visualized properly by Blue Ocean, most notably.

That said, https://github.com/jenkinsci/pipeline-model-definition-plugin/pull/292 - this would allow nesting parallel within a stage's stages if and only if the stage isn't itself already within a parallel. I think that it won't visualize particularly nicely in Blue Ocean (i.e., it may just visualize the whole nested stage with a parallel on its own, without visualizing its children independently), but I think we can live with that anyway.

torsten.kleiber@ikb.de (JIRA)

unread,
Oct 17, 2018, 1:09:02 AM10/17/18
to jenkinsc...@googlegroups.com

I don't agree that this should not be rendered! Visualization of a pipeline for end user is important! Every other job control software has no problem with such parallelisation's. And for optimization of build time I need a lot of parallelisation's. 

So I would request the corresponding changes in blue ocean too after implementation of this change.

In the meantime I run in several issues in Jenkins, where implementing one change in some plugin contradicts other restrictions in another  plugin.

onodera@mail.ru (JIRA)

unread,
Nov 22, 2018, 7:58:03 AM11/22/18
to jenkinsc...@googlegroups.com

I agree that this will be very useful. I use sequential stages to lock a single resource (DB) for multiple stages (as suggested in JENKINS-43336)

 

For example, a pipeline like this one is currently impossible:

pipeline {
  agent any
  stages {
    stage ('Non-production') {
      options {
        lock 'DB1'
      }
      stages {
        stage ('Deploy') {
          steps {
            echo 'Deploy'
          }
        }
        stage ('Test') {
          parallel {
            stage ('Unit testing') {
              steps {
                echo 'Unit'
              }
            }
            stage ('E2E testing') {
              steps {
                echo 'E2E'
              }
            }
          }
        }
      }
    }    
    stage ('Production') {
      stages {
        stage ('Deploy') {
          steps {
            echo 'Deploy'
          }
        }
      }
    }
  }
}

andrew.bayer@gmail.com (JIRA)

unread,
Dec 10, 2018, 12:01:03 PM12/10/18
to jenkinsc...@googlegroups.com
 

Merged, releasing in 1.3.4 momentarily.

Change By: Andrew Bayer
Status: In Review Resolved
Resolution: Fixed

bitwiseman@gmail.com (JIRA)

unread,
Oct 22, 2019, 11:25:54 PM10/22/19
to jenkinsc...@googlegroups.com
Liam Newman closed an issue as Fixed
 

Bulk closing resolved issues.

Change By: Liam Newman
Status: Resolved Closed
This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f)
Atlassian logo
Reply all
Reply to author
Forward
0 new messages