[JIRA] (JENKINS-41334) Support parallel execution of stages in Declarative

6 views
Skip to first unread message

andrew.bayer@gmail.com (JIRA)

unread,
Jan 23, 2017, 12:34:01 PM1/23/17
to jenkinsc...@googlegroups.com
Andrew Bayer created an issue
 
Jenkins / Story JENKINS-41334
Support parallel execution of stages in Declarative
Issue Type: Story Story
Assignee: Andrew Bayer
Components: pipeline-model-definition-plugin
Created: 2017/Jan/23 5:33 PM
Priority: Critical Critical
Reporter: Andrew Bayer

Note - this is a post-1.0 feature.

Issues like JENKINS-41198 and JENKINS-40699 are among the drivers for this - in the Declarative model, parallel doesn't quite fit in smoothly. We need a better answer for this so that more complicated parallel execution of stages is possible within the model.

I'd previously dabbled with a stage execution dependency graph, originally in Plumber, but am easing away from that now. While I love the idea, I can't find a comfortable way to make it work without it being required for every stage, which is a non-option - i.e., I don't think it's a good idea to always require that every single stage has a marker for what stage(s) it can run before or after. So now I'm leaning in the direction of nested stages sections, like this:

pipeline {
  agent any
  
  stages {
    stage('first') {
      steps {
        echo 'first, non-parallel stage'
      }
    }

    stage('top-parallel') {
      stages {
        stage('first-parallel') {
          steps {
            echo 'First of the parallel stages without further nesting'
            sleep 60
          }
        }
        stage('second-parallel') {
          stages {
            stage('first-nested-parallel') {
              steps {
                 echo 'the first of the nested parallel stages'
                 sleep 30
              }
           }
           stage('second-nested-parallel') {
              steps {
                 echo 'the second of the nested parallel stages'
                 sleep 30
              }
           }
        }
     }
  }
}

So in this scenario, stage('first') runs first. When it completes, stage('top-parallel') starts and immediately goes into its nested stages in parallel. stage('first-parallel') starts and goes for 60 seconds, while stage('second-parallel') starts at the same time and descends into its nested stages in parallel as well.

I don't know yet where I'd allow agent and friends, but within a stage, you would need to have one and only one of steps or stages - i.e., a stage either could have steps it executes or it could be a container for parallel stages.

Actually implementing this will need to be done in tandem with Blue Ocean visualization, of course. And this is for now just raw thoughts, but I wanted to get it written down.

Add Comment Add Comment
 
This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c)
Atlassian logo

andrew.bayer@gmail.com (JIRA)

unread,
Jan 23, 2017, 3:27:01 PM1/23/17
to jenkinsc...@googlegroups.com
Andrew Bayer commented on Story JENKINS-41334
 
Re: Support parallel execution of stages in Declarative

Very preliminary initial work up at https://github.com/jenkinsci/pipeline-model-definition-plugin/pull/98 - right now it's just the infrastructure/parsing/schema/etc for making this work without actually testing the new parallelism directly, 'cos I wanted to see what else blew up as a result. =)

andrew.bayer@gmail.com (JIRA)

unread,
Jan 23, 2017, 3:27:01 PM1/23/17
to jenkinsc...@googlegroups.com
Andrew Bayer started work on Story JENKINS-41334
 
Change By: Andrew Bayer
Status: Open In Progress

andrew.bayer@gmail.com (JIRA)

unread,
Jan 23, 2017, 3:49:01 PM1/23/17
to jenkinsc...@googlegroups.com

cc @vivek James Dumay Keith Zantow Patrick Wolf Michael Neale for your thoughts. Again, remember this is not targeted for 1.0 so don't freak out. =)

pwolf@cloudbees.com (JIRA)

unread,
Jan 23, 2017, 3:55:02 PM1/23/17
to jenkinsc...@googlegroups.com

Nesting and intermixing of stages and stage seems overly complicated but I'm not sure I have something better right now. At least with parallel the fact that is something different visually alerts you that something different is going on. With this, you have to track indentation and number of } to know what to expect.

andrew.bayer@gmail.com (JIRA)

unread,
Jan 23, 2017, 4:01:05 PM1/23/17
to jenkinsc...@googlegroups.com

Yeah, I get what you mean. I'm open to calling it something other than stages but not parallel since that's already a meaningful term in Scripted Pipeline.

kzantow@cloudbees.com (JIRA)

unread,
Jan 23, 2017, 4:08:01 PM1/23/17
to jenkinsc...@googlegroups.com

Well, I understand the possible issue if this is basically run through the existing groovy interpreter, but I'd like to see something like:

stage('Build') {
  parallel { // this indicates running stages in parallel
    stage('Windows') {
       ...
    }
    stage('Linux') {
      ...
    }
  }
}

Or more simply, just have a rule that all stages within a stage must finish before it completes, and add a parallel flag for nested stages?

stage('Build', parallel) {
  stage('Windows') {
     ...
  }
  stage('Linux') {
    ...
  }
}

... just some thoughts on this. I'm not a big fan of huge nesting, I already feel like it's getting a bit too nested for my tastes in places. Might as well just use LISP if we wanted that!

andrew.bayer@gmail.com (JIRA)

unread,
Jan 23, 2017, 4:16:03 PM1/23/17
to jenkinsc...@googlegroups.com

So I am vehemently opposed to adding more arguments to stage because I'm still not happy about there being any arguments to stage. =) Not to mention that would be a pain to translate between Groovy syntax and JSON syntax. Keith Zantow's first example is basically the one I've proposed but with a different name for the block (parallel vs stages) and, well, I kinda have that working, so my bias is definitely in that direction, modulo naming. We can technically handle parallel as the name, just as we handle stage when it's not really StageStep on the backend, I just prefer to avoid duplication of names if possible.

pwolf@cloudbees.com (JIRA)

unread,
Jan 23, 2017, 4:25:03 PM1/23/17
to jenkinsc...@googlegroups.com

I think the first example from Keith Zantow is pretty easy to grok but it does reuse parallel in a different way than in steps . If we take the original example from Andrew Bayer in description we get:

pipeline {
  agent any
  
  stages {
    stage('first') {
      steps {
        echo 'first, non-parallel stage'
      }
    }

    stage('top-parallel') {
      parallel {
        stage('first-parallel') {
          steps {
            echo 'First of the parallel stages without further nesting'
            sleep 60
          }
        }
        stage('second-parallel') {
          parallel {
            stage('first-nested-parallel') {
              steps {
                 echo 'the first of the nested parallel stages'
                 sleep 30
              }
           }
           stage('second-nested-parallel') {
              steps {
                 echo 'the second of the nested parallel stages'
                 sleep 30
              }
           }
        }
     }
  }
}

andrew.bayer@gmail.com (JIRA)

unread,
Jan 23, 2017, 4:36:04 PM1/23/17
to jenkinsc...@googlegroups.com

Worth mentioning that my indentation sucked. =)

jdumay@cloudbees.com (JIRA)

unread,
Feb 1, 2017, 12:29:01 AM2/1/17
to jenkinsc...@googlegroups.com

Andrew Bayer BTW I have not forgotten about this. Will come back to it when I come up for air...

roidelapluie@inuits.eu (JIRA)

unread,
Feb 2, 2017, 3:38:03 AM2/2/17
to jenkinsc...@googlegroups.com

Would also love to be able to code those parallel:

pipeline {
  stages {
    def parallelstages = [:]

    paralellstages['foo'] = {
      stage {
        agent {docker {hello} }
      }
    }
    parallel parallelstages
  }
}

roidelapluie@inuits.eu (JIRA)

unread,
Feb 2, 2017, 3:39:01 AM2/2/17
to jenkinsc...@googlegroups.com
Julien Pivotto edited a comment on Story JENKINS-41334
Would also love to be able to code those parallel:

{code}

pipeline {
  stages {
    def parallelstages = [:]

    paralellstages['foo'] = {
      stage {
        agent {docker
{ ' hello ' } }
      }
    }
    parallel parallelstages
  }
}
{code}

roidelapluie@inuits.eu (JIRA)

unread,
Feb 2, 2017, 3:41:03 AM2/2/17
to jenkinsc...@googlegroups.com
Julien Pivotto updated an issue
 
Change By: Julien Pivotto
Comment:
Would also love to be able to code those parallel:

{code}
pipeline {
  stages {
    def parallelstages = [:]

    paralellstages['foo'] = {
      stage {
        agent {docker 'hello' }
      }
    }
    parallel parallelstages
  }
}
{code}

yannick@koehler.name (JIRA)

unread,
Feb 6, 2017, 12:58:02 PM2/6/17
to jenkinsc...@googlegroups.com
Yannick Koehler commented on Story JENKINS-41334
 
Re: Support parallel execution of stages in Declarative

I am interested in a scenario that looks like this:

stages {
stage('extract') {
}
stage('build') {
parallel {
stage('build product A') {
stage('build module set 1') {
}
stage('build module set 2') {
}
}
stage('build product B') {
}
}
}
stage('finalize') {
}
}

The outcome order would be that build only occurs after extract, that finalize only occurs after all build sub-stage are completed. That build product A and build procduct B are done in parallel. That build module set 2 only occurs after build module set 1 has completed (since it has some dependencies).

Now, I feel that using "stages" inside a "stage" or a "stages" tag make no sense. Also, indenting those "stage" looks messy.

A suggestion would be to do this operation like Make does, by stating the stages first, then adjusting the dependencies later, i.e.:

stages {
stage('extract') { }
stage('build product A') { }
stage('build module set 1') { }
stage('build module set 2') { }
stage('build product B') { }
stage('finalize') { }
}

This would be default create a set of dependencies such as

<target>: <dependencies>
'build product A': 'extract'
'build module set 1': 'build product A'
'build module set 2': 'build module set 1'
'build product B': 'build module set 2'
'finalize': 'build product B'

Same as today, the author can then change those dependencies from the default to what he wants:

stages {
stage('extract') { }
stage('build product A') { }
stage('build module set 1') { }
stage('build module set 2') { }
stage('build product B')

{ dependencies: [ 'extract' ] }


stage('finalize')

{ dependencies: ['build product A', 'build product B'] }

}

This effectively transforming the dependency table to

<target>: <dependencies>
'build product A': 'extract'
'build module set 1': 'build product A'
'build module set 2': 'build module set 1'
'build product B': 'extract'
'finalize': 'build product A', 'build product B'

Using this set of dependency, the parallelism used is imply in the dependency, as soon as a target dependency are met, the associated stage can start.

This is backward compatible, get rid of the "parallel" declarative statement and add a minimum set of keywords "dependencies" to the current DSL.

Anyway, just a need of mine and an opinion to share on how to do it.

yannick@koehler.name (JIRA)

unread,
Feb 6, 2017, 1:00:04 PM2/6/17
to jenkinsc...@googlegroups.com
Yannick Koehler edited a comment on Story JENKINS-41334

yannick@koehler.name (JIRA)

unread,
Feb 6, 2017, 1:00:04 PM2/6/17
to jenkinsc...@googlegroups.com
Yannick Koehler edited a comment on Story JENKINS-41334
I am interested in a scenario that looks like this:

{ { noformat}
  stages {
    stage('extract') {
    }
    stage('build') {
      parallel {
        stage('build product A') {
          stage('build module set 1') {
          }
          stage('build module set 2') {
          }
        }
        stage('build product B') {
        }
      }
    }
    stage('finalize') {
    }
  }
{noformat } }

yannick@koehler.name (JIRA)

unread,
Feb 6, 2017, 1:01:01 PM2/6/17
to jenkinsc...@googlegroups.com
Yannick Koehler edited a comment on Story JENKINS-41334
I am interested in a scenario that looks like this:

{noformat}
  stages {
    stage('extract') {
    }
    stage('build') {
      parallel {
        stage('build product A') {
          stage('build module set 1') {
          }
          stage('build module set 2') {
          }
        }
        stage('build product B') {
        }
      }
    }
    stage('finalize') {
    }
  }
{noformat}

The outcome order would be that build only occurs after extract, that finalize only occurs after all build sub-stage are completed.  That build product A and build procduct B are done in parallel.  That build module set 2 only occurs after build module set 1 has completed (since it has some dependencies).

Now, I feel that using "stages" inside a "stage" or a "stages" tag make no sense.  Also, indenting those "stage" looks messy.

A suggestion would be to do this operation like Make does, by stating the stages first, then adjusting the dependencies later, i.e.:

{noformat}
  stages {
    stage('extract') { }

    stage('build product A') { }
    stage('build module set 1') { }
    stage('build module set 2') { }
    stage('build product B') { }
    stage('finalize') { }
  }
{noformat}

This would be default create a set of dependencies such as

{noformat}
   <target>: <dependencies>
   'build product A': 'extract'
   'build module set 1': 'build product A'
   'build module set 2': 'build module set 1'
   'build product B': 'build module set 2'
   'finalize': 'build product B'
{noformat}

Same as today, the author can then change those dependencies from the default to what he wants:

{noformat}
  stages {
    stage('extract') { }

    stage('build product A') { }
    stage('build module set 1') { }
    stage('build module set 2') { }
    stage('build product B') {
      dependencies: [ 'extract' ]
    }
    stage('finalize') {
      dependencies: ['build product A', 'build product B']
    }
  }
{noformat}

This effectively transforming the dependency table to

{noformat}
   <target>: <dependencies>
   'build product A': 'extract'
   'build module set 1': 'build product A'
   'build module set 2': 'build module set 1'
   'build product B': 'extract'
   'finalize': 'build product A', 'build product B'
{noformat}

Using this set of dependency, the parallelism used is imply in the dependency, as soon as a target dependency are met, the associated stage can start.

This is backward compatible, get rid of the "parallel" declarative statement and add a minimum set of keywords "dependencies" to the current DSL.

Anyway, just a need of mine and an opinion to share on how to do it.

bitwiseman@gmail.com (JIRA)

unread,
Feb 17, 2017, 4:50:06 PM2/17/17
to jenkinsc...@googlegroups.com

Andrew Bayer

I think name-wise I'm against reusing stages to mean "parallel" without some extra marking. Children of stages are executed in serial at top level, it would be strange to make them execute in parallel at the non-root level. The behavior should not change so radically with some indicator, something like stages (parallel: true)? That would mean I could use the same marker for the top level stages.

Any plans to have some sort of design meeting?

yannick@koehler.name (JIRA)

unread,
Feb 17, 2017, 8:29:05 PM2/17/17
to jenkinsc...@googlegroups.com

Could be wrong, but the concept of Jenkins stage was always about concurrency... To me, parallel and concurrent is pretty much the same thing. Before stage concurrency was limited to different build same stage name. In my proposal, concurrency is between different stage within the same build.

Maybe this shows that we may need to augment stage to allow control of exactly how the stage's concurrency is defined, same build, between builds, same stage, different stage name...

Note also that in my proposal, stage is not used to mean parallel, without explicit additionnal dependencies keywords, the stages are serialized. Only when a dependency indicate the 2 stage share the same dependency that they can then be run concurrently (or in parallel).

Having said all that, I am not against adding a "parallel: true", or... restore the "concurrency" option and allowing to specify a maximum value to that same concurrency.

    stage('build product B') { 
      dependencies: [ 'extract' ]
      concurrency: infinite
    }

andrew.bayer@gmail.com (JIRA)

unread,
Feb 27, 2017, 12:38:02 PM2/27/17
to jenkinsc...@googlegroups.com

scm_issue_link@java.net (JIRA)

unread,
Mar 21, 2017, 7:37:07 PM3/21/17
to jenkinsc...@googlegroups.com

Code changed in jenkins
User: Andrew Bayer
Path:
rfc/JENKINS-41334-parallel-stage-execution.md
http://jenkins-ci.org/commit/pipeline-model-definition-plugin/c033644c282e0f1727fdb0b6beddaff6a5819723
Log:
Merge pull request #126 from abayer/parallel-stages-rfc

RFC: JENKINS-41334: Parallel stage execution

Compare: https://github.com/jenkinsci/pipeline-model-definition-plugin/compare/00dc23399128...c033644c282e

This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e)
Atlassian logo

scm_issue_link@java.net (JIRA)

unread,
Mar 21, 2017, 7:37:07 PM3/21/17
to jenkinsc...@googlegroups.com

bitwiseman@gmail.com (JIRA)

unread,
Oct 22, 2019, 11:25:33 PM10/22/19
to jenkinsc...@googlegroups.com
Liam Newman closed an issue as Fixed
 

Bulk closing resolved issues.

Change By: Liam Newman
Status: Resolved Closed
This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f)
Atlassian logo
Reply all
Reply to author
Forward
0 new messages