[JIRA] (JENKINS-54010) Support two levels of parallelity in stages

0 views
Skip to first unread message

daniel.kurzynski@sap.com (JIRA)

unread,
Oct 11, 2018, 4:20:02 AM10/11/18
to jenkinsc...@googlegroups.com
Daniel Kurzynski created an issue
 
Jenkins / Bug JENKINS-54010
Support two levels of parallelity in stages
Issue Type: Bug Bug
Assignee: Unassigned
Attachments: parallel-stages.PNG
Components: blueocean-plugin
Created: 2018-10-11 08:19
Priority: Minor Minor
Reporter: Daniel Kurzynski

Support scripted parallel blocks in parallel stages in a declarative pipeline: 

Example pipeline:
```
pipeline {
agent none
stages {
stage('Parallel Stage') {
parallel {
stage('Stage 1') {
steps

{ echo "Stage 1" }

}
stage('Stage 2') {
steps {
script {
parallel (
"Stage 2.1.":

{ echo "Stage 2.1." }

,
"Stage 2.2.":

{ echo "Stage 2.2." }

)
}
}
}
}
}
}
}
```

Currently it is not properly visualized in blue ocean as shown in parallel-stages.png

Add Comment Add Comment
 
This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)

daniel.kurzynski@sap.com (JIRA)

unread,
Oct 11, 2018, 4:22:01 AM10/11/18
to jenkinsc...@googlegroups.com
Daniel Kurzynski updated an issue
Change By: Daniel Kurzynski
Support scripted parallel blocks in parallel stages in a declarative pipeline .

Example pipeline:
``` {code:java}
pipeline {
    agent none
    stages {
        stage('Parallel Stage') {
            parallel {
                stage('Stage 1') {
                    steps {
                     echo "Stage 1"
                    }
                }
                stage('Stage 2') {
                    steps {
                     script {
                     parallel (
                     "Stage 2.1.": {
                     echo "Stage 2.1."
                     },
                     "Stage 2.2.": {
                     echo "Stage 2.2."
                     }
                     )
                     }
                    }
                }
            }
        }
    }
}
``` {code}

Currently it is not properly visualized in blue ocean as shown in parallel-stages.png

daniel.kurzynski@sap.com (JIRA)

unread,
Oct 11, 2018, 4:30:03 AM10/11/18
to jenkinsc...@googlegroups.com
Daniel Kurzynski updated an issue
Change By: Daniel Kurzynski
Attachment: parallel-stages-new.png

daniel.kurzynski@sap.com (JIRA)

unread,
Oct 11, 2018, 4:31:01 AM10/11/18
to jenkinsc...@googlegroups.com

daniel.kurzynski@sap.com (JIRA)

unread,
Oct 11, 2018, 4:31:02 AM10/11/18
to jenkinsc...@googlegroups.com
Daniel Kurzynski updated an issue
Support scripted parallel blocks in parallel stages in a declarative pipeline.


Example pipeline:
{code:java}
pipeline {
    agent none
    stages {
        stage('Parallel Stage') {
            parallel {
                stage('Stage 1') {
                    steps {
                     echo "Stage 1"
                    }
                }
                stage('Stage 2') {
                    steps {
                     script {
                     parallel (
                     "Stage 2.1.": {
                     echo "Stage 2.1."
                     },
                     "Stage 2.2.": {
                     echo "Stage 2.2."
                     }
                     )
                     }
                    }
                }
            }
        }
    }
}
{code}

Currently it is not properly visualized in blue ocean as shown in parallel-stages.png


 

The idea how it should be visualized is shown in parallel-stages-new.png

phil.segal@gmail.com (JIRA)

unread,
Oct 11, 2018, 5:01:03 AM10/11/18
to jenkinsc...@googlegroups.com
Phil Segal commented on Bug JENKINS-54010
 
Re: Support two levels of parallelity in stages

I would hope this would also support the same for scripted pipelines too.

phil.segal@gmail.com (JIRA)

unread,
Oct 11, 2018, 5:01:03 AM10/11/18
to jenkinsc...@googlegroups.com
Phil Segal assigned an issue to Unassigned
 
Change By: Phil Segal
Assignee: Phil Segal

phil.segal@gmail.com (JIRA)

unread,
Oct 11, 2018, 5:01:04 AM10/11/18
to jenkinsc...@googlegroups.com
Phil Segal assigned an issue to Phil Segal

tristan.cladet@siemens.com (JIRA)

unread,
Oct 18, 2018, 8:57:02 AM10/18/18
to jenkinsc...@googlegroups.com
Tristan Cladet commented on Bug JENKINS-54010
 
Re: Support two levels of parallelity in stages

+100  Today, I am stucked to run different kind of tests (python, qtest, ...) in parallel on two configurations (Windows and Linux) in parallel.

di.gaus@gmx.de (JIRA)

unread,
Nov 16, 2018, 4:01:02 AM11/16/18
to jenkinsc...@googlegroups.com

This picture demonstrates it better... running parallel within parallel without a step before makes no sense since you could just run them in the outer parallel task.

di.gaus@gmx.de (JIRA)

unread,
Nov 16, 2018, 4:25:05 AM11/16/18
to jenkinsc...@googlegroups.com
Dirk Gausmann edited a comment on Bug JENKINS-54010
This picture demonstrates it better... running parallel within parallel without a step before makes no sense since you could just run them in the outer parallel task.

!pipeline_mockup.png!

di.gaus@gmx.de (JIRA)

unread,
Nov 16, 2018, 4:27:02 AM11/16/18
to jenkinsc...@googlegroups.com
Dirk Gausmann edited a comment on Bug JENKINS-54010
This picture demonstrates it better... running parallel within parallel without a step before makes no sense since you could just run them in the outer parallel task.

ahmed@sghaier.com (JIRA)

unread,
Jan 11, 2019, 8:06:03 AM1/11/19
to jenkinsc...@googlegroups.com

Dirk Gausmann Actually the complete idea from Ben Dean would be really nice to have. Check this comment here:

https://issues.jenkins-ci.org/browse/JENKINS-38442?focusedCommentId=292647&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-292647

It would be nice to have that not only for the declarative, but also the scripted pipeline.

Sometimes the test suites to cover are not statically declared and can depend on different things for different branches, hens the need for the scripted pipeline.

mircea.albu@gmail.com (JIRA)

unread,
Feb 12, 2019, 9:02:03 AM2/12/19
to jenkinsc...@googlegroups.com

We need this for the scripted pipeline as well.

rishirt.us@gmail.com (JIRA)

unread,
Apr 23, 2019, 1:46:02 PM4/23/19
to jenkinsc...@googlegroups.com

Is there any progress on this task? My team has a highly parallel flow that requires multiple layers of nested parallels. It would make life easier for debugging purposes if this feature were implemented.

thomas.william@logmein.com (JIRA)

unread,
Apr 24, 2019, 8:55:05 AM4/24/19
to jenkinsc...@googlegroups.com

same here - i have a 50+ different build configs to run in parallel, a dozen per OS (Linux, MacOS, iOS, Android, Windows . . .) - Blue Ocean graph is pretty useless this way . . .

shahaf.d@samsung.com (JIRA)

unread,
May 5, 2019, 7:38:03 AM5/5/19
to jenkinsc...@googlegroups.com

This feature will help us a lot.

Multiple levels of parallelizm are currently supported only by creating new jobs, which is a poor solution given that Jenkins groovy supports it out of the box.

I'm sure this feature will help a lot of users since many applications test themselves on multiple platforms today.

dntsaygoodbye@gmail.com (JIRA)

unread,
May 5, 2019, 3:43:03 PM5/5/19
to jenkinsc...@googlegroups.com

Shahaf Duenyas I'm not sure that you are talking about the BlueOcean plugin.

Jenkins pipeline already supports multiple levels of parallelism, this is only about Blue Ocean's UI

shahaf.d@samsung.com (JIRA)

unread,
May 6, 2019, 4:24:05 AM5/6/19
to jenkinsc...@googlegroups.com

Alaiksei Savanchuk - I am talking about the BlueOcean plugin.

We work with BlueOcean and we have multiple levels of parallelizm in our pipeline, and the only method to show it correctly using the GUI is creating multiple jobs for the different paltforms, otherwise it shows all of the parallel steps in the internal parallel block in the same level of the parallel steps of the external parallel block which is obviously wrong, furthermore, it also shows the serial steps of the internal job as parallel.

 

For example - we have the following pipeline:

Parallel:

    Linux:

        Build

        Parallel:

            Fixed tests

            Custom tests

    Windows:

        Build

 

We currently create a different job for windows and linux pipes and trigger them from the master job.

When trying to run it in a single job, BlueOcean shows it as follows:

Parallel:

    Linux

    Fixed tests

    Custom tests

    Windows

 

I would very much like to see it as shown in JENKINS-38442 (pipeline_mockup.png)

mathias@nedrebo.org (JIRA)

unread,
May 19, 2019, 4:38:04 PM5/19/19
to jenkinsc...@googlegroups.com

Our team is also hoping for a solution to this, the setup we have is similar to what Shahaf Duenyas describes, starting with some sequential pre-work, then splitting into windows and linux build, then ideally further splitting several times down the pipeline (multiple tests that each is parameterized over OS and HW configurations and fully parallelizable).

Right now our pipeline takes three times as long as it should because we really like to have the user friendly visualization we get from Blue Ocean. This is of cause not ideal because time to failure is longer that it should be and the dev cycle time suffers.

I think this issue should add support for any levels of parallelism, not hard code it for any arbitrarily magic number of parallelism.

The issue should be renamed to "Support nested parallel stages"

I also agree that the "pipeline_mockup.png" in JENKINS-38442 looks like the natural way to render this and it should scale well. Rendering the X(1 for example) top level parallel stages different then the N-X inner levels, as I have seen proposed, seems like a hack tailored to some specific use case.

rishirt.us@gmail.com (JIRA)

unread,
May 20, 2019, 11:57:04 PM5/20/19
to jenkinsc...@googlegroups.com
Rishi Thakkar updated an issue
 
Change By: Rishi Thakkar
Attachment: Screen Shot 2019-05-14 at 9.49.06 AM.png

rishirt.us@gmail.com (JIRA)

unread,
May 20, 2019, 11:59:03 PM5/20/19
to jenkinsc...@googlegroups.com
Rishi Thakkar updated an issue
Change By: Rishi Thakkar
Attachment: Screen Shot 2019-05-20 at 8.55.46 PM.png

rishirt.us@gmail.com (JIRA)

unread,
May 21, 2019, 12:01:15 AM5/21/19
to jenkinsc...@googlegroups.com
Rishi Thakkar commented on Bug JENKINS-54010
 
Re: Support two levels of parallelity in stages

My set up is also very similar to the one that Shahaf Duenyas has described.

 

A possible solution is to combine all nested stages into one node. When you click on this node, it expands and shows you everything else that was under it. This way the view is fairly clean and intuitive to use. See following images for examples:

 

In the image below lets imagine I clicked on build 1:

The UI will then transition to show the following nested pipeline:

 

This basically steps away from only supporting nested parallels to just supporting nested pipelines.

 

Is there anyone who can be considered the owner of this issue? It would be great if we could come up with a plan and start executing. As  Shahaf Duenyas  said, when you have 50+ builds in parallel, the use is pretty much useless without nesting support.

rishirt.us@gmail.com (JIRA)

unread,
May 21, 2019, 12:01:25 AM5/21/19
to jenkinsc...@googlegroups.com
Rishi Thakkar edited a comment on Bug JENKINS-54010
My set up is also very similar to the one that [~shahafd] has described.


A possible solution is to combine all nested stages into one node. When you click on this node, it expands and shows you everything else that was under it. This way the view is fairly clean and intuitive to use. See following images for examples:

In the image below lets imagine I clicked on build 1:

!Screen Shot 2019-05-14 at 9.49.06 AM.png|width=169,height=304!


The UI will then transition to show the following nested pipeline:

!Screen Shot 2019-05-20 at 8.55.46 PM.png|width=451,height=198!


 

This basically steps away from only supporting nested parallels to just supporting nested pipelines.

Is there anyone who can be considered the owner of this issue? It would be great if we could come up with a plan and start executing. As  [~shahafd]  said, when you have 50+ builds in parallel, the use UI is pretty much useless without nesting support.

rishirt.us@gmail.com (JIRA)

unread,
May 21, 2019, 12:01:25 AM5/21/19
to jenkinsc...@googlegroups.com
Rishi Thakkar edited a comment on Bug JENKINS-54010
My set up is also very similar to the one that [~shahafd] has described.

 

A possible solution is to combine all nested stages into one node. When you click on this node, it expands and shows you everything else that was under it. This way the view is fairly clean and intuitive to use. See following images for examples:

 

In the image below lets imagine I clicked on build 1:

!Screen Shot 2019-05-14 at 9.49.06 AM.png|width=169,height=304!

The UI will then transition to show the following nested pipeline:

!Screen Shot 2019-05-20 at 8.55.46 PM.png|width=451,height=198!

 

This basically steps away from only supporting nested parallels to just supporting nested pipelines.

 

Is there anyone who can be considered the owner of this issue? It would be great if we could come up with a plan and start executing. As  [~shahafd]  said, when you have 50+ builds in parallel, the use is pretty much useless without nesting support.

rishirt.us@gmail.com (JIRA)

unread,
May 21, 2019, 12:51:11 AM5/21/19
to jenkinsc...@googlegroups.com

Could someone from CloudBees Inc. possibly take a look at this? I have seen a few other tickets related to parallel visualization in blue ocean owned by you guys.

mathias@nedrebo.org (JIRA)

unread,
May 22, 2019, 5:56:02 AM5/22/19
to jenkinsc...@googlegroups.com
Mathias Nedrebø edited a comment on Bug JENKINS-54010
Our team is also hoping for a solution to this, the setup we have is similar to what [~shahafd] describes, starting with some sequential pre-work, then splitting into windows and linux build, then ideally further splitting several times down the pipeline (multiple tests that each is parameterized over OS and HW configurations and fully parallelizable).

Right now our pipeline takes three times as long as it should because we really like to have the user friendly visualization we get from Blue Ocean. This is of
cause course not ideal because time to failure is longer that it should be and the dev cycle time suffers.


I think this issue should add support for any levels of parallelism, not hard code it for any arbitrarily magic number of parallelism.

*The issue should be renamed to "Support nested parallel stages"*


I also agree that the "pipeline_mockup.png" in JENKINS-38442 looks like the natural way to render this and it should scale well. Rendering the X(1 for example) top level parallel stages different then the N-X inner levels, as I have seen proposed, seems like a hack tailored to some specific use case.

ianfixes@gmail.com (JIRA)

unread,
May 22, 2019, 9:43:04 AM5/22/19
to jenkinsc...@googlegroups.com
Ian Katz commented on Bug JENKINS-54010

The essence of this issue is a mismatch between the ability to express a nested (and valid) workflow in the Jenkinsfile and the UI's ability to render it.  I don't know of a limit on nesting in the Jenkinsfile, so it follows that there shouldn't be such a limit in the UI either.

 

To me, the significance of this issue is that BlueOcean is used for CI – its job is to help us execute, understand, and troubleshoot some complex procedures in dev/release.  Failing to represent the procedure itself undermines this entire concept.

rishirt.us@gmail.com (JIRA)

unread,
May 22, 2019, 9:48:12 AM5/22/19
to jenkinsc...@googlegroups.com

Is it possible to bump the priority of this issue to a blocker or do we need someone special to do that?

rishirt.us@gmail.com (JIRA)

unread,
May 28, 2019, 1:44:13 PM5/28/19
to jenkinsc...@googlegroups.com
Rishi Thakkar updated an issue
Change By: Rishi Thakkar
Priority: Critical Blocker

rishirt.us@gmail.com (JIRA)

unread,
May 28, 2019, 1:45:06 PM5/28/19
to jenkinsc...@googlegroups.com
 
Re: Support two levels of parallelity in stages

I am bumping priority to blocker. Please let me know if this is something that I was not allowed to do.

Hopefully, this will bring a little more attention to this issue.

luckyhk.lee@samsung.com (JIRA)

unread,
Jul 15, 2019, 10:30:03 AM7/15/19
to jenkinsc...@googlegroups.com

I am looking forward parallel in parallel feature, whole code of declarative syntax.

trym.bremnes@gmail.com (JIRA)

unread,
Aug 14, 2019, 1:38:04 AM8/14/19
to jenkinsc...@googlegroups.com

This is a very important issue for us to be resolved. Without this our pipeline takes so much longer to complete, since almost everything has to be done in series instead of parallel. 

jeremiefauchergoulet@hotmail.com (JIRA)

unread,
Nov 7, 2019, 8:09:03 PM11/7/19
to jenkinsc...@googlegroups.com

I'm surprised this isn't supported by the UI.

We develop firmware and cross-compile to different architecture using agent of different OS. This equates to a lot of builds in parallels, with internal stages in parallel as well (such as static code analysis, running tests, etc.).

Parallelism is a must otherwise build times are way too long and a pain for developers who want rapid cycles of feedback and not wait for a build to complete.

This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f)
Atlassian logo

shahaf.d@samsung.com (JIRA)

unread,
Nov 10, 2019, 3:06:04 AM11/10/19
to jenkinsc...@googlegroups.com

Is there any reason why this issue doesn't get prioritized ?

It is already marked as a blocker for 6 months, and was critical for another 6 months earlier.

Can you please update on the status of this request ? Is it in development ? Do you plan to add this feature soon ?

It is very crucial for many people using BlueOcean plugin.

ianfixes@gmail.com (JIRA)

unread,
Nov 10, 2019, 2:05:04 PM11/10/19
to jenkinsc...@googlegroups.com
Ian Katz commented on Bug JENKINS-54010

I think a better question to ask would be "what technical hurdles block this request".  My guess is that when you run things in parallel, each parallel item counts as a "stage" to the UI, and so what we are really discussing here is a need to support sub-stages – something that seems a bit more complicated than a UI fix.

Is that accurate? 

rpuvvada2@gmail.com (JIRA)

unread,
Nov 22, 2019, 6:18:11 PM11/22/19
to jenkinsc...@googlegroups.com

+1 I'm currently running 20 parallel stages in my pipeline for speed but the BlueOcean visualization of a tall column of nodes isn't the most user friendly. Nested parallel would be one solution to this. I'd also be happy with a purely UI construct to categorize my stages for better visualization. This has been mentioned in a series of tickets resolved as dupes that ultimately end up here. An example based on the description:

 

pipeline {
    agent none
    stages {
        stage('Parallel Stage') {
            parallel {
                stage('Stage 1') {
                    steps {
                        echo "Stage 1"
                    }
                }
                category('Stage 2') {
                    stage("Stage 2.1"): {
                        steps {
                            echo "Stage 2.1."
                        }
                    },
                    stage(Stage 2.2") {
                        steps {
                            echo "Stage 2.2."
                        }
                    }
                }
            }
        }
    }
}

rockmax.rajat@gmail.com (JIRA)

unread,
Dec 18, 2019, 3:01:22 AM12/18/19
to jenkinsc...@googlegroups.com

Blueocean team - Any updates on this? Are you guys working on this issue since its a blocker for us?

 

  • Adobe Developer

shanidar1@gmail.com (JIRA)

unread,
Dec 18, 2019, 3:51:06 AM12/18/19
to jenkinsc...@googlegroups.com
Shani Dar updated an issue
 
Change By: Shani Dar
Attachment: image-2019-12-18-10-50-54-224.png

shanidar1@gmail.com (JIRA)

unread,
Dec 18, 2019, 4:10:21 AM12/18/19
to jenkinsc...@googlegroups.com
Shani Dar updated an issue
Change By: Shani Dar
Attachment: Selection_090.png

shanidar1@gmail.com (JIRA)

unread,
Dec 18, 2019, 4:15:26 AM12/18/19
to jenkinsc...@googlegroups.com
Shani Dar updated an issue
Change By: Shani Dar
Attachment: Selection_093.png

shanidar1@gmail.com (JIRA)

unread,
Dec 18, 2019, 4:18:04 AM12/18/19
to jenkinsc...@googlegroups.com
Shani Dar commented on Bug JENKINS-54010
 
Re: Support two levels of parallelity in stages

What I did as a workaround is building a parallel stage dynamically - 

We deploy 5 services for two clouds, we want to deploy all in parallel.  Let's say each service takes 5 min to deploy, we had to wait 10 minutes because we couldn't run two cloud stages in parallel that runs multiple services stages also in parallel:

So I built one parallel stage with all 10 services, and named each step with a relevant prefix:

 

stage('Deploy') {
                    steps {
                        script {
                            def parallelStagesMap = [:]
                                AWS_CLUSTER_NAME.collect {
                                   parallelStagesMap.put("AWS ${it}", buildUtils.Deploy('AWS', it))
                                }
                                GCP_CLUSTER_NAME.collect {
                                   parallelStagesMap.put("GCP ${it}", buildUtils.Deploy('GCP', it))
                                }

                             parallel parallelStagesMap
                        }
                    }
                }

 

 

 

 

ianfixes@gmail.com (JIRA)

unread,
Dec 18, 2019, 9:19:21 AM12/18/19
to jenkinsc...@googlegroups.com
Ian Katz commented on Bug JENKINS-54010

That's a good workaround. I can't speak for everyone here, but my personal interest in this issue is more to do with the first attached screenshot in the original issue – the ability to have multiple sequential stages shown in each parallel branch.

ianfixes@gmail.com (JIRA)

unread,
Dec 18, 2019, 9:19:33 AM12/18/19
to jenkinsc...@googlegroups.com
Ian Katz edited a comment on Bug JENKINS-54010
That's a good workaround.  I can't speak for everyone here, but my personal interest in this issue is more to do with the first attached screenshot in the original issue -- the ability to have multiple sequential stages shown in each parallel branch.      In other words, to not have it all squish horizontally into a single stage that represents the entire branch.

shanidar1@gmail.com (JIRA)

unread,
Dec 18, 2019, 9:38:05 AM12/18/19
to jenkinsc...@googlegroups.com

Ian Katz I agree, I'm also waiting for this solution, that's what I did meanwhile until this issue will be resolved..

rvnair.235@gmail.com (JIRA)

unread,
Feb 27, 2020, 5:17:04 AM2/27/20
to jenkinsc...@googlegroups.com

Is there a plan to update the visualization as shown in parallel-stages-new.png ? Or is it available in already in any of the new releases ?

This message was sent by Atlassian Jira (v7.13.12#713012-sha1:6e07c38)
Atlassian logo
Reply all
Reply to author
Forward
0 new messages