[JIRA] [pipeline-stage-view-plugin] (JENKINS-33185) Visualize parallel steps within a Pipeline Stage

388 views
Skip to first unread message

pwolf@cloudbees.com (JIRA)

unread,
Feb 26, 2016, 1:02:01 PM2/26/16
to jenkinsc...@googlegroups.com
Patrick Wolf created an issue
 
Jenkins / Story JENKINS-33185
Visualize parallel steps within a Pipeline Stage
Issue Type: Story Story
Assignee: Antonio Muñiz
Attachments: parallel.png
Components: pipeline-stage-view-plugin
Created: 26/Feb/16 6:01 PM
Priority: Minor Minor
Reporter: Patrick Wolf

1) Parallel steps are the common workflow case
2) Build logs for parallel steps are not informative
3) A user would expect Stage View to somehow address the issue

Current state:

  • stage declarations within parallel sections are not being considered as parallel
  • steps within one parallelized step are being mixed => no info in the view (see the screenshot)

Test script (move stage declarations where you want):

for (int i=0; i< 2; ++i) {
  stage "Stage #"+i
  print 'Hello, world $i!'
}

stage "Stage Parallel"
def branches = [:]
for (int i = 0; i < numHelloMessages.toInteger(); i++) {
  branches["split${i}"] = {
    stage "Stage parallel- #"+i
    node('remote') {
     echo  'Starting sleep'
     sleep 10
     echo  'Finished sleep'
    }
  }
}
parallel branches

comment from JGlick:
So concretely: the stage view would, for a given build (row), continue to display a linear sequence of stages (columns) with their associated times and so on; but each cell (stage of a build) would be some rendering of an st-planar graph, with structure determined according to the presence of steps of interest like parallel or a proposed label. Sort of like the existing Workflow Steps, but rendered as a graph rather than a tree table, restricted to displaying certain steps, and divided by stage.
A simpler request would be to only display branches of one top-level parallel step, if any. (I.e., an st-planar graph which is either trivial—one vertex—or where all vertices other than the source and sink are directly connected to both the source and the sink.) This more limited UI would match that of the Build Pipeline plugin.
Note that there is also a request JENKINS-29892 which would allow several stages to share concurrency semantics, but I think that need not have any effect on visualization; it would just be an internal change in the stage step.
Add Comment Add Comment
 
This message was sent by Atlassian JIRA (v6.4.2#64017-sha1:e244265)
Atlassian logo

jdumay@cloudbees.com (JIRA)

unread,
Feb 26, 2016, 7:41:03 PM2/26/16
to jenkinsc...@googlegroups.com
James Dumay updated an issue

Tables make a lot of sense when you want scalability and easy comparison between each row (for example, comparing duration, which seems to be a key feature of the stage view). Adding parallel steps in a "cell" will destroy that scalability and start making it negatively information dense.

We've been thinking of representing single pipeline results like the screenshot below. Timings would either be a detail on hover or shown on a seperate graph (graphs are way better at visualising time comparisons).

For each run of the pipeline we would show something like:

Example of stage duration comparison graph:
Change By: James Dumay
Attachment: stage-duration.png
Attachment: pipeline.png

jdumay@cloudbees.com (JIRA)

unread,
Feb 26, 2016, 7:42:04 PM2/26/16
to jenkinsc...@googlegroups.com
James Dumay edited a comment on Story JENKINS-33185
 
Re: Visualize parallel steps within a Pipeline Stage
I am asking for feedback so we can move forward on this issue
Tables make a lot of sense when you want scalability and easy comparison between each row (for example, comparing duration, which seems to be a key feature of the stage view). Adding parallel steps in a "cell" will destroy that scalability and start making it negatively information dense.

We've been thinking of representing single pipeline results like the screenshot below. Timings would either be a detail on hover or shown on a seperate graph (graphs are way better at visualising time comparisons). 

For each run of the pipeline we would show something like:
!pipeline.png |thumbnail !


Example of stage duration comparison graph:
!stage-duration.png |thumbnail !

jdumay@cloudbees.com (JIRA)

unread,
Feb 26, 2016, 7:45:02 PM2/26/16
to jenkinsc...@googlegroups.com
James Dumay edited a comment on Story JENKINS-33185
I am asking for feedback so we can move forward on this issue

Tables make a lot of sense when you want  scalability  scanability  and  easy  comparison between each row (for example, comparing duration, which seems to be a key feature of the stage view). Adding parallel steps in a "cell" will destroy that  scalability  scanability  and start making  it  the view  negatively information dense.


We've been thinking of representing single pipeline results like the screenshot below. Timings would either be a detail on hover or shown on a seperate graph (graphs are way better at visualising time comparisons). 

For each run of the pipeline we would show something like:
!pipeline.png!


Example of stage duration comparison graph:
!stage-duration.png!

jdumay@cloudbees.com (JIRA)

unread,
Feb 26, 2016, 7:46:01 PM2/26/16
to jenkinsc...@googlegroups.com
James Dumay edited a comment on Story JENKINS-33185
I am asking for feedback so we can move forward on this issue

Tables make a lot of sense when you want scanability and comparison between each row (for example, comparing duration, which seems to be a key feature of the stage view). Adding parallel steps in a "cell" will destroy that scanability and start making the view negatively information dense.


We've been thinking of representing single pipeline results like the screenshot below. Timings would either be a detail on hover or shown on a seperate graph (graphs are way better at visualising time comparisons). 

For each run of the pipeline we would show something like:
!pipeline.png!

Example of stage duration comparison graph : . X=Pipeline runs going forward in time and Y=duration of each stage
!stage-duration.png!

tyler@monkeypox.org (JIRA)

unread,
Mar 1, 2016, 1:40:02 PM3/1/16
to jenkinsc...@googlegroups.com

James Dumay I like your graph/timeline proposal, it makes sense to me.

amuniz@cloudbees.com (JIRA)

unread,
Mar 2, 2016, 3:51:08 AM3/2/16
to jenkinsc...@googlegroups.com
Antonio Muñiz assigned an issue to Unassigned
 
Change By: Antonio Muñiz
Assignee: Antonio Muñiz

svanoort@cloudbees.com (JIRA)

unread,
Apr 5, 2016, 3:11:03 PM4/5/16
to jenkinsc...@googlegroups.com
Sam Van Oort commented on Story JENKINS-33185
 
Re: Visualize parallel steps within a Pipeline Stage

I like the notion of something like the first one as an expansion option to get details. I think it might make sense to integrate it like so: hook into the front-end extension point API (JS), add a widget to indicate presence of parallel stages within a stage, then mouseover or click expands to the mini flow-graph for a stage.

jglick@cloudbees.com (JIRA)

unread,
Apr 8, 2016, 3:24:02 PM4/8/16
to jenkinsc...@googlegroups.com

While such a view could be implemented using the currently available flow graphs, restricting itself to display of parallel branches, JENKINS-26107 would offer more flexibility.

svanoort@cloudbees.com (JIRA)

unread,
Apr 8, 2016, 5:46:03 PM4/8/16
to jenkinsc...@googlegroups.com

svanoort@cloudbees.com (JIRA)

unread,
Apr 8, 2016, 5:46:04 PM4/8/16
to jenkinsc...@googlegroups.com
Sam Van Oort edited a comment on Story JENKINS-33185

isctsf@gmail.com (JIRA)

unread,
Apr 28, 2016, 7:54:02 AM4/28/16
to jenkinsc...@googlegroups.com

I would be happy with something more simple such as this:

https://wiki.jenkins-ci.org/download/attachments/69764097/screenshot2.png?version=1&modificationDate=1400172472000

...where the box indicating the stage simply displays the number of parallel steps that it will run. Bonus points for each step having it's own progress bar!

michael.neale@gmail.com (JIRA)

unread,
May 1, 2016, 7:59:02 PM5/1/16
to jenkinsc...@googlegroups.com

I am not sure Philip if those items inside the stages in that deploy pipeline view are parallel or just jobs (steps?)

mario.goorden@gmail.com (JIRA)

unread,
May 2, 2016, 9:08:05 AM5/2/16
to jenkinsc...@googlegroups.com

It would be nice if each job has it's own 'live' logging view which is accessible from the single pipeline view as suggested by James Dumay.
Now all logging of each parallel job end up in one log file, which makes analyzing a specific job tricky.

isctsf@gmail.com (JIRA)

unread,
May 2, 2016, 9:32:01 AM5/2/16
to jenkinsc...@googlegroups.com

Michael Neale isn't a "step" still just a step, regardless if it is defined to run in parallel or not?

From a basic user-feedback perspective, what I want to see is:
1. Tell me what stages are defined, which one is running now, and what time it has taken or estimated to complete. (I think the existing graphics and animations are all adequate in this area).
2. If there are multiple steps inside the stage, tell me what step we are on, what time it has taken or estimated to complete. (This is what I am missing)

Since pipeline stage-progress works from left to right, I'd like to see step-progress displayed from top to bottom for that specific pipeline animation.

It doesn't concern me that the pipeline animation would then become be a dynamic height (as it is already dynamic length). Maybe it's a good option to make a check-box in the job configuration that I can click that shows or hides the steps inside the stages.

I tend to work with large and complex pipelines (Currently on a project with > 880 git repos) so when I'm introducing CI/CD to different audiences these graphical front ends are really important when it comes to demo time so that the white-board concepts make sense with what I'm creating in the tools.

uncodead@gmail.com (JIRA)

unread,
May 4, 2016, 10:37:03 AM5/4/16
to jenkinsc...@googlegroups.com

I missed this feature in my workflow. I have steps in parallel and i would like to see parallel in a same column

peter.gottinger@sap.com (JIRA)

unread,
May 11, 2016, 1:21:02 AM5/11/16
to jenkinsc...@googlegroups.com

I would also love to see the visualization of parallel build steps in the plugin.

jdumay@cloudbees.com (JIRA)

unread,
May 12, 2016, 11:58:05 AM5/12/16
to jenkinsc...@googlegroups.com

We will be releasing a new pipeline visualisation this month into open source. Its not related to the stage view plugin but we could look at improving it to fit here too. Ill update this ticket when we can show more.

jdumay@cloudbees.com (JIRA)

unread,
May 12, 2016, 11:58:07 AM5/12/16
to jenkinsc...@googlegroups.com
James Dumay edited a comment on Story JENKINS-33185
We will be releasing a new pipeline visualisation  that handles parallel  this month into open source. Its not related to the stage view plugin but we could look at improving it to fit here too. Ill update this ticket when we can show more.

DanaGoyette@gmail.com (JIRA)

unread,
May 12, 2016, 3:04:02 PM5/12/16
to jenkinsc...@googlegroups.com

I hope you'll also include visualization for steps within a parallel block.

I'd like to be able to replace Matrix Jobs with equivalent workflows / pipelines, and that requires one big parallel block with a bunch of rows of the same steps (with different parameters).

andrew.bayer@gmail.com (JIRA)

unread,
May 12, 2016, 3:07:05 PM5/12/16
to jenkinsc...@googlegroups.com

Dana Goyette - that's exactly what this is about. =)

DanaGoyette@gmail.com (JIRA)

unread,
May 12, 2016, 3:38:02 PM5/12/16
to jenkinsc...@googlegroups.com

Some of the concepts above (such as the "on click" screenshot) just didn't seem tailored well to that case; hence my comment. That's all.

wynne.jg@gmail.com (JIRA)

unread,
May 19, 2016, 8:31:12 AM5/19/16
to jenkinsc...@googlegroups.com

I agree with Dana Goyette here. Not sure how feasible/scalable it would be, but here's a mockup that I think could work well with the Pipeline Stage View Plugin
http://imgur.com/riSCG2y!http://imgur.com/riSCG2y!

wynne.jg@gmail.com (JIRA)

unread,
May 19, 2016, 8:33:11 AM5/19/16
to jenkinsc...@googlegroups.com
John Wynne edited a comment on Story JENKINS-33185
I agree with [~danagoyette] here. Not sure how feasible/scalable it would be, but here's a mockup that I think could work well with the Pipeline Stage View Plugin
 http://imgur.com/riSCG2y !http://imgur.com/riSCG2y!

wynne.jg@gmail.com (JIRA)

unread,
May 19, 2016, 8:33:11 AM5/19/16
to jenkinsc...@googlegroups.com
John Wynne edited a comment on Story JENKINS-33185
I agree with [~danagoyette] here. Not sure how feasible/scalable it would be, but here's a mockup that I think could work well with the Pipeline Stage View Plugin
!http:// i. imgur.com/riSCG2y .png !

svanoort@cloudbees.com (JIRA)

unread,
May 19, 2016, 10:19:11 AM5/19/16
to jenkinsc...@googlegroups.com

John Wynne I really like the look of this – I was planning to go more or less that route when adding this (I am still working on some of the backend components needed to support this functionality at an API level).

I might play with the formatting a bit to ensure the parallel branches within a stage are still associated to the stage they belong to (maybe create an outline around the stage).

kieranwebber@gmail.com (JIRA)

unread,
May 19, 2016, 10:52:08 AM5/19/16
to jenkinsc...@googlegroups.com

John Wynne That's the exact image i had mentally when first looking around this subject so +1 for a design along the lines of that

andrew.melo@gmail.com (JIRA)

unread,
May 19, 2016, 10:58:02 AM5/19/16
to jenkinsc...@googlegroups.com

How would that visualization look if you have tons of things running in parallel that aren't necessarily related? For instance, I run the packaging step immediately, with the tests in parallel (to reduce the overall latency). Would I end up with one very tall column?

svanoort@cloudbees.com (JIRA)

unread,
May 19, 2016, 11:33:02 AM5/19/16
to jenkinsc...@googlegroups.com

Andrew Melo At first it would probably do that for the initial version (tall windows), just to get a minimum-viable implementation out.
One enhancement beyond that would be to switch to an accordion view if the number of parallel branches was bigger than some threshold, showing just a count of parallel branches at the top and a button to expand.

Alternately, we might show basic stats or icons at the top of the rollup bar to give a fast visual indicator of branch status, something like: Succeeded / Running / Failed / Total - that approach scales well to numerous branches... but for now, high concurrency within parallel blocks is somewhat limited.

svanoort@cloudbees.com (JIRA)

unread,
May 19, 2016, 8:02:02 PM5/19/16
to jenkinsc...@googlegroups.com

Worth noting if you do display parallel branches in this fashion: it is critical to sort parallel forks by name (sorted alphabetically) since the actual number of branches and evaluation order can be completely random, so we must impose absolute ordering. This is because branches are received in a hashmap, and iterator order is dependent on the hash.

barry.oneill@spartasystems.com (JIRA)

unread,
Jul 27, 2016, 10:56:04 AM7/27/16
to jenkinsc...@googlegroups.com

Is there any possibility that this issue will be picked up? We're stuck in the old pipeline world and would love to use the new 2.0 pipeline, but people here are very fond of the visual representation of the parallel jobs that the older pipeline plugins offer.
This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c)
Atlassian logo

john.d.ament@gmail.com (JIRA)

unread,
Jul 27, 2016, 11:02:03 AM7/27/16
to jenkinsc...@googlegroups.com

In addition, since the old pipeline doesn't work in Jenkins 2, that blocks upgrades to newer Jenkins releases

sgiterman@gmail.com (JIRA)

unread,
Jul 27, 2016, 5:52:03 PM7/27/16
to jenkinsc...@googlegroups.com
gitt commented on Story JENKINS-33185

You can try the new Jenkins Pipeline GUI: https://jenkins.io/projects/blueocean/. It has already support for the parallel steps visualization.

It is available in the experimental update center: http://updates.jenkins-ci.org/experimental/update-center.json

sgiterman@gmail.com (JIRA)

unread,
Jul 27, 2016, 5:55:17 PM7/27/16
to jenkinsc...@googlegroups.com
gitt edited a comment on Story JENKINS-33185
You can try the new Jenkins Pipeline GUI: https://jenkins.io/projects/blueocean/. It has already support for the parallel steps visualization.
The Project Blue Ocean is still in alpha, but is under active development: https://jenkins.io/blog/2016/07/19/blue-ocean-update/. It is available in the experimental update center: http://updates.jenkins-ci.org/experimental/update-center.json .

GitHub: https://github.com/jenkinsci/blueocean-plugin

svanoort@cloudbees.com (JIRA)

unread,
Jul 27, 2016, 7:23:20 PM7/27/16
to jenkinsc...@googlegroups.com
Sam Van Oort assigned an issue to Sam Van Oort
 
Jenkins / Story JENKINS-33185
Change By: Sam Van Oort
Assignee: Sam Van Oort

svanoort@cloudbees.com (JIRA)

unread,
Jul 27, 2016, 7:23:21 PM7/27/16
to jenkinsc...@googlegroups.com
Sam Van Oort started work on Story JENKINS-33185
 
Change By: Sam Van Oort
Status: Open In Progress

barry.oneill@spartasystems.com (JIRA)

unread,
Jul 28, 2016, 8:48:02 AM7/28/16
to jenkinsc...@googlegroups.com
Barry O'Neill commented on Story JENKINS-33185
 
Re: Visualize parallel steps within a Pipeline Stage

Wow, looks slick. I'm not sure it's stable enough for us to use for our main CI just yet, but that definitely is more along the lines of what we're looking for. Thanks for the update!

svanoort@cloudbees.com (JIRA)

unread,
Jul 30, 2016, 12:19:03 PM7/30/16
to jenkinsc...@googlegroups.com

John Ament Wait, what's the issue you're experiencing with Jenkins 2.0? This is supposed to be 100% compatible, and I haven't seen any JIRAs opened about an issue. If there is one, it will be a very high priority fix.

Barry O'Neill I've updated status to reflect that we are indeed working towards this. It's not forgotten, not abandoned, just far more work to implement than anyone would expect without seeing the code or edge cases. I think currently I've currently invested ~1-2 months.

The challenge is that we did not have a common set of APIs for decomposing the directed acyclic graphs from a pipeline execution, and have hit multiple problems like this one that are painful without them. So, I've had to write those components to provide the functionality we need. And they need to handle a lot of nasty, complex edge cases to work correctly – for example: nested parallel blocks, where both parallel blocks are incomplete.

svanoort@cloudbees.com (JIRA)

unread,
Jul 30, 2016, 12:20:05 PM7/30/16
to jenkinsc...@googlegroups.com
Sam Van Oort edited a comment on Story JENKINS-33185
[~johnament]  Wait, what's the issue you're experiencing with Jenkins 2.0?  This is supposed to be 100% compatible, and I haven't seen any JIRAs opened about an issue.  If there is one, it will be a very high priority fix.

[~barryoneill]  I've updated status to reflect that we are indeed working towards this.  It's not forgotten, not abandoned, just far more work to implement than anyone would expect without seeing the code or edge cases.  I think currently I've currently invested ~1-2 months.

The challenge is that we did not have a common set of APIs for decomposing the directed acyclic graphs from a pipeline execution, and have hit multiple problems like this one that are painful without them.  So, I've had to write those components to provide the functionality we need.  And they need to handle a lot of nasty, complex edge cases to work correctly -- for example: nested parallel blocks, where both parallel blocks are incomplete.      

The current plan is that the BlueOcean UI linked above will also consume the same APIs to provide additional functionality.

raphink@gmail.com (JIRA)

unread,
Sep 8, 2016, 7:16:11 AM9/8/16
to jenkinsc...@googlegroups.com

Slawa Giterman the Jenkins Pipeline GUI is great, but it still doesn't show steps inside each parallel branch, even though the syntax is accepted. Do you know if that is planned?

piotrzsl@gmail.com (JIRA)

unread,
Sep 19, 2016, 4:23:44 PM9/19/16
to jenkinsc...@googlegroups.com

I see 2 issues with that plugin in latest version:

1. Step time is invalid, like it stop counting when all paraell tasks are started, it looks like step finish but in real job is still waiting for finish of parallel taks until move to next step.
2. No option to scroll to see older builds, it shows only last ~20, but cannot see older one. Would be nice to see same info when entering a specific build page.

Except that, I like it..

svanoort@cloudbees.com (JIRA)

unread,
Sep 19, 2016, 4:28:06 PM9/19/16
to jenkinsc...@googlegroups.com

Piotr Zegar Do you mean the Blue-Ocean plugin? Stage View? Pipeline view plugin? Build Pipeline Plugin?

These are all different things.

svanoort@cloudbees.com (JIRA)

unread,
Sep 19, 2016, 5:03:17 PM9/19/16
to jenkinsc...@googlegroups.com
Sam Van Oort stopped work on Story JENKINS-33185
 
Change By: Sam Van Oort
Status: In Progress Open

sorin.sbarnea@gmail.com (JIRA)

unread,
Oct 4, 2016, 6:33:09 AM10/4/16
to jenkinsc...@googlegroups.com

What would be the current expect way to define a job with multiple parallel tasks if we cannot use stage inside it? How we are going to distinguish between "parallel-build-foo" and "parallel-build-bar"?

aburdukovskiy@blizzard.com (JIRA)

unread,
Oct 4, 2016, 1:26:05 PM10/4/16
to jenkinsc...@googlegroups.com
Anton B commented on Story JENKINS-33185

I imagine that since the parallel step takes a map, the keys can act as identifiers for the parallel tasks.

svanoort@cloudbees.com (JIRA)

unread,
Oct 4, 2016, 1:33:14 PM10/4/16
to jenkinsc...@googlegroups.com

Sorin Sbarnea The answer from Anton B is correct. The block-scoped version of the stage step, i.e. 

stage ('myStageName') {
  // do stuff here
}

can also be included in parallels. However there is not a UI visualization for that approach yet.

mneale@cloudbees.com (JIRA)

unread,
Oct 4, 2016, 6:51:07 PM10/4/16
to jenkinsc...@googlegroups.com

The name of the parallel "branches" do indeed look like stages, and provide and identifier (this shows up int he log as a prefix, and also in blue ocean visualisation as branches of execution).

Sam Van Oort I wouldn't recomment putting stages inside of things until we decide how we want to visualise it.

svanoort@cloudbees.com (JIRA)

unread,
Oct 4, 2016, 7:02:27 PM10/4/16
to jenkinsc...@googlegroups.com

Indeed – the branch name is the identifier in this case, it's up in the air if stages inside parallels will be supported in the UI or not. The syntax allows it, but just because you can doesn't mean you should!

george.tutoveanu@gmail.com (JIRA)

unread,
Oct 11, 2016, 3:40:12 AM10/11/16
to jenkinsc...@googlegroups.com
George Tutoveanu updated an issue
 
Change By: George Tutoveanu
{color:red}colored text{color} 1) Parallel steps are the common workflow case
2) Build logs for parallel steps are not informative
3) A user would expect Stage View to somehow address the issue

Current state:
- stage declarations within parallel sections are not being considered as parallel
- steps within one parallelized step are being mixed => no info in the view (see the screenshot)

Test script (move stage declarations where you want):
{noformat}
for (int i=0; i< 2; ++i) {
  stage "Stage #"+i
  print 'Hello, world $i!'
}

stage "Stage Parallel"
def branches = [:]
for (int i = 0; i < numHelloMessages.toInteger(); i++) {
  branches["split${i}"] = {
    stage "Stage parallel- #"+i
    node('remote') {
     echo  'Starting sleep'
     sleep 10
     echo  'Finished sleep'
    }
  }
}
parallel branches
{noformat}

comment from JGlick:
So concretely: the stage view would, for a given build (row), continue to display a linear sequence of stages (columns) with their associated times and so on; but each cell (stage of a build) would be some rendering of an st-planar graph, with structure determined according to the presence of steps of interest like parallel or a proposed label. Sort of like the existing Workflow Steps, but rendered as a graph rather than a tree table, restricted to displaying certain steps, and divided by stage.
A simpler request would be to only display branches of one top-level parallel step, if any. (I.e., an st-planar graph which is either trivial—one vertex—or where all vertices other than the source and sink are directly connected to both the source and the sink.) This more limited UI would match that of the Build Pipeline plugin.
Note that there is also a request JENKINS-29892 which would allow several stages to share concurrency semantics, but I think that need not have any effect on visualization; it would just be an internal change in the stage step.

george.tutoveanu@gmail.com (JIRA)

unread,
Oct 11, 2016, 3:41:56 AM10/11/16
to jenkinsc...@googlegroups.com

jakub.rusiecki@gmail.com (JIRA)

unread,
Nov 13, 2016, 6:20:10 AM11/13/16
to jenkinsc...@googlegroups.com
Jakub Rusiecki commented on Story JENKINS-33185
 
Re: Visualize parallel steps within a Pipeline Stage

To recap visualization needs I's suggest swimlines (it is simpler that rectangles in rectangles..) :
1. branches of a stage
Stage can fork into parallel branches (as described above) - these need parallel visualization by a) names described in parallel statement and b) their times.
Stage time shall be actual time incl. branches.
Visualization should make swimline for each branch below stage they fork.
2. branch stages
Each branch may have stages - time of branch shall include sum of it's steps and stages.
Visualization should make swimline for each stage below branch they are included in.
3. sub-stages
Each stage (e.g. A) can have sub-stages (e.g. A1 and A2) - Visualization of substages (A1, A2) as swimline below main stage (A) so to show that their times do not add but run in parallel.

jakub.rusiecki@gmail.com (JIRA)

unread,
Nov 13, 2016, 6:20:21 AM11/13/16
to jenkinsc...@googlegroups.com
Jakub Rusiecki edited a comment on Story JENKINS-33185
To recap visualization needs I' s d suggest swimlines (it is simpler that rectangles in rectangles..) :
1. *branches of a stage*

Stage can fork into parallel branches (as described above) - these need parallel visualization by a) names described in parallel statement and b) their times.
Stage time shall be actual time incl. branches.
Visualization should make swimline for each branch below stage they fork.
2. *branch stages*
Each branch may have stages - time of branch shall include sum of it's steps and stages.
Visualization should make swimline for each stage below branch they are included in.
3. *sub-stages*

Each stage (e.g. A) can have sub-stages (e.g. A1 and A2) - Visualization of substages (A1, A2) as swimline below main stage (A) so to show that their times do not add but run in parallel.

jakub.rusiecki@gmail.com (JIRA)

unread,
Nov 13, 2016, 6:22:15 AM11/13/16
to jenkinsc...@googlegroups.com
Jakub Rusiecki edited a comment on Story JENKINS-33185
To recap visualization needs I'd suggest swimlines (it is simpler that rectangles in rectangles..) :
1. *branches of a stage*
Stage can fork into parallel branches (as described above) - these need parallel visualization by a) names described in parallel statement and b) their times.
Stage time shall be actual time incl. branches.
Visualization should make swimline for each branch below stage they fork.
2. *branch stages*
Each branch may have stages - time of branch shall include sum of it's steps and stages.
Visualization should make swimline for each stage below branch they are included in.
3. *sub-stages*
Each stage (e.g. A) can have sub-stages (e.g. A1 and A2) - Visualization of substages (A1, A2) as one swimline for all substages one-after-another below main stage (A) so to show that their substage times do not add but run are included in parallel stage time .

lgawel@cayan.com (JIRA)

unread,
Nov 15, 2016, 4:15:07 PM11/15/16
to jenkinsc...@googlegroups.com

Do we have a timeline for this fix?

jdumay@cloudbees.com (JIRA)

unread,
Nov 20, 2016, 5:07:22 PM11/20/16
to jenkinsc...@googlegroups.com
James Dumay updated an issue
 
Change By: James Dumay
{panel:title=Support for visualizing parallel steps is unplanned|titleBGColor=lightgrey}
A common thread in the feedback we have received is to represent parallel steps in the Stage View as cells within the Stage column.

The use cases supported by Pipeline Stage View, such as visualizing stages across multiple runs and detecting changes in execution time or stages that fail regularly, work well due to the use of a table to visualize this data. Tables are excellent at allowing the eye to scan over a lot of related data.

There has some debate within CloudBees on how parallel could be added to Pipeline Stage View without breaking the advantages that a table brings to the visualization. We believe that adding additional cells within the table column would reduce the usability of the overall experience as scanning many rows becomes much more difficult.

From a technical point of view it would be non-trivial to retrofit the Stage View with the parallel information.

For the time being solving this problem has been delayed by CloudBees and we encourage any community members who are interested in solving this problem to pick it up.

Over the last year CloudBees and the Jenkins community have been heavily invested in bringing a new user interface to Jenkins that is designed for Pipeline called [Blue Ocean|https://jenkins.io/projects/blueocean/].

Blue Ocean [offers an entirely new way of visualizing Pipelines|https://jenkins.io/projects/blueocean/#pipelines], including parallel steps, and makes it much easier to navigate the Pipelines log by Stage and Step.

We aim that over time that many of the use cases you have come to enjoy in Stage View will be available in Blue Ocean. We would like to encourage people interested in this feature to try out Blue Ocean and provide us feedback.

Thanks,
James Dumay
jdu...@cloudbees.com
{panel}


*Original Request*

jdumay@cloudbees.com (JIRA)

unread,
Nov 20, 2016, 5:08:40 PM11/20/16
to jenkinsc...@googlegroups.com
James Dumay updated an issue
{panel:title=Support for visualizing parallel steps is unplanned|titleBGColor=lightgrey}
A common thread in the feedback we have received is to represent parallel steps in the Stage View as cells within the Stage column.

The use cases supported by Pipeline Stage View, such as visualizing stages across multiple runs and detecting changes in execution time or stages that fail regularly, work well due to the use of a table to visualize this data. Tables are excellent at allowing the eye to scan over a lot of related data.

There has some debate within CloudBees on how parallel could be added to Pipeline Stage View without breaking the advantages that a table brings to the visualization. We believe that adding additional cells within the table column would reduce the usability of the overall experience as scanning many rows becomes much more difficult.

From a technical point of view it would be non-trivial to retrofit the Stage View with the parallel information.

For the time being solving this problem has been delayed by CloudBees and we encourage any community members who are interested in solving this problem to pick it up.

Over the last year CloudBees and the Jenkins community have been heavily invested in bringing a new user interface to Jenkins that is designed for Pipeline called [Blue Ocean|https://jenkins.io/projects/blueocean/].

Blue Ocean [offers an entirely new way of visualizing Pipelines|https://jenkins.io/projects/blueocean/#pipelines], including parallel steps, and makes it much easier to navigate the Pipelines log by Stage and Step.

jdumay@cloudbees.com (JIRA)

unread,
Nov 20, 2016, 5:10:04 PM11/20/16
to jenkinsc...@googlegroups.com
James Dumay updated an issue
{panel:title=Support for visualizing parallel steps is unplanned|titleBGColor=lightgrey}
A common thread in the feedback we have received is to represent parallel steps in the Stage View as cells within the Stage column.

The use cases supported by Pipeline Stage View, such as visualizing stages across multiple runs and detecting changes in execution time or stages that fail regularly, work well due to the use of a table to visualize this data. Tables are excellent at allowing the eye to scan over a lot of related data.

There has some debate within CloudBees on how parallel could be added to Pipeline Stage View without breaking the advantages that a table brings to the visualization. We believe that adding additional cells within the table column would reduce the usability of the overall experience as scanning many rows becomes much more difficult.

From a technical point of view it would be non-trivial to retrofit the Stage View with the parallel information.

For the time being solving this problem has been delayed by CloudBees and we encourage any community members who are interested in solving this problem to pick it up.

Over the last year CloudBees and the Jenkins community have been heavily invested in bringing a new user interface to Jenkins that is designed for Pipeline called _* [Blue Ocean|https://jenkins.io/projects/blueocean/] *_ .

https://jenkins.io/projects/blueocean/

 Blue Ocean _* [offers an entirely new way of visualizing Pipelines|https://jenkins.io/projects/blueocean/#pipelines] , *_ including parallel steps, and makes it much easier to navigate the Pipelines log by Stage and Step.

jdumay@cloudbees.com (JIRA)

unread,
Nov 20, 2016, 5:10:06 PM11/20/16
to jenkinsc...@googlegroups.com
James Dumay updated an issue
{panel:title=Support for visualizing parallel steps is unplanned|titleBGColor=lightgrey}
A common thread in the feedback we have received is to represent parallel steps in the Stage View as cells within the Stage column.

The use cases supported by Pipeline Stage View, such as visualizing stages across multiple runs and detecting changes in execution time or stages that fail regularly, work well due to the use of a table to visualize this data. Tables are excellent at allowing the eye to scan over a lot of related data.

There has some debate within CloudBees on how parallel could be added to Pipeline Stage View without breaking the advantages that a table brings to the visualization. We believe that adding additional cells within the table column would reduce the usability of the overall experience, as scanning many rows becomes much more difficult.


From a technical point of view it would be non-trivial to retrofit the Stage View with the parallel information.

For the time being solving this problem has been delayed by CloudBees and we encourage any community members who are interested in solving this problem to pick it up.

Over the last year CloudBees and the Jenkins community have been heavily invested in bringing a new user interface to Jenkins that is designed for Pipeline called _*[Blue Ocean|https://jenkins.io/projects/blueocean/]*_.

Blue Ocean _*[offers an entirely new way of visualizing Pipelines including parallel steps |https://jenkins.io/projects/blueocean/#pipelines including parallel steps ]*_, and makes it much easier to navigate the Pipelines log by Stage and Step.

jdumay@cloudbees.com (JIRA)

unread,
Nov 20, 2016, 5:11:06 PM11/20/16
to jenkinsc...@googlegroups.com
James Dumay updated an issue
{panel:title=Support for visualizing parallel steps is unplanned|titleBGColor=lightgrey}
A common thread in the feedback we have received is to represent parallel steps in the Stage View as cells within the Stage column.

The use cases supported by Pipeline Stage View, such as visualizing stages across multiple runs and detecting changes in execution time or stages that fail regularly, work well due to the use of a table to visualize this data. Tables are excellent at allowing the eye to scan over a lot of related data.

There has some debate within CloudBees on how parallel could be added to Pipeline Stage View without breaking the advantages that a table brings to the visualization. We believe that adding additional cells within the table column would reduce the usability of the overall experience, as scanning many rows becomes much more difficult.

From a technical point of view it would be non-trivial to retrofit the Stage View with the parallel information.

For the time being solving this problem has been delayed by CloudBees and we encourage any community members who are interested in solving this problem to pick it up.

Over the last year CloudBees and the Jenkins community have been heavily invested in bringing a new user interface to Jenkins that is designed for Pipeline called _*[Blue Ocean|https://jenkins.io/projects/blueocean/]*_.

Blue Ocean _*[offers an entirely new way of visualizing Pipelines including parallel steps|https://jenkins.io/projects/blueocean/#pipelines]*_ , and makes it much easier to navigate the Pipelines log by Stage and Step.

We aim that over time that many of the use cases you have come to enjoy in Stage View will be available in Blue Ocean. We would like to encourage people interested in this feature to try out Blue Ocean and provide us feedback.

You can find out more about Blue Ocean at https://jenkins.io/projects/blueocean/

jdumay@cloudbees.com (JIRA)

unread,
Nov 20, 2016, 5:12:05 PM11/20/16
to jenkinsc...@googlegroups.com
James Dumay updated an issue
{panel:title=Support for visualizing parallel steps is unplanned|titleBGColor=lightgrey}
A common thread in the feedback we have received is to represent parallel steps in the Stage View as cells within the Stage column.

The use cases supported by Pipeline Stage View, such as visualizing stages across multiple runs and detecting changes in execution time or stages that fail regularly, work well due to the use of a table to visualize this data. Tables are excellent at allowing the eye to scan over a lot of related data.

There has some debate within CloudBees on how parallel could be added to Pipeline Stage View without breaking the advantages that a table brings to the visualization. We believe that adding additional cells within the table column would reduce the usability of the overall experience, as scanning many rows becomes much more difficult.

From a technical point of view it would be non-trivial to retrofit the Stage View with the parallel information.

For the time being solving this problem has been delayed by CloudBees and . However, we encourage any community members who are interested in solving this problem to pick it up.

jdumay@cloudbees.com (JIRA)

unread,
Nov 20, 2016, 5:12:07 PM11/20/16
to jenkinsc...@googlegroups.com
James Dumay assigned an issue to CloudBees Inc.
Change By: James Dumay
Assignee: Sam Van Oort CloudBees Inc.

jdumay@cloudbees.com (JIRA)

unread,
Nov 20, 2016, 5:12:11 PM11/20/16
to jenkinsc...@googlegroups.com
James Dumay updated an issue
{panel:title=Support for visualizing parallel steps is unplanned|titleBGColor=lightgrey}
A common thread in the feedback we have received is to represent parallel steps in the Stage View as cells within the Stage column.

The use cases supported by Pipeline Stage View, such as visualizing stages across multiple runs and detecting changes in execution time or stages that fail regularly, work well due to the use of a table to visualize this data. Tables are excellent at allowing the eye to scan over a lot of related data.

There has some debate within CloudBees on how parallel could be added to Pipeline Stage View without breaking the advantages that a table brings to the visualization. We believe that adding additional cells within the table column would reduce the usability of the overall experience , as scanning many rows becomes much more difficult.


From a technical point of view it would be non-trivial to retrofit the Stage View with the parallel information.

For the time being solving this problem has been delayed by CloudBees and we encourage any community members who are interested in solving this problem to pick it up.


Over the last year CloudBees and the Jenkins community have been heavily invested in bringing a new user interface to Jenkins that is designed for Pipeline called _*[Blue Ocean|https://jenkins.io/projects/blueocean/]*_.

Blue Ocean _*[offers an entirely new way of visualizing Pipelines|https://jenkins.io/projects/blueocean/#pipelines]*_ including parallel steps, and makes it much easier to navigate the Pipelines log by Stage and Step.

We aim that over time that many of the use cases you have come to enjoy in Stage View will be available in Blue Ocean. We would like to encourage people interested in this feature to try out Blue Ocean and provide us feedback.

jdumay@cloudbees.com (JIRA)

unread,
Nov 20, 2016, 5:14:41 PM11/20/16
to jenkinsc...@googlegroups.com
James Dumay updated an issue
{panel:title=Support for visualizing parallel steps is unplanned|titleBGColor=lightgrey}
A common thread in the feedback we have received is to represent parallel steps in the Stage View as cells within the Stage column.

The use cases supported by Pipeline Stage View, such as visualizing stages across multiple runs and detecting changes in execution time or stages that fail regularly, work well due to the use of a table to visualize this data. Tables are excellent at allowing the eye to scan over a lot of related data.

There has some debate within CloudBees on how parallel could be added to Pipeline Stage View without breaking the advantages that a table brings to the visualization. We believe that adding additional cells within the table column would reduce the usability of the overall experience, as scanning many rows becomes much more difficult.


From a technical point of view it would be non-trivial to retrofit the Stage View with the parallel information.

For the time being solving this problem has been delayed by CloudBees and we encourage any community members who are interested in solving this problem to pick it up.

Over the last year CloudBees and the Jenkins community have been heavily invested in bringing a new user interface to Jenkins that is designed for Pipeline called _*[Blue Ocean|https://jenkins.io/projects/blueocean/]*_.

Blue Ocean _*[offers an entirely new way of visualizing Pipelines|https://jenkins.io/projects/blueocean/#pipelines ]*_ including parallel steps ]*_ , and makes it much easier to navigate the Pipelines log by Stage and Step.

elgalu3@gmail.com (JIRA)

unread,
Nov 21, 2016, 5:28:14 AM11/21/16
to jenkinsc...@googlegroups.com
Leo Gallucci commented on Story JENKINS-33185
 
Re: Visualize parallel steps within a Pipeline Stage

This missing feature might be one of main reasons why Jenkins is on HOLD in one of the most popular tech radars:
https://www.thoughtworks.com/radar/tools

jdumay@cloudbees.com (JIRA)

unread,
Nov 21, 2016, 5:32:07 AM11/21/16
to jenkinsc...@googlegroups.com

Leo Gallucci take a look at Blue Ocean - we're building an entirely new user experience for CD pipelines, including a visual editor for Pipeline.

jdumay@cloudbees.com (JIRA)

unread,
Nov 21, 2016, 5:33:07 AM11/21/16
to jenkinsc...@googlegroups.com
James Dumay updated an issue
Change By: James Dumay
Attachment: successful-pipeline.png

jdumay@cloudbees.com (JIRA)

unread,
Nov 21, 2016, 5:34:30 AM11/21/16
to jenkinsc...@googlegroups.com
James Dumay edited a comment on Story JENKINS-33185
 
Re: Visualize parallel steps within a Pipeline Stage
[~elgalu] take a look at [Blue Ocean|https://jenkins.io/projects/blueocean/] - we're building an entirely new user experience for CD pipelines, including a [visual editor for Pipeline|https://jenkins.io/blog/2016/09/19/blueocean-beta-declarative-pipeline-pipeline-editor/].

Here's a preview that includes parallel visualisation:
!successful-pipeline.png|thumbnail!

jdumay@cloudbees.com (JIRA)

unread,
Nov 21, 2016, 5:34:31 AM11/21/16
to jenkinsc...@googlegroups.com

jdumay@cloudbees.com (JIRA)

unread,
Nov 21, 2016, 5:35:07 AM11/21/16
to jenkinsc...@googlegroups.com
James Dumay edited a comment on Story JENKINS-33185
[~elgalu] take a look at [Blue Ocean|https://jenkins.io/projects/blueocean/] - we're building an entirely new user experience for CD pipelines, including a [visual editor for Pipeline|https://jenkins.io/blog/2016/09/19/blueocean-beta-declarative-pipeline-pipeline-editor/].

Here's a preview screenshot that includes parallel visualisation:
!successful-pipeline.png|thumbnail!

jdumay@cloudbees.com (JIRA)

unread,
Nov 21, 2016, 5:35:12 AM11/21/16
to jenkinsc...@googlegroups.com
James Dumay edited a comment on Story JENKINS-33185
[~elgalu] take a look at [Blue Ocean|https://jenkins.io/projects/blueocean/] - we're building an entirely new user experience for CD pipelines, including a [visual editor for Pipeline|https://jenkins.io/blog/2016/09/19/blueocean-beta-declarative-pipeline-pipeline-editor/].

Here's a preview that includes parallel visualisation:
!successful-pipeline.png |thumbnail !

sorin.sbarnea@gmail.com (JIRA)

unread,
Nov 21, 2016, 5:38:15 AM11/21/16
to jenkinsc...@googlegroups.com

Leo Gallucci Please remember that ThoughtWorks has its own CI/CD solutions so I would count their tech radar a biased one. Still, I have to confess that I partially support your view, I would love to be able to migrate OpenStack build jobs to pipelines but I doubt it would be possible in the next 1-2 years, and that's because they are not mature enough and because there were some design decisions regarding not supporting old jobs, making progressive transitions hardly possible.

Shortly, future is bright for small companies with simple workflows as they would be able to benefit from Jenkins pipelines. Others are kinda screwed as I don't see any migration scenario working at this moment.

elgalu3@gmail.com (JIRA)

unread,
Nov 21, 2016, 8:01:12 AM11/21/16
to jenkinsc...@googlegroups.com
Leo Gallucci updated an issue
 
Change By: Leo Gallucci
Attachment: blue-ocean-parallel-generated-view.png

elgalu3@gmail.com (JIRA)

unread,
Nov 21, 2016, 8:01:22 AM11/21/16
to jenkinsc...@googlegroups.com
Leo Gallucci commented on Story JENKINS-33185
 
Re: Visualize parallel steps within a Pipeline Stage

Yes Sorin Sbarnea I also thought about them being biased, I assisted their tech radar presentation and the topic came, the presenter assured they are not biased and they make a big effort to be objective and explained all the reasons behind it.

James Dumay Blue Ocean looked promising, I installed it. Why a visual editor? shouldn't the view be generated out of the Jenkinsfile? I thought we want pipeline as code.
So I tried to write `parallel` Jenkinsfiles and the generated view looks weird, both examples:

Left example

node() {
  stage('Build') {
    println 'I prepare the build for the parallel steps'
  }

  parallel (
   "chrome" : { println 'running tests in chrome' },
   "firefox" : { println 'running tests in firefox' },
   "edge" : { println 'running tests in edge' }
  )
}

Right example

node() {
  stage('Build') {
    println 'I prepare the build for the parallel steps'
  }

  parallel (
   "chrome" : { stage('chrome') { println 'running tests in chrome' } },
   "firefox" : { stage('firefox') { println 'running tests in firefox' } },
   "edge" : { stage('edge') { println 'running tests in edge' } }
  )
}

diemol@gmail.com (JIRA)

unread,
Nov 21, 2016, 8:17:19 AM11/21/16
to jenkinsc...@googlegroups.com
Diego Molina updated an issue
 
Change By: Diego Molina
Priority: Minor Major

diemol@gmail.com (JIRA)

unread,
Nov 21, 2016, 8:17:39 AM11/21/16
to jenkinsc...@googlegroups.com
Diego Molina updated an issue
Change By: Diego Molina
Priority: Major Minor

crummynz@gmail.com (JIRA)

unread,
Nov 21, 2016, 8:20:04 AM11/21/16
to jenkinsc...@googlegroups.com
node() {
  stage('Build') {
    println 'I prepare the build for
 the parallel steps'
  }

  stage('Test') {
   parallel (
   
"chrome" : { println 'running tests in chrome' },
   "firefox" : { println 'running tests in firefox' },
   "edge" : { println 'running tests in edge' }
  )
  }
}

elgalu3@gmail.com (JIRA)

unread,
Nov 21, 2016, 8:30:12 AM11/21/16
to jenkinsc...@googlegroups.com
Leo Gallucci updated an issue
 
Change By: Leo Gallucci
Attachment: blue-ocean-parallel-WORKS.png

elgalu3@gmail.com (JIRA)

unread,
Nov 21, 2016, 8:30:53 AM11/21/16
to jenkinsc...@googlegroups.com

diemol@gmail.com (JIRA)

unread,
Nov 21, 2016, 8:31:05 AM11/21/16
to jenkinsc...@googlegroups.com

I guess it is better to read a bit how it works before Leo Gallucci

attila.strba@esrlabs.com (JIRA)

unread,
Nov 21, 2016, 9:26:04 AM11/21/16
to jenkinsc...@googlegroups.com

Guys one more question, as far as I understood from Jesse Glick post here https://issues.jenkins-ci.org/browse/JENKINS-28293?focusedCommentId=277255&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-277255 the parallel steps visualisation should be also supported if stages are nested under the parallel step since stage accepts block parameter, correct?

Does this mean that the Blue Ocean plugin has to be still updated in order the visualisation displays it correctly?

pieterjan.busschaert@gmail.com (JIRA)

unread,
Nov 21, 2016, 9:29:06 AM11/21/16
to jenkinsc...@googlegroups.com

Does/will the visualization in BlueOcean support sub-stages in parallel?

I think we have a very common use-case, but it's not really supported in the current BlueOcean gui:

parallel (
 "win7-vs2012" : {node("vs2012") {stage("checkout") {...}; stage("build") {...}; stage("test") {...};} },
 "win10-vs2015" : {node ("vs2015") {stage("checkout") {...}; stage("build") {...}; stage("test") {...};} },
 "linux-gcc5" : {node ("gcc5") {stage("checkout") {...}; stage("build") {...}; stage("test") {...};} }
)
stage("email notifications") {...}

crummynz@gmail.com (JIRA)

unread,
Nov 21, 2016, 9:43:11 AM11/21/16
to jenkinsc...@googlegroups.com
malcolm crum updated an issue
 
Change By: malcolm crum
Attachment: Screen Shot 2016-11-21 at 15.41.33.png

crummynz@gmail.com (JIRA)

unread,
Nov 21, 2016, 9:43:33 AM11/21/16
to jenkinsc...@googlegroups.com
 
                                                            

                                                        
 
node() {
 stage('Build') 
 { println 'I prepare the build for the parallel steps' } 
 stage('Test') {
 parallel (
 "win7-vs2012" : { stage("checkout") { }; stage("build") { }; stage("test") { } },
 "win10-vs2015" : { stage("checkout") { }; stage("build") { }; stage("test") { }},
 "linux-gcc5" : { stage("checkout") { }; stage("build") { }; stage("test") { } }
 )
 }
 }
 {/code} 
I didn't see the stages in the parallel steps. I don't think that's supported in blue ocean right now.  
 		 
                                                
 
                                            
                                        
 		 
                                
crummynz@gmail.com (JIRA)
unread,
Nov 21, 2016, 9:43:55 AM11/21/16
to jenkinsc...@googlegroups.com


     
         
         
         
         
     
    
 
        
 
             
            
                
 
                     
                        
 
                            
                                
 
                                       
                                     malcolm crum edited a comment on Story JENKINS-33185  
                                
 
                            
                        
 		 
                
 
                
 
                     
                        
 
                            
 
                                     
                                        
 
                                            
                                                
 
                                                     [~pjaytycy], I just tested the following code:

{code}

node() {
  stage('Build') {
    println 'I prepare the build for the parallel steps'
  }

  stage('Test') {
   parallel (
 "win7-vs2012" : { stage("checkout") { }; stage("build") { }; stage("test") { } },
 "win10-vs2015" : { stage("checkout") { }; stage("build") { }; stage("test") { }},
 "linux-gcc5" : { stage("checkout") { }; stage("build") { }; stage("test") { } }
)
  }
}
{
 / code}

I didn't see the stages in the parallel steps. I don't think that's supported in blue ocean right now . :

   !Screen Shot 2016-11-21 at 15.41.33.png|thumbnail!  		 
                                                
 
                                            
                                        
 		 
                                
gboccard@gmail.com (JIRA)
unread,
Nov 21, 2016, 9:50:06 AM11/21/16
to jenkinsc...@googlegroups.com


     
         
         
         
         
     
    
svanoort@cloudbees.com (JIRA)
unread,
Nov 21, 2016, 10:53:06 AM11/21/16
to jenkinsc...@googlegroups.com


     
         
         
         
         
     
    
 
                                     
                                        
 
                                            
                                                
 
                                                     
Ping James Dumay and Vivek Pandey for questions/requests about Blue Ocean and parallels/nesting. 
 		 
                                                
 
                                            
                                        
 		 
                                
jdumay@cloudbees.com (JIRA)
unread,
Nov 21, 2016, 4:31:27 PM11/21/16
to jenkinsc...@googlegroups.com


     
         
         
         
         
     
    
 
                                     
                                        
 
                                            
                                                
 
                                                     
malcolm crum nested stages (anywhere where there is a outer and inner stage) as you've suggested can't be visualised in Blue Ocean as we haven't been able to design a visualisation that meets all users requirements and expectations. There have been many solutions suggested but none that fit. You can follow and vote for it at JENKINS-38442.
 		 
                                                
 
                                            
                                        
 		 
                                
jdumay@cloudbees.com (JIRA)
unread,
Nov 21, 2016, 4:31:49 PM11/21/16
to jenkinsc...@googlegroups.com


     
         
         
         
         
     
    
 
        
 
             
            
                
 
                     
                        
 
                            
                                
 
                                       
                                     James Dumay edited a comment on Story JENKINS-33185  
                                
 
                            
                        
 		 
                
 
                
 
                     
                        
 
                            
 
                                     
                                        
 
                                            
                                                
 
                                                     [~crummy] nested stages (anywhere where there is a outer and inner stage) as you've suggested can't be visualised in Blue Ocean as we haven't been able to  design a visualisation that meets all users requirements and expectations. There have been many solutions suggested but none that fit. You can follow and vote for it at JENKINS-38442.  For the time being, we ignore the inner stages but show the steps within.  
                                                
 
                                            
                                        
 		 
                                
jdumay@cloudbees.com (JIRA)
unread,
Nov 21, 2016, 4:32:04 PM11/21/16
to jenkinsc...@googlegroups.com

                
 
                     
                        
 
                            
                                
 
                                       
                                     James Dumay edited a comment on Story JENKINS-33185  
                                
 
                            
                        
 		 
                
 
                
 
                                     
                                        
 
                                            
                                                
 
                                                     [~crummy]  [~pjaytycy]  nested stages (anywhere where there is a outer and inner stage) as you've suggested can't be visualised in Blue Ocean as we haven't been able to  design a visualisation that meets all users requirements and expectations. There have been many solutions suggested but none that fit. You can follow and vote for it at JENKINS-38442. For the time being, we ignore the inner stages but show the steps within.  
                                                
 
                                            
                                        
 		 
                                
jdumay@cloudbees.com (JIRA)
unread,
Nov 21, 2016, 4:51:05 PM11/21/16
to jenkinsc...@googlegroups.com


     
         
         
         
         
     
    
 
                                     
                                        
 
                                            
                                                
 
                                                     
Giacomo Boccardo the only ticket you have quoted that is unscheduled is JENKINS-39847. Aside, I think we do support parellel very well - Blue Ocean is the only place where parellel visualisation is supported so far  
 
As you can see, its a hard problem to solve visually when there are so many variations in the way that parallel can be used within Pipeline Script. The challenge with creating a visualization for Pipeline is to create a visualisation for turing complete programming language. That isn't practically possible under all circumstances.
 		 
                                                
 
                                            
                                        
 		 
                                
mneale@cloudbees.com (JIRA)
unread,
Nov 22, 2016, 12:45:55 AM11/22/16
to jenkinsc...@googlegroups.com


     
         
         
         
         
     
    
 
        
 
             
            
                
 
                     
                        
 
                            
                                
 
                                       
                                     Michael Neale updated an issue  
                                
 
                            
                        
 		 
                
 
                
 
                     
                        
 
                            
                                
 
                                     
                                      
                                
 
                                
 
                                     
                                        
 
                                            

                                        
 		 
                                
 
                                
 
                                     
                                        
 
                                            
                                                
 
                                                    Change By: 
                                                     Michael Neale  
                                                
 
                                                
 
                                                    Attachment: 
                                                     stages.png  
                                                
 
                                            
                                        
 		 
                                
mneale@cloudbees.com (JIRA)
unread,
Nov 22, 2016, 12:45:55 AM11/22/16
to jenkinsc...@googlegroups.com


     
         
         
         
         
     
    
 
        
 
             
            
                
 
                     
                        
 
                            
                                
 
                                       
                                     Michael Neale commented on Story JENKINS-33185  
                                
 
                            
                        
 		 
                
 
                
 
                     
                        
 
                            
                                
 
                                     
                                      
                                
 
                                
 
                                     
                                        
 
                                            
                                                
 
                                                      Re: Visualize parallel steps within a Pipeline Stage   
                                                
 
                                            
                                        
 		 
                                
 
                                
 
                                     
                                        
 
                                            
                                                
 
                                                     
malcolm crum Malcolm - could you sketch out what you would hope that the test stage above would look like? could be on a metaphorical back of a napkin (or literal napkin, although they can be hard to draw on). 
 
My guess is something like: 
 
  
 		 
                                                
 
                                            
                                        
 		 
                                
andrew.melo@gmail.com (JIRA)
unread,
Nov 22, 2016, 1:20:11 AM11/22/16
to jenkinsc...@googlegroups.com


     
         
         
         
         
     
    
 
                                     
                                        
 
                                            
                                                
 
                                                     
I understand completely it's hard to try and arrange a display for the results of a turing-complete programming language, but perhaps the compromise is allowing the programmer define where things should go, instead of having the language try to divine out the positioning post-facto. For instance (pseudo code, please be gentle)
 
stage("checkout",0,0) 
 { git checkout } 
stage("build",1,0) 
 { build windows} 
stage("build",1,1)
 { build linux} 
stage("deploy",2,0) 
 { deploy windows} 
stage("deploy",2,1)
 { deploy linux} 
Since the person writing the code knows (ostensibly) where things should go, you could just trust them to make the right grid-like display of each node/step
 		 
                                                
 
                                            
                                        
 		 
                                
jdumay@cloudbees.com (JIRA)
unread,
Nov 22, 2016, 2:09:30 AM11/22/16
to jenkinsc...@googlegroups.com


     
         
         
         
         
     
    
 
                                     
                                        
 
                                            
                                                
 
                                                     
Andrew Melo being able to deliver a consistent visualization is one of the drivers of the new Declarative Pipeline syntax. 
 
Declarative Pipeline is more structured. For example, you can only nest a single parellel within a stage. There is no nesting of stages but parallel branches can have names that do what a lot of people on the comments of this ticket have done by defining a stage within a parellel.
 
To use Leo Gallucci's example, in Pipeline Script:
 
                                                        

                                                            
 
                                                                
node() {
  stage('Build') {
    println 'I prepare the build for
 the parallel steps'
  }

  parallel (
   "chrome" : { stage('chrome') { println 'running tests in chrome' } },
   "firefox" : { stage('firefox') { println 'running tests in firefox' } },
   "edge" : { stage('edge') { println 'running tests in edge' } }
  )
}

 
                                                            

                                                        
 
Would be the following in Declarative:
 
                                                        

                                                            
 
                                                                
pipeline {
  agent label:''" // same as node { ... }
  stages {
    stage('build') {
      steps {
        println 'I prepare the build for the parallel steps'
      }
    }
    stage('browser testing') {
      parallel (
        'chrome': {
          println 'running tests in chrome'
        },
        'firefox': {
          println 'running tests in firefox'
        },
        'edge': {
          println 'running tests in edge'
        }
      )
    }
  }
}

 
                                                            

                                                        
 
Now the more awesome part is that we can figure out how the Pipeline is displayed by reading the model vs executing it. That allows us to describe all the stages, steps and parallel branches before execution (making a linter feasible), build your own tooling to modify or construct Jenkinsfiles and even provides a foundation for the Jenkins project to build a Pipeline Editor.
 
It may not be as flexible as the scripting variant that you've come to know but it will give much more consistent results across many of the pain points discussed here. All this does not mean that Blue Ocean won't be trying to visualize your Jenkinsfiles written in Pipeline Script. However, we recognise that due to the nature of the beast there are going to be some things that just won't be possible to visualize when using Script.
 		 
                                                
 
                                            
                                        
 		 
                                
jdumay@cloudbees.com (JIRA)
unread,
Nov 22, 2016, 2:09:30 AM11/22/16
to jenkinsc...@googlegroups.com


     
         
         
         
         
     
    
 
        
 
             
            
                
 
                     
                        
 
                            
                                
 
                                       
                                     James Dumay edited a comment on Story JENKINS-33185  
                                
 
                            
                        
 		 
                
 
                
 
                     
                        
 
                            
 
                                     
                                        
 
                                            
                                                
 
                                                     [~perilousapricot] being able to deliver a consistent visualization is one of the drivers of the new [Declarative Pipeline|https://jenkins.io/blog/2016/09/19/blueocean-beta-declarative-pipeline-pipeline-editor/] syntax. 

Declarative Pipeline is more structured. For example, you can only [nest a single parellel  block  within a stage|https://github.com/jenkinsci/pipeline-model-definition-plugin/wiki/Parallelism#split-up-a-test-suite]. There is no nesting of stages but parallel branches can have names that do what a lot of people on the comments of this ticket have done by defining a stage within a parellel.

To use [~elgalu]'s example, in Pipeline Script:
{code}

node() {
  stage('Build') {
    println 'I prepare the build for the parallel steps'
  }

  parallel (
   "chrome" : { stage('chrome') { println 'running tests in chrome' } },
   "firefox" : { stage('firefox') { println 'running tests in firefox' } },
   "edge" : { stage('edge') { println 'running tests in edge' } }
  )
}
{code}


Would be the following in Declarative:
{code}

pipeline {
  agent label:''" // same as node { ... }
  stages {
    stage('build') {
      steps {
        println 'I prepare the build for the parallel steps'
      }
    }
    stage('browser testing') {
      parallel (
        'chrome': {
          println 'running tests in chrome'
        },
        'firefox': {
          println 'running tests in firefox'
        },
        'edge': {
          println 'running tests in edge'
        }
      )
    }
  }
}
{code}

Now the more awesome part is that we can figure out how the Pipeline is displayed by reading the model vs executing it. That allows  us to describe all the stages, steps and parallel branches before execution (making [a linter|https://github.com/jenkinsci/pipeline-model-definition-plugin/pull/58] feasible), build your own tooling to modify or construct Jenkinsfiles and even provides a foundation for the Jenkins project to build a [Pipeline Editor|https://jenkins.io/images/post-images/blueocean/pipeline-editor.png].


It may not be as flexible as the scripting variant that you've come to know but it will give much more consistent results across many of the pain points discussed here. All this does not mean that Blue Ocean won't be trying to visualize your Jenkinsfiles written in Pipeline Script. However, we recognise that due to the nature of the beast there are going to be some things that just won't be possible to visualize when using Script.
jdumay@cloudbees.com (JIRA)
unread,
Nov 22, 2016, 2:12:05 AM11/22/16
to jenkinsc...@googlegroups.com
It may not be as flexible as the scripting variant that you've come to know but it will give much more consistent results across many of the pain points discussed here  (however, you can use Script from a shared library in Declarative as an escape hatch) .  

  All this does not mean that Blue Ocean won't be trying to visualize your Jenkinsfiles written in Pipeline Script.  It's important to us that it works for the majority case.  However, we recognise that due to the nature of the beast there are going to be some things that just won't be possible to visualize when using Script. 		 
                                                
 
                                            
                                        
 		 
                                
jdumay@cloudbees.com (JIRA)
unread,
Nov 22, 2016, 2:13:04 AM11/22/16
to jenkinsc...@googlegroups.com
andrew.melo@gmail.com (JIRA)
unread,
Nov 22, 2016, 2:44:19 AM11/22/16
to jenkinsc...@googlegroups.com


     
         
         
         
         
     
    
 
                                     
                                        
 
                                            
                                                
 
                                                     
I like it! My only hitch would be trying to describe the dependency that wasn't a series of stages (forgive my pseudo-code) where things would block unnecessarily:
 
stage("checkout") 
 { git checkout XXX } 
node("linux-ubuntu") { stage("compile") 
 { XXX} 
 }
 node("linux-centos") { stage ("compile") 
 { XXX } 
 ; stage ("test") 
 { YYY } 
 }
 
It's a contrived example, but I currently do something like:
 state("checkout") 
 { git checkout XXX } 
stage("compile") {
 parallel {
 node ("linux-ubuntu") 
 { XXX } 
 node ("linux-centos") 
 { XXX } 
}
 stage ("test") {
 node ("linux-centos") 
 { YYY } 
}
 
which means every part of "compile" has to finish before the "test" phase happens, even if there are available executors. If there's a rewrite in a declarative language, I'd hope that the "test" parts could at least start before all the "compile" stages start. I know it's not what's exposed in the current or blue ocean implementations, but it would be nice to fire off additional nodes/stages without waiting on unnecessary parallel branches. 
 		 
                                                
 
                                            
                                        
 		 
                                
andrew.melo@gmail.com (JIRA)
unread,
Nov 22, 2016, 2:46:03 AM11/22/16
to jenkinsc...@googlegroups.com
 
                                     
                                        
 
                                            
                                                
 
                                                     
I think the visualization is a second-order optimization, but at least firing off the subordinate nodes would speed things up a lot. And ,as a script writer, if I could hint to you who goes where visually, that would simplify your job
 		 
                                                
 
                                            
                                        
 		 
                                
jdumay@cloudbees.com (JIRA)
unread,
Nov 22, 2016, 2:48:04 AM11/22/16
to jenkinsc...@googlegroups.com


     
         
         
         
         
     
    
 
                                     
                                        
 
                                            
                                                
 
                                                     
Andrew Melo BTW you can wrap code in JIRA in {code} some code {code} to preserve formatting
 		 
                                                
 
                                            
                                        
 		 
                                
It is loading more messages.
0 new messages