[JIRA] (JENKINS-39914) Declarative pipeline does not allow loops to reuse code (verbatim or parametrized same-pattern) in the Jenkinsfile script

4 views
Skip to first unread message

jimklimov@gmail.com (JIRA)

unread,
Nov 21, 2016, 10:06:02 AM11/21/16
to jenkinsc...@googlegroups.com
Jim Klimov created an issue
 
Jenkins / Bug JENKINS-39914
Declarative pipeline does not allow loops to reuse code (verbatim or parametrized same-pattern) in the Jenkinsfile script
Issue Type: Bug Bug
Assignee: Andrew Bayer
Components: pipeline-model-definition-plugin
Created: 2016/Nov/21 3:05 PM
Priority: Minor Minor
Reporter: Jim Klimov

I am learning the Pipeline plugin by trying to wrap a portable project build and test routine into this paradigm. I started with the Multibranch approach and Declarative DSL syntax, and found that I can not make loops there. This issue splits to several use-cases, so I'll try to untangle this furball below:

1) Part of the test routine is also very patternized and parallelizable - it has to run several 'make $distcheckargs' jobs for several distcheck scenarios predefined in the project Makefile. I did manage to loop over this one as an array of strings, and ultimately even managed to build a list into a variable and then parallel() it, but only by forfeiting the DDSL syntax and using a script{} step.

2) The overall pipeline-contents code is same for different Unixish platforms supported by that project (including rather obscure and ancient ones) - so I want the resulting Jenkinsfile to loop over my agents with respective labels (VMs and physical hosts with those OSes), and push the same pipeline commands to them. So far I could not achieve this - I can not use script{} or parallel{} or variable assignment and loops at high levels in the pipeline code; I can not (in DDSL) nest stages to separate the OSes and contain CI steps; I can not wrap the pipeline{} block into a for-loop because pipeline{} apparently must be at the top level in a file, otherwise keywords defined for DDSL become invalid markup for the common DSL...

I gather something of this sort is a relatively standard FOSS use-case, so it should be supported by DDSL (and if it happens to be implemented already - then it should get more visibly and explicitly documented and exemplified).

Add Comment Add Comment
 
This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c)
Atlassian logo

jimklimov@gmail.com (JIRA)

unread,
Nov 21, 2016, 10:15:03 AM11/21/16
to jenkinsc...@googlegroups.com
Jim Klimov updated an issue
Change By: Jim Klimov
I am learning the Pipeline plugin by trying to wrap a portable project build and test routine into this paradigm. I started with the Multibranch approach and Declarative DSL syntax, and found that I can not make loops there. This issue splits to several use-cases, so I'll try to untangle this furball below:

1) Part of the test routine is also very patternized and parallelizable - it has to run several 'make $distcheckargs' jobs for several distcheck scenarios predefined in the project Makefile. I did manage to loop over this one as an array of strings, and ultimately even managed to build a list into a variable and then parallel() it, but only by forfeiting the DDSL syntax and using a script{} step.

2) The overall pipeline-contents code is same for different Unixish platforms supported by that project (including rather obscure and ancient ones) - so I want the resulting Jenkinsfile to loop over my agents with respective labels (VMs and physical hosts with those OSes), and push the same pipeline commands to them. So far I could not achieve this - I can not use script{} or parallel{} or variable assignment and loops at high levels in the pipeline code; I can not (in DDSL) nest stages to separate the OSes and contain CI steps; I can not wrap the pipeline{} block into a for-loop because pipeline{} apparently must be at the top level in a file, otherwise keywords defined for DDSL become invalid markup for the common DSL and apparently I have to rewrite the prepared DDSL code into equivalent DSL ...


I gather something of this sort is a relatively standard FOSS use-case, so it should be supported by DDSL (and if it happens to be implemented already - then it should get more visibly and explicitly documented and exemplified).

andrew.bayer@gmail.com (JIRA)

unread,
Nov 21, 2016, 10:52:02 AM11/21/16
to jenkinsc...@googlegroups.com
Andrew Bayer commented on Bug JENKINS-39914
 
Re: Declarative pipeline does not allow loops to reuse code (verbatim or parametrized same-pattern) in the Jenkinsfile script

Can you give me some pseudo code example so for what you'd like to be able to do in Declarative but can't currently?

jimklimov@gmail.com (JIRA)

unread,
Nov 21, 2016, 10:56:01 AM11/21/16
to jenkinsc...@googlegroups.com

Yes, just a few minutes ago I succeeded in making a job which works and launches (or rather queues) the runs I want, link posted to IRC and here :

FWIW, This pipeline rolls for me as both a loop over nodelabels, and inside a loop over distcheck-like Make recipes. But it is no longer DDSL, at least not for the most part (the "environment{}" block is a handy one to keep intact though ) - http://pastebin.com/dSdNX4rh

andrew.bayer@gmail.com (JIRA)

unread,
Nov 21, 2016, 11:06:01 AM11/21/16
to jenkinsc...@googlegroups.com

That's...impressively deranged. =) Nested parallels and stages are things we've been avoiding due in part to visualization challenges in Blue Ocean, but I'll use this as further evidence that we need to figure that visualization out! I have been thinking about a matrix section at the top that would allow you to run the normally defined stages in parallel across multiple configurations of other top level steps (I.e., agent, environment, etc) but that's most likely not going to make it into 1.0. For now, you're probably best off using standard Pipeline script, but as I said, I'll keep this use case in mind going forward.

jimklimov@gmail.com (JIRA)

unread,
Nov 21, 2016, 11:09:01 AM11/21/16
to jenkinsc...@googlegroups.com

Thanks for the... compliment, I guess And good luck cooking up another great product

andrew.bayer@gmail.com (JIRA)

unread,
Dec 19, 2016, 4:00:01 PM12/19/16
to jenkinsc...@googlegroups.com

martin.sander@sap.com (JIRA)

unread,
Mar 14, 2018, 11:47:02 AM3/14/18
to jenkinsc...@googlegroups.com
Martin Sander commented on Bug JENKINS-39914
 
Re: Declarative pipeline does not allow loops to reuse code (verbatim or parametrized same-pattern) in the Jenkinsfile script

Andrew Bayer are you still considering this?

very simple use case that I would like to see

instead of

        stage('distribute the image') {
            parallel {
                stage('copy to eastus') {
                    steps {
                        sh 'bash -xe distribute-image.sh eastus'
                    }
                }
                stage('copy to australiaeast') {
                    steps {
                        sh 'bash -xe distribute-image.sh australiaeast'
                    }
                }
            }
        }

I want

        stage('distribute the image') {
            parallel {
                for (location in ["eastus", "australiaeast"]) {
                    stage("copy to $location") {
                        steps {
                            sh "bash -xe distribute-image.sh $location"
                        }
                    }
                }
            }
        }

The syntax checker complains that only "stage" calls are allowed inside parallel - but unwinding the loop, that is the only thing that happens..
So shouldn't this be possible at least in theory?

This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e)
Atlassian logo

andrew.bayer@gmail.com (JIRA)

unread,
Mar 14, 2018, 12:07:05 PM3/14/18
to jenkinsc...@googlegroups.com

bitwiseman@gmail.com (JIRA)

unread,
May 27, 2019, 5:40:02 PM5/27/19
to jenkinsc...@googlegroups.com
Liam Newman closed an issue as Won't Fix
 
Change By: Liam Newman
Status: Resolved Closed
This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)
Reply all
Reply to author
Forward
0 new messages