[JIRA] (JENKINS-40986) Matrix structure for Declarative Pipeline

28 views
Skip to first unread message

jan.sprinz@posteo.de (JIRA)

unread,
Sep 19, 2018, 11:18:05 AM9/19/18
to jenkinsc...@googlegroups.com
Jan Sprinz commented on Improvement JENKINS-40986
 
Re: Matrix structure for Declarative Pipeline

Any news on this? Seems like most of the prerequisites are out of the way and this would be an absolute killer feature to have for declarative pipelines.

I know it's way to early to think about visualization, but i'm not sure it's a good idea to include this in the normal blueocean pipeline view, since it would get really cluttered really fast. Maybe there could be a popup presenting something similar to the visualization of matrix builds in classic jenkins? I have a job with seven dimensions, i won't even try to imagine what it would look like to have all of those displayed in the normal pipeline view...
Add Comment Add Comment
 
This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)

andrew.bayer@gmail.com (JIRA)

unread,
Dec 7, 2018, 9:16:03 AM12/7/18
to jenkinsc...@googlegroups.com
Andrew Bayer updated an issue
 
Jenkins / Improvement JENKINS-40986
Change By: Andrew Bayer
Sprint: Declarative backlog

seckler@in.tum.de (JIRA)

unread,
Mar 22, 2019, 3:57:03 AM3/22/19
to jenkinsc...@googlegroups.com
Steffen Seckler commented on Improvement JENKINS-40986
 
Re: Matrix structure for Declarative Pipeline

could we up the priority of this?
seeing that most major other CI tools have this kind of feature, it should be implemented in jenkins rather soon...

jan.sprinz@posteo.de (JIRA)

unread,
Mar 24, 2019, 3:29:03 AM3/24/19
to jenkinsc...@googlegroups.com

Yes please! It's a must-have, really. The workaround of using loops in a scripted block for this is ugly and error-prone... All i really want for Christmas Easter is declarative Jenkins matrices.

jan.sprinz@posteo.de (JIRA)

unread,
Mar 24, 2019, 3:31:04 AM3/24/19
to jenkinsc...@googlegroups.com
Jan Sprinz updated an issue
 
Change By: Jan Sprinz
Priority: Minor Major

trejkaz@trypticon.org (JIRA)

unread,
May 24, 2019, 2:46:02 AM5/24/19
to jenkinsc...@googlegroups.com
trejkaz commented on Improvement JENKINS-40986
 
Re: Matrix structure for Declarative Pipeline

I drafted up what I would hope this sort of thing would look like, and ended up with something like:

pipeline {
    // ...
    stages {
        // ...
        matrixBuild {
            matrix [label: 'debian',  prettyName: 'Debian'],
                   [label: 'ubuntu',  prettyName: 'Ubuntu'],
                   [label: 'centos',  prettyName: 'CentOS'],
                   [label: 'macos',   prettyName: 'macOS'],
                   [label: 'windows', prettyName: 'Windows']

            template {
                stage("Test on ${prettyName}") {
                    agent {
                        label name
                    }
                    steps {
                        unstash 'compile-artifacts'
                        unstash 'dot-gradle'

                        gradle tasks: 'check', switches: '--stacktrace'
                    }
                    post {
                        always {
                            junit '*/build/test-results/**/*.xml'
                        }
                    }
                }
            }
        }
        // ...
    }
}

The values in the maps for matrix would just become available in the context for template, and if you wanted to put whole closures in as the values, that's fine too.

Practicality, though, no idea at all. I was going to try to do it as a pipeline library some day if free time ever came up.

bitwiseman@gmail.com (JIRA)

unread,
May 27, 2019, 5:38:03 PM5/27/19
to jenkinsc...@googlegroups.com
Liam Newman assigned an issue to Liam Newman
 
Change By: Liam Newman
Assignee: Andrew Bayer Liam Newman

batbko@gmail.com (JIRA)

unread,
Sep 21, 2019, 11:53:03 AM9/21/19
to jenkinsc...@googlegroups.com
Denis Dudarev commented on Improvement JENKINS-40986
 
Re: Matrix structure for Declarative Pipeline

It would be great if not only 2d matrices would be supported(more than 2 axis should be supported). As well only sparsable matrices would be useful.

 

Use case:

  • User has combination of architectures ('x86',  'x86_64', 'armv7', 'armv8')
  • User has combination of platforms ('win32', 'linux', 'osx', 'android')
  • User has combination of compilers ('clang', 'gcc', 'cl', 'icc')
  • User desired that combination '.*''linux|osx|android''cl' doesn't makes sense
  • User desired that combination 'arm.''win32|osx''.' doesn't makes sense
  • User desired that combination '.*''osx''gcc' is not supported yet

 

It could be regexp syntax, as I offered before, or more declarative like in original "Matrix job". IMHO, only this approach could make sense in 90% usecases.

 

Also, there should be a way to skip stage/step on given axises combination.

 

Use case:

  • User has combination of architectures ('x86',  'x86_64', 'armv7', 'armv8')
  • User has combination of platforms ('win32', 'linux', 'osx', 'android')
  • User has combination of compilers ('clang', 'gcc', 'cl', 'icc')
  • User has stage build
  • User has stage build MSI package
  • User desired that build stage makes sense in any combination of axises
  • User desired that build MSI doesn't make sense on '.''linux|osx|android''.'

 

Please let me know, if there is any design draft of upcoming API.
This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f)
Atlassian logo

batbko@gmail.com (JIRA)

unread,
Sep 21, 2019, 11:54:03 AM9/21/19
to jenkinsc...@googlegroups.com
Denis Dudarev edited a comment on Improvement JENKINS-40986
It would be great if not only 2d matrices would be supported(more than 2 axis should be supported). As well only sparsable matrices would be useful.

 

Use case:
* User has combination of architectures ('x86',  'x86_64', 'armv7', 'armv8')
* User has combination of platforms ('win32', 'linux', 'osx', 'android')
* User has combination of compilers ('clang', 'gcc', 'cl', 'icc')
* User desired that combination '.*' \ -'linux|osx|android' \ -'cl' doesn't makes sense
* User desired that combination 'arm.*' \ -'win32|osx' \ -'.*' doesn't makes sense
* User desired that combination '.*' \ -'osx' \ -'gcc' is not supported yet


 

It could be regexp syntax, as I offered before, or more declarative like in original "Matrix job". IMHO, only this approach could make sense in 90% usecases.

 

Also, there should be a way to skip stage/step on given axises combination.

 

Use case:
* User has combination of architectures ('x86',  'x86_64', 'armv7', 'armv8')
* User has combination of platforms ('win32', 'linux', 'osx', 'android')
* User has combination of compilers ('clang', 'gcc', 'cl', 'icc')
* User has stage build
* User has stage build MSI package
* User desired that build stage makes sense in any combination of axises
* User desired that build MSI doesn't make sense on '.*' \ -'linux|osx|android' \ -'.*'


 

Please let me know, if there is any design draft of upcoming API.

batbko@gmail.com (JIRA)

unread,
Sep 21, 2019, 11:59:02 AM9/21/19
to jenkinsc...@googlegroups.com
Denis Dudarev edited a comment on Improvement JENKINS-40986
It would be great if not only 2d matrices would be supported(more than 2 axis should be supported). As well only sparsable matrices would be useful.

 
Use User case:
 {code:sh}
* User has combination of architectures ('x86',  
'x86_64', 'armv7', 'armv8')

* User has combination of platforms ('win32', 'linux', 'osx', 'android')
* User has combination of compilers ('clang', 'gcc', 'cl', 'icc')
* User desired that combination '.*' \ -'linux|osx|android' \ -'cl' doesn't makes make sense
* User desired that combination 'arm.*' \ -'win32|osx' \ -'.*' doesn't makes make sense
* User desired that combination '.*' \ -'osx' \ -'gcc' is not supported yet {code}


 

It could be regexp syntax, as I offered before, or more declarative like in original "Matrix job". IMHO, only this approach could make sense in 90% usecases user cases .


 

Also, there should be a way to skip stage/step on given axises combination.

 

Use User case:
 {code:sh}
* User has combination of architectures ('x86',  
'x86_64', 'armv7', 'armv8')

* User has combination of platforms ('win32', 'linux', 'osx', 'android')
* User has combination of compilers ('clang', 'gcc', 'cl', 'icc')
* User has stage build
* User has stage build MSI package
* User desired that build stage makes sense in any combination of axises
* User desired that build MSI doesn for ' t make sense on ' .*' \ -'linux|osx|android' \ -'.*' doesn't make sense
{code}
 

Please let me know, if there is any design draft of upcoming API.

batbko@gmail.com (JIRA)

unread,
Sep 21, 2019, 12:00:02 PM9/21/19
to jenkinsc...@googlegroups.com
Denis Dudarev edited a comment on Improvement JENKINS-40986
It would be great if not only 2d matrices would be supported(more than 2 axis should be supported). As well only sparsable matrices would be useful.

 

User case:
{code:sh}
* User has combination of architectures ('x86',  'x86_64', 'armv7', 'armv8')
* User has combination of platforms ('win32', 'linux', 'osx', 'android')
* User has combination of compilers ('clang', 'gcc', 'cl', 'icc')
* User desired that combination '.*'-'linux|osx|android'-'cl' doesn't make sense
* User desired that combination 'arm.*'-'win32|osx'-'.*' doesn't make sense

* User desired that combination '.*'-'osx'-'gcc' is not supported yet{code}


 

It could be regexp syntax, as I offered before, or more declarative like in original "Matrix job". IMHO, only this approach could make sense in 90% user cases.


 

Also, there should be a way to skip stage/step on given axises combination.


User case:
{code:sh}
* User has combination of architectures ('x86',  'x86_64', 'armv7', 'armv8')
* User has combination of platforms ('win32', 'linux', 'osx', 'android')
* User has combination of compilers ('clang', 'gcc', 'cl', 'icc')
* User has stage build
* User has stage build MSI package
* User desired that build stage makes sense in any combination of axises
* User desired that build MSI for '.*'-'linux|osx|android'-'.*' doesn't make sense
{code}

 Please [~abayer], please let me know, if there is any design draft of upcoming API.

batbko@gmail.com (JIRA)

unread,
Sep 21, 2019, 12:05:04 PM9/21/19
to jenkinsc...@googlegroups.com
Denis Dudarev edited a comment on Improvement JENKINS-40986
It would be great if not only 2d matrices would be supported(more than 2 axis should be supported). As well only sparsable matrices would be useful.

 

User case:
{code:sh}
* User has combination of architectures ('x86',  'x86_64', 'armv7', 'armv8')
* User has combination of platforms ('win32', 'linux', 'osx', 'android')
* User has combination of compilers ('clang', 'gcc', 'cl', 'icc')
* User desired that combination '.*'-'linux|osx|android'-'cl' doesn't make sense
* User desired that combination 'arm.*'-'win32|osx'-'.*' doesn't make sense
* User desired that combination '.*'-'osx'-'gcc' is not supported yet{code}


 

It could be regexp syntax, as I offered before, or more declarative like in original "Matrix job". IMHO, only this approach could make sense in 90% user cases.

 

Also, there should be a way to skip stage/step on given axises combination.


User case:
{code:sh}
* User has combination of architectures ('x86',  'x86_64', 'armv7', 'armv8')
* User has combination of platforms ('win32', 'linux', 'osx', 'android')
* User has combination of compilers ('clang', 'gcc', 'cl', 'icc')
* User has stage ' build '
* User has stage ' build MSI package '
* User desired that ' build '
stage makes sense in any combination of axises
* User desired that ' build MSI package' for '.*'-'linux|osx|android'-'.*' doesn't make sense
{code}

[~abayer], please let me know, if there is any design draft of upcoming API.

batbko@gmail.com (JIRA)

unread,
Sep 21, 2019, 9:07:05 PM9/21/19
to jenkinsc...@googlegroups.com
Denis Dudarev edited a comment on Improvement JENKINS-40986
It would be great if not only 2d matrices would be supported(more than 2 axis should be supported). As well only sparsable matrices would be useful.

 

User case:
{code:sh}
* User has combination of architectures ('x86',  'x86_64', 'armv7', 'armv8')
* User has combination of platforms ('win32', 'linux', 'osx', 'android')
* User has combination of compilers ('clang', 'gcc', 'cl', 'icc')
* User desired that combination '.*'-'linux|osx|android'-'cl' doesn't make sense
* User desired that combination 'arm.*'-'win32|osx'-'.*' doesn't make sense
* User desired that combination '.*'-'osx'-'gcc' is not supported yet{code}


 

It could be regexp syntax, as I offered before, or more declarative imperative like in original "Matrix job". IMHO, only this approach could make sense in 90% user cases.


 

Also, there should be a way to skip stage/step on given axises combination.


User case:
{code:sh}
* User has combination of architectures ('x86',  'x86_64', 'armv7', 'armv8')
* User has combination of platforms ('win32', 'linux', 'osx', 'android')
* User has combination of compilers ('clang', 'gcc', 'cl', 'icc')
* User has stage 'build'
* User has stage 'build MSI package'
* User desired that 'build' stage makes sense in any combination of axises
* User desired that 'build MSI package' for '.*'-'linux|osx|android'-'.*' doesn't make sense
{code}

[~abayer], please let me know, if there is any design draft of upcoming API.

bitwiseman@gmail.com (JIRA)

unread,
Nov 18, 2019, 11:36:06 AM11/18/19
to jenkinsc...@googlegroups.com

You can pull 1.5.0-beta1 from the experimental update site now to take a look at matrix.

serhii.shepel@gmail.com (JIRA)

unread,
Nov 18, 2019, 3:28:06 PM11/18/19
to jenkinsc...@googlegroups.com

Liam Newman could you please provide some info regarding usage and possibilities for the bindings...

I can see a bunch of test cases here: (https://github.com/jenkinsci/pipeline-model-definition-plugin/blob/335e44ff6d19cfd43f58be210d9af81e4491cdec/pipeline-model-definition/src/test/java/org/jenkinsci/plugins/pipeline/modeldefinition/MatrixTest.java), but a bit more details would be nice...

seckler@in.tum.de (JIRA)

unread,
Nov 19, 2019, 7:08:03 AM11/19/19
to jenkinsc...@googlegroups.com

serhii.shepel@gmail.com (JIRA)

unread,
Nov 19, 2019, 7:11:04 AM11/19/19
to jenkinsc...@googlegroups.com

Ah right!
Thanks Steffen Seckler

batbko@gmail.com (JIRA)

unread,
Nov 19, 2019, 7:13:04 AM11/19/19
to jenkinsc...@googlegroups.com

Thank you for great feature!

I could only see 2 axis cases in tests and json files. Is this a limit for there?

Anyway I need better instructions to check it out. For example - sample Jenkinsfile with 3axis build would be very useful.

batbko@gmail.com (JIRA)

unread,
Nov 19, 2019, 7:17:04 AM11/19/19
to jenkinsc...@googlegroups.com

Hm. I've reviewed json file more detailed. It seems to be quite agile implementation, covering most of cases above. So what I need (and most users I believe) is a Jenkinsfile example, to review end-user syntax.

andrew.bayer@gmail.com (JIRA)

unread,
Nov 19, 2019, 7:40:03 AM11/19/19
to jenkinsc...@googlegroups.com

batbko@gmail.com (JIRA)

unread,
Nov 19, 2019, 7:51:04 AM11/19/19
to jenkinsc...@googlegroups.com

Got it, thanks!

batbko@gmail.com (JIRA)

unread,
Nov 19, 2019, 8:00:05 AM11/19/19
to jenkinsc...@googlegroups.com

Results of my humble review:

  • It seems to be usable, it covers user cases
  • Excludes syntax is quite huge, but I have no fast good idea for much compact one without imperative expressions from original matrix functionality, it could be very annoying to avoid all uninterested combinations with axis count >2, especially in cases where *most* of combinations are uninterested (e.g. at the start of a project)

simlarouche@gmail.com (JIRA)

unread,
Dec 9, 2019, 10:37:04 AM12/9/19
to jenkinsc...@googlegroups.com

+1 on the exclude syntax, it's taking a lot of space.
Also, can the values field of the axis block be dynamic instead of using a filter parameter?
I have a use case with axes with a lot of values depending on a parameter, so if I can't act on the matrix values, my only option is to handle it in a when block?
Also, in most cases, I would rather use a global variable to store my axes values as well, makes it easier to maintain and I could re-use it in another matrix block further down in the script.
But overall I'm very happy to see a matrix block in declarative pipeline.

wohali@apache.org (JIRA)

unread,
Dec 16, 2019, 9:06:03 PM12/16/19
to jenkinsc...@googlegroups.com

One enhancement that might be nice, if possible, is to have the values for an axis name be supported as non-literals. I don't seem to be able to make this work in the current version. Sorry if this isn't the right place, but you were asking for examples above...

 

Example:

pipeline {
  environment {
     LOW_ERLANG_VER = '19.3.6.8'
     MID_ERLANG_VER = '20.3.8.24'
     HIGH_ERLANG_VER = '22.2'
{{  }}}

  stages {
    // ...
    // Other stages that need the values above
    // ...
    stage("Build") {
      matrix {
        axes {
          axis {
            name 'ERLANG_VERSION'
            values "{$env.LOW_ERLANG_VER}", "${env.MID_ERLANG_VER}", "${env.HIGH_ERLANG_VER}"
{{          }}}
        } // axes
        // ...
        // do the build with ${ERLANG_VERSION}
        // ...
      } // matrix
    } // stage
  } // stages
} // pipeline

if that makes sense.

bitwiseman@gmail.com (JIRA)

unread,
Mar 4, 2020, 12:34:03 PM3/4/20
to jenkinsc...@googlegroups.com

Joan Touzet
This totally makes sense. An issue has been opened for it. https://issues.jenkins-ci.org/browse/JENKINS-61047
This message was sent by Atlassian Jira (v7.13.12#713012-sha1:6e07c38)
Atlassian logo
Reply all
Reply to author
Forward
0 new messages