Nested parallel dynamically

[Fabian Cenedese]

Hi

I'm trying to create a pipeline which should have nested parallel
blocks. In detail, I have a number of stages which should run on
a number of agents. However the information where each stage
should run is dynamic as it comes from another job (upstream).
So declarative pipeline is not possible and doesn't support
nested parallels anyway, I believe.
To prevent all stages running on separate executors (and so
blocking the execution of other jobs) I'd like to combine the
stages for one agent into a parallel pipeline and then run all
those agent pipelines in parallel again. It should be the same
as if I had all stages in a single pipeline running on one agent,
using one executor except that now multiple agents are used,
each one running its own parallel pipeline. It's important that
they are really parallel because the job should use as little
time as possible. That's also the reason to use multiple agents
instead of just one.

I looked on the net and found examples for scripted pipelines
or nested parallel stages, but I couldn't find an example that
I could make work. I only used declarative pipelines before so
I'm not that versed in scripted pipelines.

Can someone point me to an example that goes in my direction?

Thanks a lot

bye Fabi

[Ivan Fernandez Calvo]

We resolved the dynamic matrix support by implementing a matrix step in a shared library,

the matrix axis are arrays that can come from files if you want 

pipeline {
  agent any

  stages {
    stage('Matrix sample') {
      steps {

        matrix(
          agent: 'linux',
          axes:[
            axis('VAR_NAME_00', [ 1, 2 ]),
            axis('VAR_NAME_01', [ 'a', 'b', 'c', 'd', 'e' ])
          ],
          excludes: [
            axis('VAR_NAME_00', [ 1 ]),
            axis('VAR_NAME_01', [ 'd', 'e' ]),
          ]
          ) {
            echo "${VAR_NAME_00} - ${VAR_NAME_01}"
          }

        }
      }
    }
  }

https://github.com/elastic/apm-pipeline-library/blob/main/vars/matrix.groovy

https://github.com/elastic/apm-pipeline-library/blob/main/vars/matrix.txt

[Fabian Cenedese]

At 18:12 17.05.2023, you wrote:

>We resolved the dynamic matrix support by implementing a matrix step in a shared library,
>the matrix axis are arrays that can come from files if you want
>
>pipeline {
> agent any
>
> stages {
> stage('Matrix sample') {
> steps {
>
> matrix(
> agent: 'linux',
> axes:[
> axis('VAR_NAME_00', [ 1, 2 ]),
> axis('VAR_NAME_01', [ 'a', 'b', 'c', 'd', 'e' ])
> ],
> excludes: [
> axis('VAR_NAME_00', [ 1 ]),
> axis('VAR_NAME_01', [ 'd', 'e' ]),
> ]
> ) {
> echo "${VAR_NAME_00} - ${VAR_NAME_01}"
> }
>
> }
> }
> }
> }
>
>https://github.com/elastic/apm-pipeline-library/blob/main/vars/matrix.groovy
>https://github.com/elastic/apm-pipeline-library/blob/main/vars/matrix.txt

Thanks for your example, I'll take a look at it. Does it show up in the pipeline
stage view just like a regular job? I found that if the stage order is not each time
the same then the stage view will reset with each build, so only the last one is
visible. Maybe the matrix can help me with that.

bye Fabi