How to properly use parallels in pipelines

[Chris Overend]

parallel (

    'windows': {
        stage('windows tests') {
            parallel (
                    stage('some_tests_1') {
                        parallel (
                            some_tests_1, # is a chunk of tests
                        )
                     }
                    stage('some_tests_2') {
                        parallel (
                            some_tests_2,  # is a chunk of tests
                        )
                    }
                    stage('some_tests_3') {
                        parallel (
                            some_tests_3  # is a chunk of tests
                        )
                    }
            )
    }
    'linux': ...same as above

Currently we call flows calling flows and the above displays how it would look in a pipeline.

from: https://github.com/jenkinsci/pipeline-examples/blob/master/docs/BEST_PRACTICES.md

Nested parallel blocks can lead to swamping your available executors, as each execution of the first parallel block calls multiple executions of the second parallel block, and so on. In general, think carefully about your parallelism and your available executors when using parallel.

With this in mind we could

parallel (

    'windows': {
        stage('windows tests') {
            parallel (
                some_tests_1, # is a chunk of tests
                some_tests_2,  # is a chunk of tests
                some_tests_3  # is a chunk of tests
            )
        }
    }
    'linux': ...same as above

Also from https://github.com/jenkinsci/pipeline-examples/blob/master/docs/BEST_PRACTICES.md

Don’t put stages in parallel blocks - that just goes weird, breaking a lot of logic in the Stage View and elsewhere. Save yourself the pain - don't do it!

So how do you display this work flow properly if we should not use stages in parallel?

[Nick Le Mouton]

I've been playing around with this as well, there doesn't seem to be a lot of documentation around using parallel in declarative pipelines, but this seems to work for us:

    stage('Code analysis') {

      steps {

        parallel (

          lint: {

            //run lint

          },

          phpmd: {

            //run phpmd

          },

          phpcs: {

            //run phpcs

          }

        )

      }

     

 

Or if you're using older scripted pipelines then it'd be something like:

    stage('Code analysis') {
      parallel (
        phpmd: {
          //run phpmd
        },
        phpcs: {
          //run phpcs
        },
        phploc: {
          //run phploc
        },
      )
    }

I hope this helps

Nick Le Mouton