trying to get Jenkins pipeline to run across multiple nodes in parallel

[Stephen DiMilla]

I'm new to using pipelines.

I'm trying to execute the same commands against a number of nodes in parallel

Ideally I will pass a list of machines to run the same code against. Below I am 

hardcoding the machine names (machine1 and machine2) for the purposes of this example.

I found the following in Jenkins Pipeline examples:

----------------------------

def labels = ['precise', 'trusty'] // labels for Jenkins node types we will build on
def builders = [:]
for (x in labels) {
    def label = x // Need to bind the label variable before the closure - can't do 'for (label in labels)'

    // Create a map to pass in to the 'parallel' step so we can fire all the builds at once
    builders[label] = {
      node(label) {
        // build steps that should happen on all nodes go here
      }
    }
}

parallel builders
----------------------------
So I adapted it to be the following:
----------------------------
pipeline {
    stages {
        stage('Stage1') {
            steps {
                def labels = ['machine1','machine2'] // labels for Jenkins node types we will build on
                def builders = [:]
                for (x in labels) {
                    def label = x // Need to bind the label variable before the closure - can't do 'for (label in labels)'

                    // Create a map to pass in to the 'parallel' step so we can fire all the builds at once
                    builders[label] = {
                        node(label) {
                            sh '''
                                echo "hostname:`hostname`"
                                echo "whoami:`whoami`"
                
                                '''
                        }
                    }
                }
                parallel builders
            }
        }
    }
}
----------------------------

But when I use it I get the following error:

Running in Durability level: MAX_SURVIVABILITY
org.codehaus.groovy.control.MultipleCompilationErrorsException: startup failed:
WorkflowScript: 5: Expected a step @ line 5, column 17.
                   def labels = ['machine1','machine2'] // labels for Jenkins node types we will build on
                   ^

WorkflowScript: 6: Expected a step @ line 6, column 17.
                   def builders = [:]
                   ^

WorkflowScript: 7: Expected a step @ line 7, column 17.
                   for (x in labels) {
                   ^

General error during semantic analysis: There's no @DataBoundConstructor on any constructor of class org.jenkinsci.plugins.workflow.cps.steps.ParallelStep

org.kohsuke.stapler.NoStaplerConstructorException: There's no @DataBoundConstructor on any constructor of class org.jenkinsci.plugins.workflow.cps.steps.ParallelStep

----------------------------

I can't figure a way around this.

So I was googling around and found this code that does work

pipeline {
    agent none
    stages {
        stage('stage1') {
            failFast true
            parallel {
                stage('S1') {
                    agent {
                        label "machine1"
                    }
                    steps {
                        sh '''
                            echo "hostname: `hostname`"
                            echo "whoami:`whoami`"
                
                        '''
                    }
                }
                stage('S2') {
                    agent {
                        label "machine2"
                    }
                    steps {
                        sh '''
                            echo "hostname: `hostname`"
                            echo "whoami:`whoami`"
                
                        '''
                    }
                }
            }

        }
    }
}

But it's not dynamic, I need to have a stage per machine. Does anyone have a suggestion

about how I can achieve what I have in my original code?

I want to dynamically decide what nodes to run the code on.

[Mark Waite]

As far as I can tell, dynamically defining the number of parallel stages will require that you use the scripted Pipeline syntax rather than the declarative Pipeline syntax.

There is a stackoverflow article that shows dynamically defined steps being used in a declarative Pipeline.  It uses either a Pipeline shared library or a snippet of scripted Pipeline at the end of the declarative Pipeline definition to generate the steps.

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/b2ae506c-023e-4604-896e-50b4eac6d817%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Thanks!

Mark Waite

[Ivan Fernandez Calvo]

to use groovy in dleclarative you have to use the `script` step to warp your code

 

pipeline {

  stages {

    stage('Stage1') {

      steps {

        script { // YOU NEED TO USE THE SCRIPT BLOCK

          def labels = ['machine1', 'machine2'] // labels for Jenkins node types we will build on

          def builders = [: ]

          for (x in labels) {

            def label = x // Need to bind the label variable before the closure - can't do 'for (label in labels)'

            // Create a map to pass in to the 'parallel' step so we can fire all the builds at once

            builders[label] = {

              node(label) {

                sh ''

                '

                echo "hostname:`hostname`"

                echo "whoami:`whoami`"

                ''

                '

              }

            }

          }

          parallel builders

        }

      }

    }

  }

}

or define a function then call it, I personally like it much is cleaner

pipeline {

  stages {

    stage('Stage1') {

      steps {

        parallelTasks()

      }

    }

  }

}

def parallelTasks() {