Generating parallel flow with Build Flow Plugin

[Rodrigo García Peláez]

Hi,

I'm trying to use build flow plugin https://wiki.jenkins-ci.org/display/JENKINS/Build+Flow+Plugin to run the same job multiple times in parallel but with different parameters that I produce from a given String parameter in the job

I was trying something like this:

parallel (

{

  params["BACKEND_SERVER"].split(",").each()

  { build( "GENERIC_DEPLOYMENT",BACKEND_BUILD_TAG:params["BACKEND_BUILD_TAG"],BACKEND_SERVER:it) }

}

)

The console for the last run I tried shows this:

Started by user pelaezr
[EnvInject] - Loading node environment variables.
Building on master in workspace /Users/jenkins/.jenkins/jobs/MULTI/workspace
parallel {
    Schedule job GENERIC_DEPLOYMENT
    Build GENERIC_DEPLOYMENT #19 started
    GENERIC_DEPLOYMENT #19 completed 
    Schedule job GENERIC_DEPLOYMENT
    Build GENERIC_DEPLOYMENT #20 started
    GENERIC_DEPLOYMENT #20 completed 
}
Notifying upstream projects of job completion
Finished: SUCCESS

But it doesn't seem to have run in parallel since it took 11 minutes and normally the GENERIC_DEPLOYMENT job takes about 5. This GENERIC_DEPLOYMENT should allow concurrent builds since it has ticked the check box with "Execute concurrent builds if necessary"

Jenkins is configured at the moment with a max number of executors of 4 and no other jobs were running

What am I missing? 

Thanks and regards

Rodrigo

[Rodrigo García Peláez]

Any ideas?

When I run sequentially the GENERIC_DEPLOYMENT job quickly I get the job to run twice in parallel. The quite period is 0 for this job. But when I do it from the Build Flow job below it runs them sequentially

Thanks

Rodrigo

[Tom De Vylder]

On 16 Jan 2014, at 19:15, Rodrigo García Peláez <rodrigoga...@gmail.com> wrote:

> Any ideas?
>
> When I run sequentially the GENERIC_DEPLOYMENT job quickly I get the job to run twice in parallel. The quite period is 0 for this job. But when I do it from the Build Flow job below it runs them sequentially
>
> Thanks
> Rodrigo
>

Hi,

In order to make your jobs run parallel you need to split them into different blocks.
E.g.:

parallel (
{
build( “GENERIC_DEPLOYMENT" )
},
{
build( “BACKEND_BUILD_TAG:params["BACKEND_BUILD_TAG"]" )
},
{
build( “BACKEND_SERVER:it)" )
}
)

Regards,
Tom De Vylder

[Rodrigo García Peláez]

Hi,

Thanks for your response.

My problem is that I'm trying to build it dynamically depending on the parameters. I can receive a list of servers to deploy to so I want to run the job to deploy to each server in parallel. That list of servers can be different every time, as it's passed manually with an extended choice parameter, triggered manually.

Any ideas of how I can achieve that?

Thanks and regards

Rodrigo

[stuartrowe]

Hi Rodrigo,

BuildFlow's parallel function takes a list or map of closures - it will
execute each in parallel.

For your situation, you should iterate over your list of servers, creating a
closure for each and adding it to list. This list can the be passed as an
argument to parallel().

e.g.

// create a closure for the deploy job for each server
def serverDeployJobs = []
for (server in servers) {
def deployJob = {
def jobParams = [:]
// set up params for deploy job on current server here...

// call build
build(jobParams, DeployProjectName)
}
serverDeployJobs.add(deployJob)
}

// schedule deploy jobs in parallel
parallel(serverDeployJobs)

Hope that helps,
Stuart



--
View this message in context: http://jenkins-ci.361315.n4.nabble.com/Generating-parallel-flow-with-Build-Flow-Plugin-tp4687604p4691098.html
Sent from the Jenkins users mailing list archive at Nabble.com.

[Rodrigo García Peláez]

Hi Stuart,

I'm not very familiar with groovy but I think it makes sense :)  

I'll give that a go

To be honest I hadn't tried again to sort this out because in the end I created a fixed job to deploy to the servers I had to, but it will be useful if I get it working for the future

Thanks for your help! I'll post a reply here after I try it

Regards

Rodrigo

[Rodrigo García Peláez]

Hi,

I forgot to reply to this. Just wanted to confirm your suggested solution works Stuart, thanks for that and sorry for late response!

Rodrigo

[Derek Goss]

I can't upvote this enough, this does the trick. I was stuck for a while on building jobs in parallel based on a dynamically-populated job list.

Nice work.

[Jay Neese]

I am using a similar configuration as OP, except that multiple different jobs must be ran in parallel.  I have two separate build jobs (test1 and test2)  that are being fed into the parallel function and rather than running jobs test1 and test2, it is running test2 twice.  In my case the jobs will be different, but the parameter (version) that is passed to the jobs will not change. 

Here is the code:

version = "4.1.0"
println "Deploying version " + version
def components =  ["test1,test2"]
println components
def component = components.get(0)
println component
def values = component.split(',')
def jobsInParallel = [];
for ( myComponent  in values ) {
  println myComponent
  def parallelJob = {
    def jobParams = [:]
    jobParams = myComponent
    println jobParams
    build (jobParams,Version:version)
  }
  jobsInParallel.add(parallelJob)
}
parallel(jobsInParallel)

And here is the Console output:

Deploying version 4.1.0
[test1,test2]
test1,test2
test1
test2
parallel {
test2
test2
    Schedule job test2
    Schedule job test2
    Build test2 #1 started
    Build test2 #1 started
    test2 #1 completed 
    test2 #1 completed 
}
Finished: SUCCESS

Any ideas on why the test1 job isn't being included in the parallel build?  Any help would be appreciated. 

Thank You, 

Jay

[Jay Neese]

Just to follow up, I found a solution.  The code looks like this:

version = "4.1.0"
println "Deploying version " + version

def jobsList =  ["test1,test2"] - list of jobs, but first element has all jobs
def jobNamesString= jobsList.get(0) // Get the comma delimited string of job names
def jobs= jobNamesString.split(',') // Creates an array of jobNames that can be iterated
// construct and collect closures for LATER execution
buildClosures = []
for (int i = 0; i < jobs.size(); i++) {
   def jobName = jobs[i]
   def curClosure = {
        build(jobName,Version:version)
    }
    buildClosures.add(curClosure)
}


// execute the closures in buildClosures in parallel
parallel(buildClosures)

And here is the Console output:

Deploying version 4.1.0

parallel {

    Schedule job test1

    Schedule job test2

    Build test1 #12 started

    Build test2 #10 started

    test1 #12 completed 

    test2 #10 completed 

}

Finished: SUCCESS

This post was a big help:  http://jenkins-ci.361315.n4.nabble.com/BUILD-FLOW-parallel-closure-simple-example-td4699743.html