Run same Jenkins job on different repositories in parallel using Declarative Pipeline
37 views
Skip to first unread message
Roman O
Jun 11, 2020, 9:49:15 AM6/11/20
to Jenkins Users
My goal is to run the same job check-single-repo on multiple repositories in parallel.
Below pipeline doesn't seem to achieve the goal:
pipeline {
agent any
options {
ansiColor('xterm')
}
parameters {
extendedChoice description: '', multiSelectDelimiter: ',', name: 'REPO_NAMES', quoteValue: false, saveJSONParameterToFile: false, type: 'PT_CHECKBOX', value: 'repo1,repo2,repo3', visibleItemCount: 3
}
stages {
stage('Prepare repos to run on') {
steps {
script {
repo_names = params.REPO_NAMES.split(',')
def single_repo_jobs = [:]
for (repo_name in repo_names) {
println repo_name
single_repo_jobs[repo_name] = {
stage(repo_name) {
catchError {
build job: 'check-single-repo',
parameters:
[
string(name:'REPO_NAME', value: repo_name)
] }
}
}
}
println single_repo_jobs
parallel single_repo_jobs
}
}
}
}
} Its output:
...
repo1
repo2
repo3
{repo1=org.jenkinsci.plugins.workflow.cps.CpsClosure2@3a396959, repo2=org.jenkinsci.plugins.workflow.cps.CpsClosure2@1a4b5000, repo3=org.jenkinsci.plugins.workflow.cps.CpsClosure2@1d034ac}
[Pipeline] parallel
[Pipeline] { (Branch: repo1)
[Pipeline] { (Branch: repo2)
[Pipeline] { (Branch: repo3)
[Pipeline] stage
[Pipeline] { (repo3)
[Pipeline] stage
[Pipeline] { (repo3)
[Pipeline] stage
[Pipeline] { (repo3)
[Pipeline] catchError
[Pipeline] {
[Pipeline] catchError
[Pipeline] {
[Pipeline] catchError
[Pipeline] {
[Pipeline] build (Building check-single-repo)
Scheduling project: check-single-repo
[Pipeline] build (Building check-single-repo)
Scheduling project: check-single-repo
[Pipeline] build (Building check-single-repo)
Scheduling project: check-single-repo
Starting building: check-single-repo #230
Starting building: check-single-repo #230
Starting building: check-single-repo #230
[Pipeline] }
check-single-repo #230 repo3 completed with status UNSTABLE (propagate: false to ignore)
[Pipeline] }
check-single-repo #230 repo3 completed with status UNSTABLE (propagate: false to ignore)
[Pipeline] }
check-single-repo #230 repo3 completed with status UNSTABLE (propagate: false to ignore)
...As it invokes only one pipeline #230 on repo3
How to run the same Jenkins job on different repositories in parallel using Declarative Pipeline?
Message has been deleted
Roman O
Jun 11, 2020, 10:00:13 AM6/11/20
to Jenkins Users
pipeline {
agent any
options {
ansiColor('xterm')
}
parameters {
extendedChoice description: '', multiSelectDelimiter: ',', name: 'REPO_NAMES', quoteValue: false, saveJSONParameterToFile: false, type: 'PT_CHECKBOX', value: 'repo1,repo2,repo3', visibleItemCount: 3
}
stages {
stage('Prepare repos to run on') {
steps {
script {
def repo_names = params.REPO_NAMES.split(',')
def single_repo_jobs = repo_names.collectEntries {
["${it}" :
{
catchError {
stage(it) {
build job: 'check-single-repo',
parameters:
[
string(name:'REPO_NAME', value: "${it}")
]
}
}
}
]
}
parallel single_repo_jobs
}
}
}
}
}
Devin Nusbaum
Jun 11, 2020, 10:22:22 AM6/11/20
to Jenkins Users
The problem in your original code is that the stages you create (more specifically the closures for the stages) reference the mutable repo_name variable directly, and so after you create all the stages in the loop, repo_name’s value is the final repo in repo_names, so when the stages are executed they all use the same value of repo_name.
You need to modify the script so that the stages do not refer to the loop variable directly, usually by creating a copy of the loop variable, but approaches like using collectEntries or passing the loop variable as an argument to a closure that creates another closure would also fix the issue.
On Jun 11, 2020, at 10:00, Roman O <warri...@gmail.com> wrote:
agent any
options {
ansiColor('xterm')
}
parameters {
extendedChoice description: '', multiSelectDelimiter: ',', name: 'REPO_NAMES', quoteValue: false,saveJSONParameterToFile: false, type: 'PT_CHECKBOX', value: 'repo1,repo2,repo3', visibleItemCount: 3
}
stages {
stage('Prepare repos to run on') {
steps {
script {
def repo_names = params.REPO_NAMES.split(',')
def single_repo_jobs = repo_names.collectEntries {
["${it}" :
{
catchError {
stage(it) {
build job: 'check-single-repo',
parameters:
[
string(name:'REPO_NAME', value: "${it}")
]
}
}
}
]
}
parallel single_repo_jobs
}
}
}
}
}
On Thursday, June 11, 2020 at 4:49:15 PM UTC+3, Roman O wrote:
My goal is to run the same job check-single-repo on multiple repositories in parallel.Below pipeline doesn't seem to achieve the goal:
agent any
options {
ansiColor('xterm')
}
parameters {
extendedChoice description: '', multiSelectDelimiter: ',', name: 'REPO_NAMES', quoteValue: false,saveJSONParameterToFile: false, type: 'PT_CHECKBOX', value: 'repo1,repo2,repo3', visibleItemCount: 3
--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/d094696f-9984-4e0e-9189-66c06aee8dado%40googlegroups.com.
monger_39
Jun 17, 2020, 3:55:42 AM6/17/20
to jenkins...@googlegroups.com
Hi,
not sure if your problem is now solved, so ..
Try changing your code to use:
w pipeline doesn't seem to achieve the goal:
pipeline {
agent any
options {
ansiColor('xterm')
}
parameters {
extendedChoice description: '', multiSelectDelimiter: ',', name: 'REPO_NAMES', quoteValue: false, saveJSONParameterToFile: false, type: 'PT_CHECKBOX', value: 'repo1,repo2,repo3', visibleItemCount: 3
}
stages {
stage('Prepare repos to run on') {
steps {
script {
repo_names = params.REPO_NAMES.split(',')
def single_repo_jobs = [:]
for (repo_name in repo_names) {
def theRepoName = "$repo_name" // This assignment is important
println theRepoName
single_repo_jobs[ theRepoName ] = {
stage( theRepoName ) {
catchError {
build job: 'check-single-repo',
parameters:
[
string(name:'REPO_NAME', value: theRepoName )
] }
}
}
}
println single_repo_jobs
parallel single_repo_jobs
}
}
}
}
}I keep forgetting the exact reason for this extra assignment needed but do know it's needed. I'm sure it's
properly explained and documented, but remains non-intuitive.
gr M
The following code worked for me after following the example from docs:
pipeline { agent any options { ansiColor('xterm') } parameters { extendedChoice description: '', multiSelectDelimiter: ',', name: 'REPO_NAMES', quoteValue: false, saveJSONParameterToFile: false, type: 'PT_CHECKBOX', value: 'repo1,repo2,repo3', visibleItemCount: 3 } stages { stage('Prepare repos to run on') { steps { script { def repo_names = params.REPO_NAMES.split(',') def single_repo_jobs = repo_names.collectEntries { ["${it}" : { catchError { stage(it) { build job: 'check-single-repo', parameters: [ string(name:'REPO_NAME', value: "${it}") ] } } } ] } parallel single_repo_jobs } }
On Thursday, June 11, 2020 at 4:49:15 PM UTC+3, Roman O wrote:
--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/507c0997-e718-4f7a-ab5d-05b598f3e736o%40googlegroups.com.
Reply all
Reply to author
Forward
0 new messages