Pipeline with multiple Git repositories - how to checkout?

[Torsten Reinhard]

Hi,

I have a pipeline doing a build and afterwards a deployment to the TEST environment.

For the build I need to

// checkout sources for building
checkout scm

and later on I need to

//checkout configuration for TEST environment
checkout scm (config)

I wasn´t able to configure a working directory for each repo - and the "Multiple SCMs plugin" is deprecated with a hint to the Pipeline plugin.
How can I setup the Pipeline job so it´s using "Pipeline script from SCM" AND additionally some files/configs from a different repo?

I know about the workaround with subtrees and also this issue: https://issues.jenkins-ci.org/browse/JENKINS-13228,
but I´m wondering why help points to the Pipeline plugin ?

Any sample setup available ?

Thanx for helping,

Torsten

[Mark Waite]

The "checkout" step in a Jenkinsfile will allow you checkout multiple repositories from a single Pipeline job.  It works quite well.  Use the "Pipeline Syntax" link from pipeline job pages to complete parameters for the "checkout" step.

As an example, I have a small repository that contains a series of branches that I use to define the jobs which will checkout and build from multiple repositories.  That allows me to have a small repository which defines how I want to build many other repositories.

Mark Waite

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/0aa76f95-1c3c-43bb-a3a7-01b45a3eb282%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Torsten Reinhard]

But than you need the locations (and branches) of the repositories inside the Jenkinsfile (or a resource read/parsed from the Jenkinsfile), right?

Something like:

// checkout repo1 (origin)
checkout([$class: 'GitSCM', branches: [[name: '*/master']],
doGenerateSubmoduleConfigurations: false, extensions: [[$class:
'RelativeTargetDirectory', relativeTargetDir: 'origin']], gitTool:
'Default', submoduleCfg: [], userRemoteConfigs: [[credentialsId:
'Bitbucket_Access', name: 'origin', url:
'ssh://g...@server01.mycompany.net:7999/path/to/repo1.git']]])

// checkout repo1-config (origin-config)
checkout([$class: 'GitSCM', branches: [[name: '*/master']],
doGenerateSubmoduleConfigurations: false, extensions: [[$class:
'RelativeTargetDirectory', relativeTargetDir: 'origin-config']], gitTool:
'Default', submoduleCfg: [], userRemoteConfigs: [[credentialsId:
'Bitbucket_Access', name: 'origin', url:
'ssh://g...@server01.mycompany.net:7999/path/to/repo1-config.git']]])



Compared to just

checkout scm

which will work for multi-branches, the above approach doesnt seem to scale or am I missing something?

I would like an approach where both the repo1 and the repo1-config are configured in the Pipeline job - and the Jenkinsfile will be more generic (without hard coded URL´s inside)

Your thoughts ?

Regards, Torsten

[Mark Waite]

Yes, the pipeline approach I'm suggesting will require that each branch has a Jenkinsfile, and the contents of that Jenkinsfile must specify which repositories are used to checkout that pipeline.

I think that is desirable, since I want the source code in the repository to control the definition of the Jenkins job.

I don't want a pipeline job where the pipeline is defined inside the Jenkins job.  A pipeline in the Jenkins job is separate from the source code, and that feels like the wrong direction to me.

Mark Waite

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/a692a6fb-a34f-4f14-8354-1cf9387ef98a%40googlegroups.com.

[anillingutla]

Mark,

After 2 years i ran in to this again.. 

my repo is something like below - 

masterrepo -

  |___repo1

     |__Jenkinsfile

  |___repo2

     |__Jenkinsfile

  |___repo3

     |__Jenkinsfile

  |__Jenkinsfile

We are planning to create multiple chef-cookbook repos (submodules) under mainrepo. Main Jenkins file is used for main pipeline.

but our goal is to checkout each of the repo and run  build,test cookbooks. and finally if all the tests are successful, we promote the code to staging env.

Iam little hesitant if we re designing the right way.  I see Torsten Reinhard has talked about the similar concept but hardcoding the git repos in checkout step. 

Can we "checkoutscm" repo1 and run the tests and followed by "checkoutscm" for repo2 and son on until all our dependency cookbooks are tested and then promote the code to staging and deploy the code to staging environment (vRA / vRO). Its a vRealize environment. please suggest.

- Anil Lingutla  

[Mark Waite]

Yes, Pipeline can do that, and do it quite well.

Mark Waite

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/2a739f96-545e-4d46-a4d3-90aaeecec99a%40googlegroups.com.

[niristotle okram]

Neither chef nor git conforms to your repository structure. 

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/2a739f96-545e-4d46-a4d3-90aaeecec99a%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Sent from mobile device, excuse typos if any.

[anillingutla]

Can you please elaborate. We thought of making sub module structure. And run checkout each submodule run tests. 

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/CAPzcO4gAHi_i3O%3DnpjPCffXBJ-0Z1TJsxnKEO7ea1vvrtnAEpA%40mail.gmail.com.

[Jacques van der Merwe]

I know this is fairly old thread but I have an issue

when using the multi repo checkout it seems to remove the previous repo checkout before doing the new one. 

I have tried checking all the repos required for my build using the Jenkins Declarative "checkout" command

checkout([$class: 'GitSCM', branches: [[name: "${gitBranch}"]], doGenerateSubmoduleConfigurations: false, extensions: [[$class: 'CloneOption', timeout: 120]], submoduleCfg: [], userRemoteConfigs: [
     [credentialsId: "${credentialsId}", url: "${mainProjectGITURL}"],
     [credentialsId: "${credentialsId}", url: "${list[0]}"],
     [credentialsId: "${credentialsId}", url: "${list[1]}"],
     [credentialsId: "${credentialsId}", url: "${list[2]}"],
     [credentialsId: "${credentialsId}", url: "${list[3]}"]
     ]])

With this config I can see on my build server each project are being checked out but only the last project is availbale after this checkout.

I also tried doing in a loop using "git" only

   def list = "${includedProjectsGITURLS}".split("\n")
    echo "Number of repos : " + list.size()
    echo "Checking out main project (${gitBranch}) :  " + mainProjectGITURL + "\n";
    git credentialsId: 'blablabla', url: "${mainProjectGITURL}"
    for (String gitSubURL:list)
 {
    echo "Checking out sub project (${gitBranch}) :  " + gitSubURL + "\n";
        git credentialsId: 'blablabla', url: "${gitSubURL}"
 }
The result is the same, making me think "checkout" is just wrapper for GIT

I also tried checking each repo into its own sub folder but the result is the same

any ideas how I can checkout multiple repos in Jenkins declarative script for single build

[Mark Waite]

Git takes full control of the directory for each checkout.  Thus, the last checkout "wins" by containing the content of only that checkout.

Perform a checkout command for each repository in a separate directory.  Create the directory for each checkout with the dir() step or the ws() step, and place the checkout inside the block from that step.  Something like this:

dir('dir-1-for-repo-1') {

  checkout([$class: 'GitSCM', branches: [[name: gitBranch]], extensions: [[$class: 'CloneOption', timeout: 120]], 

          userRemoteConfigs: [[credentialsId: "${credentialsId}", url: mainProjectGITURL ]  ]])

}

dir('dir-2-for-repo-2') {

  checkout([$class: 'GitSCM', branches: [[name: gitBranch]], extensions: [[$class: 'CloneOption', timeout: 120]], 

          userRemoteConfigs: [[credentialsId: "${credentialsId}", url:  list[0] ]  ]])

}

On Tue, May 28, 2019 at 4:14 AM Jacques van der Merwe <vane...@gmail.com> wrote:

I know this is fairly old thread but I have an issue

when using the multi repo checkout it seems to remove the previous repo checkout before doing the new one. 

I have tried checking all the repos required for my build using the Jenkins Declarative "checkout" command

checkout([$class: 'GitSCM', branches: [[name: "${gitBranch}"]], doGenerateSubmoduleConfigurations: false, extensions: [[$class: 'CloneOption', timeout: 120]], submoduleCfg: [], userRemoteConfigs: [
     [credentialsId: "${credentialsId}", url: "${mainProjectGITURL}"],
     [credentialsId: "${credentialsId}", url: "${list[0]}"],
     [credentialsId: "${credentialsId}", url: "${list[1]}"],
     [credentialsId: "${credentialsId}", url: "${list[2]}"],
     [credentialsId: "${credentialsId}", url: "${list[3]}"]
     ]])

With this config I can see on my build server each project are being checked out but only the last project is availbale after this checkout.

Multiple userRemoteConfigs in a single checkout command will perform a single checkout of one of the remote configs.  It won't iterate over the list of remote configs and definitely won't create the separate subdirectories that would be needed to retain a checkout of multiple repositories.

Multiple userRemoteConfigs in a single checkout are allowed as a historical use that was allowed for those rare cases where multiple locations contained the same repository and any one of the repositories could be used.  Multiple userRemoteConfigs won't help in this case and they don't help in almost all cases.

 

I also tried doing in a loop using "git" only

   def list = "${includedProjectsGITURLS}".split("\n")
    echo "Number of repos : " + list.size()
    echo "Checking out main project (${gitBranch}) :  " + mainProjectGITURL + "\n";
    git credentialsId: 'blablabla', url: "${mainProjectGITURL}"
    for (String gitSubURL:list)
 {
    echo "Checking out sub project (${gitBranch}) :  " + gitSubURL + "\n";
        git credentialsId: 'blablabla', url: "${gitSubURL}"
 }
The result is the same, making me think "checkout" is just wrapper for GIT

The git step is a simplified form of the checkout step.  If your case does not need the full capability of the checkout step, you are welcome to use the git step.  I find that I almost always want the checkout step because it allows me to control things more precisely.

 

I also tried checking each repo into its own sub folder but the result is the same

We'll need more explanation of what was tried in this case, since that is the technique that works for many users.

Thanks,

Mark Waite

 

any ideas how I can checkout multiple repos in Jenkins declarative script for single build

On Wednesday, December 21, 2016 at 7:25:43 PM UTC+2, Torsten Reinhard wrote:

Hi,

I have a pipeline doing a build and afterwards a deployment to the TEST environment.

For the build I need to

// checkout sources for building
checkout scm

and later on I need to

//checkout configuration for TEST environment
checkout scm (config)

I wasn´t able to configure a working directory for each repo - and the "Multiple SCMs plugin" is deprecated with a hint to the Pipeline plugin.
How can I setup the Pipeline job so it´s using "Pipeline script from SCM" AND additionally some files/configs from a different repo?

I know about the workaround with subtrees and also this issue: https://issues.jenkins-ci.org/browse/JENKINS-13228,
but I´m wondering why help points to the Pipeline plugin ?

Any sample setup available ?

Thanx for helping,

Torsten

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/0d41e558-853d-4cbc-8b8f-fbcdfffedb57%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Thanks!

Mark Waite

[Sravya Kancherla]

Hi,

I have Pipeline for multiple Git repositories to checkout and i want to skip one repo at particular branch

eg: repo1,repo2,repo3,repo4.....

i want to skip repo3 when branch is re-5.1 

[john levin]

You should use git submodules to handle multiple repositories.

Check the submodule documentation for more info.

/John

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/a3a16754-74b5-4022-88ea-eb9e0ac19555n%40googlegroups.com.

[Sravya Kancherla]

Hi Join 

I have pipeline for multiple git repositories  as shown below, i want to skip a stage

stage (RM) {

steps{

     checkout([$class: 'GitSCM', branches: [[name: gitBranch]], extensions: [[$class: 'CloneOption', timeout: 120]], 

          userRemoteConfigs: [[credentialsId: "${credentialsId}", url: mainProjectGITURL1 ]  ]])

}

}
stage (RMC) {

           steps{

     checkout([$class: 'GitSCM', branches: [[name: gitBranch]], extensions: [[$class: 'CloneOption', timeout: 120]], 

          userRemoteConfigs: [[credentialsId: "${credentialsId}", url: mainProjectGITURL2 ]  ]])

}

}

I want to skip stage RMC when branch is re-5.1

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/CAGJXh7o_iovEq2jqrA2f3dW7ajtansTJ7iW%3D8CtmvpnLExxBCg%40mail.gmail.com.

[Jérôme Godbout]

Not sure you can put the if before the whole stage (I’m using scripted pipeline, not declared one du to limitation). but Having stage that appear or disappear make the GUI stage display do weird think into scripted pipeline for the job table. I often just skip the inner content of the steps and display a trace into the console to keep a trace the step was skipped, anyway the build time for that step will be nearly 0ms.

 

steps {

  if(gitBranch != ‘re-5.1’) {

     … do your step code here

  }

}

 

This way the job display may stay the same when skipping or not the steps.

 

Might not be what you want, just throwing an idea here.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/CA%2BfgoYc%2Bky%3D7u%2BxCTSCyPZT1SL7Odg%3D%3DzZF3_a71OLibpDhY3w%40mail.gmail.com.