Sharing git repos amongst multibranch workspaces?

[Jonathan del Strother]

Is there any way to get the multibranch plugin to share its git checkouts?   It's fetching a fresh clone from github for every single branch, which seems excessive when I have 30 branches and a 500MB repo.  I'm struggling to get it working because the initial checkouts keep getting killed due to the 10 minute timeout.

I wondered if there was an option to either have a single .git repo shared amongst the workspaces (perhaps with git-worktree), or a local repo that can be cloned from, to avoid the repeated transfer from github.com.

[Mark Waite]

The git plugin supports a reference repository from its "Advanced clone options".  A reference repository reduces the clone time and the disk space for the history portion of a repository clone by creating pointers into a bare repository on the same computer.

The git plugin supports shallow clone from its "Advanced clone options".  A shallow clone reduces the clone time and disk space for the history portion of a repository clone by only cloning a certain number of commits rather than the entire history.  With git versions after 1.9, I believe it can still push new commits from a shallow clone.

The git plugin allows sparse checkouts from the "Sparse checkout" additional behaviour.  Sparse checkouts reduce disc space for the working directory portion of the clone by using a subset of the working directory to be checked out instead of the full working directory.

Each of those options can be assigned in a Jenkinsfile.  The pipeline snippet generator knows how to create those options when you choose git in the snippet generator.

I still haven't found the "best" way to handle those additional git plugin options when dealing with merges between long-lived git branches.  When I assign those types of options, I seem to also be required to assign a branch to be built, though I don't want that git "branch to be built" to merge between long-lived branches.

Mark Waite

On Sun, Jun 5, 2016 at 6:10 AM Jonathan del Strother <jdelst...@gmail.com> wrote:

Is there any way to get the multibranch plugin to share its git checkouts?   It's fetching a fresh clone from github for every single branch, which seems excessive when I have 30 branches and a 500MB repo.  I'm struggling to get it working because the initial checkouts keep getting killed due to the 10 minute timeout.

I wondered if there was an option to either have a single .git repo shared amongst the workspaces (perhaps with git-worktree), or a local repo that can be cloned from, to avoid the repeated transfer from github.com.

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/23e7bf3e-513d-4720-ac70-506bae1f8b22%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Jonathan del Strother]

Ahh, I didn't realise I could specify the reference repo in the Jenkinsfile - I was looking for it in the multibranch job's configuration.

So rather than "checkout scm" in the Jenkinsfile I want something like this - ?

  checkout([$class: 'GitSCM', branches: [[name: '*']], doGenerateSubmoduleConfigurations: false, extensions: [[$class: 'CloneOption', depth: 0, noTags: false, reference: '/foo', shallow: false]], submoduleCfg: [], userRemoteConfigs: [[credentialsId: '...', url: '....']]])

Is there any way of re-using the existing 'scm' object there?   I'm not familiar with Java or Groovy, was hoping to do something along the lines of:

  scm.extensions.push([$class: 'CloneOption', reference: '/foo'])

  checkout scm

but I'm struggling to get that working.

[Mark Waite]

That's a great question about reusing the existing scm object, and it sounds like a great idea.  Unfortunately, I'm not yet familiar enough with groovy to answer.  Maybe other more experienced pipeline users will know the answer?

Mark Waite

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/54894249-e010-4f0f-b52b-e3186022a78a%40googlegroups.com.

[Jonathan del Strother]

I can't figure out how I might use Jenkinsfile to specify options on the initial clone operation - AFAICT, the clone occurs before any steps in my Jenkinsfile pipeline take place, right?   With the multibranch plugin, I can only seem to control options on the checkout.   Am I missing something?

On Sunday, 5 June 2016 13:55:49 UTC+1, Mark Waite wrote:

[Alex Ehlke]

Ticket is here: https://issues.jenkins-ci.org/browse/JENKINS-33273 This is also a complete blocker for us :(

[Mark Waite]

I can see how it can be a blocker for large enough repositories.  For many users, it may be enough that they use a reference repository.

Of course, since it is a blocker for you, a pull request with a proposed implementation would certainly be of interest to the community and would resolve your blocker.

The mercurial plugin has the ability to use a cached copy on the master node, so you might refer to its technique as a reference.  I'm not sure it is exactly the same, but it might be applicable.

Alternately, you could propose a pull request which introduces the concept of repositories caches on agents, with an "automatic reference" to the repository cache from jobs that "opt-in" to using the repository cache.  There are several other very good suggestions in JENKINS-33273 worth considering as well.

Thanks,

Mark Waite

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/2c4c14ba-d478-41b7-b0fd-49199852d0e4%40googlegroups.com.